Guenadi N Jilevski's Oracle BLOG

Oracle RAC, DG, EBS, DR and HA DBA BLOG

Build Oracle RAC 11.2.0.3 on Oracle Solaris 11 11.11 using Oracle VirtualBox

Build Oracle RAC 11.2.0.3 on Oracle Solaris 11 11.11 using Oracle VirtualBox

In the article you will have a look at how to use some Oracle VirtualBox features to build two node Oracle 11gR2 11.2.0.3 RAC system on Oracle Solaris 11 11.11 x86-64. For information about a similar two node Oracle RAC 11.2 setup on Solaris 10 x86-64 using VirtualBox click here. The article will emphasize on the Solaris 11 management options and configurations required to meet Oracle 11.2.0.3 RAC installation prerequisites.

The following software will be used:

  1. Oracle 11.2.0.3 for Solaris (x86-64)– patch 10404530. Download from MOS here.
  2. Oracle Solaris 11 (x86-64). I used Oracle Solaris 11 11/11 Live Media for x86. Download from here.
  3. Oracle VM VirtualBox 4.1.8 Download from here.

There will be two virtual machines Sol1 and Sol2, each of them will be configured with

  • 4GB RAM
  • 160GB bootable disk
  • NIC – NAT: for access to Internet
  • NIC – bridged for public interface in RAC with address 192.168.2.21/22 (first IP 192.168.2.21 on sol1 and second IP 192.168.2.22 on node sol2). These are public interface in RAC.
  • NIC – bridged for private interface in RAC with address 10.10.2.21/22 (first IP 10.10.2.21 on sol1 and second IP 10.10.2.22 on node sol2). These are private interface in RAC.
  • NIC – Host Only : for FTP/SSH/telnet/SCP access from the host OS to the guest OS. (first IP 192.168.56.51 on sol1 and second IP 192.168.56.51on node sol2).
  • 5 10GB attached shared disks for the ASM storage.
  • Sol1 VM will run Solaris 11 guest with hostname sol1. Sol2 VM will run Solaris 11 guest with hostname sol2.

Note. For access to the Guest OS (Solaris) from the Host (MS Windows Vista 64) via ssh/scp/telnet/ftp.

For the host OS to be able to access the guest OS a Host Only adapter is required and the corresponding IP on the guest OS should be within the IP subnet of the VirtualBox Host-Only adapter. On the host (Windows in this case) we have.

Ethernet adapter VirtualBox Host-Only Network:

Connection-specific DNS Suffix . :

Description . . . . . . . . . . . : VirtualBox Host-Only Ethernet Adapter

Physical Address. . . . . . . . . : 08-00-27-00-3C-B7

DHCP Enabled. . . . . . . . . . . : No

Autoconfiguration Enabled . . . . : Yes

IPv4 Address. . . . . . . . . . . : 192.168.56.1(Preferred)

Subnet Mask . . . . . . . . . . . : 255.255.255.0

Default Gateway . . . . . . . . . :

NetBIOS over Tcpip. . . . . . . . : Enabled

Thus, the IP, corresponding to the Host-Only NIC, inside the guest (Solaris in this case) should be within the 192.168.56.* subnet. Should the VirtualBox Host-Only adapter IP change, make sure that the IP within the guest OS is within the same subnet as the IP of the VirtualBox Host-Only adapter.

Examples of accessing Solaris 11 from a cygwin on Windows using ssh/scp.

bash-3.2$ uname -a

CYGWIN_NT-6.0-WOW64 userpc 1.7.5(0.225/5/3) 2010-04-12 19:07 i686 Cygwin

bash-3.2$

bash-3.2$ ssh -X gjilevski@192.168.56.52

The authenticity of host ‘192.168.56.52 (192.168.56.52)’ can’t be established.

RSA key fingerprint is 1b:f8:c2:74:cf:29:4b:e8:0d:a6:d8:f6:d9:51:92:72.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added ‘192.168.56.52’ (RSA) to the list of known hosts.

Password:

Warning: untrusted X11 forwarding setup failed: xauth key data not generated

Warning: No xauth data; using fake authentication data for X11 forwarding.

Last login: Mon Feb 6 22:22:49 2012

Oracle Corporation SunOS 5.11 11.0 November 2011

gjilevski@sol2:~$ uname -a

SunOS sol2 5.11 11.0 i86pc i386 i86pc

gjilevski@sol2:~$ ls /u01/sh.sh

/u01/sh.sh

gjilevski@sol2:~$ exit

logout

Connection to 192.168.56.52 closed.

bash-3.2$ scp gjilevski@192.168.56.52:/u01/sh.sh .

Password:

sh.sh 100% 682 0.7KB/s 00:00

bash-3.2$ uname -a

CYGWIN_NT-6.0-WOW64 userpc 1.7.5(0.225/5/3) 2010-04-12 19:07 i686 Cygwin

bash-3.2$

Same approach can be used for telnet or ftp access from host OS if ftp or telnet services are enabled on the guest Solaris OS.

First you will create the first virtual machine and configure network interfaces and install Solaris software and meet the Oracle RAC installation prerequisites. You will than attach the shared disks to the first virtual machine and prepare the shared storage for ASM. Last you will clone the boot disk of the first VM to create the second VM and plug the shared storage and change the IPs and hostname. In this article you will have role separation with two different accounts one (grid) for ASM storage management and the second (oracle) for RDBMS installation. In order to support role separation you will create additional groups for (asmadmin, asmdba, asmoper, oinstall) and ( dba. oinstall, oper). At the end you will install GI, RDBMS and create a two node RAC database. You will see how to fix the problems that were encountered during the installation.

  1. Create the first VM Sol1

    Oracle Solaris 11 was released late 2011. In order to obtain detailed information for installation options click here or access Solaris 11 documentation click here.

    I am using Oracle Solaris 11 11/11 Live Media for x86. It performs default installation using Automatic network management. After Solaris 11 installation, I will switch to Manual network management and modify IP addresses and hostnames.

    Select New and click Next. Enter the name of the VM (Sol1) and press Next.


Select 4096 MB for the RAM of the VM and press Next to continue.


Select create a new disk for the boot disk and press Next to continue.


Select Dynamically expanding storage and press Next to continue.


Select 160GB (not 16GB) and press Next to continue. (16GB is not sufficient to install Solaris and Oracle GI and RDBMS).


Press Next to continue.


Press Finish.

This concludes the VM creation. After that select the VM click settings and add the four NIC as specified. First to be NAT, bridged, bridged and Host-Only.

Create the disks to be used as shared.

VBoxManage createhd –filename d:\vb\asm1.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm2.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm3.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm4.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm5.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm6.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm7.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm8.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename d:\vb\asm9.vdi –size 10240 –format VDI –variant Fixed

    VBoxManage createhd –filename d:\vb\asm10.vdi –size 10240 –format VDI –variant Fixed

Attach the shared disks to the VM and mark them as shared. Note that for a disk to be shared it must be fixed.

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 1 –device 0 –type hdd –medium d:\vb\asm1.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 2 –device 0 –type hdd –medium d:\vb\asm2.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 3 –device 0 –type hdd –medium d:\vb\asm3.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 4 –device 0 –type hdd –medium d:\vb\asm4.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 5 –device 0 –type hdd –medium d:\vb\asm5.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 6 –device 0 –type hdd –medium d:\vb\asm6.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 7 –device 0 –type hdd –medium d:\vb\asm7.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 8 –device 0 –type hdd –medium d:\vb\asm8.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 9 –device 0 –type hdd –medium d:\vb\asm9.vdi –mtype shareable

VBoxManage storageattach Sol1 –storagectl “SATA Controller” –port 10 –device 0 –type hdd –medium d:\vb\asm10.vdi –mtype shareable

VBoxManage modifyhd d:\vb\asm1.vdi –type shareable

VBoxManage modifyhd d:\vb\asm2.vdi –type shareable

VBoxManage modifyhd d:\vb\asm3.vdi –type shareable

VBoxManage modifyhd d:\vb\asm4.vdi –type shareable

VBoxManage modifyhd d:\vb\asm5.vdi –type shareable

VBoxManage modifyhd d:\vb\asm6.vdi –type shareable

VBoxManage modifyhd d:\vb\asm7.vdi –type shareable

VBoxManage modifyhd d:\vb\asm8.vdi –type shareable

VBoxManage modifyhd d:\vb\asm9.vdi –type shareable

VBoxManage modifyhd d:\vb\asm10.vdi –type shareable

The configuration will look like the graphical image below.


Now that the shared disks are attached and the virtual NICs are specified let’s start the VM for Solaris installation. After the VM is created and started specify the media for Solaris installation. The iso image file can be either staged in a directory on the host OS or on DVD. Start the VM and specify the installation media containing the Solaris iso. In this case the iso media is on a Windows directory and you press on the yellow folder with green arrow to select it and press Next to continue.

Note that although the image is for Solaris 10 but you must use Solaris 11 iso file and will work the same way.


Press Next to continue.


Press Finish to start. Once the system boots up Select Solaris 11 and press Return.


Select keyboard.


Select language.


Wait until the virtual OS is launched in memory. Note that Automatic Network is selected. In order to start the real installation click Install Oracle Solaris.


Press Next.


Select the 160GB disk. I did not partition the disk but later added in swap space for the OUI to succeed. The default swap is 1GB and OUI asks for minimum 4GB in order to install 11.2.0.3 RAC.


Select your time zone.


Create an account with administrative privileges. Starting with Solaris 11 root is defined as a role and you need a user to log in. A role cannot log you in. Specify the hostname.


Review and press Install.


Wait for the install to completes.



Reboot once installation completes.


Select Solaris 11.


Login in with the previously created user.


Select login session.


Finally you are complete the Solaris 11 installation.


  1. Installing Guest Additions

After login as yourself, on the VirtualBox console select Devices->Install Guest Additions… The Virtual Editions media gets mounted as a virtual CD. Select OK.


Here you have a chance to change the root password. Initially, use the default (solaris) which is expired and need a change.


Reboot the Solaris guest VM.

  1. Shared Folders configuration

Shared folders are configured as follows.


VirtualBox allows you to mount on the guest OS folders from the host OS.

If using latest VirtualBox 4.1 and later you can automatically add shared folders and VirtualBox mounts them for you.


In case you are on a previous VirtualBox release, as root user create directories for each folder and use the following command to mount.

pfexec mount -F vboxfs Solaris10 /whatever-you-want

In my case I created two directories /software and /OracleVMServer and mounted the shared folders as follow.

pfexec mount -F vboxfs software /software

pfexec mount -F vboxfs OracleVMServer /OracleVMServer

To create a permanent mount points that persist a reboot add the following line to the /etc/vfstab file.

software – /software vboxfs – yes –

OracleVMServer – /OracleVMServer vboxfs – yes –

  1. Setting the shared storage for ASM and SWAP

The default configured swap space was 1GB. For Oracle 11.2 installation 4GB minimum is required. Using the following commands drop and recreate the swap with desired size 5GB.

root@sol1:~# swap -d /dev/zvol/dsk/rpool/swap

root@sol1:~# zfs volsize=5G rpool/swap

root@sol1:~# swap -a /dev/zvol/dsk/rpool/swap

root@sol1:~# swap -s

total: 259856k bytes allocated + 35808k reserved = 295664k used, 6669044k available

root@sol1:~# swap -l

swapfile dev swaplo blocks free

/dev/zvol/dsk/rpool/swap 195,2 8 10485752 10485752

root@sol1:~#

There are ten shared disks attached to the Solaris VM. Those disks are visible as c1t1d0, c1t2d0,c1t3d0,c1t4d0, c1t5d0, c1t6d0, c1t7d0, c1t8d0, c1t9d0 and c1t10d0. You need to format and prepare them for the ASM with privileges (0660) and ownership (grid:asmadmin). Each disk is formatted and a partition is created starting from 1 cylinder. The available disks can be seen using the format command as shown below. Than for each disk you invoke fdisk and accept Solaris as default type of partition. Then you partition with partition command and select 0 partition to start from first cylinder and enter size 9.95GB (remember 10GB was the original size) and print the partition table. You repeat the procedure for all ten shared disks. The flow of commands is as follow.

root@sol1:~# format

Searching for disks…done

AVAILABLE DISK SELECTIONS:

0. c1t0d0 <DEFAULT cyl 20883 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@0,0

1. c1t1d0 <DEFAULT cyl 1303 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@1,0

2. c1t2d0 <DEFAULT cyl 1303 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@2,0

3. c1t3d0 <DEFAULT cyl 1303 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@3,0

4. c1t4d0 <DEFAULT cyl 1303 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@4,0

5. c1t5d0 <DEFAULT cyl 1303 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@5,0

6. c1t6d0 <DEFAULT cyl 1302 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@6,0

7. c1t7d0 <DEFAULT cyl 1302 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@7,0

8. c1t8d0 <DEFAULT cyl 1302 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@8,0

9. c1t9d0 <DEFAULT cyl 1302 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@9,0

10. c1t10d0 <DEFAULT cyl 1302 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@a,0

Specify disk (enter its number): 4

selecting c1t4d0

[disk formatted]

FORMAT MENU:

disk – select a disk

type – select (define) a disk type

partition – select (define) a partition table

current – describe the current disk

format – format and analyze the disk

fdisk – run the fdisk program

repair – repair a defective sector

label – write label to the disk

analyze – surface analysis

defect – defect list management

backup – search for backup labels

verify – read and display labels

save – save new disk/partition definitions

inquiry – show vendor, product and revision

volname – set 8-character volume name

!<cmd> – execute <cmd>, then return

quit

format> format>

format> fdisk

No fdisk table exists. The default partition for the disk is:

a 100% “SOLARIS System” partition

Type “y” to accept the default partition, otherwise type “n” to edit the

partition table.

y

format>

format> partition

PARTITION MENU:

0 – change `0′ partition

1 – change `1′ partition

2 – change `2′ partition

3 – change `3′ partition

4 – change `4′ partition

5 – change `5′ partition

6 – change `6′ partition

7 – change `7′ partition

select – select a predefined table

modify – modify a predefined partition table

name – name the current table

print – display the current table

label – write partition map and label to the disk

!<cmd> – execute <cmd>, then return

quit

partition> 0

Part Tag Flag Cylinders Size Blocks

0 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]:

Enter partition permission flags[wm]:

Enter new starting cyl[0]: 1

Enter partition size[0b, 0c, 1e, 0.00mb, 0.00gb]: 9.95g

partition> print

Current partition table (unnamed):

Total disk cylinders available: 1302 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks

0 unassigned wm 1 – 1299 9.95GB (1299/0/0) 20868435

1 unassigned wm 0 0 (0/0/0) 0

2 backup wu 0 – 1301 9.97GB (1302/0/0) 20916630

3 unassigned wm 0 0 (0/0/0) 0

4 unassigned wm 0 0 (0/0/0) 0

5 unassigned wm 0 0 (0/0/0) 0

6 unassigned wm 0 0 (0/0/0) 0

7 unassigned wm 0 0 (0/0/0) 0

8 boot wu 0 – 0 7.84MB (1/0/0) 16065

9 unassigned wm 0 0 (0/0/0) 0

partition>

partition> label

Ready to label disk, continue? y

partition>quit

In this case the whole disk c1t4d0 was partitioned as one partition and made available for ASM. Repeat the same steps for all remaining nine shared disks. Since slice 0 was used you need to make those available to ASM by setting the permissions and ownership as show below.

chown grid:asmadmin /dev/rdsk/c1t1d0s0

chmod 660 /dev/rdsk/c1t1d0s0

chown grid:asmadmin /dev/rdsk/c1t10d0s0

chmod 660 /dev/rdsk/c1t10d0s0

chown grid:asmadmin /dev/rdsk/c1t9d0s0

chmod 660 /dev/rdsk/c1t9d0s0

chown grid:asmadmin /dev/rdsk/c1t8d0s0

chmod 660 /dev/rdsk/c1t8d0s0

chown grid:asmadmin /dev/rdsk/c1t7d0s0

chmod 660 /dev/rdsk/c1t7d0s0

chown grid:asmadmin /dev/rdsk/c1t6d0s0

chmod 660 /dev/rdsk/c1t6d0s0

chown grid:asmadmin /dev/rdsk/c1t5d0s0

chmod 660 /dev/rdsk/c1t5d0s0

chown grid:asmadmin /dev/rdsk/c1t4d0s0

chmod 660 /dev/rdsk/c1t4d0s0

chown grid:asmadmin /dev/rdsk/c1t3d0s0

chmod 660 /dev/rdsk/c1t3d0s0

chown grid:asmadmin /dev/rdsk/c1t2d0s0

chmod 660 /dev/rdsk/c1t2d0s0

For more information refer to Oracle documentation here.

  1. Creating Solaris users and groups

Oracle starting with 11gR2 is enabling a role separation when installing Oracle GI and Oracle RDBMS. The idea is Oracle GI will have a user grid as ASM administrator and Oracle RDBMS will have a user oracle as RDBMS administrator. Both users share oinstall group so that both Oracle GI and Oracle RDBMS share the same inventory. However each admin has different groups. The grid user is a member of asmadmin, asmdba, asmoper and the primary group is oinstall. The oracle user is a member of dba, asmdba and oper and the primary group is oinstall. Users and groups are created with the following commands.

groupadd -g 1000 oinstall

groupadd -g 1020 asmadmin

groupadd -g 1021 asmdba

groupadd -g 1022 asmoper

groupadd -g 1031 dba

groupadd -g 1032 oper

useradd -u 1100 -g oinstall -G asmoper,asmadmin,asmdba -d /export/home/grid -m grid

useradd -u 1101 -g oinstall -G oper,dba,asmdba -d /export/home/oracle -m oracle

User grid and user oracle separate the responsibilities between storage administrator and Oracle DBA.

  1. Create the directories for ORACLE_BASE and ORACLE_HOME for Oracle GI and Oracle RDBMS

    mkdir -p /u01/app/11.2.0/grid

    mkdir -p /u01/app/grid

    chown -R grid:oinstall /u01

    mkdir -p /u01/app/oracle

    chown oracle:oinstall /u01/app/oracle

    chmod -R 775 /u01

    mkdir -p /u01/app/11.2.0/grid

    chown grid:oinstall /u01/app/11.2.0/grid

    chmod -R 775 /u01/app/11.2.0/grid

    mkdir -p /u01/app/oracle/product/11.2.0/db_1

    chown -R oracle:oinstall /u01/app/oracle

    chmod -R 775 /u01/app/oracle

  1. Setup Solaris kernel parameters.

Create two projects for Oracle GI and Oracle RDBMS users respectively. Set the share memory parameters.

projadd -U grid -K “project.max-shm-memory=(priv,6g,deny)” user.grid

projmod -sK “project.max-sem-nsems=(priv,512,deny)” user.grid

projmod -sK “project.max-sem-ids=(priv,128,deny)” user.grid

projmod -sK “project.max-shm-ids=(priv,128,deny)” user.grid

projmod -sK “project.max-shm-memory=(priv,6g,deny)” user.grid

projadd -U oracle -K “project.max-shm-memory=(priv,6g,deny)” user.oracle

projmod -sK “project.max-sem-nsems=(priv,512,deny)” user.oracle

projmod -sK “project.max-sem-ids=(priv,128,deny)” user.oracle

projmod -sK “project.max-shm-ids=(priv,128,deny)” user.oracle

projmod -sK “project.max-shm-memory=(priv,6g,deny)” user.oracle

/usr/sbin/projmod -sK “process.max-file-descriptor=(priv,65536,deny)” user.oracle

/usr/sbin/projmod -sK “process.max-file-descriptor=(priv,65536,deny)” user.grid

If the max file descriptors are not set properly there might be errors starting OUI.

Set the TCP and UDP kernel parameters

/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000

/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500

/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000

/usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500

Put in /etc/inittab the following lines for the TCP and UDP parameters to persist across reboot.

tm::sysinit:/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000 > /dev/console

tm::sysinit:/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500 > /dev/console

tm::sysinit:/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000 > /dev/console

tm::sysinit:/usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500 > /dev/console

  1. Automatic SSH configuration

Oracle recommends using OUI to setup ssh user equivalence where users can connect across the node of the cluster without a password. To avoid errors while attaching $OH when the remote node closes a connection prematurely set in /etc/ssh/sshd_config file LoginGraceTime 0. After changing the file restart ssh service.

    svcadm restart ssh

  1. Enable Core file creation

Make sure that Core Dumps are enabled. To check if core dumps are enabled use coreadm command as follows.

# coreadm

global core file pattern:

global core file content: default

init core file pattern: core

init core file content: default

global core dumps: disabled

per-process core dumps: enabled

global setid core dumps: disabled

per-process setid core dumps: disabled

global core dump logging: disabled

#

As root user make the following directory. And enable core dumps as show below.

mkdir -p /var/cores

coreadm -g /var/cores/%f.%n.%p.%t.core -e global -e global-setid -e log -d process -d proc-setid

Verify the result.

# coreadm

global core file pattern: /var/cores/%f.%n.%p.%t.core

global core file content: default

init core file pattern: core

init core file content: default

global core dumps: enabled

per-process core dumps: disabled

global setid core dumps: enabled

per-process setid core dumps: disabled

global core dump logging: enabled

#

  1. Network Time protocol Settings

You have two options for time synchronization: an operating system configured network time protocol (NTP), or Oracle Cluster Time Synchronization Service. Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services. If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server. So there are two options

  1. Disable NTP and rely entirely on CTSS:

As root execute the command below:

/usr/sbin/svcadm disable ntp

  1. Configure NTP and use it together with CTSS:

As root edit the /etc/inet/ntp.conf file to add “slewalways yes” and “disable pll” to the file. After you make these changes, restart xntpd using the command

/usr/sbin/svcadm restart ntp.

NTP is disabled in the installation reflected in article.

  1. Create the following profiles

For user grid

umask 022

ORACLE_BASE=/u01/app/grid

ORACLE_HOME=/u01/app/11.2.0/grid

ORACLE_SID=+ASM1

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_H OME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s unlimited

ulimit -v unlimited

if [ -t 0 ]; then

stty intr ^C

fi

For user oracle

umask 022

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

ORACLE_SID=D11G

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s unlimited

ulimit -v unlimited

if [ -t 0 ]; then

stty intr ^C

fi

  1. Set up file descriptors

Edit the /etc/system to add the following parameters.

# Hard limit on file descriptors for single process

set rlim_fd_max = 65536

# Soft limit on the file descriptors for a single process

set rlim_fd_cur = 65536

  1. Require packages and patches

For Solaris 10 x86-64 the following packages and patches are required.

SUNWarc

SUNWbtool

SUNWcsl

SUNWhea

SUNWlibC

SUNWlibm

SUNWlibms

SUNWsprot

SUNWtoo

SUNWi1of (ISO8859-1)

SUNWi1cs (ISO8859-15)

SUNWi15cs

SUNWxwfnt

119961-05 or later

119964-14 or later

120754-06 or later

139556-08 or later

139575-03 or later

141415-04 or later

141445-09 or later [11.2.0.2]

Since I am on Solaris 11 I did

root@sol2:~# pkg install SUNWarc SUNWbtool SUNWhea SUNWlibC SUNWlibm SUNWlibms SUNW SUNWtoo SUNWi1of SUNWi1cs SUNWi15cs SUNWxwfnt SUNWcsl

Creating Plan

pkg install: The following pattern(s) did not match any allowable packages. Try

using a different matching pattern, or refreshing publisher information:

    SUNWi1of

    SUNW

    SUNWxwfnt

root@sol2:~#

Solaris 11 is supported and certified for Oracle 11.2.0.3. Despite, that I did not find information about packages and patches for Oracle 11.2.0.3 OUI confirmed later that installed packages and patches suffice for Oracle 11.2.0.3 RAC installation on Solaris 11.

  1. Configure IP addresses and confirm hostname

    The performed Oracle Solaris 11 11/11 Live Media installation provides Automatic Network Management. In order to manually set up IP addresses I switched to Manual Network management using the following command.

    root@sol1:~# netadm enable -p ncp DefaultFixed

    Enabling ncp ‘DefaultFixed’

    root@sol1:~#

    After switch to manual Network Management I have.

    gjilevski@sol1:~$ dladm show-phys

    LINK MEDIA STATE SPEED DUPLEX DEVICE

    net1 Ethernet unknown 0 unknown e1000g1

    net2 Ethernet unknown 0 unknown e1000g2

    net0 Ethernet unknown 0 unknown e1000g0

    net3 Ethernet unknown 0 unknown e1000g3

    gjilevski@sol1:~$

    In Solaris 11 IP addresses are set as follows:

    ipadm create-ip net0

    ipadm create-addr -T dhcp net0/addr

    ipadm create-ip net1

    ipadm create-addr -T static -a local=192.168.2.21/24 net1/addr

    ipadm create-ip net2

    ipadm create-addr -T static -a local=10.10.2.21/24 net2/addr

    ipadm create-ip net3

    ipadm create-addr -T static -a local=192.168.56.51/24 net3/addr

    At the end I have:

    root@sol1:~# dladm show-phys

    LINK MEDIA STATE SPEED DUPLEX DEVICE

    net1 Ethernet up 1000 full e1000g1

    net2 Ethernet up 1000 full e1000g2

    net0 Ethernet up 1000 full e1000g0

    net3 Ethernet up 1000 full e1000g3

    root@sol1:~#

    Hostname was configured during Solaris 11 installation.

  2. Modify /etc/hosts file to have entries for the public, private and scan addresses similarly to the output below

    gjilevski@sol2:~$ cat /etc/hosts

    #

    # Copyright 2009 Sun Microsystems, Inc. All rights reserved.

    # Use is subject to license terms.

    #

    # Internet host table

    #

    ::1 localhost

    127.0.0.1 localhost

    192.168.2.21 sol1 sol1.gj.com

    192.168.2.22 sol2 sol2.gj.com

    192.168.2.31 sol1-vip sol1-vip.gj.com

    192.168.2.32 sol2-vip sol2-vip.gj.com

    192.168.2.51 scan-sol scan-sol.gj.com

    10.10.2.21 sol1-priv sol1-priv.gj.com

    10.10.2.22 sol2-priv sol2-priv.gj.com

    192.168.56.51 sol1-exp sol1-exp.gj.com

    192.168.56.52 sol2-exp sol2-exp.gj.com

    gjilevski@sol2:~$

  1. Clone the boot disk of Sol1 VM to create a second VM corresponding to the second node.

    d:\VB>VBoxManage createhd –filename d:\vb\asm_5.vdi –size 10240 –format VDI –

    -variant Fixed

    0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%

    Disk image created. UUID: 0df463ae-587a-4f62-bcd1-ac7dbab86f8f

    d:\VB>

Create a second VM called Sol2 with the same resource as Sol1 but specify the cloned disk as a bootable disk. Attach the shared disks to the second VM using commands as described below.

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 1 –device 0 –type hdd –medium d:\vb\asm1.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 2 –device 0 –type hdd –medium d:\vb\asm2.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 3 –device 0 –type hdd –medium d:\vb\asm3.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 4 –device 0 –type hdd –medium d:\vb\asm4.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 5 –device 0 –type hdd –medium d:\vb\asm5.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 6 –device 0 –type hdd –medium d:\vb\asm6.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 7 –device 0 –type hdd –medium d:\vb\asm7.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 8 –device 0 –type hdd –medium d:\vb\asm8.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 9 –device 0 –type hdd –medium d:\vb\asm9.vdi –mtype shareable

VBoxManage storageattach Sol2 –storagectl “SATA Controller” –port 10 –device 0 –type hdd –medium d:\vb\asm10.vdi –mtype shareable

VBoxManage modifyhd d:\vb\asm1.vdi –type shareable

VBoxManage modifyhd d:\vb\asm2.vdi –type shareable

VBoxManage modifyhd d:\vb\asm3.vdi –type shareable

VBoxManage modifyhd d:\vb\asm4.vdi –type shareable

VBoxManage modifyhd d:\vb\asm5.vdi –type shareable

VBoxManage modifyhd d:\vb\asm6.vdi –type shareable

VBoxManage modifyhd d:\vb\asm7.vdi –type shareable

VBoxManage modifyhd d:\vb\asm8.vdi –type shareable

VBoxManage modifyhd d:\vb\asm9.vdi –type shareable

VBoxManage modifyhd d:\vb\asm10.vdi –type shareable

  1. Modify the hostname and IPs on the cloned VM Sol2

In order to avoid collision with IP start only the second cloned VM and change the IP addresses and hostname. Initially both machines apparently have same IPs and hostname. As root on Sol2 VM execute the following commands.

In order to modify the hostname to sol2 execute the following command to change the host and delete the interfaces and reboot the node.

svccfg -s svc:/system/identity:node setprop config/nodename =sol2

svcadm refresh svc:/system/identity:node

svcadm restart svc:/system/identity:node

ipadm delete-ip net0

ipadm delete-ip net1

ipadm delete-ip net2

ipadm delete-ip net3

After restart as root execute the following commands to setup the IP addresses.

ipadm create-ip net0

ipadm create-addr -T dhcp net0/addr

ipadm create-ip net1

ipadm create-addr -T static -a local=192.168.2.22/24 net1/addr

ipadm create-ip net2

ipadm create-addr -T static -a local=10.10.2.22/24 net2/addr

ipadm create-ip net3

ipadm create-addr -T static -a local=192.168.56.52/24 net3/addr

root@sol2:~# dladm show-phys

LINK MEDIA STATE SPEED DUPLEX DEVICE

net1 Ethernet up 1000 full e1000g1

net2 Ethernet up 1000 full e1000g2

net0 Ethernet up 1000 full e1000g0

net3 Ethernet up 1000 full e1000g3

root@sol2:~#

  1. Reboot the cloned Sol2 and fire up the Sol1
  2. Use OUI to setup ssh user equivalence for user grid and oracle.
  3. Run cluvfy to verify the prerequisites.

    Use cluvfy from the stage directory to make sure that prerequisites are met.

    ./runcluvfy.sh stage -post hwos -n sol1,sol2 –verbose

    /runcluvfy.sh stage -pre crsinst -n sol1,sol2 -verbose

  4. Start OUI for Oracle GI installation.

Oracle 11.2.0.3 on Solaris 11 requires that variable AWT_TOOLKIT be exported.

export AWT_TOOLKIT=XToolkit

Failure to do so will produce the following error.

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2012-02-04_05-50-46AM. Please wait …grid@sol1:/u01/stage/grid$ Exception in thread “main” java.lang.UnsatisfiedLinkError: /tmp/OraInstall2012-02-04_05-50-46AM/jdk/jre/lib/amd64/motif21/libmawt.so: ld.so.1: java: fatal: libXm.so.4: open failed: No such file or directory

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1753)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1649)

    at java.lang.Runtime.load0(Runtime.java:769)

    at java.lang.System.load(System.java:968)

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1753)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1670)

    at java.lang.Runtime.loadLibrary0(Runtime.java:822)

    at java.lang.System.loadLibrary(System.java:993)

    at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)

    at java.security.AccessController.doPrivileged(Native Method)

    at java.awt.Toolkit.loadLibraries(Toolkit.java:1509)

    at java.awt.Toolkit.<clinit>(Toolkit.java:1530)

    at com.jgoodies.looks.LookUtils.isLowResolution(LookUtils.java:484)

    at com.jgoodies.looks.LookUtils.<clinit>(LookUtils.java:249)

    at com.jgoodies.looks.plastic.PlasticLookAndFeel.<clinit>(PlasticLookAndFeel.java:135)

    at java.lang.Class.forName0(Native Method)

    at java.lang.Class.forName(Class.java:242)

    at javax.swing.SwingUtilities.loadSystemClass(SwingUtilities.java:1779)

    at javax.swing.UIManager.setLookAndFeel(UIManager.java:453)

    at oracle.install.commons.util.Application.startup(Application.java:780)

    at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:165)

    at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:182)

    at oracle.install.commons.base.driver.common.Installer.startup(Installer.java:348)

    at oracle.install.ivw.crs.driver.CRSInstaller.startup(CRSInstaller.java:98)

    at oracle.install.ivw.crs.driver.CRSInstaller.main(CRSInstaller.java:105)

After issuing xhost + from a root connected terminal session and setting a DISPLAY=:0 and AWT_TOOLKIT=XToolkit in a session logged as grid run OUI. Select Skip software update and press Next to continue.


Select Install and configure GI


Select typical installation.


Specify the SCAN name and enter data for the second node.


Make sure that interfaces are properly selected.


Select ASM as storage option.


Specify dusk group name, redundancy and disks. I selected HIGH redundancy.


Specify inventory location.


Examine the prerequisites check.


Review the summary and press Install.


Wait for the OUI to prompt for the scripts to be run as root.


root@sol1:/u01/app/11.2.0/grid# ./root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of “dbhome” have not changed. No need to overwrite.

The contents of “oraenv” have not changed. No need to overwrite.

The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /var/opt/oracle/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

User ignored Prerequisites during installation

OLR initialization – successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

Adding Clusterware entries to inittab

CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘sol1’

CRS-2676: Start of ‘ora.mdnsd’ on ‘sol1’ succeeded

CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘sol1’

CRS-2676: Start of ‘ora.gpnpd’ on ‘sol1’ succeeded

CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘sol1’

CRS-2672: Attempting to start ‘ora.gipcd’ on ‘sol1’

CRS-2676: Start of ‘ora.gipcd’ on ‘sol1’ succeeded

CRS-2676: Start of ‘ora.cssdmonitor’ on ‘sol1’ succeeded

CRS-2672: Attempting to start ‘ora.cssd’ on ‘sol1’

CRS-2672: Attempting to start ‘ora.diskmon’ on ‘sol1’

CRS-2676: Start of ‘ora.diskmon’ on ‘sol1’ succeeded

CRS-2676: Start of ‘ora.cssd’ on ‘sol1’ succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk 054f135ad3644fbabff3f058ef28ff75.

Successful addition of voting disk 834ba35a0f6c4fe0bf04700b9edc2bb2.

Successful addition of voting disk 3a99cb045bd94f5ebf038d4b4584228c.

Successful addition of voting disk 133e3427001c4f0dbfa957587866693a.

Successful addition of voting disk 90b3a7e49b004ff3bf58d1aa6eb91bbb.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

— —– —————– ——— ———

1. ONLINE 054f135ad3644fbabff3f058ef28ff75 (/dev/rdsk/c4t1d0s0) [DATA]

2. ONLINE 834ba35a0f6c4fe0bf04700b9edc2bb2 (/dev/rdsk/c4t2d0s0) [DATA]

3. ONLINE 3a99cb045bd94f5ebf038d4b4584228c (/dev/rdsk/c4t3d0s0) [DATA]

4. ONLINE 133e3427001c4f0dbfa957587866693a (/dev/rdsk/c4t4d0s0) [DATA]

5. ONLINE 90b3a7e49b004ff3bf58d1aa6eb91bbb (/dev/rdsk/c4t5d0s0) [DATA]

Located 5 voting disk(s).

CRS-2672: Attempting to start ‘ora.asm’ on ‘sol1’

CRS-2676: Start of ‘ora.asm’ on ‘sol1’ succeeded

CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘sol1’

CRS-2676: Start of ‘ora.DATA.dg’ on ‘sol1’ succeeded

Configure Oracle Grid Infrastructure for a Cluster … succeeded

root@sol1:/u01/app/11.2.0/grid#

root@sol2:/u01/app/11.2.0/grid# ./root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Creating /usr/local/bin directory…

Copying dbhome to /usr/local/bin …

Copying oraenv to /usr/local/bin …

Copying coraenv to /usr/local/bin …

Creating /var/opt/oracle/oratab file…

Entries will be added to the /var/opt/oracle/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

OLR initialization – successful

Adding Clusterware entries to inittab

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node sol1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

Configure Oracle Grid Infrastructure for a Cluster … succeeded

root@sol2:/u01/app/11.2.0/grid#

At Final configuration step CLUVFY fails.


INFO: ERROR:

INFO: PRVG-1101 : SCAN name “scan-sol” failed to resolve

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for “scan-sol” (IP address: 192.168.2.51) failed

INFO: ERROR:

INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name “scan-sol”

As this is the only error in the log I ignored it. The error is due to not properly defined SCAN in DNS.

Exit the OUI.


Verify that GI is properly installed and configured.

grid@sol2:/u01/app/11.2.0/grid/log/sol2$ crsctl check cluster -all

**************************************************************

sol1:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

sol2:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

grid@sol2:/u01/app/11.2.0/grid/log/sol2$ crsctl stat res -t

——————————————————————————–

NAME TARGET STATE SERVER STATE_DETAILS

——————————————————————————–

Local Resources

——————————————————————————–

ora.DATA.dg

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.LISTENER.lsnr

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.asm

ONLINE ONLINE sol1 Started

ONLINE ONLINE sol2 Started

ora.gsd

OFFLINE OFFLINE sol1

OFFLINE OFFLINE sol2

ora.net1.network

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.ons

ONLINE ONLINE sol1

ONLINE ONLINE sol2

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE sol1

ora.cvu

1 ONLINE ONLINE sol1

ora.oc4j

1 ONLINE ONLINE sol1

ora.scan1.vip

1 ONLINE ONLINE sol1

ora.sol1.vip

1 ONLINE ONLINE sol1

ora.sol2.vip

1 ONLINE ONLINE sol2

grid@sol2:/u01/app/11.2.0/grid/log/sol2$ crsctl stat res -t -init

——————————————————————————–

NAME TARGET STATE SERVER STATE_DETAILS

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.asm

1 ONLINE ONLINE sol2 Started

ora.cluster_interconnect.haip

1 ONLINE ONLINE sol2

ora.crf

1 ONLINE ONLINE sol2

ora.crsd

1 ONLINE ONLINE sol2

ora.cssd

1 ONLINE ONLINE sol2

ora.cssdmonitor

1 ONLINE ONLINE sol2

ora.ctssd

1 ONLINE ONLINE sol2 ACTIVE:0

ora.diskmon

1 OFFLINE OFFLINE

ora.evmd

1 ONLINE ONLINE sol2

ora.gipcd

1 ONLINE ONLINE sol2

ora.gpnpd

1 ONLINE ONLINE sol2

ora.mdnsd

1 ONLINE ONLINE sol2

grid@sol2:/u01/app/11.2.0/grid/log/sol2$

  1. Oracle RDBMS installation only

Log in as oracle user. Set and export the following variables.

export AWT_TOOLKIT=XToolkit

export DISPLAY=:0.0

Run the installer. Ignore the E-mail notification and press Next to continue.


Select skip software updates.


Select install software only.


Select both nodes and Oracle RAC database.


Select languages.


Select EE.


Select location.


Select groups.


Examine the checks.


Separate checks confirmed that I can continue.

root@sol1:/u01/app/11.2.0/grid/bin# ./crsctl stat res ora.net1.network

NAME=ora.net1.network

TYPE=ora.network.type

TARGET=ONLINE , ONLINE

STATE=ONLINE on sol1, ONLINE on sol2

root@sol1:/u01/app/11.2.0/grid/bin# ./crsctl stat res ora.sol1.vip

NAME=ora.sol1.vip

TYPE=ora.cluster_vip_net1.type

TARGET=ONLINE

STATE=ONLINE on sol1

root@sol1:/u01/app/11.2.0/grid/bin# ./crsctl stat res ora.sol2.vip

NAME=ora.sol2.vip

TYPE=ora.cluster_vip_net1.type

TARGET=ONLINE

STATE=ONLINE on sol2

root@sol1:/u01/app/11.2.0/grid/bin#

Ignore examine Summary and run installation.


Run the scripts.

Exit the installer.


  1. Create a cluster database using dbca

If you intend to have separate disk groups as FRA create it. In my case I created the following disk groups.


While logged in as oracle user and properly set environment variables run dbca. Select RAC database.


Select Create a database.


Select a template.


Select Admin managed and both nodes.


Select OEM option.


Specify password(s).


Error!


Look at:

Dbca Does Not Show ASM Diskgroup Information [ID 1286434.1]

ASM Diskgroup Can Not Be Shown When Creating Database With DBCA [ID 1269734.1]

Turned out permission issue. Add setuid and setgid bit for oracle binary under $GRID_HOME/bin:

$ chmod 6755 $GRID_HOME/bin/oracle

The oracle binary for both GRID_HOME/bin and RDBMS ORACLE_HOME/bin should have 6755 permission, eg: -rwsr-s—x

Select a Disk Group.



Select FRA and Archiving options.


Select sample schemas.


Select parameters.


Press Next.


Select Create.


Review the summary.


Modify the .profile to include ORACLE_UNQNAME and start dbconsole.


Modify passwords and take note of the OEM DC URL.


At the end database is up and running.

gjilevski@sol1:/u01/app/11.2.0/grid/bin$ ./crsctl stat res -t

——————————————————————————–

NAME TARGET STATE SERVER STATE_DETAILS

——————————————————————————–

Local Resources

——————————————————————————–

ora.DATA.dg

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.DATADG.dg

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.LISTENER.lsnr

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.asm

ONLINE ONLINE sol1 Started

ONLINE ONLINE sol2 Started

ora.gsd

OFFLINE OFFLINE sol1

OFFLINE OFFLINE sol2

ora.net1.network

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.ons

ONLINE ONLINE sol1

ONLINE ONLINE sol2

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE sol2

ora.cvu

1 ONLINE ONLINE sol2

ora.d11g.db

1 ONLINE ONLINE sol1 Open

2 ONLINE ONLINE sol2 Open

ora.oc4j

1 ONLINE ONLINE sol2

ora.scan1.vip

1 ONLINE ONLINE sol2

ora.sol1.vip

1 ONLINE ONLINE sol1

ora.sol2.vip

1 ONLINE ONLINE sol2

gjilevski@sol1:/u01/app/11.2.0/grid/bin$

oracle@sol2:~$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Feb 6 18:24:05 2012

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME

———– ————————————————————

     1 sol1.gj.com:D11G1

     2 sol2.gj.com:D11G2

SQL>

SQL>

OEM status is as follows for the cluster database.


OEM status is as follows for the cluster.



February 7, 2012 - Posted by | oracle

40 Comments »

  1. Terrific post. Thanks.

    Comment by Substrate Ratio Mushrooms | February 15, 2012 | Reply

  2. Packages SUNWi1of and SUNWxwfnt are believed to be replaced by SUNWfont-xorg-core and SUNWfont-xorg-iso8859-1.

    Comment by JasonH | February 28, 2012 | Reply

    • Hi,

      Thanks. This is an area that the documentation do not cover clearly as far as Oracle 11.2 and Solaris 11 is concerned.

      Regards,

      Comment by gjilevski | February 28, 2012 | Reply

  3. how can use datapump,export,import,expdp,impdp,patching,migration,performence tunning using addm,awr,statspack,silentmode installation in rac

    Comment by Patel Nir | April 13, 2012 | Reply

    • Hi,

      In the same manner and fashion as in a single instance database using Oracle 11.2.

      Regards,

      Comment by gjilevski | April 13, 2012 | Reply

  4. How to create grid & oracle profile

    Comment by orgil | July 19, 2012 | Reply

  5. Facing problem??? root.sh

    Performing root user operation for Oracle 11g

    The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME= /u01/app/11.2.0/grid

    Enter the full pathname of the local bin directory: [/usr/local/bin]:
    Creating /usr/local/bin directory…
    Copying dbhome to /usr/local/bin …
    Copying oraenv to /usr/local/bin …
    Copying coraenv to /usr/local/bin …

    Creating /var/opt/oracle/oratab file…
    Entries will be added to the /var/opt/oracle/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
    Creating trace directory
    User ignored Prerequisites during installation
    OLR initialization – successful
    root wallet
    root wallet cert
    root cert export
    peer wallet
    profile reader wallet
    pa wallet
    peer wallet keys
    pa wallet keys
    peer cert request
    pa cert request
    peer cert
    pa cert
    peer root cert TP
    profile reader root cert TP
    pa root cert TP
    peer pa cert TP
    pa peer cert TP
    profile reader pa cert TP
    profile reader peer cert TP
    peer user cert
    pa user cert
    Adding Clusterware entries to inittab
    CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘sol1’
    CRS-2676: Start of ‘ora.mdnsd’ on ‘sol1’ succeeded
    CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘sol1’
    CRS-2676: Start of ‘ora.gpnpd’ on ‘sol1’ succeeded
    CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘sol1’
    CRS-2672: Attempting to start ‘ora.gipcd’ on ‘sol1’
    CRS-2676: Start of ‘ora.cssdmonitor’ on ‘sol1’ succeeded
    CRS-2676: Start of ‘ora.gipcd’ on ‘sol1’ succeeded
    CRS-2672: Attempting to start ‘ora.cssd’ on ‘sol1’
    CRS-2672: Attempting to start ‘ora.diskmon’ on ‘sol1’
    CRS-2676: Start of ‘ora.diskmon’ on ‘sol1’ succeeded
    CRS-2676: Start of ‘ora.cssd’ on ‘sol1’ succeeded

    ASM created and started successfully.

    Disk Group DATA created successfully.

    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user ‘root’, privgrp ‘root’..
    Operation successful.
    CRS-4256: Updating the profile
    Successful addition of voting disk 9d1a1c1813e84fadbf0b3afc395f6ced.
    Successfully replaced voting disk group with +DATA.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 9d1a1c1813e84fadbf0b3afc395f6ced (/dev/rdsk/c3t2d0s0) [DATA]
    Located 1 voting disk(s).
    CRS-2672: Attempting to start ‘ora.asm’ on ‘sol1’
    CRS-2676: Start of ‘ora.asm’ on ‘sol1’ succeeded
    CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘sol1’
    CRS-2676: Start of ‘ora.DATA.dg’ on ‘sol1’ succeeded
    /u01/app/11.2.0/grid/bin/srvctl start nodeapps -n sol1 … failed
    FirstNode configuration failed at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 9196.
    /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

    Comment by orgil | July 31, 2012 | Reply

    • Hi,

      Seems that the culptit is ‘/u01/app/11.2.0/grid/bin/srvctl start nodeapps -n sol1 … failed’

      Check for more details the logs. Are all installation prerequistes met?

      Start from there..

      Regards,

      Comment by gjilevski | July 31, 2012 | Reply

      • Cluvfy -post failed but it was ignored GRID installation step.

        grid@sol1:/software/grid$ ./runcluvfy.sh stage -post hwos -n sol1,sol2 -verbose

        Performing post-checks for hardware and operating system setup

        Checking node reachability…

        Check: Node reachability from node “sol1”
        Destination Node Reachable?
        ———————————— ————————
        sol1 yes
        sol2 yes
        Result: Node reachability check passed from node “sol1”

        Checking user equivalence…

        Check: User equivalence for user “grid”
        Node Name Status
        ———————————— ————————
        sol2 passed
        sol1 passed
        Result: User equivalence check passed for user “grid”

        Checking node connectivity…

        Checking hosts config file…
        Node Name Status
        ———————————— ————————
        sol2 passed
        sol1 passed

        Verification of the hosts config file successful

        Interface information for node “sol2”
        Name IP Address Subnet Gateway Def. Gateway HW Address MTU
        —— ————— ————— ————— ————— —————– ——
        net0 10.0.2.15 10.0.2.0 10.0.2.15 10.0.2.2 08:00:27:86:FC:EA 1500
        net1 192.168.2.22 192.168.2.0 192.168.2.22 10.0.2.2 08:00:27:87:87:D6 1500
        net2 10.10.2.22 10.10.2.0 10.10.2.22 10.0.2.2 08:00:27:1C:3F:79 1500
        net3 192.168.56.52 192.168.56.0 192.168.56.52 10.0.2.2 08:00:27:1F:2D:75 1500

        Interface information for node “sol1”
        Name IP Address Subnet Gateway Def. Gateway HW Address MTU
        —— ————— ————— ————— ————— —————– ——
        net0 10.0.2.15 10.0.2.0 10.0.2.15 10.0.2.2 08:00:27:FB:F6:A9 1500
        net1 192.168.2.21 192.168.2.0 192.168.2.21 10.0.2.2 08:00:27:54:E3:20 1500
        net2 10.10.2.21 10.10.2.0 10.10.2.21 10.0.2.2 08:00:27:95:C3:21 1500
        net3 192.168.56.51 192.168.56.0 192.168.56.51 10.0.2.2 08:00:27:29:F1:F3 1500

        Check: Node connectivity of subnet “10.0.2.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol2[10.0.2.15] sol1[10.0.2.15] yes
        Result: Node connectivity passed for subnet “10.0.2.0” with node(s) sol2,sol1

        Check: TCP connectivity of subnet “10.0.2.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol1:10.0.2.15 sol2:10.0.2.15 failed

        ERROR:
        PRVF-7617 : Node connectivity between “sol1 : 10.0.2.15” and “sol2 : 10.0.2.15” failed
        Result: TCP connectivity check failed for subnet “10.0.2.0”

        Check: Node connectivity of subnet “192.168.2.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol2[192.168.2.22] sol1[192.168.2.21] yes
        Result: Node connectivity passed for subnet “192.168.2.0” with node(s) sol2,sol1

        Check: TCP connectivity of subnet “192.168.2.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol1:192.168.2.21 sol2:192.168.2.22 passed
        Result: TCP connectivity check passed for subnet “192.168.2.0”

        Check: Node connectivity of subnet “10.10.2.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol2[10.10.2.22] sol1[10.10.2.21] yes
        Result: Node connectivity passed for subnet “10.10.2.0” with node(s) sol2,sol1

        Check: TCP connectivity of subnet “10.10.2.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol1:10.10.2.21 sol2:10.10.2.22 passed
        Result: TCP connectivity check passed for subnet “10.10.2.0”

        Check: Node connectivity of subnet “192.168.56.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol2[192.168.56.52] sol1[192.168.56.51] yes
        Result: Node connectivity passed for subnet “192.168.56.0” with node(s) sol2,sol1

        Check: TCP connectivity of subnet “192.168.56.0”
        Source Destination Connected?
        —————————— —————————— —————-
        sol1:192.168.56.51 sol2:192.168.56.52 passed
        Result: TCP connectivity check passed for subnet “192.168.56.0”

        Interfaces found on subnet “10.0.2.0” that are likely candidates for VIP are:
        sol2 net0:10.0.2.15
        sol1 net0:10.0.2.15

        Interfaces found on subnet “192.168.2.0” that are likely candidates for a private interconnect are:
        sol2 net1:192.168.2.22
        sol1 net1:192.168.2.21

        Interfaces found on subnet “10.10.2.0” that are likely candidates for a private interconnect are:
        sol2 net2:10.10.2.22
        sol1 net2:10.10.2.21

        Interfaces found on subnet “192.168.56.0” that are likely candidates for a private interconnect are:
        sol2 net3:192.168.56.52
        sol1 net3:192.168.56.51
        Checking subnet mask consistency…
        Subnet mask consistency check passed for subnet “10.0.2.0”.
        Subnet mask consistency check passed for subnet “192.168.2.0”.
        Subnet mask consistency check passed for subnet “10.10.2.0”.
        Subnet mask consistency check passed for subnet “192.168.56.0”.
        Subnet mask consistency check passed.

        Result: Node connectivity check failed

        Checking multicast communication…

        Checking subnet “10.0.2.0” for multicast communication with multicast group “230.0.1.0”…
        Check of subnet “10.0.2.0” for multicast communication with multicast group “230.0.1.0” passed.

        Checking subnet “192.168.2.0” for multicast communication with multicast group “230.0.1.0”…
        Check of subnet “192.168.2.0” for multicast communication with multicast group “230.0.1.0” passed.

        Checking subnet “10.10.2.0” for multicast communication with multicast group “230.0.1.0”…
        Check of subnet “10.10.2.0” for multicast communication with multicast group “230.0.1.0” passed.

        Checking subnet “192.168.56.0” for multicast communication with multicast group “230.0.1.0”…
        Check of subnet “192.168.56.0” for multicast communication with multicast group “230.0.1.0” passed.

        Check of multicast communication passed.

        Checking for multiple users with UID value 0
        Result: Check for multiple users with UID value 0 passed
        Check: Time zone consistency
        Result: Time zone consistency check passed

        Checking shared storage accessibility…

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c1t0d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t0d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t1d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t2d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t3d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t4d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t5d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t6d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t7d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c3t0d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c3t1d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c3t2d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c3t3d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c3t4d0s0 on nodes:
        sol2,sol1

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c3t5d0s0 on nodes:
        sol2,sol1

        No shared storage found

        Shared storage check failed on nodes “sol2,sol1”

        Post-check for hardware and operating system setup was unsuccessful on all the nod

        Comment by orgil | July 31, 2012

      • Hi,

        It seems two potential problems

        1. Network configuration – PRVF-7617 : Node connectivity between “sol1 : 10.0.2.15″ and “sol2 : 10.0.2.15″ failed
        2. Shared storage –

        WARNING:
        sol1:Unable to determine the sharedness of /dev/dsk/c2t0d0s0 on nodes:

        I would suggest to see if these two are really a problem or apppear as a problem.

        You could BOTH manually check the networking and shared storage and use

        ./runcluvfy.sh stage -post hwos -n sol1,sol2 –verbose

        ./runcluvfy.sh stage -pre crsinst -n sol1,sol2 -verbose

        After you doublecheck repeat the installation and examine the logs

        Regards,

        Comment by gjilevski | July 31, 2012

    • hi,i got the same error when i run root.sh, do you fix this now? how to fix it?

      Comment by hbh | September 11, 2012 | Reply

  6. Hi everyone

    First of all I request you guys to [b]PLEASE DO HELP ME RESOLVE THIS ISSUE – Thanks!![/b]

    I am trying to install Oracle Grid Infrastructure 11gR2 11.0.2.0.3 on Oracle Solaris 11.11 x86 64bit installed using Live Media on Oracle VirtualBox 4.2.6.

    After setting up the cluster nodes I ran the cluster verification utility which came out with 100% success.

    However during the execution of root.sh I got the following error:

    [b]/u01/app/11.2.0/grid/root.sh[/b]

    CRS-4266: Voting file(s) successfully replaced
    ##  STATE    File Universal Id                File Name Disk group
    
     1. ONLINE   4a3543261bc34fc9bf6e41b5d7fa04c8 (/dev/rdsk/c8t2d0s1) [OCR]
    Located 1 voting disk(s).
    CRS-2672: Attempting to start 'ora.asm' on 'sol1'
    CRS-2676: Start of 'ora.asm' on 'sol1' succeeded
    CRS-2672: Attempting to start 'ora.OCR.dg' on 'sol1'
    CRS-2676: Start of 'ora.OCR.dg' on 'sol1' succeeded
    /u01/app/11.2.0/grid/bin/srvctl start nodeapps -n sol1 ... failed
    FirstNode configuration failed at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 9196.
    /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

    [size=3]I re-ran this command to see the actual errors[/size]

    [b]/u01/app/11.2.0/grid/bin/srvctl start nodeapps -n sol1[/b]

    PRCR-1013 : Failed to start resource ora.net1.network
    PRCR-1064 : Failed to start resource ora.net1.network on node sol1
    CRS-2674: Start of 'ora.net1.network' on 'sol1' failed
    PRCR-1079 : Failed to start resource ora.sol1.vip
    CRS-2674: Start of 'ora.net1.network' on 'sol1' failed
    CRS-2632: There are no more servers to try to place resource 'ora.sol1.vip' on that would satisfy its placement policy
    PRCR-1013 : Failed to start resource ora.ons
    PRCR-1064 : Failed to start resource ora.ons on node sol1
    CRS-2674: Start of 'ora.net1.network' on 'sol1' failed

    [size=3]I ran the following commands to see the SAN and SCAN-Listners status[/size]

    [b]srvctl status scan[/b]

    SCAN VIP scan1 is enabled
    SCAN VIP scan1 is not running
    SCAN VIP scan2 is enabled
    SCAN VIP scan2 is not running
    SCAN VIP scan3 is enabled
    SCAN VIP scan3 is not running

    [b]srvctl status scan_listener[/b]

    SCAN Listener LISTENER_SCAN1 is enabled
    SCAN listener LISTENER_SCAN1 is not running
    SCAN Listener LISTENER_SCAN2 is enabled
    SCAN listener LISTENER_SCAN2 is not running
    SCAN Listener LISTENER_SCAN3 is enabled
    SCAN listener LISTENER_SCAN3 is not running

    [b]srvctl config scan_listener[/b]

    SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
    SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
    SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

    [size=3]Here are the contents of my /etc/hosts and /etc/resolv.conf files[/size]

    [b]cat /etc/hosts[/b]

    # Public
    192.168.56.101   sol1 	     sol1.localdomain loghost
    192.168.56.102   sol2 	     sol2.localdomain
    192.168.56.160   solfiler    solfiler.localdomain
    
    # VIP
    192.168.56.111   sol1-vip    sol1-vip.localdomain
    192.168.56.112   sol2-vip    sol2-vip.localdomain
    
    # Private Interconnect
    192.168.0.101    sol1-priv   sol1-priv.localdomain
    192.168.0.102    sol2-priv   sol2-priv.localdomain
    
    # Scan
    #192.168.56.201  sol-scan    sol-scan.localdomain
    #192.168.56.202  sol-scan    sol-scan.localdomain
    #192.168.56.203  sol-scan    sol-scan.localdomain

    [b]cat /etc/resolv.conf[/b]

    # _AUTOGENERATED_FROM_SMF_V1_
    #
    # WARNING: THIS FILE GENERATED FROM SMF DATA.
    #   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
    # See resolv.conf(4) for details.
    
    search	localdomain
    nameserver	192.168.56.101
    nameserver	192.168.56.102
    nameserver	192.168.1.1

    [size=3]Here are the network details[/size]

    [b]dladm show-phys[/b]

    LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
    net1              Ethernet             up         1000   full      e1000g1
    net2              Ethernet             up         1000   full      e1000g2
    net0              Ethernet             up         1000   full      e1000g0

    [b]ipadm show-addr[/b]

    ADDROBJ           TYPE     STATE        ADDR
    lo0/v4            static   ok           127.0.0.1/8
    net0/v4           dhcp     ok           10.0.2.15/24
    net1/addr         static   ok           192.168.56.101/24
    net2/addr         static   ok           192.168.0.101/24
    lo0/v6            static   ok           ::1/128
    net0/v6           addrconf ok           fe80::a00:27ff:fe5f:f484/10

    [size=3]I did the nslookup on my SCAN name[/size]

    [b]nslookup sol-scan[/b]

    Name:	sol-scan.localdomain
    Address: 192.168.56.202
    Name:	sol-scan.localdomain
    Address: 192.168.56.203
    Name:	sol-scan.localdomain
    Address: 192.168.56.201

    [size=3]I ran the following commands to see if the installation process is reading my SCAN configuration properly[/size]

    [b]srvctl stop scan_listener
    srvctl stop scan

    srvctl config scan[/b]

    SCAN name: sol-scan, Network: 1/192.168.56.0/255.255.255.0/net1
    SCAN VIP name: scan1, IP: /sol-scan/192.168.56.203
    SCAN VIP name: scan2, IP: /sol-scan/192.168.56.201
    SCAN VIP name: scan3, IP: /sol-scan/192.168.56.202

    So it is reading the SCAN information well therefore there is no problem there

    [size=3]I tried to start the SCAN Listeners again and ended up with the same errors[/size]

    [b]srvctl start scan_listener[/b]

    PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr
    CRS-2674: Start of 'ora.net1.network' on 'sol1' failed
    CRS-2632: There are no more servers to try to place resource 'ora.LISTENER_SCAN1.lsnr' on that would satisfy its placement policy
    PRCR-1079 : Failed to start resource ora.LISTENER_SCAN2.lsnr
    CRS-2674: Start of 'ora.net1.network' on 'sol1' failed
    CRS-2632: There are no more servers to try to place resource 'ora.LISTENER_SCAN2.lsnr' on that would satisfy its placement policy
    PRCR-1079 : Failed to start resource ora.LISTENER_SCAN3.lsnr
    CRS-2674: Start of 'ora.net1.network' on 'sol1' failed
    CRS-2632: There are no more servers to try to place resource 'ora.LISTENER_SCAN3.lsnr' on that would satisfy its placement policy

    If it helps, I am using the default automatic network management of Solaris 11.11 however I changed the IP addresses of net1 and net2 manually to meet my network configuration requirements for the clusterware.

    I followed every step of prerequisites for installation on solaris 11.

    After verifying everything that I could have as detailed above I am now clueless as to where things are going wrong

    Could it be the automatic network management of Solaris 11.11 that is not allowing creation of VIP SCAN IPs ???

    Once again [b] I REQUEST YOU GUYS TO PLEASE DO HELP ME FIX THIS PROBLEM[/b].

    Regards
    Terry.

    Comment by Terry | January 30, 2013 | Reply

    • Hi Terry,

      I would first look at cluvfy utility output in verbose mode ( post hwos & pre crsinst).

      ./runcluvfy.sh stage -post hwos -n sol1,sol2 –verbose

      ./runcluvfy.sh stage -pre crsinst -n sol1,sol2 -verbose

      After that I would look at why the network resources are failing.

      As far as I remember I ran in manual network management mode. It worked fine for me.

      Best Regards,

      Comment by gjilevski | January 30, 2013 | Reply

  7. Hi Guenadi,

    Its nice to hear from you !!

    Here is a bit of information on the interfaces I am using

    Net0 is NAT Adapter for internet connectivity using DHCP
    Net1 is HostOnly Adapter for Public connectivity which you can see in my /etc/hosts
    Net2 is HostOnly Adapter for Private Interconnect which you can see in my /etc/hosts

    As you suggested I ran the cluvfy utility, results are listed below :

    1. ./runcluvfy.sh stage -post hwos -n sol1,sol2 -verbose >> cluvfy_post.log

    vi cluvfy_post.log

    Performing post-checks for hardware and operating system setup
    
    Checking node reachability...
    
    Check: Node reachability from node "Sol1"
      Destination Node                      Reachable?
      ------------------------------------  ------------------------
      sol1                                  yes
      sol2                                  yes
    Result: Node reachability check passed from node "Sol1"
    
    
    Checking user equivalence...
    
    Check: User equivalence for user "grid"
      Node Name                             Status
      ------------------------------------  ------------------------
      sol2                                  passed
      sol1                                  passed
    Result: User equivalence check passed for user "grid"
    Version of exectask could not be retrieved from node "sol2"
    
    Checking node connectivity...
    
    Checking hosts config file...
      Node Name                             Status
      ------------------------------------  ------------------------
      sol1                                  passed
    
    Verification of the hosts config file successful
    
    
    Interface information for node "sol1"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     net0   10.0.2.15       10.0.2.0        10.0.2.15       10.0.2.2        08:00:27:5F:F4:84 1500
     net1   192.168.56.101  192.168.56.0    192.168.56.101  10.0.2.2        08:00:27:78:CA:E2 1500
     net2   192.168.0.101   192.168.0.0     192.168.0.101   10.0.2.2        08:00:27:44:8E:13 1500
     net2   169.254.187.71  169.254.0.0     169.254.187.71  10.0.2.2        08:00:27:44:8E:13 1500
    
    
    Check: Node connectivity of subnet "10.0.2.0"
    Result: Node connectivity passed for subnet "10.0.2.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "10.0.2.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:10.0.2.15                  sol1:10.0.2.15                  passed
    Result: TCP connectivity check passed for subnet "10.0.2.0"
    
    
    Check: Node connectivity of subnet "192.168.56.0"
    Result: Node connectivity passed for subnet "192.168.56.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "192.168.56.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:192.168.56.101             sol1:192.168.56.101             passed
    Result: TCP connectivity check passed for subnet "192.168.56.0"
    
    
    Check: Node connectivity of subnet "192.168.0.0"
    Result: Node connectivity passed for subnet "192.168.0.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "192.168.0.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:192.168.0.101              sol1:192.168.0.101              passed
    Result: TCP connectivity check passed for subnet "192.168.0.0"
    
    
    Check: Node connectivity of subnet "169.254.0.0"
    Result: Node connectivity passed for subnet "169.254.0.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "169.254.0.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:169.254.187.71             sol1:169.254.187.71             passed
    Result: TCP connectivity check passed for subnet "169.254.0.0"
    
    
    Interfaces found on subnet "10.0.2.0" that are likely candidates for VIP are:
    sol1 net0:10.0.2.15
    
    Interfaces found on subnet "169.254.0.0" that are likely candidates for VIP are:
    sol1 net2:169.254.187.71
    
    Interfaces found on subnet "192.168.56.0" that are likely candidates for a private interconnect are:
    sol1 net1:192.168.56.101
    
    Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
    sol1 net2:192.168.0.101
    
    Result: Node connectivity check passed
    
    Checking multicast communication...
    
    Checking subnet "10.0.2.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "10.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Checking subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Checking subnet "169.254.0.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "169.254.0.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Check of multicast communication passed.
    
    Checking for multiple users with UID value 0
    Result: Check for multiple users with UID value 0 passed
    Check: Time zone consistency
    Result: Time zone consistency check passed
    
    Checking shared storage accessibility...
    
      Disk                                  Sharing Nodes (1 in count)
      ------------------------------------  ------------------------
      /dev/dsk/c1t0d0s0                     sol1
      /dev/dsk/c1t1d0s0                     sol1
      /dev/dsk/c2t0d0s0                     sol1
      /dev/dsk/c7t0d0s0                     sol1
      /dev/dsk/c7t1d0s0                     sol1
      /dev/dsk/c8t2d0s0                     sol1
      /dev/dsk/c8t3d0s0                     sol1
      /dev/dsk/c8t4d0s0                     sol1
    
    
    Shared storage check was successful on nodes "sol1"
    
    Post-check for hardware and operating system setup was unsuccessful.
    
    Post-check for hardware and operating system setup was unsuccessful.
    Checks did not pass for the following node(s):
            sol2

    Post-check for sol2 did not work because root.sh script failed on sol1 therefore it did not make it to sol2.

    2. ./runcluvfy.sh stage -pre crsinst -n sol1,sol2 -verbose >> cluvfy_pre.log

    vi cluvfy_pre.log

    Performing pre-checks for cluster services setup
    
    Checking node reachability...
    
    Check: Node reachability from node "Sol1"
      Destination Node                      Reachable?
      ------------------------------------  ------------------------
      sol1                                  yes
      sol2                                  yes
    Result: Node reachability check passed from node "Sol1"
    
    
    Checking user equivalence...
    
    Check: User equivalence for user "grid"
      Node Name                             Status
      ------------------------------------  ------------------------
      sol2                                  passed
      sol1                                  passed
    Result: User equivalence check passed for user "grid"
    Version of exectask could not be retrieved from node "sol2"
    
    ERROR:
    Remote execution files could not be copied to "/tmp/CVU_11.2.0.3.0_grid/" on the following nodes:
            sol2
    Verification will proceed with nodes:
            sol1
    
    Checking node connectivity...
    
    Checking hosts config file...
      Node Name                             Status
      ------------------------------------  ------------------------
      sol1                                  passed
    
    Verification of the hosts config file successful
    
    
    Interface information for node "sol1"
     Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
     ------ --------------- --------------- --------------- --------------- ----------------- ------
     net0   10.0.2.15       10.0.2.0        10.0.2.15       10.0.2.2        08:00:27:5F:F4:84 1500
     net1   192.168.56.101  192.168.56.0    192.168.56.101  10.0.2.2        08:00:27:78:CA:E2 1500
     net2   192.168.0.101   192.168.0.0     192.168.0.101   10.0.2.2        08:00:27:44:8E:13 1500
     net2   169.254.187.71  169.254.0.0     169.254.187.71  10.0.2.2        08:00:27:44:8E:13 1500
    
    
    Check: Node connectivity of subnet "10.0.2.0"
    Result: Node connectivity passed for subnet "10.0.2.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "10.0.2.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:10.0.2.15                  sol1:10.0.2.15                  passed
    Result: TCP connectivity check passed for subnet "10.0.2.0"
    
    
    Check: Node connectivity of subnet "192.168.56.0"
    Result: Node connectivity passed for subnet "192.168.56.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "192.168.56.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:192.168.56.101             sol1:192.168.56.101             passed
    Result: TCP connectivity check passed for subnet "192.168.56.0"
    
    
    Check: Node connectivity of subnet "192.168.0.0"
    Result: Node connectivity passed for subnet "192.168.0.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "192.168.0.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:192.168.0.101              sol1:192.168.0.101              passed
    Result: TCP connectivity check passed for subnet "192.168.0.0"
    
    
    Check: Node connectivity of subnet "169.254.0.0"
    Result: Node connectivity passed for subnet "169.254.0.0" with node(s) sol1
    
    
    Check: TCP connectivity of subnet "169.254.0.0"
      Source                          Destination                     Connected?
      ------------------------------  ------------------------------  ----------------
      Sol1:169.254.187.71             sol1:169.254.187.71             passed
    Result: TCP connectivity check passed for subnet "169.254.0.0"
    
    
    Interfaces found on subnet "10.0.2.0" that are likely candidates for VIP are:
    sol1 net0:10.0.2.15
    
    Interfaces found on subnet "169.254.0.0" that are likely candidates for VIP are:
    sol1 net2:169.254.187.71
    
    Interfaces found on subnet "192.168.56.0" that are likely candidates for a private interconnect are:
    sol1 net1:192.168.56.101
    
    Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are:
    sol1 net2:192.168.0.101
    
    Result: Node connectivity check passed
    
    Checking multicast communication...
    
    Checking subnet "10.0.2.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "10.0.2.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Checking subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Checking subnet "169.254.0.0" for multicast communication with multicast group "230.0.1.0"...
    Check of subnet "169.254.0.0" for multicast communication with multicast group "230.0.1.0" passed.
    
    Check of multicast communication passed.
    
    Check: Total memory
      Node Name     Available                 Required                  Status
      ------------  ------------------------  ------------------------  ----------
      sol1          4GB (4194304.0KB)         2GB (2097152.0KB)         passed
    Result: Total memory check passed
    
    Check: Available memory
      Node Name     Available                 Required                  Status
      ------------  ------------------------  ------------------------  ----------
      sol1          912.6562MB (934560.0KB)   50MB (51200.0KB)          passed
    Result: Available memory check passed
    
    Check: Swap space
      Node Name     Available                 Required                  Status
      ------------  ------------------------  ------------------------  ----------
      sol1          6GB (6291452.0KB)         4GB (4194304.0KB)         passed
    Result: Swap space check passed
    
    Check: Free disk space for "sol1:/u01/app/11.2.0/grid"
      Path              Node Name     Mount point   Available     Required      Status
      ----------------  ------------  ------------  ------------  ------------  ------------
      /u01/app/11.2.0/grid  sol1          /             83.1263GB     5.5GB         passed
    Result: Free disk space check passed for "sol1:/u01/app/11.2.0/grid"
    
    Check: Free disk space for "sol1:/tmp"
      Path              Node Name     Mount point   Available     Required      Status
      ----------------  ------------  ------------  ------------  ------------  ------------
      /tmp              sol1          /tmp          5.8123GB      1GB           passed
    Result: Free disk space check passed for "sol1:/tmp"
    
    Check: User existence for "grid"
      Node Name     Status                    Comment
      ------------  ------------------------  ------------------------
      sol1          passed                    exists(1100)
    
    Checking for multiple users with UID value 1100
    Result: Check for multiple users with UID value 1100 passed
    Result: User existence check passed for "grid"
    
    Check: Group existence for "oinstall"
      Node Name     Status                    Comment
      ------------  ------------------------  ------------------------
      sol1          passed                    exists
    Result: Group existence check passed for "oinstall"
    
    Check: Group existence for "dba"
      Node Name     Status                    Comment
      ------------  ------------------------  ------------------------
      sol1          passed                    exists
    Result: Group existence check passed for "dba"
    
    Check: Membership of user "grid" in group "oinstall" [as Primary]
      Node Name         User Exists   Group Exists  User in Group  Primary       Status
      ----------------  ------------  ------------  ------------  ------------  ------------
      sol1              yes           yes           yes           yes           passed
    Result: Membership check for user "grid" in group "oinstall" [as Primary] passed
    
    Check: Membership of user "grid" in group "dba"
      Node Name         User Exists   Group Exists  User in Group  Status
      ----------------  ------------  ------------  ------------  ----------------
      sol1              yes           yes           yes           passed
    Result: Membership check for user "grid" in group "dba" passed
    
    Check: Run level
      Node Name     run level                 Required                  Status
      ------------  ------------------------  ------------------------  ----------
      sol1          3                         3                         passed
    Result: Run level check passed
    
    Check: Hard limits for "maximum open file descriptors"
      Node Name         Type          Available     Required      Status
      ----------------  ------------  ------------  ------------  ----------------
      sol1              hard          65536         65536         passed
    Result: Hard limits check passed for "maximum open file descriptors"
    
    Check: Soft limits for "maximum open file descriptors"
      Node Name         Type          Available     Required      Status
      ----------------  ------------  ------------  ------------  ----------------
      sol1              soft          65536         1024          passed
    Result: Soft limits check passed for "maximum open file descriptors"
    
    Check: Hard limits for "maximum user processes"
      Node Name         Type          Available     Required      Status
      ----------------  ------------  ------------  ------------  ----------------
      sol1              hard          19995         16384         passed
    Result: Hard limits check passed for "maximum user processes"
    
    Check: Soft limits for "maximum user processes"
      Node Name         Type          Available     Required      Status
      ----------------  ------------  ------------  ------------  ----------------
      sol1              soft          19995         2047          passed
    Result: Soft limits check passed for "maximum user processes"
    
    Check: System architecture
      Node Name     Available                 Required                  Status
      ------------  ------------------------  ------------------------  ----------
      sol1          64-bit amd64 kernel modules  64-bit amd64 kernel modules  passed
    Result: System architecture check passed
    
    Check: Kernel version
      Node Name     Available                 Required                  Status
      ------------  ------------------------  ------------------------  ----------
      sol1          5.11                      5.11                      passed
    Result: Kernel version check passed
    
    Checking for multiple users with UID value 0
    Result: Check for multiple users with UID value 0 passed
    
    Check: Current group ID
    Result: Current group ID check passed
    
    Starting check for consistency of primary group of root user
      Node Name                             Status
      ------------------------------------  ------------------------
      sol1                                  passed
    
    Check for consistency of root user's primary group passed
    
    Starting Clock synchronization checks using Network Time Protocol(NTP)...
    
    NTP Configuration file check started...
    The NTP configuration file "/etc/inet/ntp.conf" is available on all nodes
    NTP Configuration file check passed
    
    Checking daemon liveness...
    
    Check: Liveness for "ntpd"
      Node Name                             Running?
      ------------------------------------  ------------------------
      sol1                                  yes
    Result: Liveness check passed for "ntpd"
    Check for NTP daemon or service alive passed on all nodes
    
    Checking NTP daemon command line for slewing option "'slewalways yes' & 'disable pll'"
    Check: NTP daemon command line
      Node Name                             Slewing Option Set?
      ------------------------------------  ------------------------
      sol1                                  yes
    Result:
    NTP daemon slewing option check passed
    
    Checking NTP daemon's boot time configuration, in file "/etc/inet/ntp.conf", for slewing option "'slewalways yes' & 'disable pll'"
    
    Check: NTP daemon's boot time configuration
      Node Name                             Slewing Option Set?
      ------------------------------------  ------------------------
      sol1                                  yes
    Result:
    NTP daemon's boot time configuration check for slewing option passed
    
    Checking whether NTP daemon or service is using UDP port 123 on all nodes
    
    Check for NTP daemon or service using UDP port 123
      Node Name                             Port Open?
      ------------------------------------  ------------------------
      sol1                                  yes
    
    NTP common Time Server Check started...
    NTP Time Server "131.188.3.222" is common to all nodes on which the NTP daemon is running
    NTP Time Server "193.67.79.202" is common to all nodes on which the NTP daemon is running
    Check of common NTP Time Server passed
    
    Clock time offset check from NTP Time Server started...
    Checking on nodes "[sol1]"...
    Check: Clock time offset from NTP Time Server
    
    Time Server: 131.188.3.222
    Time Offset Limit: 1000.0 msecs
      Node Name     Time Offset               Status
      ------------  ------------------------  ------------------------
    PRVF-5413 : Node "sol1" has a time offset of 2409.28 that is beyond permissible limit of 1000.0 from NTP Time Server "131.188.3.222"
      sol1          2409.28                   failed
    
    Time Server: 193.67.79.202
    Time Offset Limit: 1000.0 msecs
      Node Name     Time Offset               Status
      ------------  ------------------------  ------------------------
    PRVF-5413 : Node "sol1" has a time offset of 2366.55 that is beyond permissible limit of 1000.0 from NTP Time Server "193.67.79.202"
      sol1          2366.55                   failed
    PRVF-5424 : Clock time offset check failed
    Result: Clock synchronization check using Network Time Protocol(NTP) failed
    
    Checking Core file name pattern consistency...
    Core file name pattern consistency check passed.
    
    Checking to make sure user "grid" is not in "root" group
      Node Name     Status                    Comment
      ------------  ------------------------  ------------------------
      sol1          passed                    does not exist
    Result: User "grid" is not part of "root" group. Check passed
    
    Check default user file creation mask
      Node Name     Available                 Required                  Comment
      ------------  ------------------------  ------------------------  ----------
      sol1          0022                      0022                      passed
    Result: Default user file creation mask check passed
    Checking consistency of file "/etc/resolv.conf" across nodes
    
    Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
    File "/etc/resolv.conf" does not have both domain and search entries defined
    Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
    domain entry in file "/etc/resolv.conf" is consistent across nodes
    Checking file "/etc/resolv.conf" to make sure that only one domain entry is defined
    All nodes have one domain entry defined in file "/etc/resolv.conf"
    Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
    search entry in file "/etc/resolv.conf" is consistent across nodes
    Checking DNS response time for an unreachable node
      Node Name                             Status
      ------------------------------------  ------------------------
      sol1                                  passed
    The DNS response time for an unreachable node is within acceptable limit on all nodes
    
    File "/etc/resolv.conf" is consistent across nodes
    
    Check: Time zone consistency
    Result: Time zone consistency check passed
    
    Pre-check for cluster services setup was unsuccessful on all the nodes.

    Now I will be really grateful to you if you can please suggest me how can I fix this problem, as its taking Net0 for VIP
    which is actually DHCP based and I am excluding it in the installation.

    Thank you very much for you time.

    Regards
    Terry

    Comment by Terry | January 30, 2013 | Reply

    • Hi,

      If I were you I would double check the network setup and if the interfaces and hostnames are setup correctly.

      Good Luck. Let me know how it goes.

      Best Regards,

      Comment by gjilevski | January 30, 2013 | Reply

  8. Hi Guenadi,

    Thanks for getting back to me once again, I did add another bridged adapter, hooked it to the router and used it for Public IPs and everything went thru fine.

    Thanks for your help, appreciate it !
    Regards
    Terry

    Comment by Terry | January 31, 2013 | Reply

    • Hello Terry,
      i got the same error as you, can you please post your network configuration of Solaris 11 hosts to me?
      thank you very much.
      Best Regards,
      HeBaoHua

      Comment by HeBaoHua | February 1, 2013 | Reply

  9. HeBaoHua,

    First of all you need to understand that your public ip must resolve to your hostname, the contents of my /etc/hosts file are already listed above all I did is to change my public interface to a bridged adapter and manualy assigned it an ip that resoves to my router and then changed my public ips including VIP SCANs to that interface in my /etc/hosts and /etc/named.conf files.

    If you adapt the network model as show in this article of Guenad you should be fine.

    For more on this issuue please check out the following link:

    https://forums.oracle.com/forums/thread.jspa?threadID=2492602&stqc=true

    All the best !!
    Regards
    Terry

    Comment by Terry | February 2, 2013 | Reply

  10. HeBaoHua,

    First of all you need to understand that your public ip must resolve to your hostname, the contents of my /etc/hosts file are already listed above all I did is to change my public interface to a bridged adapter and manually assigned it an ip that resoves to my router and then changed my public ips including VIP SCANs to that interface in my /etc/hosts and DNS zone files.

    If you adapt the network model as show in this article of Guenadi you should be fine.

    For more on this issuue please check out the following link:

    https://forums.oracle.com/forums/thread.jspa?threadID=2492602&stqc=true

    All the best !!
    Regards
    Terry

    Comment by Terry | February 2, 2013 | Reply

  11. hi gjilevski,
    I need install oracle RAC11gr2 in Solaris 11 x.86 using Oracle VM 3.2.

    I tried to follow but here is for Oracle Virtualbox. You know how when using the racle vm 3.2?

    Thanks,

    Tarcísio.

    Comment by Tarcisio | July 5, 2013 | Reply

  12. Gjilevski, thanks for your return.

    I installed the Oracle Solaris 11 using this template link ioracle VM Templates for x86).

    My problem is to prepare the environment for RAC 11gR2. I am using this its step by step. I can do some things but the virtual machine is in Oracle VM 3.2 I am finding some difuldades. Eg
    1) the format command returns me all records as “Unknown Unknown-0001-cyl 2087 alt 0 hd 255 sec 63” even after formatting them (as in your procedure)
    2) I do not know how to share discs (as you did in virtualbox shared folders)

    I researched on the internet. I found your procedure and another Oracle sparc. None of them is equal to my environment.

    You know any template with solaris 11 and RAC configured?

    Thanks

    Comment by Tarcisio | July 5, 2013 | Reply

    • Hi,

      I have not installed 11gR2 RAC on Solaris 11 on VM server. I do have some guidelines for installing oracle 11gr2 RAC linux templates on VM Server 3.1. You can have a look there for an example of configuring a shared storage in VM environment.

      I am not aware of Oracle RAC 11gR2 templates for Solaris 11, but have not checked recently.

      Using Oracle VM Manager will help you with the shared storage and the overall management.

      Best Regards,

      Comment by gjilevski | July 5, 2013 | Reply

      • Gjilevski, thank you again for your attention.

        I will look its guidelines for installing Oracle RAC 11gR2 Linux templates.

        Best Regards,

        Tarcisio.

        Comment by Tarcisio | July 5, 2013

  13. Hi Gjilevski,

    I looked at her guidelines for installing Oracle RAC 11gR2 linux but I decided to follow the guidelines for solaris.

    Since I am using Oracle VM 3, I’m struggling to adapt this guidelines that this is for virtualbox.

    In my environment, I put a fixed IP to the server node 1: 192.168.2.20.

    I have some questions:

    1) “… IP addresses are set as follows” you create “…-T dhcp…” to net0, but net0 I would not have fixed IP server? In my example I put fixed IP: 192.168.2.20.

    2) to create addresses in net1, net2 and net3 (dladm create-addr …), the 192.168.2.20 stopped responding on the network.

    Could you give more details about the configuration of the network?

    Best regards,

    Tarcísio.

    Comment by Tarcisio | July 8, 2013 | Reply

  14. I don’t even understand how I ended up here, but I believed this submit was good.
    I do not recognise who you might be but certainly you’re going to a
    well-known blogger if you happen to are not already.
    Cheers!

    Comment by เสื้อคู่ | March 12, 2014 | Reply

    • Thanks for reply.

      The devices are displaying on OUI during Grid installation but end of grid installation root.sh script not able to create voting disk & it’s failing..

      I am try to start ASM manually then getting below error which indicate missing devices or not able to access the device due to lack of permission.

      SQL> startup
      ASM instance started

      Total System Global Area 283930624 bytes
      Fixed Size 2223424 bytes
      Variable Size 256541376 bytes
      ASM Cache 25165824 bytes
      ORA-15032: not all alterations performed
      ORA-15017: diskgroup “VOTINGDISK” cannot be mounted
      ORA-15063: ASM discovered an insufficient number of disks for diskgroup
      “VOTINGDISK”

      ——————————–

      below are the root.sh

      CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘vmserver1’
      CRS-2676: Start of ‘ora.mdnsd’ on ‘vmserver1’ succeeded
      CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘vmserver1’
      CRS-2676: Start of ‘ora.gpnpd’ on ‘vmserver1’ succeeded
      CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘vmserver1’
      CRS-2672: Attempting to start ‘ora.gipcd’ on ‘vmserver1’
      CRS-2676: Start of ‘ora.gipcd’ on ‘vmserver1’ succeeded
      CRS-2676: Start of ‘ora.cssdmonitor’ on ‘vmserver1’ succeeded
      CRS-2672: Attempting to start ‘ora.cssd’ on ‘vmserver1’
      CRS-2672: Attempting to start ‘ora.diskmon’ on ‘vmserver1’
      CRS-2676: Start of ‘ora.diskmon’ on ‘vmserver1’ succeeded
      CRS-2676: Start of ‘ora.cssd’ on ‘vmserver1’ succeeded

      Disk Group VOTINGDISK creation failed with the following message:
      ORA-03135: connection lost contact

      Configuration of ASM … failed

      looks like problem with shared storage….

      Please suggest..

      Thanks

      Yousuf

      Comment by Yousuf | March 17, 2014 | Reply

      • Hi,

        How many ASM disk do you have? Look at

        ORA-15063: ASM discovered an insufficient number of disks for diskgroup
        “VOTINGDISK”

        You should have either

        1. Min 3 ASM disks for normal
        2. Min 5 ASM disks for high

        Regards,

        Comment by gjilevski | March 17, 2014

      • Hi,

        If I were you, I would troubleshoot and check what exactly is in the trace/log files to identify what is going on…

        Oracle Error: ORA-15063

        Error Description:
        ASM discovered an insufficient number of disks for diskgroup “string”

        Error Cause:
        ASM was unable to find a sufficient number of disks belonging to the diskgroup to continue the operation.

        Action:
        Check that the disks in the diskgroup are present and functioning, that the owner of the ORACLE binary has read/write permission to the disks, and that the ASM_DISKSTRING initialization parameter has been set correctly. Verify that ASM discovers the appropriate disks by querying V$ASM_DISK from the ASM instance.

        Comment by gjilevski | March 17, 2014

  15. Gjilevski, Thanks for sharing the details setup..

    I am doing same configuration with same above steps..

    Facing one problem with respect to shared storage.. all other pre req checks got passed successfully including network & all.

    Only shared storage not able to detect.

    $ ./runcluvfy.sh comp ssa -n vmserver1,vmserver2 -verbose

    Verifying shared storage accessibility

    Checking shared storage accessibility…

    No shared storage found

    Shared storage check was successful on nodes “vmserver1,vmserver2”

    Verification of shared storage accessibility was successful.

    Node1:

    grid:vmserver1:/u01/software/grid$ ls -lL /dev/rdsk/c8t1d0s0 /dev/rdsk/c8t2d0s0 /dev/rdsk/c8t3d0s0
    crw-rw—- 1 grid asmadmin 217, 64 Feb 26 20:57 /dev/rdsk/c8t1d0s0
    crw-rw—- 1 grid asmadmin 217, 128 Feb 22 17:31 /dev/rdsk/c8t2d0s0
    crw-rw—- 1 grid asmadmin 217, 192 Feb 22 17:31 /dev/rdsk/c8t3d0s0
    grid:vmserver1:/u01/software/grid$

    Node2:

    root@vmserver2:/dev# ls -lL /dev/rdsk/c8t1d0s0 /dev/rdsk/c8t2d0s0 /dev/rdsk/c8t3d0s0
    crw-rw—- 1 grid asmadmin 217, 64 Feb 24 15:29 /dev/rdsk/c8t1d0s0
    crw-rw—- 1 grid asmadmin 217, 128 Feb 24 15:29 /dev/rdsk/c8t2d0s0
    crw-rw—- 1 grid asmadmin 217, 192 Feb 24 15:29 /dev/rdsk/c8t3d0s0

    permission is 660.

    OS version is Solaris 11 on VM Box.

    root@vmserver1:/dev# uname -a
    SunOS vmserver1 5.11 11.1 i86pc i386 i86pc

    Please suggest..

    Thanks

    Yousuf,

    Comment by Yousuf | March 16, 2014 | Reply

    • please suggest…

      Thanks

      Yousuf

      Comment by Yousuf | March 17, 2014 | Reply

      • Hi,

        How about OUI? What OUI says about it? It is a matter of troubleshooting the shared storage configuration…. Look into it.

        Good luck.

        Regards,

        Comment by gjilevski | March 17, 2014

  16. I have selected only one device for voting disk group because am using external redundancy..

    Comment by Yousuf | March 17, 2014 | Reply

  17. […] can see how to use Oracle VM Virtual Box to build a two node 11gR2 RAC cluster on Solaris 10 and Solaris 11. For information related to building a RAC 11gR2 cluster on OEL 5.X/6.X without GNS click here and […]

    Pingback by Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6.5 using Oracle VM Virtual Box « Guenadi N Jilevski's Oracle BLOG | May 26, 2014 | Reply

  18. […] can see how to use Oracle VM Virtual Box to build a two node 11gR2 RAC cluster on Solaris 10 and Solaris 11. For information related to building a RAC 11gR2 cluster on OEL 5.X/6.X without GNS click here and […]

    Pingback by Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6.5 using Oracle VM Virtual Box – Download « Guenadi N Jilevski's Oracle BLOG | May 26, 2014 | Reply

  19. My brother recommended I may like this website. He was once entirely right.
    This put up acdtually made my day. You cann’t believe just how so much time I had splent for
    this information! Thanks!

    Comment by Dugout Systems | May 27, 2014 | Reply

  20. There is certainly a lot to learn about this topic.
    I love all the points you made.

    Comment by Morningside Architect Dion Seminara | August 9, 2015 | Reply

  21. Thank you a lot for sharing this with all folks you actually recognize what
    you’re talking approximately! Bookmarked. Please also talk over
    with my site =). We may have a link change agreement between us

    Comment by Rubye | March 13, 2016 | Reply

  22. […] Build Oracle RAC 11.2.0.3 on Oracle Solaris 11 11.11 using … – Oracle Solaris 11 was released late 2011. In order to obtain detailed information for installation options click here or access Solaris 11 documentation click here. […]

    Pingback by How To Check Physical Memory In Solaris 11 | Imizee1 | June 5, 2016 | Reply

  23. thanks very much. I always wanted to create a RAC on virtualbox wrt Solaris. Your document helped.

    Comment by Santosh Loke | July 11, 2016 | Reply


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: