Guenadi N Jilevski's Oracle BLOG

Virtualization using Oracle VM VirtualBox for building two node Oracle RAC 11gR2 (11.2.0.2) cluster where Oracle 11.2.0.2 RDBMS coexists with 10gR2 (10.2.0.5) RDBMS based on Linux (OEL 5.4)

Virtualization using Oracle VM VirtualBox for building two node Oracle RAC 11gR2 (11.2.0.2) cluster where Oracle 11.2.0.2 RDBMS coexists with 10gR2 (10.2.0.5) RDBMS based on Linux (OEL 5.4)

In the article you will have a look at how to use some Oracle VirtualBox features to build two node Oracle 11gR2 (11.2.0.2) RAC system on Oracle Enterprise Linux (OEL 5). The setup will implement a role separation with different users for Oracle RDBMS and Oracle GI that is, user oracle and grid respectively in order to split the responsibilities between DBAs and storage administrators. Article will emphasize on the additional CRS reconfiguration necessary for Oracle 10gR2 and 11.2 databases to coexist on the same cluster running GI 11.2.0.2.

An overview to Oracle Virtualization solutions can be seen here. You can see how to use Oracle VM VirtualBox to build a two node Solaris cluster here.

You will configure Linux in an Oracle VM VirtualBox Virtual machines, install Oracle GI, Oracle RDBMS and will create a database.

The following software will be used:

1. Oracle 11gR2 (11.2.0.2) for Linux (x86-64). Patch 10098816. Download from MOS here.
   
2. Oracle Enterprise Linux (x86-64). Download from here.
   
3. Oracle VM VirtualBox 4.1.0. Download from here.
   

Two virtual machines, raclinux1 and raclinux2, will be configured each with:

• 4GB RAM
  
• 400GB bootable disk (Disk space will be dynamically allocated not a fixed size pre-allocation)
  
• NIC – NAT: for access to Internet
  
• NIC – bridged for public interface in RAC with address 192.168.20.21/22 (first IP 192.168.20.21 on raclinux1 and second IP 192.168.20.22 on node raclinux2). These are public interface in RAC.
  
• NIC – bridged for private interface in RAC with address 10.10.20.21/22 (first IP 10.10.20.21 on raclinux1 and second IP 10.10.20.22 on node raclinux2). These are private interface in RAC.
  
• NIC – Host Only : for FTP/SSH/telnet/SCP access from the host OS to the guest OS. (first IP 192.168.56.101 on raclinux1 and second IP 192.168.56.102on node raclinux2).
  
• 15 10GB attached shared disks for the ASM storage. (High Redundancy ASM disk groups will be deployed)
  

The requirements and setup of networking (Internet access from Oracle VM VirtualBox Guest OS and SSH/SCP/FTP/telnet from Host OS to Guest OS) on OEL are described here. For information for setup networking on Solaris look here.

Shared folder configuration in Oracle VM VirtualBox is considerably simpler in version 4.1.0 in that it auto-mount on startup the folder existing on the host OS as a shared folder on the Guest OS. The older manual approach described here also is applicable. In Oracle VM VirtualBox simply select the folder path and name and select auto-mount check box.

A Virtual Machine RAC1 will be created with OEL 5.4. The OEL configurations that are prerequisite for Oracle RAC 11gR2 installation will be performed. Public and private networks will be configured. Shared disks for ASM will be created and attached. Oracle ASMLib will be configured for ASM interface and the shared Oracle ASM disks will be created and labeled. For information related to RAC1 VM creation using Oracle VM virtual box click here and here. For information related to OEL 5 installation click here.

Fully configured RAC1 VM representing the raclinux1 server will be cloned to RAC2 VM representing raclinux2 server. On RAC 2 VM the network will be reconfigured to reflect the raclinux2 IPs, the new hostname and the shared disks will be attached.

The article will focus on

• Configuring RAC1 VM machine from OS perspective to meet the Oracle RAC 11gR2 installation prerequisites,
  
• Creating, attaching shared disks using Oracle VM VirtualBox and using ASMLib to prepare the shared disks for ASM installation and configuration.
  
• Cloning RAC1 VM and attaching the shared disks and fixing the network interfaces.
  
• Setup ssh user equivalence for oracle and grid user
  
• Oracle 11.2.0.2 Grid Infrastructure installation
  
• Oracle 11.2.0.2 RDBMS installation
  
• Oracle database creation using dbca
  

Configuring the first RAC1 VM machine

Add divider=10 in the duplicated title section /etc/grub.conf to speed up the performance of the Guest OEL.

[root@raclinux1 ~]# cat /etc/grub.conf

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/sda10

# initrd /initrd-version.img

#boot=/dev/sda

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title Enterprise Linux-base (2.6.18-164.el5)

root (hd0,0)

kernel /vmlinuz-2.6.18-164.el5 ro root=LABEL=/ rhgb quiet

title Enterprise Linux-base (2.6.18-164.el5)

root (hd0,0)

kernel /vmlinuz-2.6.18-164.el5 ro root=LABEL=/ rhgb quiet divider=10

initrd /initrd-2.6.18-164.el5.img

[root@raclinux1 ~]#

Configure Oracle VM VirtualBox Guest Additions by running sh ./VBoxLinuxAdditions.run from the virtual CD drive.

Modify /etc/hosts with the following IPs/hostnames

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

# Server specific

# IP addresses for RAC 2 node

#

# Public Addresses

192.168.20.21 raclinux1.gj.com raclinux1

192.168.20.22 raclinux2.gj.com raclinux2

192.168.20.23 raclinux3.gj.com raclinux3

# Private Addresse

10.10.20.21 raclinux1-priv.gj.com raclinux1-priv

10.10.20.22 raclinux2-priv.gj.com raclinux2-priv

10.10.20.23 raclinux3-priv.gj.com raclinux3-priv

# VIP Addresses

192.168.20.51 raclinux1-vip.gj.com raclinux1-vip

192.168.20.52 raclinux2-vip.gj.com raclinux2-vip

192.168.20.53 raclinux3-vip.gj.com raclinux3-vip

# SCAN

192.168.20.100 rac-scan.gj.com rac-scan

Set the interfaces.

Set the interfaces to correspond to the IP addresses in /etc/hosts. For the available interfaces the following assignment will take place.

eth0 – 192.168.20.21

eth1 – 10.10.20.21

eth2 – 192.168.56.101

eth3 – dhcp

Two ways to change the IPs of the interfaces are.

• GUI tool system-config-network
  
• Changing /etc/sysconfig/network-scripts/ifcfg-eth[1-3]
  

• /etc/sysconfig/network-scripts/ifcfg-eth0 BOOTPROTO=none IPADDR=192.168.20.21
  
• /etc/sysconfig/network-scripts/ifcfg-eth1 BOOTPROTO=none IPADDR=10.10.20.21
  
• /etc/sysconfig/network-scripts/ifcfg-eth2 BOOTPROTO=none IPADDR=192.168.56.101
  
• /etc/sysconfig/network-scripts/ifcfg-eth3 BOOTPROTO=dhcp
  

Change the hostname in /etc/sysconfig/network to raclinux1.

[root@raclinux1 network-scripts]# cat /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=no

HOSTNAME=raclinux1.gj.com

NOZEROCONF=yes

[root@raclinux1 network-scripts]#

Reboot the server for the changes to take effect.

Modify the kernel parameters or install Oracle validated RPM

net.bridge.bridge-nf-call-ip6tables = 0

net.bridge.bridge-nf-call-iptables = 0

net.bridge.bridge-nf-call-arptables = 0

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.shmall = 2097152

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048586

net.ipv4.conf.eth2.rp_filter = 2

net.ipv4.conf.eth2.rp_filter = 2

net.ipv4.conf.eth1.rp_filter = 1

net.ipv4.conf.eth0.rp_filter = 2

kernel.shmmax = 2074277888

fs.suid_dumpable = 1

Installing Oracle Validated RPM serves three purposes

• Installs all packages (RPM) required for GI, RDBMS installation
  
• Creates an oracle user and oinstall and dba roles for that user
  
• Sets kernel parameter to a proper value
  

Install all packages required for Oracle 11.2 GI and RDBMS, if not using Oracle validated rpm from the distribution media.

Binutils-2.17.50.0.6

compat-libstdc++-33-3.2.3

compat-libstdc++-33-3.2.3 (32 bit)

elfutils-libelf-0.125

elfutils-libelf-devel-0.125

gcc-4.1.2

gcc-c++-4.1.2

glibc-2.5-24

glibc-2.5-24 (32 bit)

glibc-common-2.5

glibc-devel-2.5

glibc-devel-2.5 (32 bit)

glibc-headers-2.5

ksh-20060214

libaio-0.3.106

libaio-0.3.106 (32 bit)

libaio-devel-0.3.106

libaio-devel-0.3.106 (32 bit)

libgcc-4.1.2

libgcc-4.1.2 (32 bit)

libstdc++-4.1.2

libstdc++-4.1.2 (32 bit)

libstdc++-devel 4.1.2

make-3.81

numactl-devel-0.9.8.x86_64

sysstat-7.0.2

unixODBC-2.2.11

unixODBC-2.2.11 (32 bit)

unixODBC-devel-2.2.11

unixODBC-devel-2.2.11 (32 bit)

Configure the size of the shared memory as root

umount tmpfs

mount –t tmpfs shmfs –o size=3500m /dev/shm

Make it permanent to persist reboot.

tmpfs /dev/shm tmpfs size=3500m 0 0

Set the user limits for oracle and grid in /etc/security/limits.conf to restrict the maximum number of processes for the oracle users to 16384 and maximum number of open files to 65536.

Oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

grid soft nproc 2047

grid hard nproc 16384

grid soft nofile 1024

grid hard nofile 65536

Add in /etc/pam.d/login, as per MOS Note 567524.1, the line below in order for the login program to load the pam_limits.so so that the /etc/security/limits.conf is read and limits activated and enforced.

session required pam_limits.so

Configure Network Time protocol settings

For the cluster to operate it requires time synchronization, for more information click here. In a nut shell, either NTP can be disabled and CTSS is in active mode or NTP can be enabled and CTSS can be in an observer mode.

For enabling the NTP make sure that /etc/sysconfig/ntpd has the line modified to include –x.

OPTIONS=”-x –u ntp:ntp –p /var/run/ntpd.pid”

For disabling the NTP make sure that the NTP service is stopped and disabled for auto-start and there is not configuration file.

/sbin/service ntpd stop

chkconfig ntpd off

mv /etc/ntp.conf to /etc/ntp.conf.org

In the article NTP is enabled.

Add users and groups. Oracle validated package adds only oracle user as of OEL 5. OEL 6 adds for oracle and grid users. The latter is valid as of the time of the writing.

groupadd –g 1000 oinstall

groupadd –g 1020 asmadmin

groupadd –g 1021 asmdba

groupadd –g 1031 dba

groupadd –g 1022 asmoper

groupadd –g 1023 oper

useradd –u 1100 –g oinstall –G asmadmin,asmdba,dba,asmoper grid

useradd –u 1101 –g oinstall –G dba,asmdba,oper oracle

Set the permissions and directories. Note that Oracle RDBMS directory will be created by OUI in the location specified in the profile.

mkdir –p /u01/app/11.2.0/grid

mkdir –p /u01/app/grid

mkdir –p /u01/app/oracle

chown grid:oinstall /u01/app/11.2.0/grid

chown grid:oinstall /u01/app/grid

chown oracle:oinstall /u01/app/oracle

chown –R grid:oinstall /u01

mkdir –p /u01/app/oracle

chmod –R 775 /u01/

Create the profiles for the grid and oracle users.

[grid@raclinux1 ~]$ cat .bash_profile

# .bash_profile

umask 022

ORACLE_BASE=/u01/app/grid

ORACLE_HOME=/u01/app/11.2.0/grid

ORACLE_HOSTNAME=raclinux1

ORACLE_SID=+ASM1

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH ORACLE_HOSTNAME

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit –t unlimited

ulimit –f unlimited

ulimit –d unlimited

ulimit –s unlimited

ulimit –v unlimited

if [ -t 0 ]; then

stty intr ^C

fi

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

    . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

[grid@raclinux1 ~]$

[oracle@raclinux1 ~]$ cat .bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

umask 022

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

ORACLE_HOSTNAME=raclinux1

ORACLE_SID=RACDB1

ORACLE_UNQNAME=RACDB

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH ORACLE_HOSTNAME ORACLE_UNQNAME

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit –t unlimited

ulimit –f unlimited

ulimit –d unlimited

ulimit –s unlimited

ulimit –v unlimited

if [ -t 0 ]; then

stty intr ^C

fi

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

[oracle@raclinux1 ~]$

Note that per MOS Note 1277018.1 if installing 11.2.0.2 SELINUX need not be considered. Prior to 11.2.0.1 SELINUX had to be set to disabled. Starting with 11.2.0.1 a setting SELINUX=”permissive” is supported.

Creating and configuring the shared disks

In this section you will look at how to create and attach the shared disks from Oracle VM VirtualBox perspective. Once exposed in OEL, those disks will be formatted. Last but not least ASMlib will be configured and those disks labeled.

Initially we have the following disk devices:

[root@raclinux1 dev]# ls sd*

sda sda1 sda10 sda11 sda2 sda3 sda4 sda5 sda6 sda7 sda8 sda9

[root@raclinux1 dev]#

[root@raclinux1 dev]#

Execute the script below to create the Oracle VM VirtualBox disks.

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 1 –de

vice 0 –type hdd –medium d:\vb\Linuxasm1.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 2 –de

vice 0 –type hdd –medium d:\vb\Linuxasm2.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 3 –de

vice 0 –type hdd –medium d:\vb\Linuxasm3.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 4 –de

vice 0 –type hdd –medium d:\vb\Linuxasm4.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 5 –de

vice 0 –type hdd –medium d:\vb\Linuxasm5.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 6 –de

vice 0 –type hdd –medium d:\vb\Linuxasm6.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 7 –de

vice 0 –type hdd –medium d:\vb\Linuxasm7.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 8 –de

vice 0 –type hdd –medium d:\vb\Linuxasm8.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 9 –de

vice 0 –type hdd –medium d:\vb\Linuxasm9.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 10 –d

evice 0 –type hdd –medium d:\vb\Linuxasm10.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 11 –d

evice 0 –type hdd –medium d:\vb\Linuxasm11.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 12 –d

evice 0 –type hdd –medium d:\vb\Linuxasm12.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 13 –d

evice 0 –type hdd –medium d:\vb\Linuxasm13.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 14 –d

evice 0 –type hdd –medium d:\vb\Linuxasm14.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC1 –storagectl “SATA Controller” –port 15 –d

evice 0 –type hdd –medium d:\vb\Linuxasm15.vdi –mtype shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm1.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm2.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm3.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm4.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm5.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm6.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm7.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm8.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm9.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm10.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm11.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm12.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm13.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm14.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm15.vdi –type shareable

E:\vb>

After adding the disks we have the following devices

[root@raclinux1 dev]# ls sd*

sda sda10 sda2 sda4 sda6 sda8 sdb sdd sdf sdh sdj sdl sdn sdp

sda1 sda11 sda3 sda5 sda7 sda9 sdc sde sdg sdi sdk sdm sdo

[root@raclinux1 dev]#

Format each of the shared disks using fdisk. Repeat the steps listed below for each disk from /dev/sdb to /dev/sdp.

[root@raclinux1 dev]# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel. Changes will remain in memory only,

until you decide to write them. After that, of course, the previous

content won’t be recoverable.

The number of cylinders for this disk is set to 1305.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1305, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):

Using default value 1305

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sdb1 1 1305 10482381 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@raclinux1 dev]#

Verify the devices

[root@raclinux1 dev]# ls sd*

sda sda10 sda2 sda4 sda6 sda8 sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp

sda1 sda11 sda3 sda5 sda7 sda9 sdb1 sdc1 sdd1 sde1 sdf1 sdg1 sdh1 sdi1 sdj1 sdk1 sdl1 sdm1 sdn1 sdo1 sdp1

[root@raclinux1 dev]#

Configure ASMLib

[root@raclinux1 dev]# oracleasm configure -i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets (‘[]’). Hitting <ENTER> without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid

Default group to own the driver interface []: asmadmin

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

[root@raclinux1 dev]#

[root@raclinux1 dev]# /usr/sbin/oracleasm init

Creating /dev/oracleasm mount point: /dev/oracleasm

Loading module “oracleasm”: oracleasm

Mounting ASMlib driver filesystem: /dev/oracleasm

[root@raclinux1 dev]#

Label the shared disks

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK2 /dev/sdc1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK3 /dev/sdd1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK4 /dev/sde1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK5 /dev/sdf1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK6 /dev/sdg1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK7 /dev/sdh1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK8 /dev/sdi1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK9 /dev/sdj1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK10 /dev/sdk1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]#

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK11 /dev/sdl1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK12 /dev/sdm1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK13 /dev/sdn1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK14 /dev/sdo1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]# /usr/sbin/oracleasm createdisk DISK15 /dev/sdp1

Writing disk header: done

Instantiating disk: done

[root@raclinux1 dev]#

[root@raclinux1 dev]# /usr/sbin/oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks…

Scanning system for ASM disks…

[root@raclinux1 dev]# /usr/sbin/oracleasm listdisks

DISK1

DISK10

DISK11

DISK12

DISK13

DISK14

DISK15

DISK2

DISK3

DISK4

DISK5

DISK6

DISK7

DISK8

DISK9

[root@raclinux1 dev]#

Make sure you set proper permissions for the devices.

/etc/rc.local

chmod -R 660 /dev/oracleasm

chown -R grid:oinstall /dev/oracleasm

Cloning the VM and attaching the shared disks

Starting with Oracle VM VirtualBox 4.1.0 there are few ways to clone a VM.

• Clone the VM using the GUI
  
• Clone the VM using the clonevm
  
• Clone the disk and create a new VM using the cloned disk.
  

Clone the VM using the GUI

Right click the VM and select Clone or press CTRL+O. Enter the new name, mark the check box and press Clone. Here a new RAC3 VM is created from RAC2.

Wait for the clone to complete.

Clone the VM using the clonevm

Issue VBoxManager clonevm RAC1 –-name RAC2 where RAC1 is the name of the VM that you want cloned and -–name to specify the name of the clone, RAC2 in the case.

Clone the disk and create a new VM using the cloned disk

Clone the boot disk of the RAC1.

E:\vb\RAC1>vboxmanage clonehd e:\vb\rac1\rac1.vdi e:\vb\rac2\rac2.vdi

0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%

Clone hard disk created in format ‘VDI’. UUID: 4ee173f8-0f2e-49b2-b88a-3d4dd307816d

Create a RAC2 VM using the cloned disk.

Attach the shared disks to RAC2

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 1 –de

vice 0 –type hdd –medium d:\vb\Linuxasm1.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 2 –de

vice 0 –type hdd –medium d:\vb\Linuxasm2.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 3 –de

vice 0 –type hdd –medium d:\vb\Linuxasm3.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 4 –de

vice 0 –type hdd –medium d:\vb\Linuxasm4.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 5 –de

vice 0 –type hdd –medium d:\vb\Linuxasm5.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 6 –de

vice 0 –type hdd –medium d:\vb\Linuxasm6.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 7 –de

vice 0 –type hdd –medium d:\vb\Linuxasm7.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 8 –de

vice 0 –type hdd –medium d:\vb\Linuxasm8.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 9 –de

vice 0 –type hdd –medium d:\vb\Linuxasm9.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 10 –d

evice 0 –type hdd –medium d:\vb\Linuxasm10.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 11 –d

evice 0 –type hdd –medium d:\vb\Linuxasm11.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 12 –d

evice 0 –type hdd –medium d:\vb\Linuxasm12.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 13 –d

evice 0 –type hdd –medium d:\vb\Linuxasm13.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 14 –d

evice 0 –type hdd –medium d:\vb\Linuxasm14.vdi –mtype shareable

E:\vb>VBoxManage storageattach RAC2 –storagectl “SATA Controller” –port 15 –d

evice 0 –type hdd –medium d:\vb\Linuxasm15.vdi –mtype shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm1.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm2.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm3.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm4.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm5.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm6.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm7.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm8.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm9.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm10.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm11.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm12.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm13.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm14.vdi –type shareable

E:\vb>VBoxManage modifyhd d:\vb\Linuxasm15.vdi –type shareable

E:\vb>

Fix the network interfaces on the clone

Delete the interfaces with an extension .bak and configure the IPs as per the /etc/hosts on RAC2 to reflect the settings for raclinux2. Set the hostname to raclinux2.

Set the interfaces to correspond to the IP addresses in the /etc/hosts. For the available interfaces the following assignment will take place.

eth0 – 192.168.20.22

eth1 – 10.10.20.22

eth2 – 192.168.56.102

eth3 – dhcp

Two ways to change the IPs of the interfaces are.

• GUI tool system-config-network
  
• Changing /etc/sysconfig/network-scripts/ifcfg-eth[1-3]
  

• /etc/sysconfig/network-scripts/ifcfg-eth0 BOOTPROTO=none IPADDR=192.168.20.22
  
• /etc/sysconfig/network-scripts/ifcfg-eth1 BOOTPROTO=none IPADDR=10.10.20.22
  
• /etc/sysconfig/network-scripts/ifcfg-eth2 BOOTPROTO=none IPADDR=192.168.56.102
  
• /etc/sysconfig/network-scripts/ifcfg-eth3 BOOTPROTO=dhcp
  

Change the hostname in /etc/sysconfig/network

[root@raclinux2 network-scripts]# cat /etc/sysconfig/network

NETWORKING=yes

NETWORKING_IPV6=no

HOSTNAME=raclinux2.gj.com

NOZEROCONF=yes

[root@raclinux2 network-scripts]#

Run cluster verify to check the prerequisites

Issue the following command to verify the prerequisites for GI installation.

./runcluvfy.sh stage -pre crsinst -n raclinux1,raclinux2

The output is on Annex 1. Ignore the messages for subnet 10.0.5 as they are due to dhcp IP assignment. Ignore the messages for NTP as irrelevant for a VirtualBox setup.

Setup ssh user equivalence

There are three ways to setup user equivalence.

Using OUI

Using ./sshUserSetup.sh -user oracle -hosts ‘raclinux1 raclinux2’ –advanced

[oracle@raclinux1 sshsetup]$ ./sshUserSetup.sh -user oracle -hosts ‘raclinux1 raclinux2’ -advanced

The output of this script is also logged into /tmp/sshUserSetup_2011-07-25-17-59-41.log

Hosts are raclinux1 raclinux2

user is oracle

Platform:- Linux

Checking if the remote hosts are reachable

PING raclinux1.gj.com (192.168.20.21) 56(84) bytes of data.

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=1 ttl=64 time=0.024 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=2 ttl=64 time=0.016 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=3 ttl=64 time=0.024 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=4 ttl=64 time=0.030 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=5 ttl=64 time=0.020 ms

— raclinux1.gj.com ping statistics —

5 packets transmitted, 5 received, 0% packet loss, time 3996ms

rtt min/avg/max/mdev = 0.016/0.022/0.030/0.007 ms

PING raclinux2.gj.com (192.168.20.22) 56(84) bytes of data.

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=1 ttl=64 time=0.283 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=2 ttl=64 time=0.282 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=3 ttl=64 time=0.266 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=4 ttl=64 time=0.232 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=5 ttl=64 time=0.162 ms

— raclinux2.gj.com ping statistics —

5 packets transmitted, 5 received, 0% packet loss, time 4057ms

rtt min/avg/max/mdev = 0.162/0.245/0.283/0.045 ms

Remote host reachability check succeeded.

The following hosts are reachable: raclinux1 raclinux2.

The following hosts are not reachable: .

All hosts are reachable. Proceeding further…

firsthost raclinux1

numhosts 2

The script will setup SSH connectivity from the host raclinux1.gj.com to all

the remote hosts. After the script is executed, the user can use SSH to run

commands on the remote hosts or copy files between this host raclinux1.gj.com

and the remote hosts without being prompted for passwords or confirmations.

NOTE 1:

As part of the setup procedure, this script will use ssh and scp to copy

files between the local host and the remote hosts. Since the script does not

store passwords, you may be prompted for the passwords during the execution of

the script whenever ssh or scp is invoked.

NOTE 2:

AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY

AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE

directories.

Do you want to continue and let the script make the above mentioned changes (yes/no)?

yes

The user chose yes

Please specify if you want to specify a passphrase for the private key this script will create for the local host. Passphrase is used to encrypt the private key and makes SSH much more secure. Type ‘yes’ or ‘no’ and then press enter. In case you press ‘yes’, you would need to enter the passphrase whenever the script executes ssh or scp.

The estimated number of times the user would be prompted for a passphrase is 4. In addition, if the private-public files are also newly created, the user would have to specify the passphrase on one additional occasion.

Enter ‘yes’ or ‘no’.

yes

The user chose yes

The files containing the client public and private keys already exist on the local host. The current private key may or may not have a passphrase associated with it. In case you remember the passphrase and do not want to re-run ssh-keygen, press ‘no’ and enter. If you press ‘no’, the script will not attempt to create any new public/private key pairs. If you press ‘yes’, the script will remove the old private/public key files existing and create new ones prompting the user to enter the passphrase. If you enter ‘yes’, any previous SSH user setups would be reset. If you press ‘change’, the script will associate a new passphrase with the old keys.

Press ‘yes’, ‘no’ or ‘change’

yes

The user chose yes

Creating .ssh directory on local host, if not present already

Creating authorized_keys file on local host

Changing permissions on authorized_keys to 644 on local host

Creating known_hosts file on local host

Changing permissions on known_hosts to 644 on local host

Creating config file on local host

If a config file exists already at /home/oracle/.ssh/config, it would be backed up to /home/oracle/.ssh/config.backup.

Removing old private/public keys on local host

Running SSH keygen on local host

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Generating public/private rsa key pair.

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

76:8a:0c:8c:6f:6a:9c:9e:9a:e0:ca:b2:f0:38:6c:45 oracle@raclinux1.gj.com

Creating .ssh directory and setting permissions on remote host raclinux1

THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.

The script would create ~oracle/.ssh/config file on remote host raclinux1. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.

The user may be prompted for a password here since the script would be running SSH on host raclinux1.

Warning: Permanently added ‘raclinux1,192.168.20.21’ (RSA) to the list of known hosts.

oracle@raclinux1’s password:

Done with creating .ssh directory and setting permissions on remote host raclinux1.

Creating .ssh directory and setting permissions on remote host raclinux2

THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.

The script would create ~oracle/.ssh/config file on remote host raclinux2. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.

The user may be prompted for a password here since the script would be running SSH on host raclinux2.

Warning: Permanently added ‘raclinux2,192.168.20.22’ (RSA) to the list of known hosts.

oracle@raclinux2’s password:

Done with creating .ssh directory and setting permissions on remote host raclinux2.

Copying local host public key to the remote host raclinux1

The user may be prompted for a password or passphrase here since the script would be using SCP for host raclinux1.

oracle@raclinux1’s password:

Done copying local host public key to the remote host raclinux1

Copying local host public key to the remote host raclinux2

The user may be prompted for a password or passphrase here since the script would be using SCP for host raclinux2.

oracle@raclinux2’s password:

Done copying local host public key to the remote host raclinux2

Creating keys on remote host raclinux1 if they do not exist already. This is required to setup SSH on host raclinux1.

Creating keys on remote host raclinux2 if they do not exist already. This is required to setup SSH on host raclinux2.

Updating authorized_keys file on remote host raclinux1

Updating known_hosts file on remote host raclinux1

The script will run SSH on the remote machine raclinux1. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.

Updating authorized_keys file on remote host raclinux2

Updating known_hosts file on remote host raclinux2

The script will run SSH on the remote machine raclinux2. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.

cat: /home/oracle/.ssh/known_hosts.tmp: No such file or directory

cat: /home/oracle/.ssh/authorized_keys.tmp: No such file or directory

SSH setup is complete.

————————————————————————

Verifying SSH setup

===================

The script will now run the date command on the remote nodes using ssh

to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,

THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR

PASSWORDS. If you see any output other than date or are prompted for the

password, ssh is not setup correctly and you will need to resolve the

issue and set up ssh again.

The possible causes for failure could be:

1. The server settings in /etc/ssh/sshd_config file do not allow ssh

for user oracle.

2. The server may have disabled public key based authentication.

3. The client public key on the server may be outdated.

4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.

5. User may not have passed -shared option for shared remote users or

may be passing the -shared option for non-shared remote users.

6. If there is output in addition to the date, but no password is asked,

it may be a security alert shown as part of company policy. Append the

additional text to the <OMS HOME>/sysman/prov/resources/ignoreMessages.txt file.

————————————————————————

–raclinux1:–

Running /usr/bin/ssh -x -l oracle raclinux1 date to verify SSH connectivity has been setup from local host to raclinux1.

IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.

The script will run SSH on the remote machine raclinux1. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.

Mon Jul 25 18:00:14 EEST 2011

————————————————————————

–raclinux2:–

Running /usr/bin/ssh -x -l oracle raclinux2 date to verify SSH connectivity has been setup from local host to raclinux2.

IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.

The script will run SSH on the remote machine raclinux2. The user may be prompted for a passphrase here in case the private key has been encrypted with a passphrase.

Mon Jul 25 18:00:39 EEST 2011

————————————————————————

————————————————————————

Verifying SSH connectivity has been setup from raclinux1 to raclinux1

IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.

bash: -c: line 0: unexpected EOF while looking for matching `”‘

bash: -c: line 1: syntax error: unexpected end of file

————————————————————————

————————————————————————

Verifying SSH connectivity has been setup from raclinux1 to raclinux2

IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.

bash: -c: line 0: unexpected EOF while looking for matching `”‘

bash: -c: line 1: syntax error: unexpected end of file

————————————————————————

-Verification from complete-

SSH verification complete.

[oracle@raclinux1 sshsetup]$

[oracle@raclinux1 sshsetup]$ ./sshUserSetup.sh -user oracle -hosts “raclinux1 raclinux2” -verify

The output of this script is also logged into /tmp/sshUserSetup_2011-07-25-18-01-52.log

Hosts are raclinux1 raclinux2

user is oracle

Platform:- Linux

Checking if the remote hosts are reachable

PING raclinux1.gj.com (192.168.20.21) 56(84) bytes of data.

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=1 ttl=64 time=0.015 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=2 ttl=64 time=0.020 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=3 ttl=64 time=0.724 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=4 ttl=64 time=0.017 ms

64 bytes from raclinux1.gj.com (192.168.20.21): icmp_seq=5 ttl=64 time=0.074 ms

— raclinux1.gj.com ping statistics —

5 packets transmitted, 5 received, 0% packet loss, time 4022ms

rtt min/avg/max/mdev = 0.015/0.170/0.724/0.277 ms

PING raclinux2.gj.com (192.168.20.22) 56(84) bytes of data.

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=1 ttl=64 time=0.195 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=2 ttl=64 time=0.207 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=3 ttl=64 time=0.183 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=4 ttl=64 time=0.193 ms

64 bytes from raclinux2.gj.com (192.168.20.22): icmp_seq=5 ttl=64 time=0.322 ms

— raclinux2.gj.com ping statistics —

5 packets transmitted, 5 received, 0% packet loss, time 4026ms

rtt min/avg/max/mdev = 0.183/0.220/0.322/0.051 ms

Remote host reachability check succeeded.

The following hosts are reachable: raclinux1 raclinux2.

The following hosts are not reachable: .

All hosts are reachable. Proceeding further…

firsthost raclinux1

numhosts 2

Since user has specified -verify option, SSH setup would not be done. Only, existing SSH setup would be verified.

————————————————————————

Verifying SSH setup

===================

The script will now run the date command on the remote nodes using ssh

to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,

THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR

PASSWORDS. If you see any output other than date or are prompted for the

password, ssh is not setup correctly and you will need to resolve the

issue and set up ssh again.

The possible causes for failure could be:

1. The server settings in /etc/ssh/sshd_config file do not allow ssh

for user oracle.

2. The server may have disabled public key based authentication.

3. The client public key on the server may be outdated.

4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.

5. User may not have passed -shared option for shared remote users or

may be passing the -shared option for non-shared remote users.

6. If there is output in addition to the date, but no password is asked,

it may be a security alert shown as part of company policy. Append the

additional text to the <OMS HOME>/sysman/prov/resources/ignoreMessages.txt file.

————————————————————————

–raclinux1:–

Running /usr/bin/ssh -x -l oracle raclinux1 date to verify SSH connectivity has been setup from local host to raclinux1.

IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.

Mon Jul 25 18:02:00 EEST 2011

————————————————————————

–raclinux2:–

Running /usr/bin/ssh -x -l oracle raclinux2 date to verify SSH connectivity has been setup from local host to raclinux2.

IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.

Mon Jul 25 18:02:25 EEST 2011

————————————————————————

SSH verification complete.

[oracle@raclinux1 sshsetup]$

Using manual approach

Log in as the “oracle” user on raclinux1, generate an “authorized_keys” file on raclinux1 and copy it to raclinux2 using the following commands.

su – oracle

cd ~/.ssh

cat id_rsa.pub >> authorized_keys

scp authorized_keys raclinux2:/home/oracle/.ssh/

Next, log in as the “oracle” user on raclinux2 and perform the following commands.

su – oracle

cd ~/.ssh

cat id_rsa.pub >> authorized_keys

scp authorized_keys raclinux1:/home/oracle/.ssh/

The “authorized_keys” file on both servers now contains the public keys generated on all RAC nodes.

To enable SSH user equivalency on the cluster member nodes issue the following commands on each node.

ssh raclinux1 date

ssh raclinux2 date

ssh raclinux1.gj.com date

ssh raclinux2.gj.com date

exec /usr/bin/ssh-agent $SHELL

/usr/bin/ssh-add

You should now be able to SSH and SCP between servers without entering passwords. Repeat the same procedure for grid user.

Start OUI for GI installation.

Log in as grid user and launch the OUI. Click Next.

Select Install and Configure Oracle Grid Infrastructure and click Next.

Select Typical Installation and press Next.

Enter SCAN name. Press Add.. to add a hostname and VIP name. Press Identify Network Interconnects.

Assign eth0 as public and eth1 as private interconnects.

Press Next.

Specify CRS storage type and enter the SYSASM credentials.

Specify a disk group name, redundancy and disk members. Press Next.

The idea was of a high redundancy. So here you can choose.

Specify Oracle Inventory location. Press Next to continue.

Review and press Next to continue.

Review and press Install.

Wait until prompted to execute the scripts as root.

The output of the scripts is shown bellow.

[root@raclinux1 ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@raclinux1 ~]#

[root@raclinux2 dev]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@raclinux2 dev]#

[root@raclinux1 ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@raclinux1 ~]# /u01/app/11.2.0/grid/root.sh

Running Oracle 11g root script…

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin …

Copying oraenv to /usr/local/bin …

Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

LOCAL ADD MODE

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

OLR initialization – successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

Adding daemon to inittab

ACFS-9200: Supported

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies – this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘raclinux1’

CRS-2676: Start of ‘ora.mdnsd’ on ‘raclinux1’ succeeded

CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘raclinux1’

CRS-2676: Start of ‘ora.gpnpd’ on ‘raclinux1’ succeeded

CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘raclinux1’

CRS-2672: Attempting to start ‘ora.gipcd’ on ‘raclinux1’

CRS-2676: Start of ‘ora.cssdmonitor’ on ‘raclinux1’ succeeded

CRS-2676: Start of ‘ora.gipcd’ on ‘raclinux1’ succeeded

CRS-2672: Attempting to start ‘ora.cssd’ on ‘raclinux1’

CRS-2672: Attempting to start ‘ora.diskmon’ on ‘raclinux1’

CRS-2676: Start of ‘ora.diskmon’ on ‘raclinux1’ succeeded

CRS-2676: Start of ‘ora.cssd’ on ‘raclinux1’ succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk 9855d5f82b484f42bf07ffcb6c75f1c0.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

— —– —————– ——— ———

1. ONLINE 9855d5f82b484f42bf07ffcb6c75f1c0 (/dev/oracleasm/disks/DISK1) [DATA]

Located 1 voting disk(s).

CRS-2672: Attempting to start ‘ora.asm’ on ‘raclinux1’

CRS-2676: Start of ‘ora.asm’ on ‘raclinux1’ succeeded

CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘raclinux1’

CRS-2676: Start of ‘ora.DATA.dg’ on ‘raclinux1’ succeeded

ACFS-9200: Supported

ACFS-9200: Supported

CRS-2672: Attempting to start ‘ora.registry.acfs’ on ‘raclinux1’

CRS-2676: Start of ‘ora.registry.acfs’ on ‘raclinux1’ succeeded

Configure Oracle Grid Infrastructure for a Cluster … succeeded

[root@raclinux1 ~]#

[root@raclinux2 dev]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@raclinux2 dev]# /u01/app/11.2.0/grid/root.sh

Running Oracle 11g root script…

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin …

Copying oraenv to /usr/local/bin …

Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

LOCAL ADD MODE

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

OLR initialization – successful

Adding daemon to inittab

ACFS-9200: Supported

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies – this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node raclinux1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

Configure Oracle Grid Infrastructure for a Cluster … succeeded

[root@raclinux2 dev]#

Close.

Create a diskgroup datadg

ASMCA by default crates a diskgroup with compatible.asm set to 11.2 so as a result the diskgroup can be used for a vote disk and OCR storage. If you use another way to create a disk group make sure that you set compatible.asm attribute to at least 11.2.

Let’s create a datadg disk group with high redundancy for OCR and vote disk.

Press Create. Enter disk group name. Enter disk group disks, redundancy and compatibility attributes. Press OK. Here you can select a high redundancy as was the idea.

Review.

Optionally perform the following, to move the vote disk to a disk group with a high redundancy(five disks):

Identify the vote disk and relocate it to the +datadg disk group.

[root@raclinux1 bin]# ./crsctl query css votedisk

## STATE File Universal Id File Name Disk group

— —– —————– ——— ———

1. ONLINE 9855d5f82b484f42bf07ffcb6c75f1c0 (/dev/oracleasm/disks/DISK1) [DATA]

Located 1 voting disk(s).

[root@raclinux1 bin]# ./crsctl replace votedisk +datadg

Successful addition of voting disk 00bdc433ad1d4fbebf7b02f77e66acf3.

Successful deletion of voting disk 9855d5f82b484f42bf07ffcb6c75f1c0.

Successfully replaced voting disk group with +datadg.

CRS-4266: Voting file(s) successfully replaced

[root@raclinux1 bin]# ./crsctl query css votedisk

## STATE File Universal Id File Name Disk group

— —– —————– ——— ———

1. ONLINE 00bdc433ad1d4fbebf7b02f77e66acf3 (/dev/oracleasm/disks/DISK10) [DATADG]

Located 1 voting disk(s).

[root@raclinux1 bin]#

Add an OCR mirroring.

[root@raclinux1 bin]# ./ocrconfig -add +datadg

[root@raclinux1 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 2836

Available space (kbytes) : 259284

ID : 425271827

Device/File Name : +DATA

Device/File integrity check succeeded

Device/File Name : +datadg

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@raclinux1 bin]#

In order to create a 10g database the servers in the cluster need to be pinned.

After a fresh Oracle GI 11.2 installation all servers are unpinned. After an upgrade to 11.2 all servers are pinned. Pinning a node means that the association of a node name with a node number is fixed. If a node is not pinned, its node number may change if the lease expires while it is down. The lease of a pinned node never expires.

[grid@raclinux1 stage10g]$ olsnodes -t

raclinux1 Unpinned

raclinux2 Unpinned

[grid@raclinux1 stage10g]$

[root@raclinux1 bin]# ./crsctl pin css -n raclinux1

CRS-4664: Node raclinux1 successfully pinned.

[root@raclinux1 bin]#

[root@raclinux2 bin]# ./crsctl pin css -n raclinux2

CRS-4664: Node raclinux2 successfully pinned.

[root@raclinux2 bin]#

[grid@raclinux1 stage10g]$ olsnodes -t

raclinux1 Pinned

raclinux2 Pinned

[grid@raclinux1 stage10g]$

Install RDBMS binaries

Login as oracle user and launch the OUI.

Select to Skip the Updates and press Next.

Select Install database software only and press Next.

Select all nodes and select Oracle RAC Database installation.

Select language.

Select Enterprise edition and press Next.

Enter the software and ORACLE_BASE locations and press Next.

Select the groups and press Next.

Review and press Next.

Press Install.

Wait for the prompt to run scripts as root.

Run the scripts and press OK.

Press Close.

Create 11gR2 database

Log on as oracle user. Start the dbca.

Select Oracle RAC and press Next.

Select Create a database and press Next.

Select General Purpose and press Next.

Enter the requested information as show below. Press Next.

Select the configuration show below and press Next.

Specify the passwords. Press Next to continue.

Specify memory setting and press Next.

Press Next.

Specify Create Database and press Finish.

Review.

At the end press Exit.

Install 10.2 binaries on the 11.2.0.2 cluster

Here I will briefly summarize the steps to install Oracle 10.2 binaries and patch to 10.2.0.5. A solution for Bug 8288940, when DBCA fails to create a database if ASM is used, is in patch 8288940 included with 10.2.0.5 (MOS Note 948456.1).

Download Oracle database 10.2 from here.

Download patch 8202632 from MOS.

Download patch 12419392 from MOS.

Download patch 6880880 from MOS.

Setup a profile for 10G database

[oracle@raclinux1 ~]$ cat .bash_profile10g

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

umask 022

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1

ORACLE_HOSTNAME=raclinux1

ORACLE_SID=RAC10G1

ORACLE_UNQNAME=RAC10G

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH ORACLE_HOSTNAME ORACLE_UNQNAME

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s unlimited

ulimit -v unlimited

if [ -t 0 ]; then

stty intr ^C

fi

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

[oracle@raclinux1 ~]$

Install the base Oracle 10.2.0.1

Install patch 8202632 from Disk1 directory.

Install patch 6880880 (OPatch)

Install patch 12419392 using the downloaded OPatch.

Create 10gR2 database

Log on as oracle user. Source the 10g profile and launch the dbca.

Select RAC database. Press Next.

Select Create database press Next.

Select all nodes and press Next.

Select General Purpose and press Next.

Specify RAC10G and press Next.

Specify a configuration and press Next.

Specify password and press Next.

Specify ASM and press Next.

Select a disk group and press Next.

Select OMF and press Next.

Specify FRA and archive mode and press Next.

Specify Sample schemas.

Press Next.

Specify Memory settings and press Next to continue.

Review and press Next.

Select Create a database and press Next.

Review.

Once done press OK.

Annex 1

[grid@raclinux1 grid]$ ./runcluvfy.sh stage -pre crsinst -n raclinux1,raclinux2

Performing pre-checks for cluster services setup

Checking node reachability…

Node reachability check passed from node “raclinux1”

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Node connectivity passed for subnet “192.168.20.0” with node(s) raclinux2,raclinux1

TCP connectivity check passed for subnet “192.168.20.0”

Node connectivity passed for subnet “10.10.20.0” with node(s) raclinux2,raclinux1

TCP connectivity check passed for subnet “10.10.20.0”

Node connectivity passed for subnet “192.168.156.0” with node(s) raclinux2,raclinux1

TCP connectivity check passed for subnet “192.168.156.0”

Node connectivity passed for subnet “10.0.5.0” with node(s) raclinux2,raclinux1

ERROR:

PRVF-7617 : Node connectivity between “raclinux1 : 10.0.5.15” and “raclinux2 : 10.0.5.15” failed

TCP connectivity check failed for subnet “10.0.5.0”

Interfaces found on subnet “10.0.5.0” that are likely candidates for VIP are:

raclinux2 eth3:10.0.5.15

raclinux1 eth3:10.0.5.15

Interfaces found on subnet “192.168.20.0” that are likely candidates for a private interconnect are:

raclinux2 eth0:192.168.20.22

raclinux1 eth0:192.168.20.21

Interfaces found on subnet “10.10.20.0” that are likely candidates for a private interconnect are:

raclinux2 eth1:10.10.20.22

raclinux1 eth1:10.10.20.21

Interfaces found on subnet “192.168.156.0” that are likely candidates for a private interconnect are:

raclinux2 eth2:192.168.156.102

raclinux1 eth2:192.168.156.101

Node connectivity check failed

Checking ASMLib configuration.

Check for ASMLib configuration passed.

Total memory check passed

Available memory check passed

Swap space check passed

Free disk space check passed for “raclinux2:/tmp”

Free disk space check passed for “raclinux1:/tmp”

Check for multiple users with UID value 1100 passed

User existence check passed for “grid”

Group existence check passed for “oinstall”

Group existence check passed for “dba”

Membership check for user “grid” in group “oinstall” [as Primary] passed

Membership check for user “grid” in group “dba” passed

Run level check passed

Hard limits check passed for “maximum open file descriptors”

Soft limits check passed for “maximum open file descriptors”

Hard limits check passed for “maximum user processes”

Soft limits check passed for “maximum user processes”

System architecture check passed

Kernel version check passed

Kernel parameter check passed for “semmsl”

Kernel parameter check passed for “semmns”

Kernel parameter check passed for “semopm”

Kernel parameter check passed for “semmni”

Kernel parameter check failed for “shmmax”

Check failed on nodes:

raclinux2,raclinux1

Kernel parameter check passed for “shmmni”

Kernel parameter check passed for “shmall”

Kernel parameter check passed for “file-max”

Kernel parameter check passed for “ip_local_port_range”

Kernel parameter check passed for “rmem_default”

Kernel parameter check passed for “rmem_max”

Kernel parameter check passed for “wmem_default”

Kernel parameter check passed for “wmem_max”

Kernel parameter check passed for “aio-max-nr”

Package existence check passed for “make-3.81( x86_64)”

Package existence check passed for “binutils-2.17.50.0.6( x86_64)”

Package existence check passed for “gcc-4.1.2 (x86_64)( x86_64)”

Package existence check passed for “libaio-0.3.106 (x86_64)( x86_64)”

Package existence check passed for “glibc-2.5-24 (x86_64)( x86_64)”

Package existence check passed for “compat-libstdc++-33-3.2.3 (x86_64)( x86_64)”

Package existence check passed for “elfutils-libelf-0.125 (x86_64)( x86_64)”

Package existence check passed for “elfutils-libelf-devel-0.125( x86_64)”

Package existence check passed for “glibc-common-2.5( x86_64)”

Package existence check passed for “glibc-devel-2.5 (x86_64)( x86_64)”

Package existence check passed for “glibc-headers-2.5( x86_64)”

Package existence check passed for “gcc-c++-4.1.2 (x86_64)( x86_64)”

Package existence check passed for “libaio-devel-0.3.106 (x86_64)( x86_64)”

Package existence check passed for “libgcc-4.1.2 (x86_64)( x86_64)”

Package existence check passed for “libstdc++-4.1.2 (x86_64)( x86_64)”

Package existence check passed for “libstdc++-devel-4.1.2 (x86_64)( x86_64)”

Package existence check passed for “sysstat-7.0.2( x86_64)”

Package existence check passed for “ksh-20060214( x86_64)”

Check for multiple users with UID value 0 passed

Current group ID check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)…

NTP Configuration file check started…

NTP Configuration file check passed

Checking daemon liveness…

Liveness check passed for “ntpd”

Check for NTP daemon or service alive passed on all nodes

NTP daemon slewing option check passed

NTP daemon’s boot time configuration check for slewing option passed

NTP common Time Server Check started…

PRVF-5408 : NTP Time Server “94.26.7.48” is common only to the following nodes “raclinux1”

PRVF-5408 : NTP Time Server “132.239.1.6” is common only to the following nodes “raclinux1”

PRVF-5408 : NTP Time Server “192.33.96.102” is common only to the following nodes “raclinux2”

PRVF-5408 : NTP Time Server “85.14.12.140” is common only to the following nodes “raclinux2”

Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started…

PRVF-5413 : Node “raclinux2” has a time offset of 3634.01 that is beyond permissible limit of 1000.0 from NTP Time Server “129.69.1.153”

PRVF-5413 : Node “raclinux1” has a time offset of 1777.65 that is beyond permissible limit of 1000.0 from NTP Time Server “129.69.1.153”

Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) failed

Core file name pattern consistency check passed.

User “grid” is not part of “root” group. Check passed

Default user file creation mask check passed

Checking consistency of file “/etc/resolv.conf” across nodes

File “/etc/resolv.conf” does not have both domain and search entries defined

domain entry in file “/etc/resolv.conf” is consistent across nodes

search entry in file “/etc/resolv.conf” is consistent across nodes

The DNS response time for an unreachable node is within acceptable limit on all nodes

File “/etc/resolv.conf” is consistent across nodes

Time zone consistency check passed

Starting check for Huge Pages Existence …

Check for Huge Pages Existence passed

Starting check for Hardware Clock synchronization at shutdown …

Check for Hardware Clock synchronization at shutdown passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

[grid@raclinux1 grid]$

July 29, 2011 - Posted by gjilevski | oracle