Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6.5 using Oracle VM Virtual Box

Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6.5 using Oracle VM Virtual Box

In the article you will have a look at how to use some Oracle Virtual Box features to build two node Oracle 12c (12.1.0.1) RAC system on Oracle Enterprise Linux (OEL 6.5). The setup will implement a role separation with different users for Oracle RDBMS and Oracle GI that is, user oracle and grid are used respectively in order to split the responsibilities between DBAs and storage administrators.

An overview to Oracle Virtualization solutions can be seen here. You can see how to use Oracle VM Virtual Box to build a two node 11gR2 RAC cluster on Solaris 10 and Solaris 11. For information related to building a RAC 11gR2 cluster on OEL 5.X/6.X without GNS click here and here. For information how to build a two node Oracle RAC 11gR2 11.2.0.3 with GNS (DNS, DHCP) and HAIP look here. For information how to upgrade Oracle RAC cluster 11.2.0.3 on OEL 6.1 to Oracle RAC cluster 12c on OEL6.5 look here.

In the article you will see how to configure Linux in an Oracle VM Virtual Box Virtual machines, install Oracle GI, Oracle RDBMS and will create a RAC enabled Multitenant Container Database (CDB).

The following software will be used:

Oracle VM Virtual Box 4.3.12 – download from here.
Oracle 12c (database, grid, examples) – download from OTN here.
Oracle OEL 6.5 – download from Oracle Software Delivery Cloud (former Edelivery) here.

Two virtual machines will be created and used.

OEL65A for RAC node oel65a
OEL65B for RAC node oel65b

The OEL65A/OEL65B Virtual Machines will be configured for RAC nodes each with:

4096MB RAM
80GB bootable disk (Disk space will be with a fixed size pre-allocation using LVM)
NIC – bridged for public interface in RAC with address 192.168.2.11/12 (first IP 192.168.2.11 on oel65a and second IP 192.168.2.12 on node oel65b). These are public interface in RAC.
NIC – bridged for private interface in RAC with address 10.10.10.11/12 (first IP 10.10.10.11 on oel65a and second IP 10.10.10.12 on node oel65b). These are private interface in RAC.
NIC – bridged for private interface in RAC with address 10.10.10.71/72 (first IP 10.10.10.71 on oel65a and second IP 10.10.10.72 on node oel65b). These are private interface in RAC.
NIC – bridged for private interface in RAC with address 10.10.10.81/82 (first IP 10.10.10.81 on oel65a and second IP 10.10.10.82 on node oel65b). These are private interface in RAC.
5 10GB attached shared disks for the ASM storage. (High Redundancy ASM disk group DATA will be deployed for CRS/Vote and data files).
5 2GB attached shared disks for the ASM storage. (External Redundancy ASM disk group FRA will be deployed as flash/fast recovery area).

The interfaces’ IP addresses will be as show in the table 1 below.

	VM OEL65A RAC node oel65a	VM OEL65B RAC node oel65b
eth0	192.168.2.11	192.168.2.12
eth1	10.10.10.11	10.10.10.12
eth2	10.10.10.71	10.10.10.72
eth3	10.10.10.81	10.10.10.82

The /etc/hosts file for oel65a/oel65b will be as follows:

192.168.2.11 oel65a.gj.com oel65a

10.10.10.11 oel65a-priv.gj.com oel65a-priv

192.168.2.21 oel65a-vip.gj.com oel65a-vip

10.10.10.71 oel65a-priv1.gj.com oel65a-priv1

10.10.10.81 oel65a-prib2.gj.com oel65a-priv2

192.168.2.12 oel65b.gj.com oel65b

10.10.10.12 oel65b-priv.gj.com oel65b-priv

192.168.2.22 oel65b-vip.gj.com oel65b-vip

10.10.10.72 oel65b-priv1.gj.com oel65b-priv1

10.10.10.82 oel65b-prib2.gj.com oel65b-priv2

192.168.2.51 scan.gj.com scan

192.168.2.52 scan.gj.com scan

192.168.2.53 scan.gj.com scan

The article will cover the following topics

Create an OEL65A VM with OEL 6.5 as guest OS for node oel65a.

Configure the OEL61A VM to meet the prerequisites for GI and RAC 12c deployment.
Clone OEL61A to OEL61B and attach the shared disks.
Change IP and hostname on OEL65B
Install GI 12c on oel65a and oel65b.
Install RAC RDBMS 12c on oel65a and oel65b.
Create a Multitenant Container Database (CDB) database RACDB oel65a and oel65b.
Verify database creation and create a service.

Create an OEL65A VM with OEL 6.5 as guest OS for node oel65a

In this section you will look at how to create a guest OEL 6.5 VM using Oracle VM Virtual Box.

Select New icon or Machine->New…

Select the RAM size. The final value I used is 4096MB. Larger values turned out to cause VM instability. The error below from the VM log was avoided by reducing the VM RAM from 5000MB to 4096MB.

run OUI

00:26:18.768975 VM: Raising runtime error ‘HostMemoryLow’ (fFlags=0x2)

00:26:18.770070 AssertLogRel D:\tinderbox\win-4.3\src\VBox\VMM\VMMR3\VMM.cpp(1756) int __cdecl VMMR3EmtRendezvous(struct VM *,unsigned int,int (__cdecl *)(struct VM *,struct VMCPU *,void *),void *): !pVCpu->vmm.s.fInRendezvous

00:26:18.791552 Console: VM runtime error: fatal=false, errorID=HostMemoryLow message=”Unable to allocate and lock memory. The virtual machine will be paused. Please close applications to free up memory or close the VM”

00:26:18.825633 Changing the VM state from ‘RUNNING’ to ‘GURU_MEDITATION’.

Select Create Virtual Hard Drive now

Select the drive type VDI

Select Dynamically allocated

Select the HD size of 80G or more.

The VM is created

Add the OEL 65 ISO image so that you can boot and install the OEL 65.

Configure the Clipboard and mouse integration

Configure all adapters as bridged

Start the VM and select Install

Select Skip

Press Next

Select language

Select keyboard

Select Basic Storage Devices

Select discard to reformat

Enter hostname

Enter IP addresses, Net masks and gateways. For example eth1 you will have the configuration below. Repeat the configuration for all interfaces eth0,eth2,eth3

Click Next after configuring all interfaces ethX and the hostname.

Select a city in the time zone.

Enter root password.

Select use all space

Write the changes to disk

I am going to do a full install and will pick all rpms.

I selected everything

Wait for the install to complete.

Press Forward

Continue

You can register any time. I opted for register later.

Press Forward

Create a user.

Setup date and time

I did not enable kdump

The login screen comes up after a reboot

Install Virtual Box additions

Press Run

Wait for the installation to completes

Once installation is done power reboot the VM

1.40 This concludes OEL65 installation

Configure the OEL61A VM to meet the prerequisites for GI and RAC 12c deployment.
1. Modify the UEKR3 kernel and add divider=10 by editing /etc/grub.conf. This will speed up the OEL guest VM.
[root@oel65a ~]# cat /etc/grub.conf

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/mapper/vg_oel65a-lv_root

# initrd /initrd-[generic-]version.img

#boot=/dev/sda

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)

    root (hd0,0)

    kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

    module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet divider=10

    module /initramfs-3.8.13-16.2.1.el6uek.x86_64.img

title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64.debug)

    root (hd0,0)

    kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

    module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64.debug ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

    module /initramfs-3.8.13-16.2.1.el6uek.x86_64.debug.img

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)

    root (hd0,0)

    kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

    module /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

    module /initramfs-2.6.32-431.el6.x86_64.img

title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)

    root (hd0,0)

    kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

    module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

    module /initramfs-3.8.13-16.2.1.el6uek.x86_64.img

title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64.debug)

    root (hd0,0)

    kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

    module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64.debug ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

    module /initramfs-3.8.13-16.2.1.el6uek.x86_64.debug.img

title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)

    root (hd0,0)

    kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

    module /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

    module /initramfs-2.6.32-431.el6.x86_64.img

You have new mail in /var/spool/mail/root

[root@oel65a ~]#
1. Run the command yum install oracle-rdbms-server-12cR1-preinstall. The output is in the Annex A. Note that you need to temporary enable internet access by a) configure any interface to obtain a DHCP address and b) modify the corresponding adaptor to NAT. After successful installation revert to the original configuration.
2. If you have not install all packages make sure that the following packages are installed.
binutils-2.20.51.0.2-5.11.el6 (x86_64)

compat-libcap1-1.10-1 (x86_64)

compat-libstdc++-33-3.2.3-69.el6 (x86_64)

compat-libstdc++-33-3.2.3-69.el6.i686

gcc-4.4.4-13.el6 (x86_64)

gcc-c++-4.4.4-13.el6 (x86_64)

glibc-2.12-1.7.el6 (i686)

glibc-2.12-1.7.el6 (x86_64)

glibc-devel-2.12-1.7.el6 (x86_64)

glibc-devel-2.12-1.7.el6.i686

ksh

libgcc-4.4.4-13.el6 (i686)

libgcc-4.4.4-13.el6 (x86_64)

libstdc++-4.4.4-13.el6 (x86_64)

libstdc++-4.4.4-13.el6.i686

libstdc++-devel-4.4.4-13.el6 (x86_64)

libstdc++-devel-4.4.4-13.el6.i686

libaio-0.3.107-10.el6 (x86_64)

libaio-0.3.107-10.el6.i686

libaio-devel-0.3.107-10.el6 (x86_64)

libaio-devel-0.3.107-10.el6.i686

libXext-1.1 (x86_64)

libXext-1.1 (i686)

libXtst-1.0.99.2 (x86_64)

libXtst-1.0.99.2 (i686)

libX11-1.3 (x86_64)

libX11-1.3 (i686)

libXau-1.0.5 (x86_64)

libXau-1.0.5 (i686)

libxcb-1.5 (x86_64)

libxcb-1.5 (i686)

libXi-1.3 (x86_64)

libXi-1.3 (i686)

make-3.81-19.el6

sysstat-9.0.4-11.el6 (x86_64)
1. The oracle-rdbms-server-11gR2-preinstall takes care of the oracle user. Similarly modify /etc/security/limits.conf to address the grid user.
# oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is 1024

grid soft nofile 1024

# oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is 65536

grid hard nofile 65536

# oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is 2047

grid soft nproc 2047

# oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is 16384

grid hard nproc 16384

# oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is 10240KB

grid soft stack 10240

# oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is 32768KB

grid hard stack 32768
1. Add to /etc/pam.d/login a line session required pam_limits.so
2. Enabling the Name Service Cache Daemon
chkconfig –level 35 nscd on

service nscd start
1. Configure NTP
You have two options for time synchronization: an operating system configured network time protocol (NTP), or Oracle Cluster Time Synchronization Service. Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services. If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server. So there are two options:

For enabling the NTP make sure that /etc/sysconfig/ntpd has the line modified to include -x.

OPTIONS=”-x -u ntp:ntp -p /var/run/ntpd.pid”

For disabling the NTP make sure that the NTP service is stopped and disabled for auto-start and there is not configuration file.

/sbin/service ntpd stop

chkconfig ntpd off

mv /etc/ntp.conf to /etc/ntp.conf.org

In the article NTP is enabled.
1. Set in /etc/ssh/sshd_config the following line LoginGraceTime 0
2. Create OS groups. The masked are create by oracle-rdbms-server-11gR2-preinstall
#groupadd -g 54321 oinstall

#groupadd -g 54322 dba

groupadd -g 54323 oper

groupadd -g 54324 backupdba

groupadd -g 54325 asmdba

groupadd -g 54326 dgdba

groupadd -g 54327 kmdba

groupadd -g 54328 asmadmin

groupadd -g 54329 asmoper
1. Create users.
useradd -u 54322 -g oinstall -G asmadmin,asmdba grid

##useradd -u 54321 -g oinstall -G dba,backupdba,dgdba,kmdba,asmdba asmoper oracle

usermod -g oinstall -G dba,backupdba,dgdba,kmdba,asmdba,asmoper oracle
1. Create Directories
mkdir -p /u01/app/12.1.0/grid

mkdir -p /u01/app/grid

mkdir -p /u01/app/oracle

chown -R grid:oinstall /u01

chown oracle:oinstall /u01/app/oracle

chmod -R 775 /u01/

chown oracle:oinstall /u01/app/oracle
1. List disks
[root@oel65a dev]# ls sd*

sda sda1 sda2

[root@oel65a dev]#
1. Create shared disks
From the host OS ( Windows in my case) run the following commands.

VBoxManage createhd –filename e:\vb\asm1.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm2.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm3.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm4.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm5.vdi –size 10240 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm6.vdi –size 2048 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm7.vdi –size 2048 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm8.vdi –size 2048 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm9.vdi –size 2048 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm10.vdi –size 2048 –format VDI –variant Fixed

VBoxManage createhd –filename e:\vb\asm11.vdi –size 2048 –format VDI –variant Fixed

VBoxManage storageattach OEL65a –storagectl “SATA” –port 1 –device 0 –type hdd –medium asm1.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 2 –device 0 –type hdd –medium asm2.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 3 –device 0 –type hdd –medium asm3.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 4 –device 0 –type hdd –medium asm4.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 5 –device 0 –type hdd –medium asm5.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 6 –device 0 –type hdd –medium asm6.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 7 –device 0 –type hdd –medium asm7.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 8 –device 0 –type hdd –medium asm8.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 9 –device 0 –type hdd –medium asm9.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 10 –device 0 –type hdd –medium asm10.vdi –mtype shareable

VBoxManage storageattach OEL65a –storagectl “SATA” –port 11 –device 0 –type hdd –medium asm11.vdi –mtype shareable
1. List and format the disks
Initially you will have

[root@oel65a dev]# ls -l sd*

brw-rw—-. 1 root disk 8, 0 May 12 16:37 sda

brw-rw—-. 1 root disk 8, 1 May 12 16:37 sda1

brw-rw—-. 1 root disk 8, 2 May 12 16:37 sda2

brw-rw—-. 1 root disk 8, 16 May 12 16:37 sdb

brw-rw—-. 1 root disk 8, 32 May 12 16:37 sdc

brw-rw—-. 1 root disk 8, 48 May 12 16:37 sdd

brw-rw—-. 1 root disk 8, 64 May 12 16:37 sde

brw-rw—-. 1 root disk 8, 80 May 12 16:37 sdf

brw-rw—-. 1 root disk 8, 96 May 12 16:37 sdg

brw-rw—-. 1 root disk 8, 112 May 12 16:37 sdh

brw-rw—-. 1 root disk 8, 128 May 12 16:37 sdi

brw-rw—-. 1 root disk 8, 144 May 12 16:37 sdj

brw-rw—-. 1 root disk 8, 160 May 12 16:37 sdk

brw-rw—-. 1 root disk 8, 176 May 12 16:37 sdl

[root@oel65a dev]#

For each disk from /dev/sdb to /dev/sdl perform the following procedure.

[root@oel65a dev]# fdisk /dev/sdb

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

switch off the mode (command ‘c’) and change display units to

sectors (command ‘u’).

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1305, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305):

Using default value 1305

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0xeede9dba

Device Boot Start End Blocks Id System

/dev/sdb1 1 1305 10482381 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@oel65a dev]#

At the end you will have:

[root@oel65a dev]# ls -l /dev/sd*

brw-rw—-. 1 root disk 8, 0 May 12 16:37 /dev/sda

brw-rw—-. 1 root disk 8, 1 May 12 16:37 /dev/sda1

brw-rw—-. 1 root disk 8, 2 May 12 16:37 /dev/sda2

brw-rw—-. 1 root disk 8, 16 May 12 16:46 /dev/sdb

brw-rw—-. 1 root disk 8, 17 May 12 16:46 /dev/sdb1

brw-rw—-. 1 root disk 8, 32 May 12 16:46 /dev/sdc

brw-rw—-. 1 root disk 8, 33 May 12 16:46 /dev/sdc1

brw-rw—-. 1 root disk 8, 48 May 12 16:46 /dev/sdd

brw-rw—-. 1 root disk 8, 49 May 12 16:46 /dev/sdd1

brw-rw—-. 1 root disk 8, 64 May 12 16:47 /dev/sde

brw-rw—-. 1 root disk 8, 65 May 12 16:47 /dev/sde1

brw-rw—-. 1 root disk 8, 80 May 12 16:47 /dev/sdf

brw-rw—-. 1 root disk 8, 81 May 12 16:47 /dev/sdf1

brw-rw—-. 1 root disk 8, 96 May 12 16:47 /dev/sdg

brw-rw—-. 1 root disk 8, 97 May 12 16:47 /dev/sdg1

brw-rw—-. 1 root disk 8, 112 May 12 16:48 /dev/sdh

brw-rw—-. 1 root disk 8, 113 May 12 16:48 /dev/sdh1

brw-rw—-. 1 root disk 8, 128 May 12 16:48 /dev/sdi

brw-rw—-. 1 root disk 8, 129 May 12 16:48 /dev/sdi1

brw-rw—-. 1 root disk 8, 144 May 12 16:48 /dev/sdj

brw-rw—-. 1 root disk 8, 145 May 12 16:48 /dev/sdj1

brw-rw—-. 1 root disk 8, 160 May 12 16:49 /dev/sdk

brw-rw—-. 1 root disk 8, 161 May 12 16:49 /dev/sdk1

brw-rw—-. 1 root disk 8, 176 May 12 16:49 /dev/sdl

brw-rw—-. 1 root disk 8, 177 May 12 16:49 /dev/sdl1

[root@oel65a dev]#
1. Configure ASMlib and ASmlib disks by runnin the following commands in an ordered manner.
[root@oel65a dev]# /usr/sbin/oracleasm configure -i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets (‘[]’). Hitting <ENTER> without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid

Default group to own the driver interface []: asmadmin

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

[root@oel65a dev]#

/usr/sbin/oracleasm init

[root@oel65a dev]# /usr/sbin/oracleasm init

Creating /dev/oracleasm mount point: /dev/oracleasm

Loading module “oracleasm”: oracleasm

Configuring “oracleasm” to use device physical block size

Mounting ASMlib driver filesystem: /dev/oracleasm

[root@oel65a dev]#

/usr/sbin/oracleasm createdisk DISK1 /dev/sdb1

/usr/sbin/oracleasm createdisk DISK2 /dev/sdc1

/usr/sbin/oracleasm createdisk DISK3 /dev/sdd1

/usr/sbin/oracleasm createdisk DISK4 /dev/sde1

/usr/sbin/oracleasm createdisk DISK5 /dev/sdf1

/usr/sbin/oracleasm createdisk DISK6 /dev/sdg1

/usr/sbin/oracleasm createdisk DISK7 /dev/sdh1

/usr/sbin/oracleasm createdisk DISK8 /dev/sdi1

/usr/sbin/oracleasm createdisk DISK9 /dev/sdj1

/usr/sbin/oracleasm createdisk DISK10 /dev/sdk1

/usr/sbin/oracleasm createdisk DISK11 /dev/sdl1

[root@oel65a dev]# /usr/sbin/oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks…

Scanning system for ASM disks…

[root@oel65a dev]# /usr/sbin/oracleasm listdisks

DISK1

DISK10

DISK11

DISK2

DISK3

DISK4

DISK5

DISK6

DISK7

DISK8

DISK9

[root@oel65a dev]#
1. Setting the Disk I/O Scheduler on Linux
  
  Disk I/O schedulers reorder, delay, or merge requests for disk I/O to achieve better throughput and lower latency. Linux has multiple disk I/O schedulers available, including Deadline, Noop, Anticipatory, and Completely Fair Queuing (CFQ). For best performance for Oracle ASM, Oracle recommends that you use the Deadline I/O Scheduler.
  
  On each cluster node, enter the following command to ensure that the Deadline disk I/O scheduler is configured for use:
  
  # echo deadline > /sys/block/${ASM_DISK}/queue/scheduler
  
  echo deadline > /sys/block/sdb/queue/scheduler
  
  echo deadline > /sys/block/sdc/queue/scheduler
  
  echo deadline > /sys/block/sdd/queue/scheduler
  
  echo deadline > /sys/block/sde/queue/scheduler
  
  echo deadline > /sys/block/sdf/queue/scheduler
  
  echo deadline > /sys/block/sdg/queue/scheduler
  
  echo deadline > /sys/block/sdh/queue/scheduler
  
  echo deadline > /sys/block/sdi/queue/scheduler
  
  echo deadline > /sys/block/sdj/queue/scheduler
  
  echo deadline > /sys/block/sdk/queue/scheduler
  
  echo deadline > /sys/block/sdl/queue/scheduler
2. Disable SElinux. Edit /etc/selinux/config and enable the line
  
  SELINUX=disabled
3. Disable iptables
chkconfig iptables off
1. Install the rpm running rpm -Uvh cvuqdisk-1.0.9-1.rpm
2. Set rp filtering by seting the rp_filter of the private address to 2 (loose filtering), the public address to 1 (strict filtering), using the following entries in /etc/sysctl.conf:
  
  net.ipv4.conf.eth2.rp_filter = 2
  
  net.ipv4.conf.eth2.rp_filter = 2
  
  net.ipv4.conf.eth1.rp_filter = 1
  
  net.ipv4.conf.eth0.rp_filter = 2
3. Shutdown the VM
4. This Concludes the OEL65A preparation for 12c installation.
Clone OEL61A to OEL61B.
1. Clone the OEL65a.vdi by issuing the following command.
E:\vb>VBoxManage clonehd OEL65a.vdi OEL65b.vdi

0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%

Clone hard disk created in format ‘VDI’. UUID: 5b4a256c-d6c9-4b32-8d02-c5a1b10e3

1d5

E:\vb>
1. Create OEL65b VM and add the 4 NIC as in OEL65a
2. Attach the shared disks
VBoxManage storageattach OEL65b –storagectl “SATA” –port 1 –device 0 –type hdd –medium asm1.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 2 –device 0 –type hdd –medium asm2.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 3 –device 0 –type hdd –medium asm3.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 4 –device 0 –type hdd –medium asm4.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 5 –device 0 –type hdd –medium asm5.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 6 –device 0 –type hdd –medium asm6.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 7 –device 0 –type hdd –medium asm7.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 8 –device 0 –type hdd –medium asm8.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 9 –device 0 –type hdd –medium asm9.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 10 –device 0 –type hdd –medium asm10.vdi –mtype shareable

VBoxManage storageattach OEL65b –storagectl “SATA” –port 11 –device 0 –type hdd –medium asm11.vdi –mtype shareable
1. Startup the OEL65B VM
2. This concludes OEL65B clone
Change IP and hostname on OEL65B
1. Change the hostname in /etc/sysconfig/network
[root@oel65a ~]# cat /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=oel65b.gj.com

GATEWAY=192.168.2.1

NTPSERVERARGS=iburst

[root@oel65a ~]#
1. Change the IPs of the network interfaces
  
  Issue the following command
  
  #ifconfig –a | grep eth
  
  You will see eth4,eth5,eth6,eth7
  
  I the etc/sysconfig/network-scripts you will have
  
  [root@oel65b network-scripts]# pwd
  
  /etc/sysconfig/network-scripts
  
  [root@oel65b network-scripts]# ls -l ifcfg-eth*
  
  -rw-r–r– 1 root root 294 May 13 17:09 ifcfg-eth0
  
  -rw-r–r– 1 root root 295 May 13 11:58 ifcfg-eth1
  
  -rw-r–r– 1 root root 294 May 13 11:58 ifcfg-eth2
  
  -rw-r–r– 1 root root 294 May 13 17:10 ifcfg-eth3
  
  [root@oel65b network-scripts]#
  
  Edit the ifcfg-eth0 and modify the following lines using the HWADDR from the first line of #ifconfig –a | grep eth.
  
  HWADDR=08:00:27:4F:66:B2
  
  IPADDR=10.10.10.72
  
  Edit the ifcfg-eth1 and modify the following lines using the HWADDR from the second line of #ifconfig –a | grep eth
  
  HWADDR=08:00:27:77:2B:DB
  
  IPADDR=192.168.2.12
  
  Edit the ifcfg-eth2 and modify the following lines using the HWADDR from the third line of #ifconfig –a | grep eth
  
  HWADDR=08:00:27:BC:10:F5
  
  IPADDR=10.10.10.12
  
  Edit the ifcfg-eth3 and modify the following lines using the HWADDR from the fourth line of #ifconfig –a | grep eth
  
  HWADDR=08:00:27:BC:10:F5
  
  HWADDR=08:00:27:BC:10:F5
2. Reboot the VM
3. This concludes the network reconfiguration.

Install GI 12c on oel65a and oel65b.
1. Create .bash_proile for user grid and user oracle on both oel65a and oel65b. Look at Annex A for the profiles.
2. Run ckuvfy to make sure that the prerequisites for GI installations are met. You will find the output in the Annex A.
  
  ./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b
  
  ./runcluvfy.sh stage -post hwos -n oel65a,oel65b
3. runInstall from the grid 12c stage area and configure ssh.
4. Run OUI from the grid 12c stage area
  
  Select Skip and press Next.
Select install and configure GI for a cluster.

Select Configure a standard Cluster

Select a Typical Installation

Enter the scan name and the public and virtual hostnames.

Select the interfaces and what they are used for.

Enter the software locations and ASM passwords.

Specify the pattern to use to discover the disks

Specify the 10g disk for a high redundancy disk group.

Note that the disks are as follows

Specify the inventory

Select an option to manually run the root scripts.

Wait for the checks to complete. Review and select Ignore All and press Next.

The errors are as follows

PRVF-9802 : Attempt to get udev information from node “oel65a” failed

PRVF-9802 : Attempt to get udev information from node “oel65b” failed

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

The only note ‘INS-20802 PRVF-9802 PRVF-5184 PRVF-5186 After Successful Upgrade to 11gR2 Grid Infrastructure (Doc ID 974481.1)’ suggest that the disk should be managed by UDEV.

I wonder why ASMlib does not do the job. As I am running it using Oracle VM Virtual Box I will ignore the error.

Note : ‘PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes (Doc ID 1356975.1)’ addresses the PRVF-5636 problem. I will ignore it for the same reason as stated above.

Review and press Install.

Wait until prompted to execute scripts as root.

Execute the scripts as root.

[root@oel65a ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@oel65a ~]#

[root@oel65b ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@oel65b ~]#

[root@oel65a ~]# /u01/app/12.1.0.1/grid/root.sh

Performing root user operation for Oracle 12c

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin …

Copying oraenv to /usr/local/bin …

Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/12.1.0.1/grid/crs/install/crsconfig_params

2014/05/14 15:09:12 CLSRSC-363: User ignored prerequisites during installation

OLR initialization – successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

2014/05/14 15:10:27 CLSRSC-330: Adding Clusterware entries to file ‘oracle-ohasd.conf’

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-2672: Attempting to start ‘ora.evmd’ on ‘oel65a’

CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘oel65a’

CRS-2676: Start of ‘ora.evmd’ on ‘oel65a’ succeeded

CRS-2676: Start of ‘ora.mdnsd’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘oel65a’

CRS-2676: Start of ‘ora.gpnpd’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘oel65a’

CRS-2672: Attempting to start ‘ora.gipcd’ on ‘oel65a’

CRS-2676: Start of ‘ora.cssdmonitor’ on ‘oel65a’ succeeded

CRS-2676: Start of ‘ora.gipcd’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.cssd’ on ‘oel65a’

CRS-2672: Attempting to start ‘ora.diskmon’ on ‘oel65a’

CRS-2676: Start of ‘ora.diskmon’ on ‘oel65a’ succeeded

CRS-2676: Start of ‘ora.cssd’ on ‘oel65a’ succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

CRS-2672: Attempting to start ‘ora.storage’ on ‘oel65a’

CRS-2676: Start of ‘ora.storage’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.crsd’ on ‘oel65a’

CRS-2676: Start of ‘ora.crsd’ on ‘oel65a’ succeeded

CRS-4256: Updating the profile

Successful addition of voting disk 9d8b11d713204f34bf32e2445c8c751d.

Successful addition of voting disk f791aee91e474f58bfae1efacb87c4ce.

Successful addition of voting disk 4f5bcd1ebae74f1dbf5a7edcb2c86bef.

Successful addition of voting disk d329f30451654f08bfc6b265f4007b98.

Successful addition of voting disk 0453894fdeda4f2ebf334a123938d781.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

— —– —————– ——— ———

1. ONLINE 9d8b11d713204f34bf32e2445c8c751d (/dev/oracleasm/disks/DISK1) [DATA]

2. ONLINE f791aee91e474f58bfae1efacb87c4ce (/dev/oracleasm/disks/DISK2) [DATA]

3. ONLINE 4f5bcd1ebae74f1dbf5a7edcb2c86bef (/dev/oracleasm/disks/DISK3) [DATA]

4. ONLINE d329f30451654f08bfc6b265f4007b98 (/dev/oracleasm/disks/DISK4) [DATA]

5. ONLINE 0453894fdeda4f2ebf334a123938d781 (/dev/oracleasm/disks/DISK5) [DATA]

Located 5 voting disk(s).

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘oel65a’

CRS-2673: Attempting to stop ‘ora.crsd’ on ‘oel65a’

CRS-2677: Stop of ‘ora.crsd’ on ‘oel65a’ succeeded

CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘oel65a’

CRS-2673: Attempting to stop ‘ora.evmd’ on ‘oel65a’

CRS-2673: Attempting to stop ‘ora.storage’ on ‘oel65a’

CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘oel65a’

CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘oel65a’

CRS-2677: Stop of ‘ora.storage’ on ‘oel65a’ succeeded

CRS-2673: Attempting to stop ‘ora.asm’ on ‘oel65a’

CRS-2677: Stop of ‘ora.mdnsd’ on ‘oel65a’ succeeded

CRS-2677: Stop of ‘ora.gpnpd’ on ‘oel65a’ succeeded

CRS-2677: Stop of ‘ora.evmd’ on ‘oel65a’ succeeded

CRS-2677: Stop of ‘ora.ctssd’ on ‘oel65a’ succeeded

CRS-2677: Stop of ‘ora.asm’ on ‘oel65a’ succeeded

CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘oel65a’

CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘oel65a’ succeeded

CRS-2673: Attempting to stop ‘ora.cssd’ on ‘oel65a’

CRS-2677: Stop of ‘ora.cssd’ on ‘oel65a’ succeeded

CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘oel65a’

CRS-2677: Stop of ‘ora.gipcd’ on ‘oel65a’ succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘oel65a’ has completed

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘oel65a’

CRS-2672: Attempting to start ‘ora.evmd’ on ‘oel65a’

CRS-2676: Start of ‘ora.evmd’ on ‘oel65a’ succeeded

CRS-2676: Start of ‘ora.mdnsd’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘oel65a’

CRS-2676: Start of ‘ora.gpnpd’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.gipcd’ on ‘oel65a’

CRS-2676: Start of ‘ora.gipcd’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘oel65a’

CRS-2676: Start of ‘ora.cssdmonitor’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.cssd’ on ‘oel65a’

CRS-2672: Attempting to start ‘ora.diskmon’ on ‘oel65a’

CRS-2676: Start of ‘ora.diskmon’ on ‘oel65a’ succeeded

CRS-2789: Cannot stop resource ‘ora.diskmon’ as it is not running on server ‘oel65a’

CRS-2676: Start of ‘ora.cssd’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘oel65a’

CRS-2672: Attempting to start ‘ora.ctssd’ on ‘oel65a’

CRS-2676: Start of ‘ora.ctssd’ on ‘oel65a’ succeeded

CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.asm’ on ‘oel65a’

CRS-2676: Start of ‘ora.asm’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.storage’ on ‘oel65a’

CRS-2676: Start of ‘ora.storage’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.crsd’ on ‘oel65a’

CRS-2676: Start of ‘ora.crsd’ on ‘oel65a’ succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

CRS-6017: Processing resource auto-start for servers: oel65a

CRS-6016: Resource auto-start has completed for server oel65a

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2014/05/14 15:17:53 CLSRSC-343: Successfully started Oracle clusterware stack

CRS-2672: Attempting to start ‘ora.asm’ on ‘oel65a’

CRS-2676: Start of ‘ora.asm’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘oel65a’

CRS-2676: Start of ‘ora.DATA.dg’ on ‘oel65a’ succeeded

2014/05/14 15:19:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster … succeeded

You have new mail in /var/spool/mail/root

[root@oel65a ~]#

[root@oel65b grid]# pwd

/u01/app/12.1.0.1/grid

[root@oel65b grid]# /u01/app/12.1.0.1/grid/root.sh

Performing root user operation for Oracle 12c

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin …

Copying oraenv to /usr/local/bin …

Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/12.1.0.1/grid/crs/install/crsconfig_params

2014/05/14 15:20:54 CLSRSC-363: User ignored prerequisites during installation

OLR initialization – successful

2014/05/14 15:21:21 CLSRSC-330: Adding Clusterware entries to file ‘oracle-ohasd.conf’

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Oracle High Availability Services has been started.

CRS-4133: Oracle High Availability Services has been stopped.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘oel65b’

CRS-2672: Attempting to start ‘ora.evmd’ on ‘oel65b’

CRS-2676: Start of ‘ora.mdnsd’ on ‘oel65b’ succeeded

CRS-2676: Start of ‘ora.evmd’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘oel65b’

CRS-2676: Start of ‘ora.gpnpd’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.gipcd’ on ‘oel65b’

CRS-2676: Start of ‘ora.gipcd’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘oel65b’

CRS-2676: Start of ‘ora.cssdmonitor’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.cssd’ on ‘oel65b’

CRS-2672: Attempting to start ‘ora.diskmon’ on ‘oel65b’

CRS-2676: Start of ‘ora.diskmon’ on ‘oel65b’ succeeded

CRS-2789: Cannot stop resource ‘ora.diskmon’ as it is not running on server ‘oel65b’

CRS-2676: Start of ‘ora.cssd’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘oel65b’

CRS-2672: Attempting to start ‘ora.ctssd’ on ‘oel65b’

CRS-2676: Start of ‘ora.ctssd’ on ‘oel65b’ succeeded

CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.asm’ on ‘oel65b’

CRS-2676: Start of ‘ora.asm’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.storage’ on ‘oel65b’

CRS-2676: Start of ‘ora.storage’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.crsd’ on ‘oel65b’

CRS-2676: Start of ‘ora.crsd’ on ‘oel65b’ succeeded

CRS-6017: Processing resource auto-start for servers: oel65b

CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65a’

CRS-2672: Attempting to start ‘ora.ons’ on ‘oel65b’

CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65a’ succeeded

CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘oel65a’

CRS-2677: Stop of ‘ora.scan1.vip’ on ‘oel65a’ succeeded

CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘oel65b’

CRS-2676: Start of ‘ora.scan1.vip’ on ‘oel65b’ succeeded

CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65b’

CRS-2676: Start of ‘ora.ons’ on ‘oel65b’ succeeded

CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65b’ succeeded

CRS-6016: Resource auto-start has completed for server oel65b

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

2014/05/14 15:26:57 CLSRSC-343: Successfully started Oracle clusterware stack

2014/05/14 15:27:21 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster … succeeded

[root@oel65b grid]#

The errors can be ignored. The errors are listed below

[root@oel65a logs]# cat installActions2014-05-14_02-32-31PM.log | grep ERROR

INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-7530 : Sufficient physical memory is not available on node “oel65b” [Required physical memory = 4GB (4194304.0KB)]

INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-7530 : Sufficient physical memory is not available on node “oel65a” [Required physical memory = 4GB (4194304.0KB)]

INFO: ERRORMSG(oel65b): PRVF-7530 : Sufficient physical memory is not available on node “oel65b” [Required physical memory = 4GB (4194304.0KB)]

INFO: ERRORMSG(oel65a): PRVF-7530 : Sufficient physical memory is not available on node “oel65a” [Required physical memory = 4GB (4194304.0KB)]

INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-9802 : Attempt to get udev information from node “oel65a” failed

INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-9802 : Attempt to get udev information from node “oel65b” failed

INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65b” failed

INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65a” failed

INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65b” failed

INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65a” failed

INFO: ERRORMSG(oel65b): PRVF-9802 : Attempt to get udev information from node “oel65b” failed

INFO: ERRORMSG(oel65a): PRVF-9802 : Attempt to get udev information from node “oel65a” failed

INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: ERRORMSG(oel65b): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: ERRORMSG(oel65a): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: INFO: ERROR: [Result.addErrorDescription:607] Check for integrity of file “/etc/resolv.conf” failed

INFO: INFO: ERROR: [Result.addErrorDescription:607] Check for integrity of file “/etc/resolv.conf” failed

INFO: ERRORMSG(oel65b): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: ERRORMSG(oel65b): Check for integrity of file “/etc/resolv.conf” failed

INFO: ERRORMSG(oel65a): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

INFO: ERRORMSG(oel65a): Check for integrity of file “/etc/resolv.conf” failed

INFO: ERROR:

INFO: ERROR:

INFO: ERROR:

INFO: ERROR:

INFO: ERROR:

[root@oel65a logs]#

INFO: ERROR:

INFO: PRVG-1101 : SCAN name “scan” failed to resolve

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.51) failed

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.52) failed

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.53) failed

INFO: ERROR:

INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name “scan”

INFO: Checking SCAN IP addresses…
1. Verify that GI is installed properly
[root@oel65a bin]# ./crsctl check cluster -all

**************************************************************

oel65a:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

oel65b:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

[root@oel65a bin]#

[root@oel65a bin]# ./crsctl status resource -t

——————————————————————————–

Name Target State Server State details

——————————————————————————–

Local Resources

——————————————————————————–

ora.DATA.dg

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.LISTENER.lsnr

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.asm

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b Started,STABLE

ora.net1.network

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.ons

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE oel65b STABLE

ora.LISTENER_SCAN2.lsnr

1 ONLINE ONLINE oel65a STABLE

ora.LISTENER_SCAN3.lsnr

1 ONLINE ONLINE oel65a STABLE

ora.cvu

1 ONLINE ONLINE oel65a STABLE

ora.oc4j

1 OFFLINE OFFLINE STABLE

ora.oel65a.vip

1 ONLINE ONLINE oel65a STABLE

ora.oel65b.vip

1 ONLINE ONLINE oel65b STABLE

ora.scan1.vip

1 ONLINE ONLINE oel65b STABLE

ora.scan2.vip

1 ONLINE ONLINE oel65a STABLE

ora.scan3.vip

1 ONLINE ONLINE oel65a STABLE

——————————————————————————–

[root@oel65a bin]#
1. This concludes the GI installation.
Install RAC RDBMS 12c on oel65a and oel65b
1. Login as oracle and run the OUI to install Oracle 12c binaries only.
  
  Skip security updates.
Skip software updates.

Select Install software only.

Select RAC database.

Select all nodes.

Select languages.

Select enterprise Edition.

Verify the locations.

Verify the OS groups.

Wait for the check to complete.

The errors are listed below. I can ignore them.

Single Client Access Name (SCAN) – This test verifies the Single Client Access Name configuration. Error:

–

PRVG-1101 : SCAN name “scan” failed to resolve – Cause: An attempt to resolve specified SCAN name to a list of IP addresses failed because SCAN could not be resolved in DNS or GNS using ‘nslookup’. – Action: Check whether the specified SCAN name is correct. If SCAN name should be resolved in DNS, check the configuration of SCAN name in DNS. If it should be resolved in GNS make sure that GNS resource is online.

–

PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.51) failed – Cause: Inconsistent IP address definitions found for the SCAN name identified using DNS and configured name resolution mechanism(s). – Action: Look up the SCAN name with nslookup, and make sure the returned IP addresses are consistent with those defined in NIS and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing its cache and restarting it.

–

PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.52) failed – Cause: Inconsistent IP address definitions found for the SCAN name identified using DNS and configured name resolution mechanism(s). – Action: Look up the SCAN name with nslookup, and make sure the returned IP addresses are consistent with those defined in NIS and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing its cache and restarting it.

–

PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.53) failed – Cause: Inconsistent IP address definitions found for the SCAN name identified using DNS and configured name resolution mechanism(s). – Action: Look up the SCAN name with nslookup, and make sure the returned IP addresses are consistent with those defined in NIS and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing its cache and restarting it.

Check Failed on Nodes: [oel65b, oel65a]

Verification result of failed node: oel65b Details:

–

PRVF-4664 : Found inconsistent name resolution entries for SCAN name “scan” – Cause: The nslookup utility and the configured name resolution mechanism(s), as defined in /etc/nsswitch.conf, returned inconsistent IP address information for the SCAN name identified. – Action: Check the Name Service Cache Daemon (/usr/sbin/nscd), the Domain Name Server (nslookup) and the /etc/hosts file to make sure the IP address for the SCAN names are registered correctly.

Back to Top

Verification result of failed node: oel65a Details:

–

PRVF-4664 : Found inconsistent name resolution entries for SCAN name “scan” – Cause: The nslookup utility and the configured name resolution mechanism(s), as defined in /etc/nsswitch.conf, returned inconsistent IP address information for the SCAN name identified. – Action: Check the Name Service Cache Daemon (/usr/sbin/nscd), the Domain Name Server (nslookup) and the /etc/hosts file to make sure the IP address for the SCAN names are registered correctly.

Back to Top

Task resolv.conf Integrity – This task checks consistency of file /etc/resolv.conf file across nodes

Check Failed on Nodes: [oel65b, oel65a]

Verification result of failed node: oel65b Details:

–

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b – Cause: The DNS response time for an unreachable node exceeded the value specified on nodes specified. – Action: Make sure that ‘options timeout’, ‘options attempts’ and ‘nameserver’ entries in file resolv.conf are proper. On HPUX these entries will be ‘retrans’, ‘retry’ and ‘nameserver’. On Solaris these will be ‘options retrans’, ‘options retry’ and ‘nameserver’. Make sure that the DNS server responds back to name lookup request within the specified time when looking up an unknown host name.

–

Check for integrity of file “/etc/resolv.conf” failed – Cause: Cause Of Problem Not Available – Action: User Action Not Available

Back to Top

Verification result of failed node: oel65a Details:

–

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b – Cause: The DNS response time for an unreachable node exceeded the value specified on nodes specified. – Action: Make sure that ‘options timeout’, ‘options attempts’ and ‘nameserver’ entries in file resolv.conf are proper. On HPUX these entries will be ‘retrans’, ‘retry’ and ‘nameserver’. On Solaris these will be ‘options retrans’, ‘options retry’ and ‘nameserver’. Make sure that the DNS server responds back to name lookup request within the specified time when looking up an unknown host name.

–

Check for integrity of file “/etc/resolv.conf” failed – Cause: Cause Of Problem Not Available – Action: User Action Not Available

Back to Top

Review and start the installation.

Wait until prompted to run scripts as root.

[root@oel65a bin]# /u01/app/oracle/product/12.1.0/db_1/root.sh

Performing root user operation for Oracle 12c

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/12.1.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of “dbhome” have not changed. No need to overwrite.

The contents of “oraenv” have not changed. No need to overwrite.

The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

[root@oel65a bin]#

[root@oel65b grid]# /u01/app/oracle/product/12.1.0/db_1/root.sh

Performing root user operation for Oracle 12c

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/12.1.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of “dbhome” have not changed. No need to overwrite.

The contents of “oraenv” have not changed. No need to overwrite.

The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

[root@oel65b grid]#

Exit
1. This concludes the 12c binaries installation.
Create a Multitenant Container Database (DB) database RACDB oel65a and oel65b
1. Login as grid and start asmca to create FRA disk group
Click create and select disks for the FRA disk group. Press OK to create the group.

Wait until confirmation.

Now you have the disk groups
1. Login as oracle and start dbca.
  
  Select create a database.
Select Advanced Mode

Select options that interest you.

Select a CDB option.

Select all nodes.

Select the crosschecked options

Select a password.

Specify the storage.

Skip the Vault configurations.

Configure AMM

Configure character sets.

Select what to create.

Wait for the prerequisite checks to complete.

Review the components to be installed.

Wait for the creation to complete.

You can change the passwords. Note the URL

Login to the URL

Examine what is inside.
1. This concludes the CDB creation.
Verify database creation and create a service
1. Perform Verifications
[root@oel65a bin]# ./crsctl status resource -t

——————————————————————————–

Name Target State Server State details

——————————————————————————–

Local Resources

——————————————————————————–

ora.DATA.dg

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.FRA.dg

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.LISTENER.lsnr

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.asm

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b Started,STABLE

ora.net1.network

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.ons

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE oel65b STABLE

ora.LISTENER_SCAN2.lsnr

1 ONLINE ONLINE oel65a STABLE

ora.LISTENER_SCAN3.lsnr

1 ONLINE ONLINE oel65a STABLE

ora.cvu

1 ONLINE ONLINE oel65a STABLE

ora.db12c.db

1 ONLINE ONLINE oel65a Open,STABLE

2 ONLINE ONLINE oel65b Open,STABLE

ora.oc4j

1 OFFLINE OFFLINE STABLE

ora.oel65a.vip

1 ONLINE ONLINE oel65a STABLE

ora.oel65b.vip

1 ONLINE ONLINE oel65b STABLE

ora.scan1.vip

1 ONLINE ONLINE oel65b STABLE

ora.scan2.vip

1 ONLINE ONLINE oel65a STABLE

ora.scan3.vip

1 ONLINE ONLINE oel65a STABLE

——————————————————————————–

[root@oel65a bin]#

[oracle@oel65b ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.1.0 Production on Wed May 14 21:06:19 2014

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 – 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Advanced Analytics and Real Application Testing options

SQL> set linesize 400

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME                                                             CON_ID

———– ———————————————————————————————————————————————————————————————————————————————— ———-

     1 oel65a.gj.com:DB12C1                                                     0

     2 oel65b.gj.com:DB12C2                                                     0

SQL>

[root@oel65a bin]# ./crsctl check cluster -all

**************************************************************

oel65a:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

oel65b:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

[root@oel65a bin]# ./crsctl status resource -t

——————————————————————————–

Name Target State Server State details

——————————————————————————–

Local Resources

——————————————————————————–

ora.DATA.dg

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.FRA.dg

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.LISTENER.lsnr

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.asm

ONLINE ONLINE oel65a Started,STABLE

ONLINE ONLINE oel65b Started,STABLE

ora.net1.network

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

ora.ons

ONLINE ONLINE oel65a STABLE

ONLINE ONLINE oel65b STABLE

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE oel65b STABLE

ora.LISTENER_SCAN2.lsnr

1 ONLINE ONLINE oel65a STABLE

ora.LISTENER_SCAN3.lsnr

1 ONLINE ONLINE oel65a STABLE

ora.cvu

1 ONLINE ONLINE oel65a STABLE

ora.db12c.db

1 ONLINE ONLINE oel65a Open,STABLE

2 ONLINE ONLINE oel65b Open,STABLE

ora.oc4j

1 OFFLINE OFFLINE STABLE

ora.oel65a.vip

1 ONLINE ONLINE oel65a STABLE

ora.oel65b.vip

1 ONLINE ONLINE oel65b STABLE

ora.scan1.vip

1 ONLINE ONLINE oel65b STABLE

ora.scan2.vip

1 ONLINE ONLINE oel65a STABLE

ora.scan3.vip

1 ONLINE ONLINE oel65a STABLE

——————————————————————————–

[root@oel65a bin]#

SQL> !srvctl config database -d db12c

Database unique name: DB12C

Database name: DB12C

Oracle home: /u01/app/oracle/product/12.1.0/db_1

Oracle user: oracle

Spfile: +DATA/DB12C/spfileDB12C.ora

Password file: +DATA/DB12C/orapwdb12c

Domain:

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: DB12C

Database instances: DB12C1,DB12C2

Disk Groups: FRA,DATA

Mount point paths:

Services:

Type: RAC

Start concurrency:

Stop concurrency:

Database is administrator managed

SQL>

SQL> select dbms_xdb_config.gethttpsport from dual;

GETHTTPSPORT

————

    5500

SQL> SQL>
1. Check the PDBs
  
  [oracle@oel65a ~]$ sqlplus / as sysdba
  
  SQL*Plus: Release 12.1.0.1.0 Production on Mon May 26 21:44:30 2014
  
  Copyright (c) 1982, 2013, Oracle. All rights reserved.
  
  Connected to:
  
  Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 – 64bit Production
  
  With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
  
  Advanced Analytics and Real Application Testing options
  
  SQL> set linesize 400
  
  SQL> select * from cdb_pdbs;
  
  PDB_ID PDB_NAME                                                         DBID     CON_UID GUID                 STATUS     CREATION_SCN     CON_ID
  
  ———- ——————————————————————————————————————————– ———- ———- ——————————– ————- ———— ———-
  
       3 PDB12C1                                                         2284111037 2284111037 F95F7081C42A2554E0430B02A8C0FA31 NORMAL     1933353     1
  
       2 PDB$SEED                                                         4090173063 4090173063 F95E5546BA4A17E1E0430B02A8C01013 NORMAL     1720414     1
  
       4 PDB12C2                                                         1979843754 1979843754 F95F91A081152691E0430B02A8C0D522 NORMAL     1939488     1
  
       5 PDB12C3                                                         2029633601 2029633601 F95FADC38A482785E0430B02A8C039EC NORMAL     1944720     1
  
       6 PDB12C4                                                         2035084802 2035084802 F95FCC4A123D28DEE0430B02A8C03CBF NORMAL     1951902     1
  
       7 PDB12C5                                                         2084851122 2084851122 F95FE9ADF19D29F8E0430B02A8C08DB7 NORMAL     1958038     1
  
  6 rows selected.
  
  SQL> select * from v$pdbs;
  
  CON_ID     DBID     CON_UID GUID                 NAME                 OPEN_MODE RES OPEN_TIME CREATE_SCN TOTAL_SIZE
  
  ———- ———- ———- ——————————– —————————— ———- — ————————————————————————— ———- ———-
  
       2 4090173063 4090173063 F95E5546BA4A17E1E0430B02A8C01013 PDB$SEED             READ ONLY NO    26-MAY-14 02.13.03.875 PM                         1720414    283115520
  
       3 2284111037 2284111037 F95F7081C42A2554E0430B02A8C0FA31 PDB12C1             MOUNTED             1933353        0
  
       4 1979843754 1979843754 F95F91A081152691E0430B02A8C0D522 PDB12C2             MOUNTED             1939488        0
  
       5 2029633601 2029633601 F95FADC38A482785E0430B02A8C039EC PDB12C3             MOUNTED             1944720        0
  
       6 2035084802 2035084802 F95FCC4A123D28DEE0430B02A8C03CBF PDB12C4             MOUNTED             1951902        0
  
       7 2084851122 2084851122 F95FE9ADF19D29F8E0430B02A8C08DB7 PDB12C5             MOUNTED             1958038        0
  
  6 rows selected.
  
  SQL>
2. This concludes the CDB creation

Annex A

[root@oel65a ~]# yum install oracle-rdbms-server-12cR1-preinstall

Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock

Loading support for kernel ABI

Setting up Install Process

Resolving Dependencies

–> Running transaction check

—> Package oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-11.el6 will be installed

–> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================

Package Arch Version Repository Size

==========================================================================================================================================================================

Installing:

oracle-rdbms-server-12cR1-preinstall x86_64 1.0-11.el6 public_ol6_latest 15 k

Transaction Summary

==========================================================================================================================================================================

Install 1 Package(s)

Total download size: 15 k

Installed size: 38 k

Is this ok [y/N]: Y

Downloading Packages:

Setting up and reading Presto delta metadata

Processing delta metadata

Package(s) data still to download: 15 k

oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64.rpm | 15 kB 00:00

warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY

Retrieving key from http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Importing GPG key 0xEC551F03:

Userid: “Oracle OSS group (Open Source Software group) <build@oss.oracle.com>”

From : http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Is this ok [y/N]: Y

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Installing : oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64 1/1

Verifying : oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64 1/1

Installed:

oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-11.el6

Complete!

[root@oel65a ~]#

[grid@oel65a grid]$ ./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b

Performing pre-checks for cluster services setup

Checking node reachability…

Node reachability check passed from node “oel65a”

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Node connectivity passed for subnet “10.10.10.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “10.10.10.0”

Node connectivity passed for subnet “192.168.2.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “192.168.2.0”

Node connectivity passed for subnet “192.168.122.0” with node(s) oel65a,oel65b

ERROR:

PRVF-7617 : Node connectivity between “oel65a : 192.168.122.1” and “oel65b : 192.168.122.1” failed

TCP connectivity check failed for subnet “192.168.122.0”

Interfaces found on subnet “192.168.2.0” that are likely candidates for VIP are:

oel65a eth1:192.168.2.11

oel65b eth1:192.168.2.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth0:10.10.10.71

oel65b eth0:10.10.10.72

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth2:10.10.10.11

oel65b eth2:10.10.10.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth3:10.10.10.81

oel65b eth3:10.10.10.82

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “10.10.10.0”.

Subnet mask consistency check passed for subnet “192.168.2.0”.

Subnet mask consistency check passed for subnet “192.168.122.0”.

Subnet mask consistency check passed.

ERROR:

PRVG-1172 : The IP address “192.168.122.1” is on multiple interfaces “virbr0,virbr0” on nodes “oel65a,oel65b”

Node connectivity check failed

Checking multicast communication…

Checking subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251”…

Check of subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251” passed.

Check of multicast communication passed.

Checking ASMLib configuration.

Check for ASMLib configuration passed.

Total memory check passed

Available memory check passed

Swap space check passed

Free disk space check passed for “oel65b:/usr,oel65b:/var,oel65b:/etc,oel65b:/sbin”

Free disk space check passed for “oel65a:/usr,oel65a:/var,oel65a:/etc,oel65a:/sbin”

Free disk space check passed for “oel65b:/tmp”

Free disk space check passed for “oel65a:/tmp”

Check for multiple users with UID value 54322 passed

User existence check passed for “grid”

Group existence check passed for “oinstall”

Group existence check passed for “dba”

Membership check for user “grid” in group “oinstall” [as Primary] passed

Membership check for user “grid” in group “dba” failed

Check failed on nodes:

oel65b,oel65a

Run level check passed

Hard limits check passed for “maximum open file descriptors”

Soft limits check passed for “maximum open file descriptors”

Hard limits check passed for “maximum user processes”

Soft limits check passed for “maximum user processes”

System architecture check passed

Kernel version check passed

Kernel parameter check passed for “semmsl”

Kernel parameter check passed for “semmns”

Kernel parameter check passed for “semopm”

Kernel parameter check passed for “semmni”

Kernel parameter check passed for “shmmax”

Kernel parameter check passed for “shmmni”

Kernel parameter check passed for “shmall”

Kernel parameter check passed for “file-max”

Kernel parameter check passed for “ip_local_port_range”

Kernel parameter check passed for “rmem_default”

Kernel parameter check passed for “rmem_max”

Kernel parameter check passed for “wmem_default”

Kernel parameter check passed for “wmem_max”

Kernel parameter check passed for “aio-max-nr”

Package existence check passed for “binutils”

Package existence check passed for “compat-libcap1”

Package existence check passed for “compat-libstdc++-33(x86_64)”

Package existence check passed for “libgcc(x86_64)”

Package existence check passed for “libstdc++(x86_64)”

Package existence check passed for “libstdc++-devel(x86_64)”

Package existence check passed for “sysstat”

Package existence check passed for “gcc”

Package existence check passed for “gcc-c++”

Package existence check passed for “ksh”

Package existence check passed for “make”

Package existence check passed for “glibc(x86_64)”

Package existence check passed for “glibc-devel(x86_64)”

Package existence check passed for “libaio(x86_64)”

Package existence check passed for “libaio-devel(x86_64)”

Package existence check passed for “nfs-utils”

Checking availability of ports “6200,6100” required for component “Oracle Notification Service (ONS)”

Port availability check passed for ports “6200,6100”

Check for multiple users with UID value 0 passed

Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user’s primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)…

NTP Configuration file check started…

NTP Configuration file check passed

Checking daemon liveness…

Liveness check passed for “ntpd”

Check for NTP daemon or service alive passed on all nodes

NTP common Time Server Check started…

Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started…

Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed

Core file name pattern consistency check passed.

User “grid” is not part of “root” group. Check passed

Default user file creation mask check passed

Checking integrity of file “/etc/resolv.conf” across nodes

“domain” and “search” entries do not coexist in any “/etc/resolv.conf” file

All nodes have same “search” order defined in file “/etc/resolv.conf”

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

Check for integrity of file “/etc/resolv.conf” failed

Time zone consistency check passed

Checking integrity of name service switch configuration file “/etc/nsswitch.conf” …

All nodes have same “hosts” entry defined in file “/etc/nsswitch.conf”

Check for integrity of name service switch configuration file “/etc/nsswitch.conf” passed

Checking daemon “avahi-daemon” is not configured and running

Daemon not configured check passed for process “avahi-daemon”

Daemon not running check passed for process “avahi-daemon”

Starting check for /dev/shm mounted as temporary file system …

Check for /dev/shm mounted as temporary file system passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

[grid@oel65a grid]$

[grid@oel65a grid]$ ./runcluvfy.sh stage -post hwos -n oel65a,oel65b

Performing post-checks for hardware and operating system setup

Checking node reachability…

Node reachability check passed from node “oel65a”

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Node connectivity passed for subnet “10.10.10.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “10.10.10.0”

Node connectivity passed for subnet “192.168.2.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “192.168.2.0”

Node connectivity passed for subnet “192.168.122.0” with node(s) oel65a,oel65b

ERROR:

PRVF-7617 : Node connectivity between “oel65a : 192.168.122.1” and “oel65b : 192.168.122.1” failed

TCP connectivity check failed for subnet “192.168.122.0”

Interfaces found on subnet “192.168.2.0” that are likely candidates for VIP are:

oel65a eth1:192.168.2.11

oel65b eth1:192.168.2.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth0:10.10.10.71

oel65b eth0:10.10.10.72

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth2:10.10.10.11

oel65b eth2:10.10.10.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth3:10.10.10.81

oel65b eth3:10.10.10.82

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “10.10.10.0”.

Subnet mask consistency check passed for subnet “192.168.2.0”.

Subnet mask consistency check passed for subnet “192.168.122.0”.

Subnet mask consistency check passed.

ERROR:

PRVG-1172 : The IP address “192.168.122.1” is on multiple interfaces “virbr0,virbr0” on nodes “oel65b,oel65a”

Node connectivity check failed

Checking multicast communication…

Checking subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251”…

Check of subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251” passed.

Check of multicast communication passed.

Check for multiple users with UID value 0 passed

Time zone consistency check passed

Checking shared storage accessibility…