Guenadi N Jilevski's Oracle BLOG

Oracle RAC, DG, EBS, DR and HA DBA BLOG

Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6.5 using Oracle VM Virtual Box

Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6.5 using Oracle VM Virtual Box

In the article you will have a look at how to use some Oracle Virtual Box features to build two node Oracle 12c (12.1.0.1) RAC system on Oracle Enterprise Linux (OEL 6.5). The setup will implement a role separation with different users for Oracle RDBMS and Oracle GI that is, user oracle and grid are used respectively in order to split the responsibilities between DBAs and storage administrators.

An overview to Oracle Virtualization solutions can be seen here. You can see how to use Oracle VM Virtual Box to build a two node 11gR2 RAC cluster on Solaris 10 and Solaris 11. For information related to building a RAC 11gR2 cluster on OEL 5.X/6.X without GNS click here and here. For information how to build a two node Oracle RAC 11gR2 11.2.0.3 with GNS (DNS, DHCP) and HAIP look here. For information how to upgrade Oracle RAC cluster 11.2.0.3 on OEL 6.1 to Oracle RAC cluster 12c on OEL6.5 look here.

In the article you will see how to configure Linux in an Oracle VM Virtual Box Virtual machines, install Oracle GI, Oracle RDBMS and will create a RAC enabled Multitenant Container Database (CDB).

The following software will be used:

  1. Oracle VM Virtual Box 4.3.12 – download from here.
  2. Oracle 12c (database, grid, examples) – download from OTN here.
  3. Oracle OEL 6.5 – download from Oracle Software Delivery Cloud (former Edelivery) here.

Two virtual machines will be created and used.

  1. OEL65A for RAC node oel65a
  2. OEL65B for RAC node oel65b

The OEL65A/OEL65B Virtual Machines will be configured for RAC nodes each with:

  • 4096MB RAM
  • 80GB bootable disk (Disk space will be with a fixed size pre-allocation using LVM)
  • NIC – bridged for public interface in RAC with address 192.168.2.11/12 (first IP 192.168.2.11 on oel65a and second IP 192.168.2.12 on node oel65b). These are public interface in RAC.
  • NIC – bridged for private interface in RAC with address 10.10.10.11/12 (first IP 10.10.10.11 on oel65a and second IP 10.10.10.12 on node oel65b). These are private interface in RAC.
  • NIC – bridged for private interface in RAC with address 10.10.10.71/72 (first IP 10.10.10.71 on oel65a and second IP 10.10.10.72 on node oel65b). These are private interface in RAC.
  • NIC – bridged for private interface in RAC with address 10.10.10.81/82 (first IP 10.10.10.81 on oel65a and second IP 10.10.10.82 on node oel65b). These are private interface in RAC.
  • 5 10GB attached shared disks for the ASM storage. (High Redundancy ASM disk group DATA will be deployed for CRS/Vote and data files).
  • 5 2GB attached shared disks for the ASM storage. (External Redundancy ASM disk group FRA will be deployed as flash/fast recovery area).

The interfaces’ IP addresses will be as show in the table 1 below.

VM OEL65A RAC node oel65a VM OEL65B RAC node oel65b
eth0 192.168.2.11 192.168.2.12
eth1 10.10.10.11 10.10.10.12
eth2 10.10.10.71 10.10.10.72
eth3 10.10.10.81 10.10.10.82

 

The /etc/hosts file for oel65a/oel65b will be as follows:

192.168.2.11 oel65a.gj.com oel65a

10.10.10.11 oel65a-priv.gj.com oel65a-priv

192.168.2.21 oel65a-vip.gj.com oel65a-vip

10.10.10.71 oel65a-priv1.gj.com oel65a-priv1

10.10.10.81 oel65a-prib2.gj.com oel65a-priv2

192.168.2.12 oel65b.gj.com oel65b

10.10.10.12 oel65b-priv.gj.com oel65b-priv

192.168.2.22 oel65b-vip.gj.com oel65b-vip

10.10.10.72 oel65b-priv1.gj.com oel65b-priv1

10.10.10.82 oel65b-prib2.gj.com oel65b-priv2

192.168.2.51 scan.gj.com scan

192.168.2.52 scan.gj.com scan

192.168.2.53 scan.gj.com scan

 

The article will cover the following topics

  1. Create an OEL65A VM with OEL 6.5 as guest OS for node oel65a.
  1. Configure the OEL61A VM to meet the prerequisites for GI and RAC 12c deployment.
  2. Clone OEL61A to OEL61B and attach the shared disks.
  3. Change IP and hostname on OEL65B
  4. Install GI 12c on oel65a and oel65b.
  5. Install RAC RDBMS 12c on oel65a and oel65b.
  6. Create a Multitenant Container Database (CDB) database RACDB oel65a and oel65b.
  7. Verify database creation and create a service.

 

  1. Create an OEL65A VM with OEL 6.5 as guest OS for node oel65a

In this section you will look at how to create a guest OEL 6.5 VM using Oracle VM Virtual Box.

  1. Select New icon or Machine->New…

 


  1. Select the RAM size. The final value I used is 4096MB. Larger values turned out to cause VM instability. The error below from the VM log was avoided by reducing the VM RAM from 5000MB to 4096MB.

 

run OUI

00:26:18.768975 VM: Raising runtime error ‘HostMemoryLow’ (fFlags=0x2)

00:26:18.770070 AssertLogRel D:\tinderbox\win-4.3\src\VBox\VMM\VMMR3\VMM.cpp(1756) int __cdecl VMMR3EmtRendezvous(struct VM *,unsigned int,int (__cdecl *)(struct VM *,struct VMCPU *,void *),void *): !pVCpu->vmm.s.fInRendezvous

00:26:18.791552 Console: VM runtime error: fatal=false, errorID=HostMemoryLow message=”Unable to allocate and lock memory. The virtual machine will be paused. Please close applications to free up memory or close the VM”

00:26:18.825633 Changing the VM state from ‘RUNNING’ to ‘GURU_MEDITATION’.


  1. Select Create Virtual Hard Drive now

 


  1. Select the drive type VDI

 


  1. Select Dynamically allocated

 


  1. Select the HD size of 80G or more.

 


  1. The VM is created

 


  1. Add the OEL 65 ISO image so that you can boot and install the OEL 65.

 


  1. Configure the Clipboard and mouse integration

 


  1. Configure all adapters as bridged

 


  1. Start the VM and select Install

 


  1. Select Skip

 


  1. Press Next

 


  1. Select language

 


  1. Select keyboard

 


  1. Select Basic Storage Devices

 


  1. Select discard to reformat

 


  1. Enter hostname

 


  1. Enter IP addresses, Net masks and gateways. For example eth1 you will have the configuration below. Repeat the configuration for all interfaces eth0,eth2,eth3

 


  1. Click Next after configuring all interfaces ethX and the hostname.

 


  1. Select a city in the time zone.

 


  1. Enter root password.

 


  1. Select use all space

 


  1. Write the changes to disk

 


  1. I am going to do a full install and will pick all rpms.

 


  1. I selected everything

 


  1. Wait for the install to complete.

 


  1. Press Forward

 


  1. Continue

 


  1. You can register any time. I opted for register later.

 


  1. Press Forward

 


  1. Create a user.

 


  1. Setup date and time

 


  1. I did not enable kdump

 


  1. The login screen comes up after a reboot

 


  1. Install Virtual Box additions

 


  1. Press Run

 


  1. Wait for the installation to completes

 


  1. Once installation is done power reboot the VM

 


1.40 This concludes OEL65 installation

  1. Configure the OEL61A VM to meet the prerequisites for GI and RAC 12c deployment.

    1. Modify the UEKR3 kernel and add divider=10 by editing /etc/grub.conf. This will speed up the OEL guest VM.

    [root@oel65a ~]# cat /etc/grub.conf

    # grub.conf generated by anaconda

    #

    # Note that you do not have to rerun grub after making changes to this file

    # NOTICE: You have a /boot partition. This means that

    # all kernel and initrd paths are relative to /boot/, eg.

    # root (hd0,0)

    # kernel /vmlinuz-version ro root=/dev/mapper/vg_oel65a-lv_root

    # initrd /initrd-[generic-]version.img

    #boot=/dev/sda

    default=0

    timeout=5

    splashimage=(hd0,0)/grub/splash.xpm.gz

    hiddenmenu

    title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)

        root (hd0,0)

        kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

        module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet divider=10

        module /initramfs-3.8.13-16.2.1.el6uek.x86_64.img

    title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64.debug)

        root (hd0,0)

        kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

        module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64.debug ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

        module /initramfs-3.8.13-16.2.1.el6uek.x86_64.debug.img

    title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)

        root (hd0,0)

        kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

        module /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

        module /initramfs-2.6.32-431.el6.x86_64.img

    title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64)

        root (hd0,0)

        kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

        module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

        module /initramfs-3.8.13-16.2.1.el6uek.x86_64.img

    title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-16.2.1.el6uek.x86_64.debug)

        root (hd0,0)

        kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

        module /vmlinuz-3.8.13-16.2.1.el6uek.x86_64.debug ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

        module /initramfs-3.8.13-16.2.1.el6uek.x86_64.debug.img

    title Oracle Linux Server Red Hat Compatible Kernel (2.6.32-431.el6.x86_64)

        root (hd0,0)

        kernel /tboot.gz logging=vga,serial,memory numa=off transparent_hugepage=never

        module /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_oel65a-lv_root intel_iommu=on amd_iommu=on rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 rd_LVM_LV=vg_oel65a/lv_swap crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg_oel65a/lv_root rd_NO_DM rhgb quiet

        module /initramfs-2.6.32-431.el6.x86_64.img

    You have new mail in /var/spool/mail/root

    [root@oel65a ~]#

    1. Run the command yum install oracle-rdbms-server-12cR1-preinstall. The output is in the Annex A. Note that you need to temporary enable internet access by a) configure any interface to obtain a DHCP address and b) modify the corresponding adaptor to NAT. After successful installation revert to the original configuration.
    2. If you have not install all packages make sure that the following packages are installed.

    binutils-2.20.51.0.2-5.11.el6 (x86_64)

    compat-libcap1-1.10-1 (x86_64)

    compat-libstdc++-33-3.2.3-69.el6 (x86_64)

    compat-libstdc++-33-3.2.3-69.el6.i686

    gcc-4.4.4-13.el6 (x86_64)

    gcc-c++-4.4.4-13.el6 (x86_64)

    glibc-2.12-1.7.el6 (i686)

    glibc-2.12-1.7.el6 (x86_64)

    glibc-devel-2.12-1.7.el6 (x86_64)

    glibc-devel-2.12-1.7.el6.i686

    ksh

    libgcc-4.4.4-13.el6 (i686)

    libgcc-4.4.4-13.el6 (x86_64)

    libstdc++-4.4.4-13.el6 (x86_64)

    libstdc++-4.4.4-13.el6.i686

    libstdc++-devel-4.4.4-13.el6 (x86_64)

    libstdc++-devel-4.4.4-13.el6.i686

    libaio-0.3.107-10.el6 (x86_64)

    libaio-0.3.107-10.el6.i686

    libaio-devel-0.3.107-10.el6 (x86_64)

    libaio-devel-0.3.107-10.el6.i686

    libXext-1.1 (x86_64)

    libXext-1.1 (i686)

    libXtst-1.0.99.2 (x86_64)

    libXtst-1.0.99.2 (i686)

    libX11-1.3 (x86_64)

    libX11-1.3 (i686)

    libXau-1.0.5 (x86_64)

    libXau-1.0.5 (i686)

    libxcb-1.5 (x86_64)

    libxcb-1.5 (i686)

    libXi-1.3 (x86_64)

    libXi-1.3 (i686)

    make-3.81-19.el6

    sysstat-9.0.4-11.el6 (x86_64)

    1. The oracle-rdbms-server-11gR2-preinstall takes care of the oracle user. Similarly modify /etc/security/limits.conf to address the grid user.

    # oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is 1024

    grid soft nofile 1024

    # oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is 65536

    grid hard nofile 65536

    # oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is 2047

    grid soft nproc 2047

    # oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is 16384

    grid hard nproc 16384

    # oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is 10240KB

    grid soft stack 10240

    # oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is 32768KB

    grid hard stack 32768

    1. Add to /etc/pam.d/login a line session required pam_limits.so
    2. Enabling the Name Service Cache Daemon

    chkconfig –level 35 nscd on

    service nscd start

    1. Configure NTP

    You have two options for time synchronization: an operating system configured network time protocol (NTP), or Oracle Cluster Time Synchronization Service. Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services. If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server. So there are two options:

    For enabling the NTP make sure that /etc/sysconfig/ntpd has the line modified to include -x.

    OPTIONS=”-x -u ntp:ntp -p /var/run/ntpd.pid”

    For disabling the NTP make sure that the NTP service is stopped and disabled for auto-start and there is not configuration file.

    /sbin/service ntpd stop

    chkconfig ntpd off

    mv /etc/ntp.conf to /etc/ntp.conf.org

    In the article NTP is enabled.

    1. Set in /etc/ssh/sshd_config the following line LoginGraceTime 0
    2. Create OS groups. The masked are create by oracle-rdbms-server-11gR2-preinstall

    #groupadd -g 54321 oinstall

    #groupadd -g 54322 dba

    groupadd -g 54323 oper

    groupadd -g 54324 backupdba

    groupadd -g 54325 asmdba

    groupadd -g 54326 dgdba

    groupadd -g 54327 kmdba

    groupadd -g 54328 asmadmin

    groupadd -g 54329 asmoper

    1. Create users.

    useradd -u 54322 -g oinstall -G asmadmin,asmdba grid

    ##useradd -u 54321 -g oinstall -G dba,backupdba,dgdba,kmdba,asmdba asmoper oracle

    usermod -g oinstall -G dba,backupdba,dgdba,kmdba,asmdba,asmoper oracle

    1. Create Directories

    mkdir -p /u01/app/12.1.0/grid

    mkdir -p /u01/app/grid

    mkdir -p /u01/app/oracle

    chown -R grid:oinstall /u01

    chown oracle:oinstall /u01/app/oracle

    chmod -R 775 /u01/

    chown oracle:oinstall /u01/app/oracle

    1. List disks

    [root@oel65a dev]# ls sd*

    sda sda1 sda2

    [root@oel65a dev]#

    1. Create shared disks

    From the host OS ( Windows in my case) run the following commands.

    VBoxManage createhd –filename e:\vb\asm1.vdi –size 10240 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm2.vdi –size 10240 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm3.vdi –size 10240 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm4.vdi –size 10240 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm5.vdi –size 10240 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm6.vdi –size 2048 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm7.vdi –size 2048 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm8.vdi –size 2048 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm9.vdi –size 2048 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm10.vdi –size 2048 –format VDI –variant Fixed

    VBoxManage createhd –filename e:\vb\asm11.vdi –size 2048 –format VDI –variant Fixed

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 1 –device 0 –type hdd –medium asm1.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 2 –device 0 –type hdd –medium asm2.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 3 –device 0 –type hdd –medium asm3.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 4 –device 0 –type hdd –medium asm4.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 5 –device 0 –type hdd –medium asm5.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 6 –device 0 –type hdd –medium asm6.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 7 –device 0 –type hdd –medium asm7.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 8 –device 0 –type hdd –medium asm8.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 9 –device 0 –type hdd –medium asm9.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 10 –device 0 –type hdd –medium asm10.vdi –mtype shareable

    VBoxManage storageattach OEL65a –storagectl “SATA” –port 11 –device 0 –type hdd –medium asm11.vdi –mtype shareable

    1. List and format the disks

    Initially you will have

    [root@oel65a dev]# ls -l sd*

    brw-rw—-. 1 root disk 8, 0 May 12 16:37 sda

    brw-rw—-. 1 root disk 8, 1 May 12 16:37 sda1

    brw-rw—-. 1 root disk 8, 2 May 12 16:37 sda2

    brw-rw—-. 1 root disk 8, 16 May 12 16:37 sdb

    brw-rw—-. 1 root disk 8, 32 May 12 16:37 sdc

    brw-rw—-. 1 root disk 8, 48 May 12 16:37 sdd

    brw-rw—-. 1 root disk 8, 64 May 12 16:37 sde

    brw-rw—-. 1 root disk 8, 80 May 12 16:37 sdf

    brw-rw—-. 1 root disk 8, 96 May 12 16:37 sdg

    brw-rw—-. 1 root disk 8, 112 May 12 16:37 sdh

    brw-rw—-. 1 root disk 8, 128 May 12 16:37 sdi

    brw-rw—-. 1 root disk 8, 144 May 12 16:37 sdj

    brw-rw—-. 1 root disk 8, 160 May 12 16:37 sdk

    brw-rw—-. 1 root disk 8, 176 May 12 16:37 sdl

    [root@oel65a dev]#

    For each disk from /dev/sdb to /dev/sdl perform the following procedure.

    [root@oel65a dev]# fdisk /dev/sdb

    WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

    switch off the mode (command ‘c’) and change display units to

    sectors (command ‘u’).

    Command (m for help): n

    Command action

    e extended

    p primary partition (1-4)

    p

    Partition number (1-4): 1

    First cylinder (1-1305, default 1):

    Using default value 1

    Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305):

    Using default value 1305

    Command (m for help): p

    Disk /dev/sdb: 10.7 GB, 10737418240 bytes

    255 heads, 63 sectors/track, 1305 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

    Sector size (logical/physical): 512 bytes / 512 bytes

    I/O size (minimum/optimal): 512 bytes / 512 bytes

    Disk identifier: 0xeede9dba

    Device Boot Start End Blocks Id System

    /dev/sdb1 1 1305 10482381 83 Linux

    Command (m for help): w

    The partition table has been altered!

    Calling ioctl() to re-read partition table.

    Syncing disks.

    [root@oel65a dev]#

    At the end you will have:

    [root@oel65a dev]# ls -l /dev/sd*

    brw-rw—-. 1 root disk 8, 0 May 12 16:37 /dev/sda

    brw-rw—-. 1 root disk 8, 1 May 12 16:37 /dev/sda1

    brw-rw—-. 1 root disk 8, 2 May 12 16:37 /dev/sda2

    brw-rw—-. 1 root disk 8, 16 May 12 16:46 /dev/sdb

    brw-rw—-. 1 root disk 8, 17 May 12 16:46 /dev/sdb1

    brw-rw—-. 1 root disk 8, 32 May 12 16:46 /dev/sdc

    brw-rw—-. 1 root disk 8, 33 May 12 16:46 /dev/sdc1

    brw-rw—-. 1 root disk 8, 48 May 12 16:46 /dev/sdd

    brw-rw—-. 1 root disk 8, 49 May 12 16:46 /dev/sdd1

    brw-rw—-. 1 root disk 8, 64 May 12 16:47 /dev/sde

    brw-rw—-. 1 root disk 8, 65 May 12 16:47 /dev/sde1

    brw-rw—-. 1 root disk 8, 80 May 12 16:47 /dev/sdf

    brw-rw—-. 1 root disk 8, 81 May 12 16:47 /dev/sdf1

    brw-rw—-. 1 root disk 8, 96 May 12 16:47 /dev/sdg

    brw-rw—-. 1 root disk 8, 97 May 12 16:47 /dev/sdg1

    brw-rw—-. 1 root disk 8, 112 May 12 16:48 /dev/sdh

    brw-rw—-. 1 root disk 8, 113 May 12 16:48 /dev/sdh1

    brw-rw—-. 1 root disk 8, 128 May 12 16:48 /dev/sdi

    brw-rw—-. 1 root disk 8, 129 May 12 16:48 /dev/sdi1

    brw-rw—-. 1 root disk 8, 144 May 12 16:48 /dev/sdj

    brw-rw—-. 1 root disk 8, 145 May 12 16:48 /dev/sdj1

    brw-rw—-. 1 root disk 8, 160 May 12 16:49 /dev/sdk

    brw-rw—-. 1 root disk 8, 161 May 12 16:49 /dev/sdk1

    brw-rw—-. 1 root disk 8, 176 May 12 16:49 /dev/sdl

    brw-rw—-. 1 root disk 8, 177 May 12 16:49 /dev/sdl1

    [root@oel65a dev]#

    1. Configure ASMlib and ASmlib disks by runnin the following commands in an ordered manner.

    [root@oel65a dev]# /usr/sbin/oracleasm configure -i

    Configuring the Oracle ASM library driver.

    This will configure the on-boot properties of the Oracle ASM library

    driver. The following questions will determine whether the driver is

    loaded on boot and what permissions it will have. The current values

    will be shown in brackets (‘[]’). Hitting <ENTER> without typing an

    answer will keep that current value. Ctrl-C will abort.

    Default user to own the driver interface []: grid

    Default group to own the driver interface []: asmadmin

    Start Oracle ASM library driver on boot (y/n) [n]: y

    Scan for Oracle ASM disks on boot (y/n) [y]: y

    Writing Oracle ASM library driver configuration: done

    [root@oel65a dev]#

    /usr/sbin/oracleasm init

    [root@oel65a dev]# /usr/sbin/oracleasm init

    Creating /dev/oracleasm mount point: /dev/oracleasm

    Loading module “oracleasm”: oracleasm

    Configuring “oracleasm” to use device physical block size

    Mounting ASMlib driver filesystem: /dev/oracleasm

    [root@oel65a dev]#

    /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1

    /usr/sbin/oracleasm createdisk DISK2 /dev/sdc1

    /usr/sbin/oracleasm createdisk DISK3 /dev/sdd1

    /usr/sbin/oracleasm createdisk DISK4 /dev/sde1

    /usr/sbin/oracleasm createdisk DISK5 /dev/sdf1

    /usr/sbin/oracleasm createdisk DISK6 /dev/sdg1

    /usr/sbin/oracleasm createdisk DISK7 /dev/sdh1

    /usr/sbin/oracleasm createdisk DISK8 /dev/sdi1

    /usr/sbin/oracleasm createdisk DISK9 /dev/sdj1

    /usr/sbin/oracleasm createdisk DISK10 /dev/sdk1

    /usr/sbin/oracleasm createdisk DISK11 /dev/sdl1

    [root@oel65a dev]# /usr/sbin/oracleasm scandisks

    Reloading disk partitions: done

    Cleaning any stale ASM disks…

    Scanning system for ASM disks…

    [root@oel65a dev]# /usr/sbin/oracleasm listdisks

    DISK1

    DISK10

    DISK11

    DISK2

    DISK3

    DISK4

    DISK5

    DISK6

    DISK7

    DISK8

    DISK9

    [root@oel65a dev]#

    1. Setting the Disk I/O Scheduler on Linux

       

      Disk I/O schedulers reorder, delay, or merge requests for disk I/O to achieve better throughput and lower latency. Linux has multiple disk I/O schedulers available, including Deadline, Noop, Anticipatory, and Completely Fair Queuing (CFQ). For best performance for Oracle ASM, Oracle recommends that you use the Deadline I/O Scheduler.

       

      On each cluster node, enter the following command to ensure that the Deadline disk I/O scheduler is configured for use:

       

      # echo deadline > /sys/block/${ASM_DISK}/queue/scheduler

       

      echo deadline > /sys/block/sdb/queue/scheduler

      echo deadline > /sys/block/sdc/queue/scheduler

      echo deadline > /sys/block/sdd/queue/scheduler

      echo deadline > /sys/block/sde/queue/scheduler

      echo deadline > /sys/block/sdf/queue/scheduler

      echo deadline > /sys/block/sdg/queue/scheduler

      echo deadline > /sys/block/sdh/queue/scheduler

      echo deadline > /sys/block/sdi/queue/scheduler

      echo deadline > /sys/block/sdj/queue/scheduler

      echo deadline > /sys/block/sdk/queue/scheduler

      echo deadline > /sys/block/sdl/queue/scheduler

       

    2. Disable SElinux. Edit /etc/selinux/config and enable the line

       

      SELINUX=disabled

       

    3. Disable iptables

       

    chkconfig iptables off

    1. Install the rpm running rpm -Uvh cvuqdisk-1.0.9-1.rpm
    2. Set rp filtering by seting the rp_filter of the private address to 2 (loose filtering), the public address to 1 (strict filtering), using the following entries in /etc/sysctl.conf:

       

      net.ipv4.conf.eth2.rp_filter = 2

      net.ipv4.conf.eth2.rp_filter = 2

      net.ipv4.conf.eth1.rp_filter = 1

      net.ipv4.conf.eth0.rp_filter = 2

       

       

       

    3. Shutdown the VM
    4. This Concludes the OEL65A preparation for 12c installation.

       

  2. Clone OEL61A to OEL61B.

     

    1. Clone the OEL65a.vdi by issuing the following command.

       

    E:\vb>VBoxManage clonehd OEL65a.vdi OEL65b.vdi

    0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%

    Clone hard disk created in format ‘VDI’. UUID: 5b4a256c-d6c9-4b32-8d02-c5a1b10e3

    1d5

    E:\vb>

    1. Create OEL65b VM and add the 4 NIC as in OEL65a
    2. Attach the shared disks

       

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 1 –device 0 –type hdd –medium asm1.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 2 –device 0 –type hdd –medium asm2.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 3 –device 0 –type hdd –medium asm3.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 4 –device 0 –type hdd –medium asm4.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 5 –device 0 –type hdd –medium asm5.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 6 –device 0 –type hdd –medium asm6.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 7 –device 0 –type hdd –medium asm7.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 8 –device 0 –type hdd –medium asm8.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 9 –device 0 –type hdd –medium asm9.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 10 –device 0 –type hdd –medium asm10.vdi –mtype shareable

    VBoxManage storageattach OEL65b –storagectl “SATA” –port 11 –device 0 –type hdd –medium asm11.vdi –mtype shareable

    1. Startup the OEL65B VM
    2. This concludes OEL65B clone

       

  3. Change IP and hostname on OEL65B

     

    1. Change the hostname in /etc/sysconfig/network

       

    [root@oel65a ~]# cat /etc/sysconfig/network

    NETWORKING=yes

    HOSTNAME=oel65b.gj.com

    GATEWAY=192.168.2.1

    NTPSERVERARGS=iburst

    [root@oel65a ~]#

    1. Change the IPs of the network interfaces

       

      Issue the following command

       

      #ifconfig –a | grep eth

       

      You will see eth4,eth5,eth6,eth7

       

      I the etc/sysconfig/network-scripts you will have

       

      [root@oel65b network-scripts]# pwd

      /etc/sysconfig/network-scripts

      [root@oel65b network-scripts]# ls -l ifcfg-eth*

      -rw-r–r– 1 root root 294 May 13 17:09 ifcfg-eth0

      -rw-r–r– 1 root root 295 May 13 11:58 ifcfg-eth1

      -rw-r–r– 1 root root 294 May 13 11:58 ifcfg-eth2

      -rw-r–r– 1 root root 294 May 13 17:10 ifcfg-eth3

      [root@oel65b network-scripts]#

      Edit the ifcfg-eth0 and modify the following lines using the HWADDR from the first line of #ifconfig –a | grep eth.

       

       

      HWADDR=08:00:27:4F:66:B2

      IPADDR=10.10.10.72

       

      Edit the ifcfg-eth1 and modify the following lines using the HWADDR from the second line of #ifconfig –a | grep eth

       

       

      HWADDR=08:00:27:77:2B:DB

      IPADDR=192.168.2.12

       

      Edit the ifcfg-eth2 and modify the following lines using the HWADDR from the third line of #ifconfig –a | grep eth

       

       

      HWADDR=08:00:27:BC:10:F5

      IPADDR=10.10.10.12

       

      Edit the ifcfg-eth3 and modify the following lines using the HWADDR from the fourth line of #ifconfig –a | grep eth

       

       

      HWADDR=08:00:27:BC:10:F5

      HWADDR=08:00:27:BC:10:F5

       

       

       

       

    2. Reboot the VM
    3. This concludes the network reconfiguration.

 

  1. Install GI 12c on oel65a and oel65b.

     

    1. Create .bash_proile for user grid and user oracle on both oel65a and oel65b. Look at Annex A for the profiles.
    2. Run ckuvfy to make sure that the prerequisites for GI installations are met. You will find the output in the Annex A.

       

      ./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b

      ./runcluvfy.sh stage -post hwos -n oel65a,oel65b

       

    3. runInstall from the grid 12c stage area and configure ssh.
    4. Run OUI from the grid 12c stage area

      Select Skip and press Next.


    Select install and configure GI for a cluster.


    Select Configure a standard Cluster


     

    Select a Typical Installation

     


     

    Enter the scan name and the public and virtual hostnames.

     


     

    Select the interfaces and what they are used for.

     


    Enter the software locations and ASM passwords.


    Specify the pattern to use to discover the disks


    Specify the 10g disk for a high redundancy disk group.


    Note that the disks are as follows


    Specify the inventory


    Select an option to manually run the root scripts.


    Wait for the checks to complete. Review and select Ignore All and press Next.


    The errors are as follows

    PRVF-9802 : Attempt to get udev information from node “oel65a” failed

    PRVF-9802 : Attempt to get udev information from node “oel65b” failed

    PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    The only note ‘INS-20802 PRVF-9802 PRVF-5184 PRVF-5186 After Successful Upgrade to 11gR2 Grid Infrastructure (Doc ID 974481.1)’ suggest that the disk should be managed by UDEV.

    I wonder why ASMlib does not do the job. As I am running it using Oracle VM Virtual Box I will ignore the error.

    Note : ‘PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes (Doc ID 1356975.1)’ addresses the PRVF-5636 problem. I will ignore it for the same reason as stated above.

    Review and press Install.


    Wait until prompted to execute scripts as root.


    Execute the scripts as root.


    [root@oel65a ~]# /u01/app/oraInventory/orainstRoot.sh

    Changing permissions of /u01/app/oraInventory.

    Adding read,write permissions for group.

    Removing read,write,execute permissions for world.

    Changing groupname of /u01/app/oraInventory to oinstall.

    The execution of the script is complete.

    [root@oel65a ~]#

    [root@oel65b ~]# /u01/app/oraInventory/orainstRoot.sh

    Changing permissions of /u01/app/oraInventory.

    Adding read,write permissions for group.

    Removing read,write,execute permissions for world.

    Changing groupname of /u01/app/oraInventory to oinstall.

    The execution of the script is complete.

    [root@oel65b ~]#

    [root@oel65a ~]# /u01/app/12.1.0.1/grid/root.sh

    Performing root user operation for Oracle 12c

    The following environment variables are set as:

    ORACLE_OWNER= grid

    ORACLE_HOME= /u01/app/12.1.0.1/grid

    Enter the full pathname of the local bin directory: [/usr/local/bin]:

    Copying dbhome to /usr/local/bin …

    Copying oraenv to /usr/local/bin …

    Copying coraenv to /usr/local/bin …

    Creating /etc/oratab file…

    Entries will be added to the /etc/oratab file as needed by

    Database Configuration Assistant when a database is created

    Finished running generic part of root script.

    Now product-specific root actions will be performed.

    Using configuration parameter file: /u01/app/12.1.0.1/grid/crs/install/crsconfig_params

    2014/05/14 15:09:12 CLSRSC-363: User ignored prerequisites during installation

    OLR initialization – successful

    root wallet

    root wallet cert

    root cert export

    peer wallet

    profile reader wallet

    pa wallet

    peer wallet keys

    pa wallet keys

    peer cert request

    pa cert request

    peer cert

    pa cert

    peer root cert TP

    profile reader root cert TP

    pa root cert TP

    peer pa cert TP

    pa peer cert TP

    profile reader pa cert TP

    profile reader peer cert TP

    peer user cert

    pa user cert

    2014/05/14 15:10:27 CLSRSC-330: Adding Clusterware entries to file ‘oracle-ohasd.conf’

    CRS-4133: Oracle High Availability Services has been stopped.

    CRS-4123: Oracle High Availability Services has been started.

    CRS-4133: Oracle High Availability Services has been stopped.

    CRS-4123: Oracle High Availability Services has been started.

    CRS-2672: Attempting to start ‘ora.evmd’ on ‘oel65a’

    CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.evmd’ on ‘oel65a’ succeeded

    CRS-2676: Start of ‘ora.mdnsd’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.gpnpd’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘oel65a’

    CRS-2672: Attempting to start ‘ora.gipcd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.cssdmonitor’ on ‘oel65a’ succeeded

    CRS-2676: Start of ‘ora.gipcd’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.cssd’ on ‘oel65a’

    CRS-2672: Attempting to start ‘ora.diskmon’ on ‘oel65a’

    CRS-2676: Start of ‘ora.diskmon’ on ‘oel65a’ succeeded

    CRS-2676: Start of ‘ora.cssd’ on ‘oel65a’ succeeded

    ASM created and started successfully.

    Disk Group DATA created successfully.

    CRS-2672: Attempting to start ‘ora.storage’ on ‘oel65a’

    CRS-2676: Start of ‘ora.storage’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.crsd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.crsd’ on ‘oel65a’ succeeded

    CRS-4256: Updating the profile

    Successful addition of voting disk 9d8b11d713204f34bf32e2445c8c751d.

    Successful addition of voting disk f791aee91e474f58bfae1efacb87c4ce.

    Successful addition of voting disk 4f5bcd1ebae74f1dbf5a7edcb2c86bef.

    Successful addition of voting disk d329f30451654f08bfc6b265f4007b98.

    Successful addition of voting disk 0453894fdeda4f2ebf334a123938d781.

    Successfully replaced voting disk group with +DATA.

    CRS-4256: Updating the profile

    CRS-4266: Voting file(s) successfully replaced

    ## STATE File Universal Id File Name Disk group

    – —– —————– ——— ———

    1. ONLINE 9d8b11d713204f34bf32e2445c8c751d (/dev/oracleasm/disks/DISK1) [DATA]

    2. ONLINE f791aee91e474f58bfae1efacb87c4ce (/dev/oracleasm/disks/DISK2) [DATA]

    3. ONLINE 4f5bcd1ebae74f1dbf5a7edcb2c86bef (/dev/oracleasm/disks/DISK3) [DATA]

    4. ONLINE d329f30451654f08bfc6b265f4007b98 (/dev/oracleasm/disks/DISK4) [DATA]

    5. ONLINE 0453894fdeda4f2ebf334a123938d781 (/dev/oracleasm/disks/DISK5) [DATA]

    Located 5 voting disk(s).

    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘oel65a’

    CRS-2673: Attempting to stop ‘ora.crsd’ on ‘oel65a’

    CRS-2677: Stop of ‘ora.crsd’ on ‘oel65a’ succeeded

    CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘oel65a’

    CRS-2673: Attempting to stop ‘ora.evmd’ on ‘oel65a’

    CRS-2673: Attempting to stop ‘ora.storage’ on ‘oel65a’

    CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘oel65a’

    CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘oel65a’

    CRS-2677: Stop of ‘ora.storage’ on ‘oel65a’ succeeded

    CRS-2673: Attempting to stop ‘ora.asm’ on ‘oel65a’

    CRS-2677: Stop of ‘ora.mdnsd’ on ‘oel65a’ succeeded

    CRS-2677: Stop of ‘ora.gpnpd’ on ‘oel65a’ succeeded

    CRS-2677: Stop of ‘ora.evmd’ on ‘oel65a’ succeeded

    CRS-2677: Stop of ‘ora.ctssd’ on ‘oel65a’ succeeded

    CRS-2677: Stop of ‘ora.asm’ on ‘oel65a’ succeeded

    CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘oel65a’

    CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘oel65a’ succeeded

    CRS-2673: Attempting to stop ‘ora.cssd’ on ‘oel65a’

    CRS-2677: Stop of ‘ora.cssd’ on ‘oel65a’ succeeded

    CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘oel65a’

    CRS-2677: Stop of ‘ora.gipcd’ on ‘oel65a’ succeeded

    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘oel65a’ has completed

    CRS-4133: Oracle High Availability Services has been stopped.

    CRS-4123: Starting Oracle High Availability Services-managed resources

    CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘oel65a’

    CRS-2672: Attempting to start ‘ora.evmd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.evmd’ on ‘oel65a’ succeeded

    CRS-2676: Start of ‘ora.mdnsd’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.gpnpd’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.gipcd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.gipcd’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘oel65a’

    CRS-2676: Start of ‘ora.cssdmonitor’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.cssd’ on ‘oel65a’

    CRS-2672: Attempting to start ‘ora.diskmon’ on ‘oel65a’

    CRS-2676: Start of ‘ora.diskmon’ on ‘oel65a’ succeeded

    CRS-2789: Cannot stop resource ‘ora.diskmon’ as it is not running on server ‘oel65a’

    CRS-2676: Start of ‘ora.cssd’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘oel65a’

    CRS-2672: Attempting to start ‘ora.ctssd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.ctssd’ on ‘oel65a’ succeeded

    CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.asm’ on ‘oel65a’

    CRS-2676: Start of ‘ora.asm’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.storage’ on ‘oel65a’

    CRS-2676: Start of ‘ora.storage’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.crsd’ on ‘oel65a’

    CRS-2676: Start of ‘ora.crsd’ on ‘oel65a’ succeeded

    CRS-6023: Starting Oracle Cluster Ready Services-managed resources

    CRS-6017: Processing resource auto-start for servers: oel65a

    CRS-6016: Resource auto-start has completed for server oel65a

    CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

    CRS-4123: Oracle High Availability Services has been started.

    2014/05/14 15:17:53 CLSRSC-343: Successfully started Oracle clusterware stack

    CRS-2672: Attempting to start ‘ora.asm’ on ‘oel65a’

    CRS-2676: Start of ‘ora.asm’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘oel65a’

    CRS-2676: Start of ‘ora.DATA.dg’ on ‘oel65a’ succeeded

    2014/05/14 15:19:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster … succeeded

    You have new mail in /var/spool/mail/root

    [root@oel65a ~]#

    [root@oel65b grid]# pwd

    /u01/app/12.1.0.1/grid

    [root@oel65b grid]# /u01/app/12.1.0.1/grid/root.sh

    Performing root user operation for Oracle 12c

    The following environment variables are set as:

    ORACLE_OWNER= grid

    ORACLE_HOME= /u01/app/12.1.0.1/grid

    Enter the full pathname of the local bin directory: [/usr/local/bin]:

    Copying dbhome to /usr/local/bin …

    Copying oraenv to /usr/local/bin …

    Copying coraenv to /usr/local/bin …

    Creating /etc/oratab file…

    Entries will be added to the /etc/oratab file as needed by

    Database Configuration Assistant when a database is created

    Finished running generic part of root script.

    Now product-specific root actions will be performed.

    Using configuration parameter file: /u01/app/12.1.0.1/grid/crs/install/crsconfig_params

    2014/05/14 15:20:54 CLSRSC-363: User ignored prerequisites during installation

    OLR initialization – successful

    2014/05/14 15:21:21 CLSRSC-330: Adding Clusterware entries to file ‘oracle-ohasd.conf’

    CRS-4133: Oracle High Availability Services has been stopped.

    CRS-4123: Oracle High Availability Services has been started.

    CRS-4133: Oracle High Availability Services has been stopped.

    CRS-4123: Oracle High Availability Services has been started.

    CRS-4133: Oracle High Availability Services has been stopped.

    CRS-4123: Starting Oracle High Availability Services-managed resources

    CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘oel65b’

    CRS-2672: Attempting to start ‘ora.evmd’ on ‘oel65b’

    CRS-2676: Start of ‘ora.mdnsd’ on ‘oel65b’ succeeded

    CRS-2676: Start of ‘ora.evmd’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘oel65b’

    CRS-2676: Start of ‘ora.gpnpd’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.gipcd’ on ‘oel65b’

    CRS-2676: Start of ‘ora.gipcd’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘oel65b’

    CRS-2676: Start of ‘ora.cssdmonitor’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.cssd’ on ‘oel65b’

    CRS-2672: Attempting to start ‘ora.diskmon’ on ‘oel65b’

    CRS-2676: Start of ‘ora.diskmon’ on ‘oel65b’ succeeded

    CRS-2789: Cannot stop resource ‘ora.diskmon’ as it is not running on server ‘oel65b’

    CRS-2676: Start of ‘ora.cssd’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘oel65b’

    CRS-2672: Attempting to start ‘ora.ctssd’ on ‘oel65b’

    CRS-2676: Start of ‘ora.ctssd’ on ‘oel65b’ succeeded

    CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.asm’ on ‘oel65b’

    CRS-2676: Start of ‘ora.asm’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.storage’ on ‘oel65b’

    CRS-2676: Start of ‘ora.storage’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.crsd’ on ‘oel65b’

    CRS-2676: Start of ‘ora.crsd’ on ‘oel65b’ succeeded

    CRS-6017: Processing resource auto-start for servers: oel65b

    CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65a’

    CRS-2672: Attempting to start ‘ora.ons’ on ‘oel65b’

    CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65a’ succeeded

    CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘oel65a’

    CRS-2677: Stop of ‘ora.scan1.vip’ on ‘oel65a’ succeeded

    CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘oel65b’

    CRS-2676: Start of ‘ora.scan1.vip’ on ‘oel65b’ succeeded

    CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65b’

    CRS-2676: Start of ‘ora.ons’ on ‘oel65b’ succeeded

    CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘oel65b’ succeeded

    CRS-6016: Resource auto-start has completed for server oel65b

    CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

    CRS-4123: Oracle High Availability Services has been started.

    2014/05/14 15:26:57 CLSRSC-343: Successfully started Oracle clusterware stack

    2014/05/14 15:27:21 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster … succeeded

    [root@oel65b grid]#


    The errors can be ignored. The errors are listed below

    [root@oel65a logs]# cat installActions2014-05-14_02-32-31PM.log | grep ERROR

    INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-7530 : Sufficient physical memory is not available on node “oel65b” [Required physical memory = 4GB (4194304.0KB)]

    INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-7530 : Sufficient physical memory is not available on node “oel65a” [Required physical memory = 4GB (4194304.0KB)]

    INFO: ERRORMSG(oel65b): PRVF-7530 : Sufficient physical memory is not available on node “oel65b” [Required physical memory = 4GB (4194304.0KB)]

    INFO: ERRORMSG(oel65a): PRVF-7530 : Sufficient physical memory is not available on node “oel65a” [Required physical memory = 4GB (4194304.0KB)]

    INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-9802 : Attempt to get udev information from node “oel65a” failed

    INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-9802 : Attempt to get udev information from node “oel65b” failed

    INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65b” failed

    INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65a” failed

    INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65b” failed

    INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-9802 : Attempt to get udev information from node “oel65a” failed

    INFO: ERRORMSG(oel65b): PRVF-9802 : Attempt to get udev information from node “oel65b” failed

    INFO: ERRORMSG(oel65a): PRVF-9802 : Attempt to get udev information from node “oel65a” failed

    INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: INFO: ERROR: [Result.addErrorDescription:607] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: ERRORMSG(oel65b): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: ERRORMSG(oel65a): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: INFO: ERROR: [Result.addErrorDescription:618] PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: INFO: ERROR: [Result.addErrorDescription:607] Check for integrity of file “/etc/resolv.conf” failed

    INFO: INFO: ERROR: [Result.addErrorDescription:607] Check for integrity of file “/etc/resolv.conf” failed

    INFO: ERRORMSG(oel65b): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: ERRORMSG(oel65b): Check for integrity of file “/etc/resolv.conf” failed

    INFO: ERRORMSG(oel65a): PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

    INFO: ERRORMSG(oel65a): Check for integrity of file “/etc/resolv.conf” failed

    INFO: ERROR:

    INFO: ERROR:

    INFO: ERROR:

    INFO: ERROR:

    INFO: ERROR:

    [root@oel65a logs]#

    INFO: ERROR:

    INFO: PRVG-1101 : SCAN name “scan” failed to resolve

    INFO: ERROR:

    INFO: PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.51) failed

    INFO: ERROR:

    INFO: PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.52) failed

    INFO: ERROR:

    INFO: PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.53) failed

    INFO: ERROR:

    INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name “scan”

    INFO: Checking SCAN IP addresses…


    1. Verify that GI is installed properly

       

    [root@oel65a bin]# ./crsctl check cluster -all

    **************************************************************

    oel65a:

    CRS-4537: Cluster Ready Services is online

    CRS-4529: Cluster Synchronization Services is online

    CRS-4533: Event Manager is online

    **************************************************************

    oel65b:

    CRS-4537: Cluster Ready Services is online

    CRS-4529: Cluster Synchronization Services is online

    CRS-4533: Event Manager is online

    **************************************************************

    [root@oel65a bin]#

    [root@oel65a bin]# ./crsctl status resource -t

    ——————————————————————————–

    Name Target State Server State details

    ——————————————————————————–

    Local Resources

    ——————————————————————————–

    ora.DATA.dg

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.LISTENER.lsnr

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.asm

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b Started,STABLE

    ora.net1.network

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.ons

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ——————————————————————————–

    Cluster Resources

    ——————————————————————————–

    ora.LISTENER_SCAN1.lsnr

    1 ONLINE ONLINE oel65b STABLE

    ora.LISTENER_SCAN2.lsnr

    1 ONLINE ONLINE oel65a STABLE

    ora.LISTENER_SCAN3.lsnr

    1 ONLINE ONLINE oel65a STABLE

    ora.cvu

    1 ONLINE ONLINE oel65a STABLE

    ora.oc4j

    1 OFFLINE OFFLINE STABLE

    ora.oel65a.vip

    1 ONLINE ONLINE oel65a STABLE

    ora.oel65b.vip

    1 ONLINE ONLINE oel65b STABLE

    ora.scan1.vip

    1 ONLINE ONLINE oel65b STABLE

    ora.scan2.vip

    1 ONLINE ONLINE oel65a STABLE

    ora.scan3.vip

    1 ONLINE ONLINE oel65a STABLE

    ——————————————————————————–

    [root@oel65a bin]#

    1. This concludes the GI installation.

       

  2. Install RAC RDBMS 12c on oel65a and oel65b

     

    1. Login as oracle and run the OUI to install Oracle 12c binaries only.

       

      Skip security updates.


    Skip software updates.


    Select Install software only.


    Select RAC database.


    Select all nodes.


    Select languages.


    Select enterprise Edition.


    Verify the locations.


    Verify the OS groups.


    Wait for the check to complete.

    The errors are listed below. I can ignore them.

    Single Client Access Name (SCAN) – This test verifies the Single Client Access Name configuration.  Error:

     -

    PRVG-1101 : SCAN name “scan” failed to resolve  – Cause:  An attempt to resolve specified SCAN name to a list of IP addresses failed because SCAN could not be resolved in DNS or GNS using ‘nslookup’.  – Action:  Check whether the specified SCAN name is correct. If SCAN name should be resolved in DNS, check the configuration of SCAN name in DNS. If it should be resolved in GNS make sure that GNS resource is online.

     -

    PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.51) failed  – Cause:  Inconsistent IP address definitions found for the SCAN name identified using DNS and configured name resolution mechanism(s).  – Action:  Look up the SCAN name with nslookup, and make sure the returned IP addresses are consistent with those defined in NIS and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing its cache and restarting it.

     -

    PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.52) failed  – Cause:  Inconsistent IP address definitions found for the SCAN name identified using DNS and configured name resolution mechanism(s).  – Action:  Look up the SCAN name with nslookup, and make sure the returned IP addresses are consistent with those defined in NIS and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing its cache and restarting it.

     -

    PRVF-4657 : Name resolution setup check for “scan” (IP address: 192.168.2.53) failed  – Cause:  Inconsistent IP address definitions found for the SCAN name identified using DNS and configured name resolution mechanism(s).  – Action:  Look up the SCAN name with nslookup, and make sure the returned IP addresses are consistent with those defined in NIS and /etc/hosts as configured in /etc/nsswitch.conf by reconfiguring the latter. Check the Name Service Cache Daemon (/usr/sbin/nscd) by clearing its cache and restarting it.

    Check Failed on Nodes: [oel65b,  oel65a]

    Verification result of failed node: oel65b  Details:

     -

    PRVF-4664 : Found inconsistent name resolution entries for SCAN name “scan”  – Cause:  The nslookup utility and the configured name resolution mechanism(s), as defined in /etc/nsswitch.conf, returned inconsistent IP address information for the SCAN name identified.  – Action:  Check the Name Service Cache Daemon (/usr/sbin/nscd), the Domain Name Server (nslookup) and the /etc/hosts file to make sure the IP address for the SCAN names are registered correctly.

    Back to Top

    Verification result of failed node: oel65a  Details:

     -

    PRVF-4664 : Found inconsistent name resolution entries for SCAN name “scan”  – Cause:  The nslookup utility and the configured name resolution mechanism(s), as defined in /etc/nsswitch.conf, returned inconsistent IP address information for the SCAN name identified.  – Action:  Check the Name Service Cache Daemon (/usr/sbin/nscd), the Domain Name Server (nslookup) and the /etc/hosts file to make sure the IP address for the SCAN names are registered correctly.

    Back to Top

    Task resolv.conf Integrity – This task checks consistency of file /etc/resolv.conf file across nodes

    Check Failed on Nodes: [oel65b,  oel65a]

    Verification result of failed node: oel65b  Details:

     -

    PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b  – Cause:  The DNS response time for an unreachable node exceeded the value specified on nodes specified.  – Action:  Make sure that ‘options timeout’, ‘options attempts’ and ‘nameserver’ entries in file resolv.conf are proper. On HPUX these entries will be ‘retrans’, ‘retry’ and ‘nameserver’. On Solaris these will be ‘options retrans’, ‘options retry’ and ‘nameserver’. Make sure that the DNS server responds back to name lookup request within the specified time when looking up an unknown host name.

     -

    Check for integrity of file “/etc/resolv.conf” failed  – Cause: Cause Of Problem Not Available  – Action: User Action Not Available

    Back to Top

    Verification result of failed node: oel65a  Details:

     -

    PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b  – Cause:  The DNS response time for an unreachable node exceeded the value specified on nodes specified.  – Action:  Make sure that ‘options timeout’, ‘options attempts’ and ‘nameserver’ entries in file resolv.conf are proper. On HPUX these entries will be ‘retrans’, ‘retry’ and ‘nameserver’. On Solaris these will be ‘options retrans’, ‘options retry’ and ‘nameserver’. Make sure that the DNS server responds back to name lookup request within the specified time when looking up an unknown host name.

     -

    Check for integrity of file “/etc/resolv.conf” failed  – Cause: Cause Of Problem Not Available  – Action: User Action Not Available

    Back to Top

    Review and start the installation.


    Wait until prompted to run scripts as root.



    [root@oel65a bin]# /u01/app/oracle/product/12.1.0/db_1/root.sh

    Performing root user operation for Oracle 12c

    The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME= /u01/app/oracle/product/12.1.0/db_1

    Enter the full pathname of the local bin directory: [/usr/local/bin]:

    The contents of “dbhome” have not changed. No need to overwrite.

    The contents of “oraenv” have not changed. No need to overwrite.

    The contents of “coraenv” have not changed. No need to overwrite.

    Entries will be added to the /etc/oratab file as needed by

    Database Configuration Assistant when a database is created

    Finished running generic part of root script.

    Now product-specific root actions will be performed.

    [root@oel65a bin]#

    [root@oel65b grid]# /u01/app/oracle/product/12.1.0/db_1/root.sh

    Performing root user operation for Oracle 12c

    The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME= /u01/app/oracle/product/12.1.0/db_1

    Enter the full pathname of the local bin directory: [/usr/local/bin]:

    The contents of “dbhome” have not changed. No need to overwrite.

    The contents of “oraenv” have not changed. No need to overwrite.

    The contents of “coraenv” have not changed. No need to overwrite.

    Entries will be added to the /etc/oratab file as needed by

    Database Configuration Assistant when a database is created

    Finished running generic part of root script.

    Now product-specific root actions will be performed.

    [root@oel65b grid]#

    Exit


    1. This concludes the 12c binaries installation.

       

  3. Create a Multitenant Container Database (DB) database RACDB oel65a and oel65b

     

     

    1. Login as grid and start asmca to create FRA disk group


    Click create and select disks for the FRA disk group. Press OK to create the group.


    Wait until confirmation.


    Now you have the disk groups


    1. Login as oracle and start dbca.

       

      Select create a database.

       


    Select Advanced Mode


    Select options that interest you.


    Select a CDB option.


    Select all nodes.


    Select the crosschecked options


    Select a password.


    Specify the storage.


    Skip the Vault configurations.


    Configure AMM


    Configure character sets.


    Select what to create.


    Wait for the prerequisite checks to complete.


    Review the components to be installed.


    Wait for the creation to complete.



    You can change the passwords. Note the URL


    Login to the URL


    Examine what is inside.


    1. This concludes the CDB creation.

       

  4. Verify database creation and create a service
    1. Perform Verifications

       

    [root@oel65a bin]# ./crsctl status resource -t

    ——————————————————————————–

    Name Target State Server State details

    ——————————————————————————–

    Local Resources

    ——————————————————————————–

    ora.DATA.dg

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.FRA.dg

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.LISTENER.lsnr

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.asm

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b Started,STABLE

    ora.net1.network

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.ons

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ——————————————————————————–

    Cluster Resources

    ——————————————————————————–

    ora.LISTENER_SCAN1.lsnr

    1 ONLINE ONLINE oel65b STABLE

    ora.LISTENER_SCAN2.lsnr

    1 ONLINE ONLINE oel65a STABLE

    ora.LISTENER_SCAN3.lsnr

    1 ONLINE ONLINE oel65a STABLE

    ora.cvu

    1 ONLINE ONLINE oel65a STABLE

    ora.db12c.db

    1 ONLINE ONLINE oel65a Open,STABLE

    2 ONLINE ONLINE oel65b Open,STABLE

    ora.oc4j

    1 OFFLINE OFFLINE STABLE

    ora.oel65a.vip

    1 ONLINE ONLINE oel65a STABLE

    ora.oel65b.vip

    1 ONLINE ONLINE oel65b STABLE

    ora.scan1.vip

    1 ONLINE ONLINE oel65b STABLE

    ora.scan2.vip

    1 ONLINE ONLINE oel65a STABLE

    ora.scan3.vip

    1 ONLINE ONLINE oel65a STABLE

    ——————————————————————————–

    [root@oel65a bin]#

    [oracle@oel65b ~]$ sqlplus / as sysdba

    SQL*Plus: Release 12.1.0.1.0 Production on Wed May 14 21:06:19 2014

    Copyright (c) 1982, 2013, Oracle. All rights reserved.

    Connected to:

    Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 – 64bit Production

    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

    Advanced Analytics and Real Application Testing options

    SQL> set linesize 400

    SQL> select * from v$active_instances;

    INST_NUMBER INST_NAME                                                             CON_ID

    ———– ———————————————————————————————————————————————————————————————————————————————— ———-

         1 oel65a.gj.com:DB12C1                                                     0

         2 oel65b.gj.com:DB12C2                                                     0

    SQL>

    [root@oel65a bin]# ./crsctl check cluster -all

    **************************************************************

    oel65a:

    CRS-4537: Cluster Ready Services is online

    CRS-4529: Cluster Synchronization Services is online

    CRS-4533: Event Manager is online

    **************************************************************

    oel65b:

    CRS-4537: Cluster Ready Services is online

    CRS-4529: Cluster Synchronization Services is online

    CRS-4533: Event Manager is online

    **************************************************************

    [root@oel65a bin]# ./crsctl status resource -t

    ——————————————————————————–

    Name Target State Server State details

    ——————————————————————————–

    Local Resources

    ——————————————————————————–

    ora.DATA.dg

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.FRA.dg

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.LISTENER.lsnr

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.asm

    ONLINE ONLINE oel65a Started,STABLE

    ONLINE ONLINE oel65b Started,STABLE

    ora.net1.network

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ora.ons

    ONLINE ONLINE oel65a STABLE

    ONLINE ONLINE oel65b STABLE

    ——————————————————————————–

    Cluster Resources

    ——————————————————————————–

    ora.LISTENER_SCAN1.lsnr

    1 ONLINE ONLINE oel65b STABLE

    ora.LISTENER_SCAN2.lsnr

    1 ONLINE ONLINE oel65a STABLE

    ora.LISTENER_SCAN3.lsnr

    1 ONLINE ONLINE oel65a STABLE

    ora.cvu

    1 ONLINE ONLINE oel65a STABLE

    ora.db12c.db

    1 ONLINE ONLINE oel65a Open,STABLE

    2 ONLINE ONLINE oel65b Open,STABLE

    ora.oc4j

    1 OFFLINE OFFLINE STABLE

    ora.oel65a.vip

    1 ONLINE ONLINE oel65a STABLE

    ora.oel65b.vip

    1 ONLINE ONLINE oel65b STABLE

    ora.scan1.vip

    1 ONLINE ONLINE oel65b STABLE

    ora.scan2.vip

    1 ONLINE ONLINE oel65a STABLE

    ora.scan3.vip

    1 ONLINE ONLINE oel65a STABLE

    ——————————————————————————–

    [root@oel65a bin]#

    SQL> !srvctl config database -d db12c

    Database unique name: DB12C

    Database name: DB12C

    Oracle home: /u01/app/oracle/product/12.1.0/db_1

    Oracle user: oracle

    Spfile: +DATA/DB12C/spfileDB12C.ora

    Password file: +DATA/DB12C/orapwdb12c

    Domain:

    Start options: open

    Stop options: immediate

    Database role: PRIMARY

    Management policy: AUTOMATIC

    Server pools: DB12C

    Database instances: DB12C1,DB12C2

    Disk Groups: FRA,DATA

    Mount point paths:

    Services:

    Type: RAC

    Start concurrency:

    Stop concurrency:

    Database is administrator managed

    SQL>

    SQL> select dbms_xdb_config.gethttpsport from dual;

    GETHTTPSPORT

    ————

        5500

    SQL> SQL>

    1. Check the PDBs

       

      [oracle@oel65a ~]$ sqlplus / as sysdba

       

      SQL*Plus: Release 12.1.0.1.0 Production on Mon May 26 21:44:30 2014

       

      Copyright (c) 1982, 2013, Oracle. All rights reserved.

       

       

      Connected to:

      Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 – 64bit Production

      With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

      Advanced Analytics and Real Application Testing options

       

      SQL> set linesize 400

      SQL> select * from cdb_pdbs;

       

      PDB_ID PDB_NAME                                                         DBID     CON_UID GUID                 STATUS     CREATION_SCN     CON_ID

      ———- ——————————————————————————————————————————– ———- ———- ——————————– ————- ———— ———-

           3 PDB12C1                                                         2284111037 2284111037 F95F7081C42A2554E0430B02A8C0FA31 NORMAL     1933353     1

           2 PDB$SEED                                                         4090173063 4090173063 F95E5546BA4A17E1E0430B02A8C01013 NORMAL     1720414     1

           4 PDB12C2                                                         1979843754 1979843754 F95F91A081152691E0430B02A8C0D522 NORMAL     1939488     1

           5 PDB12C3                                                         2029633601 2029633601 F95FADC38A482785E0430B02A8C039EC NORMAL     1944720     1

           6 PDB12C4                                                         2035084802 2035084802 F95FCC4A123D28DEE0430B02A8C03CBF NORMAL     1951902     1

           7 PDB12C5                                                         2084851122 2084851122 F95FE9ADF19D29F8E0430B02A8C08DB7 NORMAL     1958038     1

       

      6 rows selected.

       

      SQL> select * from v$pdbs;

       

      CON_ID     DBID     CON_UID GUID                 NAME                 OPEN_MODE RES OPEN_TIME CREATE_SCN TOTAL_SIZE

      ———- ———- ———- ——————————– —————————— ———- — ————————————————————————— ———- ———-

           2 4090173063 4090173063 F95E5546BA4A17E1E0430B02A8C01013 PDB$SEED             READ ONLY NO    26-MAY-14 02.13.03.875 PM                         1720414    283115520

           3 2284111037 2284111037 F95F7081C42A2554E0430B02A8C0FA31 PDB12C1             MOUNTED             1933353        0

           4 1979843754 1979843754 F95F91A081152691E0430B02A8C0D522 PDB12C2             MOUNTED             1939488        0

           5 2029633601 2029633601 F95FADC38A482785E0430B02A8C039EC PDB12C3             MOUNTED             1944720        0

           6 2035084802 2035084802 F95FCC4A123D28DEE0430B02A8C03CBF PDB12C4             MOUNTED             1951902        0

           7 2084851122 2084851122 F95FE9ADF19D29F8E0430B02A8C08DB7 PDB12C5             MOUNTED             1958038        0

       

      6 rows selected.

       

      SQL>

    2. This concludes the CDB creation

 

Annex A

[root@oel65a ~]# yum install oracle-rdbms-server-12cR1-preinstall

Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock

Loading support for kernel ABI

Setting up Install Process

Resolving Dependencies

–> Running transaction check

—> Package oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-11.el6 will be installed

–> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================

Package Arch Version Repository Size

==========================================================================================================================================================================

Installing:

oracle-rdbms-server-12cR1-preinstall x86_64 1.0-11.el6 public_ol6_latest 15 k

Transaction Summary

==========================================================================================================================================================================

Install 1 Package(s)

Total download size: 15 k

Installed size: 38 k

Is this ok [y/N]: Y

Downloading Packages:

Setting up and reading Presto delta metadata

Processing delta metadata

Package(s) data still to download: 15 k

oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64.rpm | 15 kB 00:00

warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY

Retrieving key from http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Importing GPG key 0xEC551F03:

Userid: “Oracle OSS group (Open Source Software group) <build@oss.oracle.com>”

From : http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Is this ok [y/N]: Y

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Installing : oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64 1/1

Verifying : oracle-rdbms-server-12cR1-preinstall-1.0-11.el6.x86_64 1/1

Installed:

oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-11.el6

Complete!

[root@oel65a ~]#

[grid@oel65a grid]$ ./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b

[grid@oel65a grid]$ ./runcluvfy.sh stage -pre crsinst -n oel65a,oel65b

Performing pre-checks for cluster services setup

Checking node reachability…

Node reachability check passed from node “oel65a”

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Node connectivity passed for subnet “10.10.10.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “10.10.10.0”

Node connectivity passed for subnet “192.168.2.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “192.168.2.0”

Node connectivity passed for subnet “192.168.122.0” with node(s) oel65a,oel65b

ERROR:

PRVF-7617 : Node connectivity between “oel65a : 192.168.122.1″ and “oel65b : 192.168.122.1″ failed

TCP connectivity check failed for subnet “192.168.122.0”

Interfaces found on subnet “192.168.2.0” that are likely candidates for VIP are:

oel65a eth1:192.168.2.11

oel65b eth1:192.168.2.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth0:10.10.10.71

oel65b eth0:10.10.10.72

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth2:10.10.10.11

oel65b eth2:10.10.10.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth3:10.10.10.81

oel65b eth3:10.10.10.82

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “10.10.10.0”.

Subnet mask consistency check passed for subnet “192.168.2.0”.

Subnet mask consistency check passed for subnet “192.168.122.0”.

Subnet mask consistency check passed.

ERROR:

PRVG-1172 : The IP address “192.168.122.1” is on multiple interfaces “virbr0,virbr0″ on nodes “oel65a,oel65b”

Node connectivity check failed

Checking multicast communication…

Checking subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251”…

Check of subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251” passed.

Check of multicast communication passed.

Checking ASMLib configuration.

Check for ASMLib configuration passed.

Total memory check passed

Available memory check passed

Swap space check passed

Free disk space check passed for “oel65b:/usr,oel65b:/var,oel65b:/etc,oel65b:/sbin”

Free disk space check passed for “oel65a:/usr,oel65a:/var,oel65a:/etc,oel65a:/sbin”

Free disk space check passed for “oel65b:/tmp”

Free disk space check passed for “oel65a:/tmp”

Check for multiple users with UID value 54322 passed

User existence check passed for “grid”

Group existence check passed for “oinstall”

Group existence check passed for “dba”

Membership check for user “grid” in group “oinstall” [as Primary] passed

Membership check for user “grid” in group “dba” failed

Check failed on nodes:

    oel65b,oel65a

Run level check passed

Hard limits check passed for “maximum open file descriptors”

Soft limits check passed for “maximum open file descriptors”

Hard limits check passed for “maximum user processes”

Soft limits check passed for “maximum user processes”

System architecture check passed

Kernel version check passed

Kernel parameter check passed for “semmsl”

Kernel parameter check passed for “semmns”

Kernel parameter check passed for “semopm”

Kernel parameter check passed for “semmni”

Kernel parameter check passed for “shmmax”

Kernel parameter check passed for “shmmni”

Kernel parameter check passed for “shmall”

Kernel parameter check passed for “file-max”

Kernel parameter check passed for “ip_local_port_range”

Kernel parameter check passed for “rmem_default”

Kernel parameter check passed for “rmem_max”

Kernel parameter check passed for “wmem_default”

Kernel parameter check passed for “wmem_max”

Kernel parameter check passed for “aio-max-nr”

Package existence check passed for “binutils”

Package existence check passed for “compat-libcap1″

Package existence check passed for “compat-libstdc++-33(x86_64)”

Package existence check passed for “libgcc(x86_64)”

Package existence check passed for “libstdc++(x86_64)”

Package existence check passed for “libstdc++-devel(x86_64)”

Package existence check passed for “sysstat”

Package existence check passed for “gcc”

Package existence check passed for “gcc-c++”

Package existence check passed for “ksh”

Package existence check passed for “make”

Package existence check passed for “glibc(x86_64)”

Package existence check passed for “glibc-devel(x86_64)”

Package existence check passed for “libaio(x86_64)”

Package existence check passed for “libaio-devel(x86_64)”

Package existence check passed for “nfs-utils”

Checking availability of ports “6200,6100” required for component “Oracle Notification Service (ONS)”

Port availability check passed for ports “6200,6100”

Check for multiple users with UID value 0 passed

Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user’s primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)…

NTP Configuration file check started…

NTP Configuration file check passed

Checking daemon liveness…

Liveness check passed for “ntpd”

Check for NTP daemon or service alive passed on all nodes

NTP common Time Server Check started…

Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started…

Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed

Core file name pattern consistency check passed.

User “grid” is not part of “root” group. Check passed

Default user file creation mask check passed

Checking integrity of file “/etc/resolv.conf” across nodes

“domain” and “search” entries do not coexist in any “/etc/resolv.conf” file

All nodes have same “search” order defined in file “/etc/resolv.conf”

PRVF-5636 : The DNS response time for an unreachable node exceeded “15000” ms on following nodes: oel65a,oel65b

Check for integrity of file “/etc/resolv.conf” failed

Time zone consistency check passed

Checking integrity of name service switch configuration file “/etc/nsswitch.conf” …

All nodes have same “hosts” entry defined in file “/etc/nsswitch.conf”

Check for integrity of name service switch configuration file “/etc/nsswitch.conf” passed

Checking daemon “avahi-daemon” is not configured and running

Daemon not configured check passed for process “avahi-daemon”

Daemon not running check passed for process “avahi-daemon”

Starting check for /dev/shm mounted as temporary file system …

Check for /dev/shm mounted as temporary file system passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

[grid@oel65a grid]$

[grid@oel65a grid]$ ./runcluvfy.sh stage -post hwos -n oel65a,oel65b

Performing post-checks for hardware and operating system setup

Checking node reachability…

Node reachability check passed from node “oel65a”

Checking user equivalence…

User equivalence check passed for user “grid”

Checking node connectivity…

Checking hosts config file…

Verification of the hosts config file successful

Node connectivity passed for subnet “10.10.10.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “10.10.10.0”

Node connectivity passed for subnet “192.168.2.0” with node(s) oel65a,oel65b

TCP connectivity check passed for subnet “192.168.2.0”

Node connectivity passed for subnet “192.168.122.0” with node(s) oel65a,oel65b

ERROR:

PRVF-7617 : Node connectivity between “oel65a : 192.168.122.1″ and “oel65b : 192.168.122.1″ failed

TCP connectivity check failed for subnet “192.168.122.0”

Interfaces found on subnet “192.168.2.0” that are likely candidates for VIP are:

oel65a eth1:192.168.2.11

oel65b eth1:192.168.2.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth0:10.10.10.71

oel65b eth0:10.10.10.72

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth2:10.10.10.11

oel65b eth2:10.10.10.12

Interfaces found on subnet “10.10.10.0” that are likely candidates for a private interconnect are:

oel65a eth3:10.10.10.81

oel65b eth3:10.10.10.82

Checking subnet mask consistency…

Subnet mask consistency check passed for subnet “10.10.10.0”.

Subnet mask consistency check passed for subnet “192.168.2.0”.

Subnet mask consistency check passed for subnet “192.168.122.0”.

Subnet mask consistency check passed.

ERROR:

PRVG-1172 : The IP address “192.168.122.1” is on multiple interfaces “virbr0,virbr0″ on nodes “oel65b,oel65a”

Node connectivity check failed

Checking multicast communication…

Checking subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251”…

Check of subnet “10.10.10.0” for multicast communication with multicast group “224.0.0.251” passed.

Check of multicast communication passed.

Check for multiple users with UID value 0 passed

Time zone consistency check passed

Checking shared storage accessibility…

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdb oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdb1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdc oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdc1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdd oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdd1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sde oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sde1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdf oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdf1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdg oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdg1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdh oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdh1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdi oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdi1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdj oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdj1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdk oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdk1 oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdl oel65a oel65b

Disk Sharing Nodes (2 in count)

———————————— ————————

/dev/sdl1 oel65a oel65b

Shared storage check was successful on nodes “oel65a,oel65b”

Checking integrity of name service switch configuration file “/etc/nsswitch.conf” …

All nodes have same “hosts” entry defined in file “/etc/nsswitch.conf”

Check for integrity of name service switch configuration file “/etc/nsswitch.conf” passed

Post-check for hardware and operating system setup was unsuccessful on all the nodes.

[grid@oel65a grid]$

[root@oel65a ~]# su – grid

[grid@oel65a ~]$ cat .bash_profile

# .bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

umask 022

ORACLE_BASE=/u01/app/grid

ORACLE_HOME=/u01/app/12.1.0.1/grid

ORACLE_HOSTNAME=oel65a

ORACLE_SID=+ASM1

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH ORACLE_HOSTNAME

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s 32768

ulimit -v unlimited

ulimit -n 4096

if [ -t 0 ]; then

stty intr ^C

fi

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

    . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

[grid@oel65a ~]$

[root@oel65a ~]# su – oracle

[oracle@oel65a ~]$ cat .bash_profile

# .bash_profile

# .bash_profile

umask 022

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1

ORACLE_HOSTNAME=oel65a

ORACLE_SID=DB12C1

ORACLE_UNQNAME=DB12C

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH ORACLE_HOSTNAME ORACLE_UNQNAME

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s 32768

ulimit -v unlimited

ulimit -n 4096

if [ -t 0 ]; then

stty intr ^C

fi

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

    . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

[oracle@oel65a ~]$

[grid@oel65b ~]$ cat .bash_profile

# .bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

umask 022

ORACLE_BASE=/u01/app/grid

ORACLE_HOME=/u01/app/12.1.0.1/grid

ORACLE_HOSTNAME=oel65b

ORACLE_SID=+ASM1

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH ORACLE_HOSTNAME

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s 32768

ulimit -v unlimited

ulimit -n 4096

if [ -t 0 ]; then

stty intr ^C

fi

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

    . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

[grid@oel65b ~]$

[root@oel65b ~]# su – oracle

[oracle@oel65b ~]$ cat .bash_profile

# .bash_profile

# .bash_profile

umask 022

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1

ORACLE_HOSTNAME=oel65b

ORACLE_SID=DB12C2

ORACLE_UNQNAME=DB12C

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH ORACLE_HOSTNAME ORACLE_UNQNAME

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s 32768

ulimit -v unlimited

ulimit -n 4096

if [ -t 0 ]; then

stty intr ^C

fi

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

    . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

[oracle@oel65b ~]$

run OUI

00:26:18.768975 VM: Raising runtime error ‘HostMemoryLow’ (fFlags=0x2)

00:26:18.770070 AssertLogRel D:\tinderbox\win-4.3\src\VBox\VMM\VMMR3\VMM.cpp(1756) int __cdecl VMMR3EmtRendezvous(struct VM *,unsigned int,int (__cdecl *)(struct VM *,struct VMCPU *,void *),void *): !pVCpu->vmm.s.fInRendezvous

00:26:18.791552 Console: VM runtime error: fatal=false, errorID=HostMemoryLow message=”Unable to allocate and lock memory. The virtual machine will be paused. Please close applications to free up memory or close the VM”

00:26:18.825633 Changing the VM state from ‘RUNNING’ to ‘GURU_MEDITATION’.

May 26, 2014 - Posted by | oracle

5 Comments »

  1. […] can read online from here or download Build two node Oracle RAC 12c (12.1.0.1) on OEL […]

    Pingback by Build two node Oracle RAC 12c (12.1.0.1) cluster on OEL 6.5 using Oracle VM Virtual Box – Download « Guenadi N Jilevski's Oracle BLOG | May 26, 2014 | Reply

  2. […] RDBMS binaries 11.2.0.4 and 12.1.0.1 and databases. More information about the latter can be found here. I would like to point out that you need to pay attention for the hardware and software […]

    Pingback by Install Oracle Enterprise Manager Cloud control 12cR3 on OEL 6.5 « Guenadi N Jilevski's Oracle BLOG | June 23, 2014 | Reply

  3. […] RDBMS binaries 11.2.0.4 and 12.1.0.1 and databases. More information about the latter can be found here. I would like to point out that you need to pay attention for the hardware and software […]

    Pingback by Install Oracle Enterprise Manager Cloud control 12cR3 on OEL 6.5 – Download « Guenadi N Jilevski's Oracle BLOG | June 23, 2014 | Reply

  4. […] Oracle RDBMS binaries 12.1.0.1 and databases. More information about the latter can be found here. I would like to point out that you need to pay attention for the hardware and software […]

    Pingback by Install Oracle Enterprise Manager Cloud control 12cR4 on OEL 6.5 « Guenadi N Jilevski's Oracle BLOG | June 23, 2014 | Reply

  5. […] Oracle RDBMS binaries 12.1.0.1 and databases. More information about the latter can be found here. I would like to point out that you need to pay attention for the hardware and software […]

    Pingback by Install Oracle Enterprise Manager Cloud control 12cR4 on OEL 6.5 – Download « Guenadi N Jilevski's Oracle BLOG | June 23, 2014 | Reply


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 773 other followers

%d bloggers like this: