Guenadi N Jilevski's Oracle BLOG

Oracle RAC, DG, EBS, DR and HA DBA BLOG

Oracle Grid Clusterware Installation for 11gR2.

Firstly, go to the staging area where the software is unzipped and run the Oracle Universal Installer (OUI):

Select Install and Configure Grid Infrastructure for a Cluster and press Next to continue:

Select Typical Installation and press Next to continue:

Select the Nodes for the Clusterware to be installed on and press Next to continue:

Here you are required to verify ssh connectivity between the nodes as you can see. Press OK to return to the main page:

Here you are provided with an option to set the private and public network interfaces. After selection eth0 as public and eth1 as private press OK:

Now, after setting up the network interfaces and verifying ssh connectivity between the nodes you can press Next to continue:

Press Next to continue and on the Specify Install location screen enter Oracle Base in the Software location. Select Cluster Registry Storage Type as ASM and enter and confirm SYSASM password. When finished press Next to continue:

Specify a name for the Diskgroup. The diskgroup in this case is called DATA. Select the candidates disks for the Data diskgroup. This will create an ASM diskgroup called DATA. When finished press Next to continue:

Verify the Inventory location and press Next to continue:

Here on this screen the installer performs all the checks for the successful completion of the installation process. As you can see, one of the kernel parameters does not meet the requirements. As this is not a mandatory parameter, it can be either fixed or ignored. If you want to ignore the parameter select ignore and press Next to continue, otherwise press Fix and Check Again:

If you have opted for Fix and Check Again you are prompted as a root user to execute the specified scripts. After running the scripts the installer will verify again the parameters. This time the prerequisites will be met and you can press Next to continue:

Press Finish to start the installation and wait until it finishes:

Press Finish to start the installation process.

Upon completion, you will be prompted to execute as root, the scripts on each of the nodes:

This completes the Grid Oracle Clusterware Installation for Oracle 11gR2.

December 31, 2009 Posted by | oracle | 2 Comments

Using Oracle 11g R1 CRS clusterware with third party non-Oracle applications

Using Oracle 11g R1 CRS clusterware with third party non-Oracle applications (xclock)

We create a script x.sh in /usr/bin on each node of the cluster with content as specified below.  The script is an action program – an executable to start/stop/check the application.

Content of x.sh

#!/bin/bash

# start/stop/check script for xclock example

# To test this change BIN_DIR to the directory where xclock is based

# and set the DISPLAY variable to a server within your network.

BIN_DIR=/usr/bin

LOG_DIR=/tmp

BIN_NAME=xclock

#DISPLAY=yourhost.domain.com:0.0

#export DISPLAY

if [ ! -d $BIN_DIR ]

then

echo “start failed”

exit 2

fi

PID1=`ps -ef | grep $BIN_NAME | grep -v grep | grep -v xclock_app | awk ‘{ print $2 }’`

case $1 in

‘start’)

if [ “$PID1” != “” ]

then

status_p1=”running”

else

if [ -x $BIN_DIR/$BIN_NAME  ]

then

umask 002

${BIN_DIR}/${BIN_NAME} & 2>${LOG_DIR}/${BIN_NAME}.log

status_p1=”started”

else

echo `basename $0`”: $BIN_NAME: Executable not found”

fi

fi

echo “$BIN_NAME: $status_p1”

;;

‘stop’)

if [ “${PID1}” != “” ]

then

kill -9 ${PID1} && echo “$BIN_NAME daemon killed”

else

echo “$BIN_NAME: no running Process!”

fi

;;

‘check’)

if [ “$PID1” != “” ]

then

echo “running”

exit 0

else

echo “not running”

exit 1

fi

;;

*)

echo “Usage: “`basename $0`” {start|stop|check}”

;;

Esac

An application profile should be added to the OCR. The main attributes are

Action Program – an executable to start/stop/check the application

Privileges – which user can start/stop the application

Resource – a resource name for your application

  1. Creating the profile

[root@raclinux1 bin]# ./crs_profile -create xclock_crs -t application -dir ./ -a /usr/bin/x.sh -r ora.raclinux1.vip

[root@raclinux1 bin]#ll

-rw-r–r– 1 root   root          758 Dec 26 10:54 xclock_crs.cap

[root@raclinux1 bin]#

[root@raclinux1 bin]# chown oracle:oinstall xclock_crs.ca

[root@raclinux1 bin]#ll

-rw-r–r– 1 oracle oinstall      758 Dec 26 10:54 xclock_crs.cap

  1. Registering the profile

[root@raclinux1 bin]# ./crs_register xclock_crs -dir ./

  1. Checking the status of the resource

[root@raclinux1 bin]# ./crs_stat | grep -A5 xclock

NAME=xclock_crs

TYPE=application

TARGET=OFFLINE

STATE=OFFLINE

  1. Setting the permissions and setting the owner

[root@raclinux1 bin]# ./crs_setperm xclock_crs -o root

[root@raclinux1 bin]# ./crs_setperm xclock_crs -u user:oracle:r-x

  1. Checking the state

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     ONLINE on raclinux1

ora.VIS.VIS2.inst                             ONLINE     ONLINE on raclinux2

ora.VIS.db                                    ONLINE     ONLINE on raclinux2

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     ONLINE on raclinux1

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     ONLINE on raclinux1

ora.raclinux2.ASM2.asm                        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ons                             ONLINE     ONLINE on raclinux2

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux2

xclock_crs                                    OFFLINE    OFFLINE

  1. Starting the resource

[root@raclinux1 bin]# ./crs_start xclock_crs

Attempting to start `xclock_crs` on member `raclinux1`

Start of `xclock_crs` on member `raclinux1` succeeded.

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     ONLINE on raclinux1

ora.VIS.VIS2.inst                             ONLINE     ONLINE on raclinux2

ora.VIS.db                                    ONLINE     ONLINE on raclinux1

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     ONLINE on raclinux1

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     ONLINE on raclinux1

ora.raclinux2.ASM2.asm                        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ons                             ONLINE     ONLINE on raclinux2

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux1

xclock_crs                                    ONLINE     ONLINE on raclinux1

  1. Stopping the resource

[root@raclinux1 bin]# ./crs_stop xclock_crs

Attempting to stop `xclock_crs` on member `raclinux1`

Killed

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     ONLINE on raclinux1

ora.VIS.VIS2.inst                             ONLINE     ONLINE on raclinux2

ora.VIS.db                                    ONLINE     ONLINE on raclinux1

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     ONLINE on raclinux1

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     ONLINE on raclinux1

ora.raclinux2.ASM2.asm                        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ons                             ONLINE     ONLINE on raclinux2

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux2

xclock_crs                                    OFFLINE    OFFLINE

[root@raclinux1 bin]#

  1. Failover

8.1          xclock and the VIP running on raclinux1

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     ONLINE on raclinux1

ora.VIS.VIS2.inst                             ONLINE     ONLINE on raclinux2

ora.VIS.db                                    ONLINE     ONLINE on raclinux1

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     ONLINE on raclinux1

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     ONLINE on raclinux1

ora.raclinux2.ASM2.asm                        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ons                             ONLINE     ONLINE on raclinux2

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux2

xclock_crs                                    ONLINE     ONLINE on raclinux1

[root@raclinux1 bin]#

8.2          Now we will pull the network cable from raclinux1. The xclock application and the VIP go offline.

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     ONLINE on raclinux1

ora.VIS.VIS2.inst                             ONLINE     ONLINE on raclinux2

ora.VIS.db                                    ONLINE     ONLINE on raclinux1

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     ONLINE on raclinux1

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     OFFLINE

ora.raclinux2.ASM2.asm                        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ons                             ONLINE     ONLINE on raclinux2

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux2

xclock_crs                                    ONLINE     OFFLINE

8.3          The  xclock  application and VIP go online on node 2

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     OFFLINE

ora.VIS.VIS2.inst                             ONLINE     OFFLINE

ora.VIS.db                                    ONLINE     OFFLINE

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     OFFLINE

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     OFFLINE

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     OFFLINE

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ASM2.asm                        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ons                             ONLINE     ONLINE on raclinux2

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux2

xclock_crs                                    ONLINE     ONLINE on raclinux2

[root@raclinux1 bin]#

Oracle does not recommend using the same VIP for more applications. In our case we used the database VIP to operate with the Apache as well. To be in compliance with Oracle recommendations we ought to create a new VIP dedicated to the Apache server and use it instead of the database IP. It should operate exactly the same as the database VIP but will have a different IP.

8.4          Node raclinux1 comes back. VIP goes back to raclinux1. The xclock   application is still present at node raclinux2.

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     OFFLINE

ora.VIS.VIS2.inst                             ONLINE     OFFLINE

ora.VIS.db                                    ONLINE     OFFLINE

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     OFFLINE

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     OFFLINE

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     OFFLINE

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     ONLINE on raclinux1

ora.raclinux2.ASM2.asm                        ONLINE     OFFLINE

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     OFFLINE

ora.raclinux2.ons                             ONLINE     OFFLINE

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux2

xclock_crs                                    ONLINE     ONLINE on raclinux2

8.5               The  xclock application also goes back to Node raclinux1 since it is declared to be dependent on node raclinux1 VIP.

[root@raclinux1 bin]# ./crs_stat.sh

HA Resourrce                                  Target     Stage

————                                  ——     —–

ora.VIS.VIS1.inst                             ONLINE     ONLINE   on raclinux1

ora.VIS.VIS2.inst                             ONLINE     ONLINE    on raclinux2

ora.VIS.db                                    ONLINE     ONLINE   on raclinux1

ora.raclinux1.ASM1.asm                        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER1_RACLINUX1.lsnr        ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_RACLINUX1.lsnr         ONLINE     ONLINE on raclinux1

ora.raclinux1.LISTENER_VIS_RACLINUX1.lsnr     ONLINE     ONLINE on raclinux1

ora.raclinux1.gsd                             ONLINE     ONLINE on raclinux1

ora.raclinux1.ons                             ONLINE     ONLINE on raclinux1

ora.raclinux1.vip                             ONLINE     ONLINE on raclinux1

ora.raclinux2.ASM2.asm                        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER1_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER2_RACLINUX2.lsnr        ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_RACLINUX2.lsnr         ONLINE     ONLINE on raclinux2

ora.raclinux2.LISTENER_VIS_RACLINUX2.lsnr     ONLINE     ONLINE on raclinux2

ora.raclinux2.gsd                             ONLINE     ONLINE on raclinux2

ora.raclinux2.ons                             ONLINE     ONLINE on raclinux2

ora.raclinux2.vip                             ONLINE     ONLINE on raclinux2

xclock_crs                                    ONLINE     ONLINE on raclinux1

December 30, 2009 Posted by | oracle | Leave a comment

Cluvfy reports Oracle CRS not installed on all nodes

Cluvfy reports Oracle CRS not installed on all nodes

While reviewing an Oracle RAC configuration I encountered following errors upon invoking cluvfy utility.

$ ./cluvfy stage -post crsinst -n raclinux1,raclinux2 -verbose

Performing post-checks for cluster services setup

Checking node reachability…

Check: Node reachability from node “raclinux1-gj”

Destination Node Reachable?

———————————— ————————

raclinux1 yes

raclinux2 yes

Result: Node reachability check passed from node “raclinux1-gj”.

Checking user equivalence…

Check: User equivalence for user “oracle”

Node Name Comment

———————————— ————————

raclinux2 passed

raclinux1 passed

Result: User equivalence check passed for user “oracle”.

ERROR:

CRS is not installed on any of the nodes.

Verification cannot proceed.

Post-check for cluster services setup was unsuccessful on all the nodes.

Cluvfy has reported that Clusterware has not been installed on the server. But this was strange and unexpected as one RAC Database was already running on these nodes and crs_stat reported the status of all CRS resources. ocrcheck also did not report any problems with the OCR. To dig further I checked Cluvfy logs located under $ORA_CRS_HOME/cv/log. It had recorded following errors

[main] [15:52:41:800] [OUIData.readInventoryData:393] ==== CRS home added: Oracle home properties:

Name : OraCRS

Type : CRS-HOME

Location : /u01/app/oracle/product/10.2/crs

Node list: [raclinux1-gj, raclinux2-gj]

; Thu Dec 24 15:52:41 GMT+08:00 2009

[main] [15:52:41:800] [OUIData.readInventoryData:401] ==== ORACLE home added: Oracle home properties:

Name : OraHome

Type : ORACLE-HOME

Location : /u01/app/oracle/product/10.2/db_1

Node list: [raclinux1, raclinux2]

; Thu Dec 24 15:52:41 GMT+08:00 2009

[main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262] CRS wasn’t found installed on node: raclinux2; Thu Dec 24 15:52:41

GMT+08:00 2009

[main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262] CRS wasn’t found installed on node: raclinux1; Thu Dec 24 15:52:41

GMT+08:00 2009

You can notice that CRS_HOME has recorded Node list as [raclinux1-gj, raclinux2-gj] and for DB_HOME it is [raclinux1, raclinux2]. Actual hostname for the nodes are raclinux1 and raclinux2. raclinux1-gj and raclinux2-gj were aliases for the two nodes.

Cluvfy uses Central Oracle Inventory to obtain ORACLE_HOME information. Checking the $ORACLE_BASE/oraInventory/ContentsXML/inventory.xml confirmed that the nodes corresponding to CRS_HOME were stored as raclinux1-gj and raclinux2-gj.

(INVENTORY)

(VERSION_INFO)

(SAVED_WITH)10.2.0.1.0(/SAVED_WITH)

(MINIMUM_VER)2.1.0.6.0(/MINIMUM_VER)

(/VERSION_INFO)

(HOME_LIST)

(HOME NAME=”OraCRS” LOC=”/u01/app/oracle/product/10.2/crs” TYPE=”O” IDX=”1″ CRS=”true”)

(NODE_LIST)

(NODE NAME=”raclinux1-gj”/)

(NODE NAME=”raclinux2-gj”/)

(/NODE_LIST)

(/HOME)

(HOME NAME=”OraHome” LOC=”/u01/app/oracle/product/10.2/db_1″ TYPE=”O” IDX=”2″)

(NODE_LIST)

(NODE NAME=”raclinux1″/)

(NODE NAME=”raclinux2″/)

(/NODE_LIST)

(/HOME)

There are few reported issues on metalink but they relate to CRS=”true” not being present for CRS_HOME entry. There is one more reported issue which was caused by incorrect permissions on /etc/oraInst.loc file.

To correct this problem, I had to run runInstaller -updateNodelist command to update the correct nodes in file. Even though some metalink notes recommend to change this file manually,  I would recommend using runInstaller command for updating inventory. Before executing the command, I ran olsnodes command to confirm that OCR stored raclinux1,raclinux2 in its repository.

runInstaller -updateNodeList -silent “CLUSTER_NODES={raclinux1,raclinux2}” ORACLE_HOME=”/u01/app/oracle/product/10.2/crs” ORACLE_HOME_NAME=”OraCRS” LOCAL_NODE=”raclinux1″ CRS=true

Starting Oracle Universal Installer…

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.

The inventory pointer is located at /var/opt/oracle/oraInst.loc

The inventory is located at /u01/app/oracle/oraInventory

‘UpdateNodeList’ was successful.

Now the cluvfy worked fine. I do not have any reasoning for why the incorrect nodes were recorded in Oracle Inventory, but above solution should take care of this.

December 27, 2009 Posted by | oracle | Leave a comment

Clusterware commands 11gR2

Beginning with Oracle 11gR2,  Oracle Clusterware stack on all nodes in a cluster can be managed from any node using the following grid infrastructure  (former CRS) commands:

$ crsctl check cluster -all – to check the status of the Clusterware on all nodes
$ crsctl stop cluster -all – to stop the Oracle Clusterware stack on all nodes
$ crsctl start cluster -all – to start the Oracle Clusterware stack on all nodes

Unlike the previous Oracle versions, beginning with Oracle 11gR2,  Oracle Clusterware stack on all nodes in a cluster can be managed from any node using the following Clusterized (Cluster Aware) commands:

$ crsctl check cluster -all – to check the status of the Clusterware on all nodes
$ crsctl stop cluster -all – to stop the Oracle Clusterware stack on all nodes
$ crsctl start cluster -all – to start the Oracle Clusterware stack on all nodes

December 22, 2009 Posted by | oracle | Leave a comment

Voting Disk Backup in Oracle 11g R2 Release

Voting Disk Backup In Oracle 11gR2 release

Voting Disk backup Procedure Change
In prior releases, backing up the voting disks using a dd command was a required postinstallation task. With Oracle Clusterware release 11.2 and later, backing up and restoring a voting disk using the dd command is not supported.

Backing up voting disks manually is no longer required, as voting disks are backed up automatically in the OCR as part of any configuration change and voting disk data is automatically restored to any added voting disks.

December 21, 2009 Posted by | oracle | Leave a comment

Installing Oracle RAC 11gR1 (11.1.0.6) RDBMS software on Linux OEL 5.3

Installing Oracle RAC 11gR1 (11.1.0.6)  RDBMS software on Linux OEL 5.3

We initiate an interactive install by logging into raclinux1 as the oracle user and start the Oracle installer:
./runInstaller.
Click on Next to continue on the Welcome screen:

Select Enterprise Edition option and press Next to continue:

Select the Oracle Base and Oracle home for the RDBMS installation. Once you have completed this, press Next to continue:

Select the Cluster Installation mode then select the additional node where the installation will take place. Make sure all nodes are selected. The current node from where the OUI is invoked is selected by default:

OUI performs all the prerequisite checks. Wait, while the checks are done. If you have any failures, correct them and re-try the tests to ensure all requirements are met before pressing the Next button:

From the Select Configuration option, Select Configure Automatic Storage Management (ASM) and enter the password for the SYS user of the ASM instance and press Next to continue. If we choose the install software only option ASM will not be created by OUI and we will need to create ASM using dbca utility after completion of the installation.

Check on the Change Disk Discovery Path… from the Configure Automatic Store Management and specify /dev/sd*.

Click Next:

Press the Next button to avoid Oracle Configuration Manager Registration:

On the Summary screen, press Install to continue:

Press Install.


Wait until the database software gets installed.

Wait until the assistants complete. And run the script root.sh from $ORACLE_HOME of each of node involved in the installation:


We log in as root Linux use and from $ORACLE_HOME directory execute the root.sh script. The output is as follows.

If we have opted not to have an ASM installed now after completion of the RDBMS install we can do it using the database configuration assistant (dbca). Please refer to  ASM creation during RDBMS database creation. We can also create an ASM independently using dbca and the steps Oracle Real Application Clusters database -> Configure Automatic Storage Management.

December 20, 2009 Posted by | oracle | 2 Comments

Resolving Gaps in Data Guard Apply Using Incremental RMAN Backup 11gr2

Resolving Gaps in Data Guard Apply Using Incremental RMAN Backup 11gr2

Recently, we had a glitch on a Data Guard (physical standby database) on infrastructure. This is not a critical database; so the monitoring was relatively lax. And that being done by an outsourcer does not help it either. In any case, the laxness resulted in a failure remaining undetected for quite some time and it was eventually discovered only when the customer complained. This standby database is usually opened for read only access from time to time.This time, however, the customer saw that the data was significantly out of sync with primary and raised a red flag. Unfortunately, at this time it had become a rather political issue.

Since the DBA in charge couldn’t resolve the problem, I was called in. In this post, I will describe the issue and how it was resolved. In summary, there are two parts of the problem:

(1) What happened
(2) How to fix it

What Happened

Let’s look at the first question – what caused the standby to lag behind. First, I looked for the current SCN numbers of the primary and standby databases. On the primary:

SQL> select current_scn from v$database;

CURRENT_SCN
———–
1447102

On the standby:

SQL> select current_scn from v$database;

CURRENT_SCN
———–
1301571

Clearly there is a difference. But this by itself does not indicate a problem; since the standby is expected to lag behind the primary (this is an asynchronous non-real time apply setup). The real question is how much it is lagging in the terms of wall clock. To know that I used the scn_to_timestamp function to translate the SCN to a timestamp:

SQL> select scn_to_timestamp(1447102) from dual;

SCN_TO_TIMESTAMP(1447102)
——————————-
18-DEC-09 08.54.28.000000000 AM

I ran the same query to know the timestamp associated with the SCN of the standby database as well (note, I ran it on the primary database, though; since it will fail in the standby in a mounted mode):

SQL> select scn_to_timestamp(1301571) from dual;

SCN_TO_TIMESTAMP(1301571)
——————————-
15-DEC-09 07.19.27.000000000 PM

This shows that the standby is two and half days lagging! The data at this point is not just stale; it must be rotten.

The next question is why it would be lagging so far back in the past. This is a 10.2 database where FAL server should automatically resolved any gaps in archived logs. Something must have happened that caused the FAL (fetch archived log) process to fail. To get that answer, first, I checked the alert log of the standby instance. I found these lines that showed the issue clearly:


Fri Dec 18 06:12:26 2009
Waiting for all non-current ORLs to be archived…
Media Recovery Waiting for thread 1 sequence 700
Fetching gap sequence in thread 1, gap sequence 700-700
… …
Fri Dec 18 06:13:27 2009
FAL[client]: Failed to request gap sequence
GAP – thread 1 sequence 700-700
DBID 846390698 branch 697108460
FAL[client]: All defined FAL servers have been attempted.

Going back in the alert log, I found these lines:

Tue Dec 15 17:16:15 2009
Fetching gap sequence in thread 1, gap sequence 700-700
Error 12514 received logging on to the standby
FAL[client, MRP0]: Error 12514 connecting to DEL1 for fetching gap sequence
Tue Dec 15 17:16:15 2009
Errors in file /opt/oracle/admin/DEL2/bdump/del2_mrp0_18308.trc:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Tue Dec 15 17:16:45 2009
Error 12514 received logging on to the standby
FAL[client, MRP0]: Error 12514 connecting to DEL1 for fetching gap sequence
Tue Dec 15 17:16:45 2009
Errors in file /opt/oracle/admin/DEL2/bdump/del2_mrp0_18308.trc:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

This clearly showed the issue. On December 15th at 17:16:15, the Managed Recovery Process encountered an error while receiving the log information from the primary. The error was ORA-12514 “TNS:listener does not currently know of service requested in connect descriptor”. This is usually the case when the TNS connect string is incorrectly specified. The primary is called DEL1 and there is a connect string called DEL1 in the standby server.

The connect string works well. Actually, right now there is no issue with the standby getting the archived logs; so there connect string is fine – now. The standby is receiving log information from the primary. There must have been some temporary hiccups causing that specific archived log not to travel to the standby. If that log was somehow skipped (could be an intermittent problem), then it should have been picked by the FAL process later on; but that never happened. Since the sequence# 700 was not applied, none of the logs received later – 701, 702 and so on – were applied either. This has caused the standby to lag behind since that time.

So, the fundamental question was why FAL did not fetch the archived log sequence# 700 from the primary. To get to that, I looked into the alert log of the primary instance. The following lines were of interest:


Tue Dec 15 19:19:58 2009
Thread 1 advanced to log sequence 701 (LGWR switch)
Current log# 2 seq# 701 mem# 0: /u01/oradata/DEL1/onlinelog/o1_mf_2_5bhbkg92_.log
Tue Dec 15 19:20:29 2009Errors in file /opt/oracle/product/10gR2/db1/admin/DEL1/bdump/del1_arc1_14469.trc:
ORA-00308: cannot open archived log ‘/u01/oraback/1_700_697108460.dbf’
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
Tue Dec 15 19:20:29 2009
FAL[server, ARC1]: FAL archive failed, see trace file.
Tue Dec 15 19:20:29 2009
Errors in file /opt/oracle/product/10gR2/db1/admin/DEL1/bdump/del1_arc1_14469.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed.
Archiver continuing
Tue Dec 15 19:20:29 2009
ORACLE Instance DEL1 – Archival Error. Archiver continuing.

These lines showed everything clearly. The issue was:

ORA-00308: cannot open archived log ‘/u01/oraback/1_700_697108460.dbf’
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory

The archived log simply was not available. The process could not see the file and couldn’t get it across to the standby site.

Upon further investigation I found that the DBA actually removed the archived logs to make some room in the filesystem without realizing that his action has removed the most current one which was yet to be transmitted to the remote site. The mystery surrounding why the FAL did not get that log was finally cleared.

Solution

Now that I know the cause, the focus was now on the resolution. If the archived log sequence# 700 was available on the primary, I could have easily copied it over to the standby, registered the log file and let the managed recovery process pick it up. But unfortunately, the file was gone and I couldn’t just recreate the file. Until that logfile was applied, the recovery will not move forward. So, what are my options?

One option is of course to recreate the standby – possible one but not technically feasible considering the time required. The other option is to apply the incremental backup of primary from that SCN number. That’s the key – the backup must be from a specific SCN number. I have described the process since it is not very obvious. The following shows the step by step approach for resolving this problem. I have shown where the actions must be performed – [Standby] or [Primary].

1. [Standby] Stop the managed standby apply process:

SQL> alter database recover managed standby database cancel;

Database altered.

2. [Standby] Shutdown the standby database

3. [Primary] On the primary, take an incremental backup from the SCN number where the standby has been stuck:

RMAN> run {
2> allocate channel c1 type disk format ‘/u01/oraback/%U.rmb’;
3> backup incremental from scn 1301571 database;
4> }

using target database control file instead of recovery catalog
allocated channel: c1
channel c1: sid=139 devtype=DISK

Starting backup at 18-DEC-09
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupset
input datafile fno=00001 name=/u01/oradata/DEL1/datafile/o1_mf_system_5bhbh59c_.dbf
… …
piece handle=/u01/oraback/06l16u1q_1_1.rmb tag=TAG20091218T083619 comment=NONE
channel c1: backup set complete, elapsed time: 00:00:06
Finished backup at 18-DEC-09
released channel: c1

4. [Primary] On the primary, create a new standby controlfile:

SQL> alter database create standby controlfile as ‘/u01/oraback/DEL1_standby.ctl’;

Database altered.

5. [Primary] Copy these files to standby host:

oracle@oradba1 /u01/oraback# scp *.rmb *.ctl oracle@oradba2:/u01/oraback
oracle@oradba2’s password:
06l16u1q_1_1.rmb 100% 43MB 10.7MB/s 00:04
DEL1_standby.ctl 100% 43MB 10.7MB/s 00:04

6. [Standby] Bring up the instance in nomount mode:

SQL> startup nomount

7. [Standby] Check the location of the controlfile:

SQL> show parameter control_files

NAME TYPE VALUE
———————————— ———– ——————————
control_files string /u01/oradata/standby_cntfile.ctl

8. [Standby] Replace the controlfile with the one you just created in primary.

9. $ cp /u01/oraback/DEL1_standby.ctl /u01/oradata/standby_cntfile.ctl

10.[Standby] Mount the standby database:

SQL> alter database mount standby database;

11.[Standby] RMAN does not know about these files yet; so you must let it know – by a process called cataloging. Catalog these files:

$ rman target=/

Recovery Manager: Release 10.2.0.4.0 – Production on Fri Dec 18 06:44:25 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: DEL1 (DBID=846390698, not open)
RMAN> catalog start with ‘/u01/oraback’;

using target database control file instead of recovery catalog
searching for all files that match the pattern /u01/oraback

List of Files Unknown to the Database
=====================================
File Name: /u01/oraback/DEL1_standby.ctl
File Name: /u01/oraback/06l16u1q_1_1.rmb

Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files…
cataloging done

List of Cataloged Files
=======================
File Name: /u01/oraback/DEL1_standby.ctl
File Name: /u01/oraback/06l16u1q_1_1.rmb

12.Recover these files:

RMAN> recover database;

Starting recover at 18-DEC-09
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /u01/oradata/DEL2/datafile/o1_mf_system_5lptww3f_.dbf
……
channel ORA_DISK_1: reading from backup piece /u01/oraback/05l16u03_1_1.rmb
channel ORA_DISK_1: restored backup piece 1
piece handle=/u01/oraback/05l16u03_1_1.rmb tag=TAG20091218T083619
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07

starting media recovery

archive log thread 1 sequence 8012 is already on disk as file /u01/oradata/1_8012_697108460.dbf
archive log thread 1 sequence 8013 is already on disk as file /u01/oradata/1_8013_697108460.dbf
… …

13. After some time, the recovery fails with the message:

archive log filename=/u01/oradata/1_8008_697108460.dbf thread=1 sequence=8009
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 12/18/2009 06:53:02
RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile ‘/u01/oradata/1_8008_697108460.dbf’
ORA-00310: archived log contains sequence 8008; sequence 8009 required
ORA-00334: archived log: ‘/u01/oradata/1_8008_697108460.dbf’

This happens because we have come to the last of the archived logs. The expected archived log with sequence# 8008 has not been generated yet.

14.At this point exit RMAN and start managed recovery process:

SQL> alter database recover managed standby database disconnect from session;

Database altered.

15.Check the SCN’s in primary and standby:

[Standby] SQL> select current_scn from v$database;

CURRENT_SCN
———–
1447474
[Primary] SQL> select current_scn from v$database;

CURRENT_SCN
———–
1447478
Now they are very close to each other. The standby has now caught up.

December 19, 2009 Posted by | oracle | 1 Comment

Installing Oracle RAC RDBMS software 11gR2

Installing Oracle RAC RDBMS software 11gR2

We have installed Oracle Clusterware 11gR2, now it is time to install the RDBMS  11gR2 software that will enable you to create a RAC database. To do this, we will need to complete the following steps involved in the RDBMS installation and the scripts that need to be run.
Firstly, from the staging area invoked OUI, type the following command and press Next to continue:

/oracle_media/database/runInstaller

Select Install Database Software Only and press Next to continue:

Select Real Application Cluster Database Installation and select the two nodes of the cluster  for the installation to proceed further. Press Next to continue:

Select Enterprise Edition and press Next to continue:

Select Oracle Base and the Software location. Press Next to continue:

Press Next to continue:

Wait until the verification completes and wait for the results. If there are errors, fix them and continue further by pressing Next:

Press Finish to start the installation process.

Wait for the installation process to complete.

Run the scripts as specified below. When finished with the scripts press OK:

Press Close to exit the installer, the installation has been successful.

The Oracle RDBMS software has completed successfully. Press Close.

December 19, 2009 Posted by | oracle | 2 Comments

Oracle 11g R2 Features

Oracle 11g R2 Features

Continuing on the previous posts, here is another gee-whiz feature of 11gR2 – the “deinstall” feature. Yes, that’s right the deinstall one. Sometimes installations fail; sometimes you have to deinstall something to clean out the server for other use. Sometimes, I did, you have to clean out beta code to install production code. A deinstall utility stops all the processes, removes all the relevant software and components (such as diskgroups), updates all config files and make all necessary modifications to the other files. All these are done without you ever bothering about remnants that may cause issues later.

You have to download the deinstall software from 11gR2 download from OTN. Choose “see all” to get to that software.

Here is the demonstration of the deinstall utility:

[oracle@oradba2 deinstall]$ ./deinstall -home /opt/oracle/product/11.2/grid1
ORACLE_HOME = /opt/oracle/product/11.2/grid1
Location of logs /opt/oracle/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################## CHECK OPERATION START ########################
Install check configuration START

Checking for existence of the Oracle home location /opt/oracle/product/11.2/grid
1
Oracle Home type selected for de-install is: SIHA
Oracle Base selected for de-install is: /opt/oracle
Checking for existence of central inventory location /opt/oracle/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /opt/oracle/produc
t/11.2/grid1

Install check configuration END

Traces log file: /opt/oracle/oraInventory/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /opt/oracle/oraInventory/logs/netd
c_check22387.log

Specify all Oracle Restart enabled listeners that are to be de-configured [LISTE
NER]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /opt/oracle/oraInventory/logs/asmcadc_
check22388.log

Automatic Storage Management (ASM) instance is detected in this Oracle home /opt/oracle/product/
11.2/grid1.
ASM Diagnostic Destination : /opt/oracle
ASM Diskgroups : +DATA1,+FRA1
Diskgroups will be dropped
De-configuring ASM will drop all the diskgroups and it's contents at cleanup time. This will aff
ect all of the databases and ACFS that use this ASM instance(s).

After some initial question and answer it shows a summary of activities and prompts you for a confirmation:

####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /opt/oracle/product/11.2/grid1
The cluster node(s) on which the Oracle home exists are: (Please input nodes seperated by ",", eg: node1,node2,...)null
Oracle Home selected for de-install is: /opt/oracle/product/11.2/grid1
Inventory Location where the Oracle home registered is: /opt/oracle/oraInventory
Following Oracle Restart enabled listener(s) will be de-configured: LISTENER
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/opt/oracle/oraInventory/logs/deinstall_deconfig2009-09-02_02-12-22-PM.out'
Any error messages from this session will be written to: '/opt/oracle/oraInventory/logs/deinstall_deconfig2009-09-02_02-12-22-PM.err'

After you press “y”, it starts the operation of a clean deinstallation. The output continues as shown below:

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /opt/oracle/oraInventory/logs/asmcadc_clean22389.log
ASM Clean Configuration START
ASM deletion in progress. This operation may take few minutes.
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /opt/oracle/oraInventory/logs/netdc_clean22390.log

De-configuring Oracle Restart enabled listener(s): LISTENER

De-configuring listener: LISTENER
    Stopping listener: LISTENER
    Listener stopped successfully.
    Unregistering listener: LISTENER
    Listener unregistered successfully.
    Deleting listener: LISTENER
    Listener deleted successfully.
Listener de-configured successfully.

De-configuring Listener configuration file...
Listener configuration file de-configured successfully.

De-configuring Naming Methods configuration file...
Naming Methods configuration file de-configured successfully.

De-configuring backup files...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

---------------------------------------->

At some point you will be asked to shutdown cssd, etc. which need root privileges. The deinstall utility shows a comamnd string you can run as root to accomplish this task:

Run the following command as the root user or the administrator on node "oradba2".

/opt/oracle/software/11gR2/deinstall/perl/bin/perl -I/opt/oracle/software/11gR2/deinstall/perl/lib -I/opt/oracle/software/11gR2/deinstall/crs/install /opt/oracle/software/11gR2/deinstall/crs/install/roothas.pl -force  -delete -paramfile /opt/oracle/software/11gR2/deinstall/response/deinstall_Ora11g_gridinfrahome1.rsp

Press Enter after you finish running the above commands

Running the command on a different terminal as root:

[root@oradba2 ~]# /opt/oracle/software/11gR2/deinstall/perl/bin/perl -I/opt/oracle/software/11gR2/deinstall/perl/lib -I/opt/oracle/software/11gR2/deinstall/crs/install /opt/oracle/software/11gR2/deinstall/crs/install/roothas.pl -force  -delete -paramfile /opt/oracle/software/11gR2/deinstall/response/deinstall_Ora11g_gridinfrahome1.rsp
2009-09-02 14:20:57: Checking for super user privileges
2009-09-02 14:20:57: User has super user privileges
2009-09-02 14:20:57: Parsing the host name
Using configuration parameter file: /opt/oracle/software/11gR2/deinstall/response/deinstall_Ora11g_gridinfrahome1.rsp
CRS-2673: Attempting to stop 'ora.cssd' on 'oradba2'
CRS-2677: Stop of 'ora.cssd' on 'oradba2' succeeded
CRS-4549: Stopping resources.
CRS-2673: Attempting to stop 'ora.diskmon' on 'oradba2'
CRS-2677: Stop of 'ora.diskmon' on 'oradba2' succeeded
CRS-4133: Oracle High Availability Services has been stopped.
ACFS-9200: Supported
Successfully deconfigured Oracle Restart stack

Now going back to the original terminal where deinstall was called from, press Enter. The output continues:

Oracle Universal Installer clean START

Detach Oracle home '/opt/oracle/product/11.2/grid1' from the central inventory on the local node : Done

Delete directory '/opt/oracle/product/11.2/grid1' on the local node : Done

The Oracle Base directory '/opt/oracle' will not be removed on local node. The directory is in use by Oracle Home '/opt/oracle/product/10.2/db1'.

The Oracle Base directory '/opt/oracle' will not be removed on local node. The directory is in use by central inventory.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

Oracle install clean START

Clean install operation removing temporary directory '/tmp/install' on node 'oradba2'

Oracle install clean END

Moved default properties file /opt/oracle/software/11gR2/deinstall/response/deinstall_Ora11g_gridinfrahome1.rsp as /opt/oracle/software/11gR2/deinstall/response/deinstall_Ora11g_gridinfrahome1.rsp1

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################
ASM instance was de-configured successfully from the Oracle home
Following Oracle Restart enabled listener(s) were de-configured successfully: LISTENER
Oracle Restart was already stopped and de-configured on node "oradba2"
Oracle Restart is stopped and de-configured successfully.
Successfully detached Oracle home '/opt/oracle/product/11.2/grid1' from the central inventory on the local node.
Successfully deleted directory '/opt/oracle/product/11.2/grid1' on the local node.
Oracle Universal Installer cleanup was successful.

Oracle install successfully cleaned up the temporary directories.
#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############

The components are cleanly deinstalled now. The directories have been cleaned up by this tool.

This was available in 11gR1 as well; but R2 just makes it very user friendly.

December 19, 2009 Posted by | oracle | Leave a comment

Creating the RAC database in Oracle 11g R1 (11.1.0.7)

Creating the RAC   database in Oracle 11g R1 (11.1.0.7)

Login as oracle user and run Database Configuration Assistant (dbca)


Select Oracle Real Application Clusters database and press Next to continue.


Select Create a Database and press Next to continue.


Select all the nodes where the clustered database will have instances and press Next to continue.


Select Custom Database and press Next to continue.


Specify the database SID and press next to continue.


Select the option to configure the Enterprise Manager and press Nest to continue


Select the same password for the Administrative users and press Next to continue.


Select the option to use Automatic Storage Management (ASM) and press next to continue.


Select the password for the sys user and select the type of parameter file that will be used for the ASM instance. Press Next to continue.


Confirm it by pressing OK.


The database configuration assistants (dbca) starts creating the ASM instance.


Here we make a selection of the disks for the DATA group for the RACDB database. We select external redundancy and ok it.

Select the disk for the FLASH diskgroup which will be used for Flash Recovery area. Select external redundancy and press OK.


Here we have the DATA and FLASH diskgroups mounted by the ASM instance. Press Next to continue.


Select DATA disk group for the Database Are and usage of Oracle Managed files. Press Next to continue.


Specify the Flash Recovery Area as an option. Specify the location of the flash recovery area and finally the flash recovery size. Press Next to continue.


Select the database components to be installed and press Next to continue.


Enable Automatic Maintenance tasks and Press Next to continue.


Examine the Database storage features and press Next to continue.


Examine the summary and press the OK to continue.


Wait until Database Configuration Assistant completes.


In case that you have run installer on a different node there will be RACDB1 instance on node raxlinux2 or +ASM2 on raclinux1. For example to reverse the instances use the following methodology. Stop the databases. After that remove the instances , then add the instances on the correct nodes. While doing it make sure you copy the correct password and spfiles. This apply both for RACDB database and for the ASM database. Please see the commands below for both RACDB and for ASM instance.

Example for the RACDB database

srvctl stop database -d RACDB
srvctl remove instance -d RACDB -i RACDB1
srvctl remove instance -d RACDB -i RACDB2
srvctl add instance -d RACDB -i RACDB1 -n raclinux1
srvctl add instance -d RACDB -i RACDB2 -n raclinux2
srvctl start database -d RACDB

Example for the ASM instance

srvctl stop asm -n raclinux1
srvctl stop asm -n raclinux2
srvctl remove asm -n raclinux2 -i +ASM1
srvctl remove asm -n raclinux1 -i +ASM2
srvctl add asm -n raclinux1 -i +ASM1 -o /u01/app/oracle/product/11.1.0/db_1
srvctl add asm -n raclinux2 -i +ASM2 -o /u01/app/oracle/product/11.1.0/db_1
srvctl start asm -n raclinux1
srvctl start asm -n raclinux2

Enterprise Manager Database Control can be recreated using the command below.

./emca -config dbcontrol db -repos recreate

Crs_stat –t –v
Name           Type           R/RA   F/FT   Target    State     Host
———————————————————————-
ora….B1.inst application    0/5    0/0    ONLINE    ONLINE    raclinux1
ora….B2.inst application    0/5    0/0    ONLINE    ONLINE    raclinux2
ora.RACDB.db   application    0/0    0/1    ONLINE    ONLINE    raclinux2
ora….SM2.asm application    0/5    0/0    ONLINE    ONLINE    raclinux1
ora….X1.lsnr application    0/5    0/0    ONLINE    ONLINE    raclinux1
ora….ux1.gsd application    0/5    0/0    ONLINE    ONLINE    raclinux1
ora….ux1.ons application    0/3    0/0    ONLINE    ONLINE    raclinux1
ora….ux1.vip application    0/0    0/0    ONLINE    ONLINE    raclinux1
ora….SM1.asm application    0/5    0/0    ONLINE    ONLINE    raclinux2
ora….X2.lsnr application    0/5    0/0    ONLINE    ONLINE    raclinux2
ora….ux2.gsd application    0/5    0/0    ONLINE    ONLINE    raclinux2
ora….ux2.ons application    0/3    0/0    ONLINE    ONLINE    raclinux2
ora….ux2.vip application    0/0    0/0    ONLINE    ONLINE    raclinux2

[oracle@raclinux2 RACDB]$ srvctl config database -d RACDB
raclinux2 RACDB1 /u01/app/oracle/product/11.1.0/db_1
raclinux1 RACDB2 /u01/app/oracle/product/11.1.0/db_1
[oracle@raclinux2 RACDB]$ srvctl status database -d RACDB
Instance RACDB1 is running on node raclinux2
Instance RACDB2 is running on node raclinux1
[oracle@raclinux2 RACDB]$

# listener.ora.raclinux2 Network Configuration File: /u01/app/oracle/product/11.1.0/db_1/network/admin/listener.ora.raclinux2
# Generated by Oracle configuration tools.

LISTENER_RACLINUX2 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2-vip)(PORT = 1521)(IP = FIRST))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.10.10.144)(PORT = 1521)(IP = FIRST))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
)
)
)
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/11.1.0/db_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

RACDB1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RACDB)
(INSTANCE_NAME = RACDB1)
)
)

RACDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux1-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RACDB)
)
)

LISTENER_RACDB2 =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux1-vip)(PORT = 1521))

LISTENER_RACDB1 =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2-vip)(PORT = 1521))

LISTENERS_RACDB =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux1-vip)(PORT = 1521))
)

RACDB2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux1-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RACDB)
(INSTANCE_NAME = RACDB2)
)
)

December 18, 2009 Posted by | oracle | Leave a comment