Guenadi N Jilevski's Oracle BLOG

Cluvfy reports Oracle CRS not installed on all nodes

Cluvfy reports Oracle CRS not installed on all nodes

While reviewing an Oracle RAC configuration I encountered following errors upon invoking cluvfy utility.

$ ./cluvfy stage -post crsinst -n raclinux1,raclinux2 -verbose

Performing post-checks for cluster services setup

Checking node reachability…

Check: Node reachability from node “raclinux1-gj”

Destination Node Reachable?

———————————— ————————

raclinux1 yes

raclinux2 yes

Result: Node reachability check passed from node “raclinux1-gj”.

Checking user equivalence…

Check: User equivalence for user “oracle”

Node Name Comment

———————————— ————————

raclinux2 passed

raclinux1 passed

Result: User equivalence check passed for user “oracle”.

ERROR:

CRS is not installed on any of the nodes.

Verification cannot proceed.

Post-check for cluster services setup was unsuccessful on all the nodes.

Cluvfy has reported that Clusterware has not been installed on the server. But this was strange and unexpected as one RAC Database was already running on these nodes and crs_stat reported the status of all CRS resources. ocrcheck also did not report any problems with the OCR. To dig further I checked Cluvfy logs located under $ORA_CRS_HOME/cv/log. It had recorded following errors

[main] [15:52:41:800] [OUIData.readInventoryData:393] ==== CRS home added: Oracle home properties:

Name : OraCRS

Type : CRS-HOME

Location : /u01/app/oracle/product/10.2/crs

Node list: [raclinux1-gj, raclinux2-gj]

; Thu Dec 24 15:52:41 GMT+08:00 2009

[main] [15:52:41:800] [OUIData.readInventoryData:401] ==== ORACLE home added: Oracle home properties:

Name : OraHome

Type : ORACLE-HOME

Location : /u01/app/oracle/product/10.2/db_1

Node list: [raclinux1, raclinux2]

; Thu Dec 24 15:52:41 GMT+08:00 2009

[main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262] CRS wasn’t found installed on node: raclinux2; Thu Dec 24 15:52:41

GMT+08:00 2009

[main] [15:52:41:800] [VerificationUtil.isCRSInstalled:1262] CRS wasn’t found installed on node: raclinux1; Thu Dec 24 15:52:41

GMT+08:00 2009

You can notice that CRS_HOME has recorded Node list as [raclinux1-gj, raclinux2-gj] and for DB_HOME it is [raclinux1, raclinux2]. Actual hostname for the nodes are raclinux1 and raclinux2. raclinux1-gj and raclinux2-gj were aliases for the two nodes.

Cluvfy uses Central Oracle Inventory to obtain ORACLE_HOME information. Checking the $ORACLE_BASE/oraInventory/ContentsXML/inventory.xml confirmed that the nodes corresponding to CRS_HOME were stored as raclinux1-gj and raclinux2-gj.

(INVENTORY)

(VERSION_INFO)

(SAVED_WITH)10.2.0.1.0(/SAVED_WITH)

(MINIMUM_VER)2.1.0.6.0(/MINIMUM_VER)

(/VERSION_INFO)

(HOME_LIST)

(HOME NAME=”OraCRS” LOC=”/u01/app/oracle/product/10.2/crs” TYPE=”O” IDX=”1″ CRS=”true”)

(NODE_LIST)

(NODE NAME=”raclinux1-gj”/)

(NODE NAME=”raclinux2-gj”/)

(/NODE_LIST)

(/HOME)

(HOME NAME=”OraHome” LOC=”/u01/app/oracle/product/10.2/db_1″ TYPE=”O” IDX=”2″)

(NODE_LIST)

(NODE NAME=”raclinux1″/)

(NODE NAME=”raclinux2″/)

(/NODE_LIST)

(/HOME)

There are few reported issues on metalink but they relate to CRS=”true” not being present for CRS_HOME entry. There is one more reported issue which was caused by incorrect permissions on /etc/oraInst.loc file.

To correct this problem, I had to run runInstaller -updateNodelist command to update the correct nodes in file. Even though some metalink notes recommend to change this file manually,  I would recommend using runInstaller command for updating inventory. Before executing the command, I ran olsnodes command to confirm that OCR stored raclinux1,raclinux2 in its repository.

runInstaller -updateNodeList -silent “CLUSTER_NODES={raclinux1,raclinux2}” ORACLE_HOME=”/u01/app/oracle/product/10.2/crs” ORACLE_HOME_NAME=”OraCRS” LOCAL_NODE=”raclinux1″ CRS=true

Starting Oracle Universal Installer…

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.

The inventory pointer is located at /var/opt/oracle/oraInst.loc

The inventory is located at /u01/app/oracle/oraInventory

‘UpdateNodeList’ was successful.

Now the cluvfy worked fine. I do not have any reasoning for why the incorrect nodes were recorded in Oracle Inventory, but above solution should take care of this.

December 27, 2009 Posted by gjilevski | oracle | Leave a comment