Guenadi N Jilevski's Oracle BLOG

Oracle RAC, DG, EBS, DR and HA DBA BLOG

Multiple OCRs and vote disks on ASM in Oracle 11gR2

Multiple OCRs and vote disks on ASM in Oracle 11gR2

Edited: 3-August-2012

Oracle Clusterware used to provide installation option for multiple locations for the OCR and vote disks prior to 11gR2. In Oracle 11gR2 ASM disk groups are used to store the OCR and vote disks. While we can use the mirroring capabilities of ASM disk groups we still can manually add additional OCR disks in Oracle 11gR2 on ASM disk groups to provide redundancy and High Availability. Here in this post we will look at adding additional OCR disks to new ASM disks groups and moving the vote disk to a disk group with high redundancy.

Initially we have Oracle GI installed on a disk group DATA with external redundancy. We created a disk group dgdup1 high redundancy with 5 disks and a disk group dgdup2 with external redundancy.

Make sure that Oracle GI is running


[root@raclinux2 ~]# cd /u01/app/11.2.0.2/grid

[root@raclinux2 grid]# cd bin

[root@raclinux2 bin]# ./crsctl check cluster -all

**************************************************************

raclinux1:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

raclinux2:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

[root@raclinux2 bin]#

Vote disks

Let’s move the vote disk to a high redundancy group.

 


[root@raclinux2 bin]#

[root@raclinux2 bin]# ./crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

1. ONLINE   7b7f9d9ae9484f2cbf7e62c164aa221b (/dev/oracleasm/disks/DISK1) [DATA]

Located 1 voting disk(s).

[root@raclinux2 bin]# ./crsctl replace  votedisk +dgdup1

Successful addition of voting disk 786515d3e2cc4fa7bfc5fd0b6e87cfeb.

Successful addition of voting disk afe5c0e3da484f62bfcc36b1c0eb4aa4.

Successful addition of voting disk 1ec83edcf92b4f55bfdc9711c49e0ddd.

Successful addition of voting disk 97278bb3f36f4f84bf5965682efe87d2.

Successful addition of voting disk 46021fd53e394fb8bf0f476f1fa210dc.

Successful deletion of voting disk 7b7f9d9ae9484f2cbf7e62c164aa221b.

Successfully replaced voting disk group with +dgdup1.

CRS-4266: Voting file(s) successfully replaced

[root@raclinux2 bin]# ./crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

1. ONLINE   786515d3e2cc4fa7bfc5fd0b6e87cfeb (/dev/oracleasm/disks/DISK11) [DGDUP1]

2. ONLINE   afe5c0e3da484f62bfcc36b1c0eb4aa4 (/dev/oracleasm/disks/DISK12) [DGDUP1]

3. ONLINE   1ec83edcf92b4f55bfdc9711c49e0ddd (/dev/oracleasm/disks/DISK13) [DGDUP1]

4. ONLINE   97278bb3f36f4f84bf5965682efe87d2 (/dev/oracleasm/disks/DISK14) [DGDUP1]

5. ONLINE   46021fd53e394fb8bf0f476f1fa210dc (/dev/oracleasm/disks/DISK15) [DGDUP1]

Located 5 voting disk(s).

[root@raclinux2 bin]#

[root@raclinux2 bin]# ./crsctl add css votedisk +data

CRS-4671: This command is not supported for ASM diskgroups.

CRS-4000: Command Add failed, or completed with errors.

[root@raclinux2 bin]# ./crsctl add css votedisk /u03/vote_acfs.dsk

CRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASM

[root@raclinux2 bin]#

As seen from above we can move a vote disk to a high redundancy disk group dgdup1 and benefit from the High Redundancy of the group. We cannot use the ‘crsctl add css votedisk’ to add a vote disk on ASM disk group or ACFS file system.
OCR disks

While we can benefit from the redundancy that the disk group storing the OCR provides we still can manually add OCR to different disk groups.

 


[root@raclinux2 bin]# cat /etc/oracle/ocr.loc

#Device/file  getting replaced by device +dgdup2

ocrconfig_loc=+DATA

local_only=false

[root@raclinux2 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          3

Total space (kbytes)     :     262120

Used space (kbytes)      :       3380

Available space (kbytes) :     258740

ID                       : 1332773503

Device/File Name         :      +DATA

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@raclinux2 bin]#

[root@raclinux2 bin]# ./ocrconfig -h

Name:

ocrconfig - Configuration tool for Oracle Cluster/Local Registry.

Synopsis:

ocrconfig [option]

option:

[-local] -export

- Export OCR/OLR contents to a file

[-local] -import          - Import OCR/OLR contents from a file

[-local] -upgrade [ []]

- Upgrade OCR from previous version

-downgrade [-version ]

- Downgrade OCR to the specified version

[-local] -backuploc        - Configure OCR/OLR backup location

[-local] -showbackup [auto|manual]  - Show OCR/OLR backup information

[-local] -manualbackup              - Perform OCR/OLR backup

[-local] -restore         - Restore OCR/OLR from physical backup

-replace -replacement

- Replace a OCR device/file <filename1> with <filename2>

-add                      - Add a new OCR device/file

-delete                   - Remove a OCR device/file

-overwrite                          - Overwrite OCR configuration on disk

-repair -add | -delete | -replace -replacement

- Repair OCR configuration on the local node

-help                               - Print out this help information

Note:

* A log file will be created in

$ORACLE_HOME/log//client/ocrconfig_.log. Please ensure

you have file creation privileges in the above directory before

running this tool.

* Only -local -showbackup [manual] is supported.

* Use option '-local' to indicate that the operation is to be performed on the Oracle Local Registry.

[root@raclinux2 bin]#

[root@raclinux2 bin]#

[root@raclinux2 bin]# ./ocrconfig -add +dgdup2

[root@raclinux2 bin]# ./ocrconfig -add +dgdup1

[root@raclinux2 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :

Version                  :          3

Total space (kbytes)     :     262120

Used space (kbytes)      :       3380

Available space (kbytes) :     258740

ID                       : 1332773503

Device/File Name         :      +DATA

Device/File integrity check succeeded

Device/File Name         :    +dgdup2

Device/File integrity check succeeded

Device/File Name         :    +dgdup1

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@raclinux2 bin]#

[root@raclinux2 bin]# cat /etc/oracle/ocr.loc

#Device/file  getting replaced by device +dgdup1

ocrconfig_loc=+DATA

ocrmirrorconfig_loc=+dgdup2

ocrconfig_loc3=+dgdup1

local_only=false[root@raclinux2 bin]#

Summary:

While we benefit from the redundancy a disk group provides we can add multiple OCR on a different disk groups. We can move the VOTE disk to a disk group with high redundancy.

To be precise vote disk in ASM has the following specifics.

No file in ASM spans a disk group including a vote file corresponding to the vote disks. Difference is that vote file is not mirrored and stripped like any other files (data, redo, control, OCR etc…) and has a fixed assignment to a failure group in a disk. Vote disk can be accessed even if ASM is not up whereas another file cannot be accessed if ASM is not up and running.

You cannot mix vote files on ASM and vote files on non-ASM. Instead you can dedicate either one ASM DG for all you cluster vote disk files or you can use non-ASM but never both except for initial migration to 11.2 when previous versions were on non-ASM storage.

For vote files you need a special number of failure groups in a disk groups as indicated below

  1. For External redundancy DG you need 1 failure group for one Vote disk file
  2. For Normal redundancy DG you need 3 failure groups for 3 vote disk file
  3. For High redundancy DG you need 5 failure groups for 5 vote disk files


October 3, 2010 - Posted by | oracle

16 Comments »

  1. Ohh..great article regarding multiple OCR and voting disks on Oracle 11g. Does it available for Oracle 10g?

    Comment by Jack Nicholson | October 5, 2010 | Reply

    • Hello,

      You can have multiple OCR and vote disks in Oracle Clusterware 10g and implement it in similar way. It is only until Oracle 11gR2 that OCR and vote disks can be on ASM disk groups.

      Regards,

      Comment by gjilevski | October 5, 2010 | Reply

  2. […] to configure multiple OCR and vote disk high availability using ASM disk groups is elaborated here. Note that vote disk can be on only one ASM disk group and the disk group itself provides […]

    Pingback by Backup and Restore of OCR in Grid Infrastructure 11G R2 11.2 « Guenadi N Jilevski's Oracle BLOG | December 20, 2010 | Reply

  3. Hi,

    Nice stuff you posted here. Thankful to you.

    I have a question here regarding OCR and Voting disk size. If I want to configure RAC env for production where there is Tera bytes of data like 3TB. then how much space I have to allocate for OCR,Voting? Minimum space requirement is 256M. Can you please update me?

    Thanks,
    S u

    Comment by su | October 18, 2011 | Reply

    • Hello,

      The size of the database(s) does not impact the OCR or VOTE disks. I suggest to make sure that sufficient disk is allocated initially by reserving 500MB for each OCR and vote disk. This value (500MB) means you will not face space constraints from activities such as disk formatting or ASM metadata.

      Best Regards,

      Comment by gjilevski | October 18, 2011 | Reply

    • Hi,

      For 280MB initial size you will need the following space.

      Minimum sizing for Oracle Clusterware configuration files using External, Nornal or High redundancy

      Redundancy Min_Number_of_disks OCR_files Voting_disk_files Total

      External 1 1X 280MB 1X 280MB 560MB

      Normal 3 2X 280MB 3X 280MB 1.4GB

      High 5 3X 280MB 5X 280MB 2.3GB

      However, you might need to buffer further and use the same methodology for an initial sizing of 500MB.

      Regards,

      Comment by gjilevski | October 19, 2011 | Reply

  4. Excellent!!! One of the best blog on this configuration I’ve seen so far.
    Thanks for putting this on your website.

    Comment by Sam | October 20, 2011 | Reply

  5. very useful note. I applied the procedure to make some changes to the existing OCR in my Exadata system.

    Comment by Sebie | December 19, 2011 | Reply

  6. Hi Guenadi,

    related to statement: “We cannot use the ‘crsctl add css votedisk’ to add a vote disk on ASM disk group or ACFS file system”

    proper explanation is we can’t use this command as votedisk can’t span multiple ASM DiskGroups.

    Votedisk can be only moved from one DG to another with the commad:
    crsctl replace votedisk

    regards,
    goran

    Comment by goran | August 1, 2012 | Reply

    • Hi,

      The statement was made in the context that vote disk is already on another ASM disk group, that is, you cannot use ‘crsctl add css votedisk’ to move from DGDUP1 to DATA. I agree with your detail elaboration.

      ACFS cannot be used for vote disk.

      for moving from ASM DG to ASM DG or vice versa use : crsctl replace votedisk
      for moving from non-ASM to ASM DG or vice versa use: crsctl add css votedisk

      Vote disk is a special file while OCR is treated as a regular file on ASM DG as far as redundancies and number of ASM disks are concerned. This a a subject to another topic…

      I will provide a better clarification and explanation.

      Regards,

      Comment by gjilevski | August 1, 2012 | Reply

      • Hi Guenadi,

        I prefer precise statements 😉 … irrespective of my comment the post is very good.

        best regards,
        goran

        Comment by goran | August 2, 2012

      • Hi,
        To be precise vote disk in ASM has the following specifics.
        No file in ASM spans a disk group including a vote file corresponding to the vote disks. Difference is that vote file is not mirrored and stripped like any other files (data, redo, control, OCR etc…) and has a fixed assignment to a failure group in a disk. Vote disk can be accessed even if ASM is not up whereas another file cannot be accessed if ASM is not up and running.
        You cannot mix vote files on ASM and vote files on non-ASM. Instead you can dedicate either one ASM DG for all you cluster vote disk files or you can use non-ASM but never both except for initial migration to 11.2 when previous versions were on non-ASM storage.
        For vote files you need a special number of failure groups in a disk groups as indicated below
        1. External redundancy 1 failure group for one Vote disk file
        2. Normal redundancy 3 failure groups for 3 vote disk file
        3. High redundancy 5 failure groups for 5 vote disk files

        Regards,

        Comment by gjilevski | August 3, 2012

  7. Nice post. I learn something new and challenging on websites I stumbleupon everyday.
    It’s always useful to read articles from other writers and use a little something from their web sites.

    Comment by general auto insurance quotes | December 12, 2012 | Reply

  8. Hi Gjilevski,

    This post is very helpful to me to understand OCR and VOTING on ASM.

    I just have a question:

    “For vote files you need a special number of failure groups in a disk groups as indicated below

    For External redundancy DG you need 1 failure group for one Vote disk file
    For Normal redundancy DG you need 3 failure groups for 3 vote disk file
    For High redundancy DG you need 5 failure groups for 5 vote disk files”

    Why it’s not enough to configure 2 FG for Normal redundancy, or 3 FG for High redundancy?

    Thanks & Regards,
    XW

    Comment by XW | December 25, 2012 | Reply

    • Hi,

      It does not meet the requirements such as:

      For External redundancy DG you need 1 failure group for one Vote disk file
      For Normal redundancy DG you need 3 failure groups for 3 vote disk file
      For High redundancy DG you need 5 failure groups for 5 vote disk files

      Vote file is a special file. Take the quorum requirements that exist for every cluster. Oracle clusterware does not make an exeption.

      Regards,

      Comment by gjilevski | December 30, 2012 | Reply

  9. Please let me know if you’re looking for a writer for
    your weblog. You have some really good posts and
    I feel I would be a good asset. If you ever want to take some of the load
    off, I’d absolutely love to write some content for your blog in exchange
    for a link back to mine. Please shoot me an e-mail if interested.
    Many thanks!

    Comment by a | June 25, 2014 | Reply


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: