Guenadi N Jilevski's Oracle BLOG

Oracle RAC, DG, EBS, DR and HA DBA BLOG

Oracle 11gR2 ACFS Replication – New feature in 11.2.0.2

Oracle 11gR2 ACFS Replication – New feature in 11.2.0.2

We will look at how to implement an ACFS replication in a two node cluster. We have a two node RAC cluster running OEL 5.4 and Oracle 11.2.0.2 GI and RDBMS. Oracle with 11.2.0.2 extends the concept of standby database to further support an asynchronous replication of ACFS from a designated primary ACFS file system to a secondary ACFS file system. This functionality provides disaster recovery by replicating an ACFS file systems across the network. The Concepts of an ACFS Replication, limitations and prerequisites are described in the Oracle documentation available here. In a nutshell we have a primary site ACFS file system and a secondary site ACFS file system that serves as a disaster recovery standby site that is used for protection. Both clustered and standalone single node ACFS can be used for replication. We simply designate one source ACFS as a primary and the target ACFS file system is the standby ACFS file system. For detailed information related to ACFS look at the post for ACFS here.

Some of the present limitations are as follows.

  • There is presently only one standby site supported for a given primary files system.
  • Only up to 8 nodes in a cluster can mount a file system.
  • There is no support for ACFS file systems with encryption or ACFS security.

 

The idea of ACFS replication as implemented as capture of the changes on the primary ACFS file system into log change files on the primary site, transferring the changes to similar log change files to the standby site and applying the changes from the logs to the standby ACFS file system. After the application of the changes the log files are purged. In order to successfully implement the ACFS Replication the ACFS file systems need to be properly sized to accommodate the changes and the network used for transferring the log should have a sufficient bandwidth. The acfsutl utility can be ran to gather for a period of time statistics for the volume of change(MB) and rate of change MB/S as specified here. This is necessary to start with properly sized ACFS file system sufficient for the replication. Also the network bandwidth is important for the replication.

For the ACFS to maintain replication version 11gR2 11.2.0.2 is required to be installed and the disk groups the ACFS file systems are based on should be created with both compatible.ASM=11.2.0.2 and compatible.ADVM=11.2.0.2 attributes. I have pre-created two groups PRIM and SEC in the two node cluster with both compatible attributes set. I have created /u05 ACFS based on PRIM data group and /u06 ACFS based on the SEC data group. The ACFS /u05 will be the primary and ACFS /u06 will be the standby.

ASMCA can be used to create the disk groups with the attributes required.


The same for the disk group the will serve as a foundation for the standby ACFS.


Here are the final ACFS file systems ( /u05 and /u06). Please note that they are mounted on both nodes of the cluster.


The following queries can be used to verify that those two attributes are set.

SELECT dg.name AS diskgroup, SUBSTR(a.name,1,18) AS name,
     SUBSTR(a.value,1,24) AS value, read_only FROM V$ASM_DISKGROUP dg,
     V$ASM_ATTRIBUTE a WHERE dg.name = 'PRIM'
     AND dg.group_number = a.group_number;

DISKGROUP                      NAME               VALUE
------------------------------ ------------------ ------------------------
READ_ON
-------
PRIM                           template.FLASHBACK 0
N
{snipped}
PRIM                           compatible.asm     11.2.0.2.0
N
DISKGROUP                      NAME               VALUE
------------------------------ ------------------ ------------------------
READ_ON
-------
PRIM                           compatible.rdbms   10.1.0.0.0
N

PRIM                           compatible.advm    11.2.0.2.0
N

PRIM                           cell.smart_scan_ca FALSE
N

{snipped}
DISKGROUP                      NAME               VALUE
------------------------------ ------------------ ------------------------
READ_ON
-------
PRIM                           template.CHANGETRA 0
N
76 rows selected.

SQL>

 

If the groups are already create make sure that those compatibility attributes are set using ‘ALTER DISKGROUP ..’ command.

Although up to eight cluster nodes are supported to mount the ACFS on the primary site, during the initial set up the primary ACFS site needs to be mounted by only one node. In our paper we will have /u05 and /u06 ACFS file system mounted on the two nodes of the cluster. ACFS based on /u05 will be primary ACFS and /u06 will be the standby ( secondary ) ACFS. The following commands will be used for mount and umount.

/bin/umount /dev/asm/prim-481 # Unmount on raclinux1,raclinux2

/bin/mount -t acfs /dev/asm/prim-481 /u05 # Mount on raclinux1,raclinux2

/bin/mount -t acfs /dev/asm/sec-351 /u06 # Mount on raclinux1,raclinux2

/bin/umount /dev/asm/sec-351 # Mount on on raclinux1,raclinux2

/sbin/mount.acfs -o all # Mount all on raclinux1, raclinux2

 

 

Let’s start the preparation.

  1. Create a user on the ASM that is granted both sysasm and sysadm privileges. In this case an user named oracle is created and a password is specified as oracle.
  2. Create a service for the primary and the secondary site.

     

    SQL> show parameter service
    
    NAME                                 TYPE        VALUE
    ------------------------------------ ----------- ------------------------------
    service_names                        string      +ASM, sec, prim, sec, prim
    SQL>
    
    [oracle@raclinux1 admin]$ cat tnsnames.ora
    # tnsnames.ora Network Configuration File: /u01/app/11.2.0.2/grid/network/admin/tnsnames.ora
    # Generated by Oracle configuration tools.
    SEC =
      (DESCRIPTION =
        (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = TCP)(HOST = rac-san)(PORT = 1521))
        )
        (CONNECT_DATA =
          (SERVICE_NAME = SEC)
        )
      )
    
    PRIM =
      (DESCRIPTION =
        (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))
        )
        (CONNECT_DATA =
          (SERVICE_NAME = PRIM)
        )
      )
    
    [oracle@raclinux1 admin]$
    
    [oracle@raclinux1 admin]$ lsnrctl status
    
    LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 03-DEC-2010 14:50:50
    
    Copyright (c) 1991, 2010, Oracle.  All rights reserved.
    
    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
    STATUS of the LISTENER
    ------------------------
    Alias                     LISTENER
    Version                   TNSLSNR for Linux: Version 11.2.0.2.0 - Production
    Start Date                03-DEC-2010 11:52:16
    Uptime                    0 days 2 hr. 58 min. 34 sec
    Trace Level               off
    Security                  ON: Local OS Authentication
    SNMP                      OFF
    Listener Parameter File   /u01/app/11.2.0.2/grid/network/admin/listener.ora
    Listener Log File         /u01/app/oracle/diag/tnslsnr/raclinux1/listener/alert/log.xml
    Listening Endpoints Summary...
      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.20.21)(PORT=1521)))
      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.20.51)(PORT=1521)))
    Services Summary...
    Service "+ASM" has 1 instance(s).
      Instance "+ASM1", status READY, has 1 handler(s) for this service...
    Service "RACDB" has 1 instance(s).
      Instance "RACDB1", status READY, has 1 handler(s) for this service...
    Service "RACDBXDB" has 1 instance(s).
      Instance "RACDB1", status READY, has 1 handler(s) for this service...
    Service "RACONE" has 1 instance(s).
      Instance "RONE_1", status READY, has 1 handler(s) for this service...
    Service "RONE" has 1 instance(s).
      Instance "RONE_1", status READY, has 1 handler(s) for this service...
    Service "RONEXDB" has 1 instance(s).
      Instance "RONE_1", status READY, has 1 handler(s) for this service...
    Service "prim" has 1 instance(s).
      Instance "+ASM1", status READY, has 1 handler(s) for this service...
    Service "sec" has 1 instance(s).
      Instance "+ASM1", status READY, has 1 handler(s) for this service...
    The command completed successfully
    [oracle@raclinux1 admin]$
    
  3. Initiate the standby ACFS while as root executing /sbin/acfsutil init standby command as follows.

     

    [root@raclinux1 bin]#

    [root@raclinux1 bin]# /sbin/acfsutil repl init standby -p oracle/oracle@prim -c sec /u06

    [root@raclinux1 bin]#

    [root@raclinux1 bin]#

    Where prim is the service created in bullet two and oracle is the user created in bullet one. Note that before starting the initiation the standby and primary site we need to have the ACFS mounted on only one node that is we need to dismount the both /u05 and /u06 on raclinux2 node.

  4. Initiate the primary ACFS while as root executing /sbin/acfsutil init primary command as follows.

     

    [root@raclinux1 bin]#

    [root@raclinux1 bin]# /sbin/acfsutil repl init primary -s oracle/oracle@sec -m /u06 -c prim /u05

    validating the remote connection

    validating the remote connection

    validating the remote connection

    acfsutil repl init: ACFS-05050: remote connection cannot be established

    acfsutil repl init: ACFS-05052: standby replication site requires reinitialization

    [root@raclinux1 bin]# /sbin/acfsutil repl init standby -p oracle/oracle@prim -c sec /u06

    [root@raclinux1 bin]#

    [root@raclinux1 bin]# /sbin/acfsutil repl init primary -s oracle/oracle@sec -m /u06 -c prim /u05

    remote connection has been established

    Registering with user specified service name-prim

    waiting for the standby replication site to initialize

    waiting for the standby replication site to initialize

    The standby replication site is initialized. ACFS replication will begin.

    [root@raclinux1 bin]#

    Note in red is the correct execution. If the service or the user is not properly created the process will have to start all over again from bullet 1.

  5. Step 4 starts the processes for ACFS replication and we can use already the ACFS replication. The replication can be validated and configuration checked with the following commands.

     

    [root@raclinux1 bin]# /sbin/acfsutil repl info -c -v /u06

    Site: Standby

    Standby status: Online

    Standby mount point: /u06

    Standby Oracle Net service name: sec

    Primary mount point: /u05

    Primary Oracle Net service name: PRIM

    Primary Oracle Net alias: oracle/****@prim

    Replicated tags:

    Log compression: Off

    Debug log level: 0

     

    [root@raclinux1 bin]#

     

    [root@raclinux1 bin]# /sbin/acfsutil repl bg info /u06

    Resource: ora.repl.transport.sec.sec.acfs

    Target State: ONLINE

    Current State: ONLINE on raclinux1

     

    Resource: ora.repl.main.sec.sec.acfs

    Target State: ONLINE

    Current State: ONLINE on raclinux1

     

    Resource: ora.repl.apply.sec.sec.acfs

    Target State: ONLINE

    Current State: ONLINE on raclinux1

     

  6. Mount primary and standby ACFS on all node of the cluster the cluster.

 

Let’s test the ACFS replication by creating a file on the primary site and re-synchronizing the standby site. On the node raclinux1 create a file on the primary /u05 ACFS.

[oracle@raclinux1 u05]$ dd if=/dev/zero of=/u05/test.dmp bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 198.774 seconds, 5.4 MB/s

[oracle@raclinux1 u05]$ ls -l

total 1050688

drwx—— 2 root root 65536 Dec 3 12:28 lost+found

-rw-r–r– 1 oracle oinstall 1073741824 Dec 3 15:03 test.dmp

[oracle@raclinux1 u05]$

[root@raclinux1 bin]# /sbin/acfsutil repl sync apply /u05

[root@raclinux1 bin]#

The above created a file in the primary /u05 and resynchronized the standby /u06. As a result the file is present on the standby ACFS /u06 and the status of the synchronization are as follows.

[root@raclinux1 bin]# /sbin/acfsutil repl info -s -v /u05

 

——————————————————-

Fri Dec 3 14:49:52 2010 – Fri Dec 3 15:47:23 2010

——————————————————-

Data replicated: 1.01GB

From writes: 1.01GB

From memory mapped updates: 0.00GB

File operations replicated: 7

Avg. rate of data replication: 17.98MB/minute

Avg. rate of file op. replication: 0.12/minute

 

Avg. time from capture to apply: 00:01:36

Min. time from capture to apply: 00:00:04

Max. time from capture to apply: 00:04:23

 

Last rlog capture to apply: 00:00:04

Last rlog apply time: Fri Dec 3 15:47:12 2010

 

Avg. time before rlog transport: 00:00:29

Min. time before rlog transport: 00:00:00

Max. time before rlog transport: 00:03:12

 

Avg. time to transport rlog: 00:00:44

Min. time to transport rlog: 00:00:00

Max. time to transport rlog: 00:03:11

 

Avg. time before rlog preprocessing: 00:00:03

Min. time before rlog preprocessing: 00:00:01

Max. time before rlog preprocessing: 00:00:08

 

Avg. time for rlog preprocessing: 00:00:00

Min. time for rlog preprocessing: 00:00:00

Max. time for rlog preprocessing: 00:00:01

 

Avg. time before applying rlog changes: 00:00:05

Min. time before applying rlog changes: 00:00:01

Max. time before applying rlog changes: 00:00:22

 

Avg. time to apply rlog changes: 00:00:12

Min. time to apply rlog changes: 00:00:00

Max. time to apply rlog changes: 00:01:16

 

Avg. time from apply to receipt: 00:00:10

Max time from apply to receipt: 00:00:19

Min time from apply to receipt: 00:00:06

 

[root@raclinux1 bin]#

 

Let’s test now creating file on both nodes on the cluster.

On node raclinux1 create the following files.

[oracle@raclinux1 u06]$ dd if=/dev/zero of=/u05/test2.dmp bs=1M count=10

10+0 records in

10+0 records out

10485760 bytes (10 MB) copied, 0.104704 seconds, 100 MB/s

[oracle@raclinux1 u06]$ ls

lost+found test2.dmp test.dmp

[oracle@raclinux1 u06]$ dd if=/dev/zero of=/u05/test3.dmp bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 245.041 seconds, 4.4 MB/s

[oracle@raclinux1 u06]$

 

On node raclinux2 create the following files.

[root@raclinux2 ~]# dd if=/dev/zero of=/u05/test5.dmp bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 324.003 seconds, 3.3 MB/s

[root@raclinux2 ~]# dd if=/dev/zero of=/u05/test6.dmp bs=1M count=1024

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 198.966 seconds, 5.4 MB/s

[root@raclinux2 ~]#

 

On node raclinux1 performed synchronization of the standby ACFS and verified that the files are present on the standby ACFS file system.

[root@raclinux1 bin]# /sbin/acfsutil repl info -s -v /u05

 

——————————————————-

Fri Dec 3 14:49:52 2010 – Fri Dec 3 16:32:57 2010

——————————————————-

Data replicated: 5.01GB

From writes: 5.01GB

From memory mapped updates: 0.00GB

File operations replicated: 11

Avg. rate of data replication: 49.77MB/minute

Avg. rate of file op. replication: 0.11/minute

 

Avg. time from capture to apply: 00:02:52

Min. time from capture to apply: 00:00:02

Max. time from capture to apply: 00:11:16

 

Last rlog capture to apply: 00:00:14

Last rlog apply time: Fri Dec 3 16:32:46 2010

 

Avg. time before rlog transport: 00:00:32

Min. time before rlog transport: 00:00:00

Max. time before rlog transport: 00:03:12

 

Avg. time to transport rlog: 00:01:04

Min. time to transport rlog: 00:00:00

Max. time to transport rlog: 00:03:11

 

Avg. time before rlog preprocessing: 00:00:04

Min. time before rlog preprocessing: 00:00:01

Max. time before rlog preprocessing: 00:00:27

 

Avg. time for rlog preprocessing: 00:00:00

Min. time for rlog preprocessing: 00:00:00

Max. time for rlog preprocessing: 00:00:02

 

Avg. time before applying rlog changes: 00:00:18

Min. time before applying rlog changes: 00:00:01

Max. time before applying rlog changes: 00:02:31

 

Avg. time to apply rlog changes: 00:00:45

Min. time to apply rlog changes: 00:00:00

Max. time to apply rlog changes: 00:06:51

 

Avg. time from apply to receipt: 00:00:11

Max time from apply to receipt: 00:00:19

Min time from apply to receipt: 00:00:06

 

[root@raclinux1 bin]#

 

ACFS Replication management

After ACFS replication was configured and tested we can look at some useful commands to check configuration, start and stop replication and pause and resume replication.

  1. Check replication configuration

     

    [root@raclinux1 bin]# /sbin/acfsutil repl info -c -v /u06

    Site: Standby

    Standby status: Online

    Standby mount point: /u06

    Standby Oracle Net service name: sec

    Primary mount point: /u05

    Primary Oracle Net service name: PRIM

    Primary Oracle Net alias: oracle/****@prim

    Replicated tags:

    Log compression: Off

    Debug log level: 0

    [root@raclinux1 bin]# /sbin/acfsutil repl info -c -v /u05

    Site: Primary

    Primary status: Online

    Primary mount point: /u05

    Primary Oracle Net service name: prim

    Standby mount point: /u06

    Standby Oracle Net service name: SEC

    Standby Oracle Net alias: oracle/****@sec

    Replicated tags:

    Log compression: Off

    Debug log level: 2

    [root@raclinux1 bin]# /sbin/acfsutil repl bg info /u06

    Resource: ora.repl.transport.sec.sec.acfs

    Target State: ONLINE

    Current State: ONLINE on raclinux1

    Resource: ora.repl.main.sec.sec.acfs

    Target State: ONLINE

    Current State: ONLINE on raclinux1

    Resource: ora.repl.apply.sec.sec.acfs

    Target State: ONLINE

    Current State: ONLINE on raclinux1

  2. Start and Stop replication. Although ACFS replication is automatically started after initiation and registered with Grid Infrastructure as a resource for automatic restart the acfsutil repl bg command can be used to start and stop the background processes and daemons implementing the replication.

     

    [root@raclinux1 bin]# /sbin/acfsutil repl bg stop /u06

    [root@raclinux1 bin]# /sbin/acfsutil repl bg start /u06

  3. Suspending and resuming ACFS replication. ACFS replication can be manually suspended and resumed. Prior to pausing the replication the sync must be executed. Pausing and resuming ACFS replication are done using the acfsuti repl [ pause | resume] /standby_fs command. Here we are going to suspend replication, resume replication create a file in the primary ACFS to synchronize with the standby ACFS file system.

     

    [root@raclinux1 bin]# sync

    [root@raclinux1 bin]# /sbin/acfsutil repl pause /u06

    [root@raclinux1 bin]# /sbin/acfsutil repl info -s -v /u05

    ——————————————————-

    Fri Dec 3 14:49:52 2010 – Fri Dec 3 16:44:18 2010

    ——————————————————-

    Data replicated: 5.01GB

    From writes: 5.01GB

    From memory mapped updates: 0.00GB

    File operations replicated: 11

    Avg. rate of data replication: 44.83MB/minute

    Avg. rate of file op. replication: 0.10/minute

    Avg. time from capture to apply: 00:02:52

    Min. time from capture to apply: 00:00:02

    Max. time from capture to apply: 00:11:16

    Last rlog capture to apply: 00:00:14

    Last rlog apply time: Fri Dec 3 16:32:46 2010

    Avg. time before rlog transport: 00:00:32

    Min. time before rlog transport: 00:00:00

    Max. time before rlog transport: 00:03:12

    Avg. time to transport rlog: 00:01:04

    Min. time to transport rlog: 00:00:00

    Max. time to transport rlog: 00:03:11

    Avg. time before rlog preprocessing: 00:00:04

    Min. time before rlog preprocessing: 00:00:01

    Max. time before rlog preprocessing: 00:00:27

    Avg. time for rlog preprocessing: 00:00:00

    Min. time for rlog preprocessing: 00:00:00

    Max. time for rlog preprocessing: 00:00:02

    Avg. time before applying rlog changes: 00:00:18

    Min. time before applying rlog changes: 00:00:01

    Max. time before applying rlog changes: 00:02:31

    Avg. time to apply rlog changes: 00:00:45

    Min. time to apply rlog changes: 00:00:00

    Max. time to apply rlog changes: 00:06:51

    Avg. time from apply to receipt: 00:00:11

    Max time from apply to receipt: 00:00:19

    Min time from apply to receipt: 00:00:06

    [root@raclinux1 bin]#

    [root@raclinux1 bin]# /sbin/acfsutil repl pause /u06

    acfsutil repl pause: ACFS-05066: ACFS replication is already paused

    [root@raclinux1 bin]# /sbin/acfsutil repl resume /u06

    [root@raclinux1 bin]#

    ## Create another 1GB file on /u05

    [root@raclinux1 bin]# /sbin/acfsutil repl sync /u05

    [root@raclinux1 bin]# /sbin/acfsutil repl sync apply /u05

    [root@raclinux1 bin]# /sbin/acfsutil repl info -s -v /u05

    ——————————————————-

    Fri Dec 3 14:49:52 2010 – Fri Dec 3 16:56:55 2010

    ——————————————————-

    Data replicated: 6.01GB

    From writes: 6.01GB

    From memory mapped updates: 0.00GB

    File operations replicated: 19

    Avg. rate of data replication: 48.44MB/minute

    Avg. rate of file op. replication: 0.15/minute

    Avg. time from capture to apply: 00:03:06

    Min. time from capture to apply: 00:00:02

    Max. time from capture to apply: 00:11:16

    Last rlog capture to apply: 00:00:49

    Last rlog apply time: Fri Dec 3 16:57:29 2010

    Avg. time before rlog transport: 00:00:31

    Min. time before rlog transport: 00:00:00

    Max. time before rlog transport: 00:03:12

    Avg. time to transport rlog: 00:00:50

    Min. time to transport rlog: 00:00:00

    Max. time to transport rlog: 00:03:11

    Avg. time before rlog preprocessing: 00:00:19

    Min. time before rlog preprocessing: 00:00:01

    Max. time before rlog preprocessing: 00:00:50

    Avg. time for rlog preprocessing: 00:00:00

    Min. time for rlog preprocessing: 00:00:00

    Max. time for rlog preprocessing: 00:00:05

    Avg. time before applying rlog changes: 00:00:47

    Min. time before applying rlog changes: 00:00:00

    Max. time before applying rlog changes: 00:03:52

    Avg. time to apply rlog changes: 00:00:35

    Min. time to apply rlog changes: 00:00:00

    Max. time to apply rlog changes: 00:06:51

    Avg. time from apply to receipt: 00:00:07

    Max time from apply to receipt: 00:00:19

    Min time from apply to receipt: 00:00:00

    [root@raclinux1 bin]#

    The file is create d in /u05 and propagated to /u06 after the replication is restarted and standby ACFS is resynchronized.

  4. Miscellaneous commands for ACFS replication. Look at acfsutil repl command.

     

Summary:

In the paper we looked at the new feature for ACFS replication introduced in Oracle 11gR2 11.2.0.2. We covered the concept of the ACFS replication based on replication logs on both primary and standby file system sites and network based change transfer from primary to standby ACFS. After Highlighting the prerequisites for ACFS replication we step by step implemented ACFS replication and demoed some basic commands for management and monitoring.

 

 

 

 

 

December 4, 2010 - Posted by | oracle

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: