Guenadi N Jilevski's Oracle BLOG

Oracle RAC, DG, EBS, DR and HA DBA BLOG

Build Active-Passive HA configuration for single instance database with Oracle GI 11.2.0.3

Build Active-Passive HA configuration for single instance database with Oracle GI 11.2.0.3

In this article you will have a look at the available options to protect a single instance database using Active/Passive configuration implemented using Oracle GI 11.2.0.3. Staring with Oracle 10.2, Oracle Clusterware provided ability to protect any third party application in a cluster by deploying Active/Passive configurations whereby only one node is active at a time and the remaining nodes are available for the active node to failover in case of the currently active node failure. For information related to building High Availability for third party applications with Oracle GI 11.2.0.3 click here. In the article you will find out how to use the GI 11.2.0.3 to set up an Active/Passive configuration for a single instance database. The following options will be covered:

  • Active/Passive configuration for single instance database with a floating VIP
  • Active/Passive configuration for single instance database without a floating VIP (SCAN only)
  • Oracle RAC One Node

Historically, cluster solutions such as HACMP, Veritas cluster, Sun cluster etc… were used to implement Active/Passive HA and failover configurations for single instance database where at any point in time only one node is running a database, listener and floating application VIP and upon node failure the applications failover to another cluster node. For accessibility to the database a floating application VIP was used to maintain the same IP regardless of the node of the cluster where Oracle database is running. Starting with 11.2 Oracle introduced the SCAN VIP that can be used to access the cluster and simplifies the configuration eliminating the Application VIP and listener failover.

Starting with Oracle 11.2 Oracle RAC One Node concept was introduced that is a standard out-off-the box single instance HA solution that can provide an Active/Passive configuration in an Oracle cluster. Oracle RAC One Node,
an additionally licensed option to the Oracle Database Enterprise Edition, is a fully supported and integrated Oracle Database failover solution running RAC enabled single instance database on one cluster node. According to MOS Note 790189.1,
Oracle RAC One Node is the preferred and recommended solution for failing over an Oracle 11g Release 2 Database going forward. Oracle continues to support alternative ways of failing over an Oracle Database but will not support action scripts and will not provide further white papers for on the matter of HA and failover configurations of a single instance Oracle databases going forward.

For information about Oracle RAC One Node installation and management click here.

The article will emphasize on building HA Active/Passive solution for database failover. In a nut shell, Oracle GI 11.2.0.3 manages resources. Database, listener and an application VIP need to be registered as cluster resources with respective profile attributes determining the restart and failover policies. The article addresses both node failure and public network failure.

The environment used in the article is based on three (3) node OEL 5 cluster using Oracle 11.2.0.3 as described here. The Active/Passive configuration will have one active and two passive standby servers.

Active/Passive configuration for single instance database without a floating VIP (SCAN only)

In this section you will look at how to register a single instance database as a resource under GI 11.2.0.3 and implement an Active/Passive configuration. A SCAN will be used to access the database and there will be no floating application VIP for access to the database. The database will be the only resource that will need to be register for management under GI 11.2.0.3.

  1. The database needs to be on a shared storage. For simplicity I installed a single instance database on shared ASM disk group DATA and DATADG using dbca.
  2. Create directories for ADRCI and audit on each cluster node where the database is supposed to run or failover.

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/alert

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/cdump

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/hm

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/incident

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/incpkg

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/ir

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/lck

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/metadata

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/metadata_dgif

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/metadata_pv

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/stage

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/sweep

    mkdir -p /u01/app/oracle/diag/rdbms/db11gr2/DB11GR2/trace

    mkdir -p /u01/app/oracle/admin/DB11GR2/adump

  3. Modify /etc/oratab on each cluster node where that database is supposed to run and include the following line.

    DB11GR2:/u01/app/oracle/product/11.2.0/db_3:N

  4. Create a pfile on each cluster node in $OH/dbs pointing to the SPFILE on ASM.

    SQL> show parameter spfile

    NAME TYPE VALUE

    ———————————— ———– ——————————

    spfile string +DATA/db11gr2/spfiledb11gr2.ora

    [oracle@raclinux2 dbs]$ cat initDB11GR2.ora

    SPFILE=’+DATA/DB11GR2/spfileDB11GR2.ora’

    [oracle@raclinux2 dbs]$

    [oracle@raclinux2 dbs]$ pwd

    /u01/app/oracle/product/11.2.0/db_3/dbs

    [oracle@raclinux2 dbs]$ scp initDB11GR2.ora raclinux1:/u01/app/oracle/product/11.2.0/db_3/dbs/

    initDB11GR2.ora 100% 41 0.0KB/s 00:00

    [oracle@raclinux2 dbs]$ scp initDB11GR2.ora raclinux3:/u01/app/oracle/product/11.2.0/db_3/dbs/

    initDB11GR2.ora 100% 41 0.0KB/s 00:00

    [oracle@raclinux2 dbs]$

  5. Create actions scripts on each node of the cluster in directory /u01/app/11.2.0.3/grid/crs/public. The main script is db11gr2_actionScript.pl calling actionstart.pl, actionStop.pl and actionClean.pl.

    [root@raclinux1 public]# cat db11gr2_actionScript.pl

    #!/usr/bin/perl

    if ($#ARGV != 0) {

     # This is an unreacheable condition

     # Oracle Clusterware would always invoke with a valid argument

    exit;

    }

    $command = $ARGV[0];

    # Start command

    if ($command eq “start”) {

    system(“su – oracle -c \”/u01/app/11.2.0.3/grid/crs/public/actionStart.pl\””);

    # steps to start the resource

    exit 0; #if the start is successfull

    # exit 1 if the start has failed

    }

    # Stop command

    if ($command eq “stop”) {

    system(“su – oracle -c \”/u01/app/11.2.0.3/grid/crs/public/actionStop.pl\””);

    # steps to stop the resource

    exit 0; # if the stop is successfull

    # exit 1 if the stop has failed

    }

    # Check command

    if ($command eq “check”) {

    $found = qx(ps -ef | grep smon | grep DB11GR2 | grep -v grep | wc -l);

    #$found = ‘ps -ef | grep smon | grep DB11GR2 | wc -l ‘;

    chomp($found);

    #print($found);

    if ($found eq “0″ ) {exit 1;} else {exit 0; }

    # steps to check the status of the resource

    # exit 0 if the resource is online

    # exit 1 otherwise

    }

    # Clean command

    if ($command eq “clean”) {

    system(“su – oracle -c \”/u01/app/11.2.0.3/grid/crs/public/actionClean.pl\””);

    # steps to force stop the resource

    exit 0; # if the force stop is successfull

    # exit 1 if the force stop has failed

    }

    [root@raclinux1 public]#

    [root@raclinux1 public]# cat actionStart.pl

    #!/usr/bin/perl

    # Stop command

    system(“export ORACLE_SID=DB11GR2

    export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3

    /u01/app/oracle/product/11.2.0/db_3/bin/sqlplus /nolog <

    connect / as sysdba

    startup

    exit

    EOF”);

    exit 0; # if the stop is successfull

    [root@raclinux1 public]#

    [root@raclinux1 public]# cat actionStop.pl

    #!/usr/bin/perl

    # Stop command

    system(“export ORACLE_SID=DB11GR2

    export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3

    /u01/app/oracle/product/11.2.0/db_3/bin/sqlplus /nolog <

    connect / as sysdba

    shutdown immediate

    exit

    EOF”);

    exit 0; # if the stop is successfull

    [root@raclinux1 public]# cat actionClean.pl

    #!/usr/bin/perl

    # Stop command

    system(“export ORACLE_SID=DB11GR2

    export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_3

    /u01/app/oracle/product/11.2.0/db_3/bin/sqlplus /nolog <

    connect / as sysdba

    shutdown immediate

    exit

    EOF”);

    exit 0; # if the stop is successfull

    [root@raclinux1 public]#

  6. Test the actionscript to make sure that they work

    [root@raclinux2 public]# ./db11gr2_actionScript.pl check

    [root@raclinux2 public]# echo $?

    0

    [root@raclinux2 public]#

    [root@raclinux2 public]# ./db11gr2_actionScript.pl stop

    SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 8 18:23:00 2012

    Copyright (c) 1982, 2011, Oracle. All rights reserved.

    SQL> Connected.

    SQL> Database closed.

    Database dismounted.

    ORACLE instance shut down.

    SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

    Data Mining and Real Application Testing options

    [root@raclinux2 public]# ./db11gr2_actionScript.pl check

    [root@raclinux2 public]# echo $?

    1

    [root@raclinux2 public]#

    [root@raclinux2 public]# ./db11gr2_actionScript.pl start

    SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 8 18:23:51 2012

    Copyright (c) 1982, 2011, Oracle. All rights reserved.

    SQL> Connected to an idle instance.

    SQL> ORACLE instance started.

    Total System Global Area 626327552 bytes

    Fixed Size 2230952 bytes

    Variable Size 452986200 bytes

    Database Buffers 167772160 bytes

    Redo Buffers 3338240 bytes

    Database mounted.

    Database opened.

    SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

    With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

    Data Mining and Real Application Testing options

    [root@raclinux2 public]# ./db11gr2_actionScript.pl check

    [root@raclinux2 public]# echo $?

    0

    [root@raclinux2 public]#

  1. Create the resource as root. Make sure that you specify correct full path for the action script. Attributes worth noting a bolded. The resource is planned to run on the list of nodes specified by HOSTING_MEMBERS. Placement policy is restricted. The resource will failover to another node if restarted more than once as RESTART_ATTEMPTS=1. The resource will run on only one cluster node at any given point in time CARDINALITY=1.

    [root@raclinux2 bin]# ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=, HOSTING_MEMBERS=raclinux1 raclinux2 raclinux3,DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=30, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network), STOP_DEPENDENCIES=hard(ora.net1.network),CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

    [root@raclinux2 bin]#

  2. Set the resource permissions

    [root@raclinux2 bin]# ./crsctl getperm resource db11gr2

    Name: db11gr2

    owner:root:rwx,pgrp:root:r-x,other::r–

    [root@raclinux2 bin]#

    [root@raclinux2 bin]# ./crsctl setperm resource db11gr2 -u user:oracle:rwx

    [root@raclinux2 bin]# ./crsctl setperm resource db11gr2 -u user:grid:rwx

    [root@raclinux2 bin]#

    [root@raclinux2 bin]# ./crsctl getperm resource db11gr2

    Name: db11gr2

    owner:root:rwx,pgrp:root:r-x,other::r–,user:oracle:rwx,user:grid:rwx

    [root@raclinux2 bin]#

  3. Test and start the resource.

    [root@raclinux1 bin]# ./crsctl start res db11gr2

    CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux1′

    CRS-2676: Start of ‘db11gr2′ on ‘raclinux1′ succeeded

    [root@raclinux1 bin]#

    [root@raclinux3 bin]# ./crsctl relocate res db11gr2

    CRS-2673: Attempting to stop ‘db11gr2′ on ‘raclinux1′

    CRS-2677: Stop of ‘db11gr2′ on ‘raclinux1′ succeeded

    CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux2′

    CRS-2676: Start of ‘db11gr2′ on ‘raclinux2′ succeeded

    You have new mail in /var/spool/mail/root

    [root@raclinux3 bin]#

    [root@raclinux3 bin]# ./crsctl relocate res db11gr2 -s raclinux2 -n raclinux3

    CRS-2673: Attempting to stop ‘db11gr2′ on ‘raclinux2′

    CRS-2677: Stop of ‘db11gr2′ on ‘raclinux2′ succeeded

    CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux3′

    CRS-2676: Start of ‘db11gr2′ on ‘raclinux3′ succeeded

    [root@raclinux3 bin]#

  4. Make sure that remote_listener register with the scan listeners.

    SQL> alter system set remote_listener=’rac-scan:1521′ scope=both;

    System altered.

    SQL>

  5. Change the tnsnames.ora alias to use scan listener.

    From

    DB11GR2 =

    (DESCRIPTION =

     (ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2)(PORT = 1521))

    (CONNECT_DATA =

    (SERVER = DEDICATED)

    (SERVICE_NAME = DB11GR2)

    )

    )

    To

    DB11GR2 =

    (DESCRIPTION =

     (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))

    (CONNECT_DATA =

    (SERVER = DEDICATED)

    (SERVICE_NAME = DB11GR2)

    )

    )

  6. Test node failure

    Before Node crash

    [root@raclinux1 bin]# ./crsctl stat res db11gr2 -t

    ——————————————————————————–

    NAME TARGET STATE SERVER STATE_DETAILS

    ——————————————————————————–

    Cluster Resources

    ——————————————————————————–

    db11gr2

     1 ONLINE ONLINE raclinux3

    [root@raclinux1 bin]#

    After Node Crash

    [root@raclinux1 bin]# ./crsctl stat res db11gr2 -t

    ——————————————————————————–

    NAME TARGET STATE SERVER STATE_DETAILS

    ——————————————————————————–

    Cluster Resources

    ——————————————————————————–

    db11gr2

     1 ONLINE ONLINE raclinux1

    [root@raclinux1 bin]#

  7. Test process death. Kill pmon and observe the RESTAR_COUNT, FAILURE_COUNT and the STATE. Oracle GI restarts ones and on the second failure fails over to available cluster node. Number of restart prior to failover can be modified by changing RESTART_ATTEMPTS value.

    [root@raclinux1 bin]# ./crsctl stat res db11gr2 -v

    NAME=db11gr2

    TYPE=cluster_resource

    LAST_SERVER=raclinux1

    STATE=ONLINE on raclinux1

    TARGET=ONLINE

    CARDINALITY_ID=1

    CREATION_SEED=400

    RESTART_COUNT=0

    FAILURE_COUNT=0

    FAILURE_HISTORY=

    ID=db11gr2 1 1

    INCARNATION=1

    LAST_RESTART=01/08/2012 23:58:36

    LAST_STATE_CHANGE=01/08/2012 23:58:36

    STATE_DETAILS=

    INTERNAL_STATE=STABLE

    [root@raclinux1 bin]#

    kill -9 pid

    [root@raclinux1 bin]# ./crsctl stat res db11gr2 -v

    NAME=db11gr2

    TYPE=cluster_resource

    LAST_SERVER=raclinux1

    STATE=ONLINE on raclinux1

    TARGET=ONLINE

    CARDINALITY_ID=1

    CREATION_SEED=400

    RESTART_COUNT=1

    FAILURE_COUNT=0

    FAILURE_HISTORY=

    ID=db11gr2 1 1

    INCARNATION=2

    LAST_RESTART=01/09/2012 00:03:37

    LAST_STATE_CHANGE=01/09/2012 00:03:37

    STATE_DETAILS=

    INTERNAL_STATE=STABLE

    [root@raclinux1 bin]#

    kill -9

    [root@raclinux1 bin]# ./crsctl stat res db11gr2 -v

    NAME=db11gr2

    TYPE=cluster_resource

    LAST_SERVER=raclinux2

    STATE=ONLINE on raclinux2

    TARGET=ONLINE

    CARDINALITY_ID=1

    CREATION_SEED=400

    RESTART_COUNT=0

    FAILURE_COUNT=1

    FAILURE_HISTORY=1326060967:raclinux1

    ID=db11gr2 1 1

    INCARNATION=3

    LAST_RESTART=01/09/2012 00:17:16

    LAST_STATE_CHANGE=01/09/2012 00:17:16

    STATE_DETAILS=

    INTERNAL_STATE=STABLE

    [root@raclinux1 bin]#

    kill -9

    [root@raclinux2 bin]# ./crsctl stat res db11gr2 -v

    NAME=db11gr2

    TYPE=cluster_resource

    LAST_SERVER=raclinux2

    STATE=ONLINE on raclinux2

    TARGET=ONLINE

    CARDINALITY_ID=1

    CREATION_SEED=400

    RESTART_COUNT=1

    FAILURE_COUNT=1

    FAILURE_HISTORY=1326060967:raclinux1

    ID=db11gr2 1 1

    INCARNATION=4

    LAST_RESTART=01/09/2012 00:49:30

    LAST_STATE_CHANGE=01/09/2012 00:49:30

    STATE_DETAILS=

    INTERNAL_STATE=STABLE

    [root@raclinux2 bin]#

    kill -9

    [root@raclinux2 bin]# ./crsctl stat res db11gr2 -v

    NAME=db11gr2

    TYPE=cluster_resource

    LAST_SERVER=raclinux1

    STATE=ONLINE on raclinux1

    TARGET=ONLINE

    CARDINALITY_ID=1

    CREATION_SEED=400

    RESTART_COUNT=0

    FAILURE_COUNT=1

    FAILURE_HISTORY=1326063300:raclinux2

    ID=db11gr2 1 1

    INCARNATION=5

    LAST_RESTART=01/09/2012 00:55:30

    LAST_STATE_CHANGE=01/09/2012 00:55:30

    STATE_DETAILS=

    INTERNAL_STATE=STABLE

    [root@raclinux2 bin]#

  8. Check Oracle Net connectivity with the db11gr2 connect sting to verify database location transparency.

Active/Passive configuration for single instance database with a floating VIP

In this section you will look at the pre Oracle 11.2, when SCAN was introduced, approach to make a database highly available in an Active/Passive configuration. The following resources will be created and registered for management under Oracle GI. Note the order of creation and registration of resources will determine their dependencies.

  1. Create and register an application VIP resource
  2. Create and register a listener resource
  3. Create and register a database resource.

Database resource depends on Listener resource. Listener resource depends on Application VIP resource. The dependencies will be specified with the resource profile attributes. The specified configuration was the only available and supported prior to Oracle 11.2 and is still supported with the caveats mentioned in Note 780189.1.

  1. Create and register an application VIP resource.

    An application VIP resource needed to be created in order to have a fixed floating IP for the clients to access the database. The clients will use this application VIP regardless of the cluster node hosting the database. For details to setting the application VIP refer to previous post here.

    appvipcfg create -network=1 -ip=192.168.20.111 -vipname=MyTestVIP -user=grid

    [root@raclinux1 bin]# ./appvipcfg create -network=1 -ip=192.168.20.111 -vipname=MyTestVIP -user=root

    Production Copyright 2007, 2008, Oracle.All rights reserved

    2011-11-12 20:09:34: Creating Resource Type

    2011-11-12 20:09:34: Executing /u01/app/11.2.0/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0/grid/crs/template/appvip.type

    2011-11-12 20:09:34: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl add type app.appvip_net1.type -basetype ora.cluster_vip_net1.type -file /u01/app/11.2.0/grid/crs/template/appvip.type

    2011-11-12 20:09:34: Create the Resource

    2011-11-12 20:09:34: Executing /u01/app/11.2.0/grid/bin/crsctl add resource MyTestVIP -type app.appvip_net1.type -attr “USR_ORA_VIP=192.168.20.111,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL=’owner:root:rwx,pgrp:root:r-x,other::r–,user:root:r-x’,HOSTING_MEMBERS=raclinux1.gj.com,APPSVIP_FAILBACK=”

    2011-11-12 20:09:34: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl add resource MyTestVIP -type app.appvip_net1.type -attr “USR_ORA_VIP=192.168.20.111,START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),ACL=’owner:root:rwx,pgrp:root:r-x,other::r–,user:root:r-x’,HOSTING_MEMBERS=raclinux1.gj.com,APPSVIP_FAILBACK=”

    [root@raclinux1 bin]#

    Add the following to /etc/hosts on each cluster node.

    192.168.20.111 raclinux-vip.gj.com raclinux-vip

  2. Create and register a listener resource
    1. Create a local listener in RDBMS $OH. Choose port 1522.

      [oracle@raclinux1 admin]$ cat listener.ora

    # listener.ora Network Configuration File: /u01/app/oracle/product/11.2.0/db_3/network/admin/listener.ora

    # Generated by Oracle configuration tools.

    LISTENER_HA =

    (DESCRIPTION_LIST =

    (DESCRIPTION =

     (ADDRESS = (PROTOCOL = TCP)(HOST = raclinux-vip)(PORT = 1522))

    )

    (DESCRIPTION =

     (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1522))

    )

    )

    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_HA = ON

    ADR_BASE_LISTENER_HA = /u01/app/oracle

    [oracle@raclinux1 admin]$ cat endpoints_listener.ora

    LISTENER_HA_RACLINUX1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.20.111)(PORT=1522)(IP=FIRST)))) # line added by Agent

    [oracle@raclinux1 admin]$

    1. Propagate the listener configuration on all cluster nodes.
    2. Create an action scrip in $GI_HOME/crs/public on each cluster node

      [oracle@raclinux1 public]$ cat LISTENER_HA_actionScript.pl

      #!/usr/bin/perl

      if ($#ARGV != 0) {

       # This is an unreacheable condition

       # Oracle Clusterware would always invoke with a valid argument

      exit;

      }

      $command = $ARGV[0];

      # Start command

      if ($command eq “start”) {

      system(“su – oracle -c \”lsnrctl start listener_ha\””);

      # steps to start the resource

      exit 0; #if the start is successfull

      # exit 1 if the start has failed

      }

      # Stop command

      if ($command eq “stop”) {

      system(“su – oracle -c \”lsnrctl stop listener_ha\””);

      # steps to stop the resource

      exit 0; #if the stop is successfull

      # exit 1 if the stop has failed

      }

      # Check command

      if ($command eq “check”) {

      $found = qx(ps -ef | grep listener_ha | grep -v grep | wc -l);

      #$found = ‘ps -ef | grep smon | grep DB11GR2 | wc -l ‘;

      chomp($found);

      #print($found);

      if ($found eq “0″ ) {exit 1;} else {exit 0; }

      # steps to check the status of the resource

      # exit 0 if the resource is online

      # exit 1 otherwise

      }

      # Clean command

      if ($command eq “clean”) {

      system(“su – oracle -c \”lsnrctl stop listener_ha\””);

      # steps to force stop the resource

      exit 0; #if the force stop is successfull

      # exit 1 if the force stop has failed

      }

      [oracle@raclinux1 public]$

    3. Test the action script and make sure that performs as expected prior to integration with Oracle GI.

      [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl check

      [root@raclinux1 public]# echo $?

      0

      [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl stop

      LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 20:52:03

      Copyright (c) 1991, 2011, Oracle. All rights reserved.

      Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522)))

      The command completed successfully

      [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl check

      [root@raclinux1 public]# echo $?

      1

      [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl start

      LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 20:52:17

      Copyright (c) 1991, 2011, Oracle. All rights reserved.

      Starting /u01/app/oracle/product/11.2.0/db_3/bin/tnslsnr: please wait…

      TNSLSNR for Linux: Version 11.2.0.3.0 – Production

      System parameter file is /u01/app/oracle/product/11.2.0/db_3/network/admin/listener.ora

      Log messages written to /u01/app/oracle/diag/tnslsnr/raclinux1/listener_ha/alert/log.xml

      Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.20.111)(PORT=1522)))

      Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1522)))

      Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522)))

      STATUS of the LISTENER

      ————————

      Alias listener_ha

      Version TNSLSNR for Linux: Version 11.2.0.3.0 – Production

      Start Date 09-JAN-2012 20:52:17

      Uptime 0 days 0 hr. 0 min. 0 sec

      Trace Level off

      Security ON: Local OS Authentication

      SNMP OFF

      Listener Parameter File /u01/app/oracle/product/11.2.0/db_3/network/admin/listener.ora

      Listener Log File /u01/app/oracle/diag/tnslsnr/raclinux1/listener_ha/alert/log.xml

      Listening Endpoints Summary…

      (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.20.111)(PORT=1522)))

      (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1522)))

      The listener supports no services

      The command completed successfully

      [root@raclinux1 public]# ./LISTENER_HA_actionScript.pl check

      [root@raclinux1 public]# echo $?

      0

      [root@raclinux1 public]#

    4. Create the listener resource. Pay attention to the attributes in bold face. Note that the listener depends on Application VIP created earlier and run on the same node.

      [root@raclinux3 bin]# ./crsctl add resource Listener -type cluster_resource -attr ” ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/Listener_actionScript.pl, DESCRIPTION=Listener, DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=60, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=hard(intermediate:MyTestVIP) pullup(intermediate:MyTestVIP), STOP_DEPENDENCIES=hard(intermediate:MyTestVIP),CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, HOSTING_MEMBERS=raclinux3 raclinux1 raclinux2, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

      [root@raclinux3 bin]#

      [root@raclinux3 bin]#

      [root@raclinux1 bin]# ./crsctl start res Listener

      CRS-2679: Attempting to clean ‘Listener’ on ‘raclinux1′

      CRS-2681: Clean of ‘Listener’ on ‘raclinux1′ succeeded

      CRS-2672: Attempting to start ‘Listener’ on ‘raclinux1′

      CRS-2676: Start of ‘Listener’ on ‘raclinux1′ succeeded

      [root@raclinux1 bin]# ./crsctl status res Listener

      NAME=Listener

      TYPE=cluster_resource

      TARGET=ONLINE

      STATE=ONLINE on raclinux1

      [root@raclinux1 bin]#

  3. Create and register a database resource
    1. Create a database as in the previous section.
    2. Repeat the steps 1 to 6 inclusive from the previous section for Active/Passive with SCAN.
    3. Create an action script. I will reuse the db11gr2_actionScript.pl from the previous section. Prepare the database as in the previous section.
    4. Teet the action scripts.
    5. Create the database resource and start the resource.

      Pay attention to the dependencies! The database needs to start on the same cluster node where the listener is started. The listener starts on the node where the application VIP starts.

      [root@raclinux3 bin]# ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=dbha, DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=60, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=’hard(ora.DATA.dg,intermediate:Listener,ora.DATADG.dg) pullup(ora.DATA.dg,intermediate:Listener,ora.DATADG.dg)’, STOP_DEPENDENCIES=’hard(intermediate:ora.DATA.dg,intermediate:ora.DATADG.dg)’,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, HOSTING_MEMBERS=raclinux3 raclinux1 raclinux2, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

      [root@raclinux3 bin]#

      [root@raclinux1 bin]#

      [root@raclinux1 bin]#./crsctl modify resource db11gr2 -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=dbha, DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=60, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=’hard(ora.DATA.dg,ora.DATADG.dg,Listener) pullup(ora.DATA.dg,ora.DATADG.dg,Listener)’, STOP_DEPENDENCIES=’hard(intermediate:ora.DATA.dg,intermediate:ora.DATADG.dg,Listener)’,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, HOSTING_MEMBERS=raclinux3 raclinux1 raclinux2, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

      [root@raclinux1 bin]#

      [root@raclinux1 bin]# ./crsctl start res db11gr2

      CRS-2672: Attempting to start ‘db11gr2′ on ‘raclinux1′

      CRS-2676: Start of ‘db11gr2′ on ‘raclinux1′ succeeded

      [root@raclinux1 bin]#

    6. Setup the listener registration due to non default port( I am using port 1522).

      SQL> show parameter listener

      NAME TYPE VALUE

      ———————————— ———– ——————————

      listener_networks string

      local_listener string

      remote_listener string rac-scan:1521

      SQL>

      SQL> alter system set local_listener=’192.168.20.111:1522′ scope=both;

      System altered.

      SQL>

    7. Setup Oracle Net

      DB11 =

      (DESCRIPTION =

      (ADDRESS = (PROTOCOL = TCP)(HOST = raclinux-vip)(PORT = 1522))

      (CONNECT_DATA =

      (SERVER = DEDICATED)

      (SERVICE_NAME = DB11GR2)

      )

      )

      Test

      [oracle@raclinux3 ~]$ lsnrctl service listener_ha

      LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 14:48:22

      Copyright (c) 1991, 2011, Oracle. All rights reserved.

      Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux-vip)(PORT=1522)))

      Services Summary…

      Service “DB11GR2″ has 1 instance(s).

       Instance “DB11GR2″, status READY, has 1 handler(s) for this service…

      Handler(s):

      “DEDICATED” established:0 refused:0 state:ready

      LOCAL SERVER

      Service “DB11GR2XDB” has 1 instance(s).

       Instance “DB11GR2″, status READY, has 1 handler(s) for this service…

      Handler(s):

       “D000″ established:0 refused:0 current:0 max:1022 state:ready

      DISPATCHER <machine: raclinux3.gj.com, pid: 10694>

      (ADDRESS=(PROTOCOL=tcp)(HOST=raclinux3.gj.com)(PORT=18626))

      The command completed successfully

      [oracle@raclinux3 ~]$

      [oracle@raclinux1 admin]$ tnsping db11

      TNS Ping Utility for Linux: Version 11.2.0.3.0 – Production on 09-JAN-2012 14:54:48

      Copyright (c) 1997, 2011, Oracle. All rights reserved.

      Used parameter files:

      Used TNSNAMES adapter to resolve the alias

      Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = raclinux-vip)(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = DB11GR2)))

      OK (110 msec)

      [oracle@raclinux1 admin]$ sqlplus system/sys1@db11

      SQL*Plus: Release 11.2.0.3.0 Production on Mon Jan 9 14:55:01 2012

      Copyright (c) 1982, 2011, Oracle. All rights reserved.

      Connected to:

      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

      With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

      Data Mining and Real Application Testing options

      SQL>

    8. Test server failure

      [root@raclinux3 bin]# ./crsctl stat res -t

      ——————————————————————————–

      NAME TARGET STATE SERVER STATE_DETAILS

      ——————————————————————————–

      Local Resources

      ——————————————————————————–

      ora.DATA.dg

       ONLINE ONLINE raclinux1

       ONLINE ONLINE raclinux2

       ONLINE ONLINE raclinux3

      ……………………………………………

      ——————————————————————————–

      Cluster Resources

      ——————————————————————————–

      Listener

       1 ONLINE ONLINE raclinux3

      MyTest

       1 ONLINE ONLINE raclinux3

      MyTestVIP

       1 ONLINE ONLINE raclinux3

      MyTestVIP1

       1 ONLINE ONLINE raclinux1

      db11gr2

       1 ONLINE ONLINE raclinux3

      ora.LISTENER_SCAN1.lsnr

       1 ONLINE ONLINE raclinux1

      …………………………………………………………….

      [root@raclinux3 bin]#

      Kill raclinux3

      [root@raclinux3 bin]# ./crsctl stat res -t

      ……………………………………………….

      ——————————————————————————–

      Cluster Resources

      ——————————————————————————–

      Listener

       1 ONLINE ONLINE raclinux1

      MyTest

       1 ONLINE ONLINE raclinux1

      MyTestVIP

       1 ONLINE ONLINE raclinux1

      MyTestVIP1

       1 ONLINE ONLINE raclinux1

      db11gr2

       1 ONLINE ONLINE raclinux1

      ……………………………………………

    9. Test process crash. Similarly to the previous section Oracle GI restarts the resource only once. On the second restart the resource fails over. Number of restart prior to failover can be modified by RESTART_ATTEMPTS.

Active/Passive configuration using Oracle RAC One Node

For information about installing and managing Oracle RAC One Node click here.

Oracle RAC One Node is RAC-enabled single instance database that runs on only one node in the cluster. In case of a failure of the node where RAC One Node is running the instance fails over to another node of the cluster. Upon Oracle RAC One Node installation the user specifies:

  1. List of nodes for the HOSTING_MEMBERS to limit the nodes where RAC One node will run in an admin-managed database.
  2. A server pool for the policy managed RAC One Node Database.

Added advantage for Oracle RAC One Node is the tight integration with srvctl utility and ability to perform an online relocation to another cluster node. For the previous two cases you cannot use srvctl utility to manage Active/Passive configurations and do not have ability for an online relocation to another cluster node. Oracle RAC One Node is an additional cost option to the Oracle Enterprise Edition and presently does not support Data Guard. In note 790189.1 Oracle explicitly states that Oracle RAC One Node will be a direction that will be followed for failing over an Oracle 11.2 database and newer.

Sources

  1. Using Oracle Clusterware to Protect Oracle Database 10g with Oracle Enterprise Manager Grid Control Integration
  2. Oracle Clusterware and Application Failover Management [ID 790189.1]

January 9, 2012 - Posted by | oracle

43 Comments »

  1. Guenadi

    There is a problem with SCAN approach.

    For instance if the database is running in node1 when the public network is down in this node then the connections to the database will fail because VIP has failed over to node2. Have you tested it?

    May be a solution is add a hard stop dependency ora.LISTENER.lsnr or ora.net1.network. I still need to test.

    Regards

    Comment by lscheng | February 4, 2012 | Reply

    • Hi Lishan,

      Good and valid point! Thank you for pointing it out!

      I did not test a public network failure. The database should follow and failover with the VIP if you intend to protect agains public network failure. From present definition

      ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=, HOSTING_MEMBERS=raclinux1 raclinux2 raclinux3,DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=30, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=, STOP_DEPENDENCIES=,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

      with a NEW dependency

      START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network),

      to a NEW resource definition

      ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=, HOSTING_MEMBERS=raclinux1 raclinux2 raclinux3,DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=30, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network), CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

      I will test and update the article. Listener does not failover.

      Best Regards,

      Comment by gjilevski | February 5, 2012 | Reply

    • Hi,

      If you modify the attributes as follows

      crsctl modify resource db11gr2 -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=, DEGREE=1, ENABLED=1, AUTO_START=restore, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=30, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network),STOP_DEPENDENCIES=hard(ora.net1.network), CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*, PLACEMENT=balanced, LOAD=1, ACTIVE_PLACEMENT=0″

      and test both node failure or public interface failure you will get a database failover. See the test results for db11gr2

      [root@raclinux1 bin]# ./crsctl stat res -t
      ——————————————————————————–
      NAME TARGET STATE SERVER STATE_DETAILS
      ——————————————————————————–
      Local Resources
      ——————————————————————————–
      ora.DATA.dg
      ONLINE ONLINE raclinux1
      ONLINE ONLINE raclinux2
      ora.DATADG.dg
      ONLINE ONLINE raclinux1
      ONLINE ONLINE raclinux2
      ora.LISTENER.lsnr
      ONLINE ONLINE raclinux1
      ONLINE OFFLINE raclinux2
      ora.asm
      ONLINE ONLINE raclinux1 Started
      ONLINE ONLINE raclinux2 Started
      ora.gsd
      OFFLINE OFFLINE raclinux1
      OFFLINE OFFLINE raclinux2
      ora.net1.network
      ONLINE ONLINE raclinux1
      ONLINE OFFLINE raclinux2
      ora.ons
      ONLINE ONLINE raclinux1
      ONLINE OFFLINE raclinux2
      ora.registry.acfs
      ONLINE ONLINE raclinux1
      ONLINE ONLINE raclinux2
      ——————————————————————————–
      Cluster Resources
      ——————————————————————————–
      Listener
      1 ONLINE ONLINE raclinux1
      MyTest
      1 ONLINE UNKNOWN raclinux1 CHECK TIMED OUT
      MyTestVIP
      1 ONLINE ONLINE raclinux1
      MyTestVIP1
      1 ONLINE ONLINE raclinux1
      db11gr2
      1 ONLINE ONLINE raclinux1
      ora.LISTENER_SCAN1.lsnr
      1 ONLINE ONLINE raclinux1
      ora.RAC10G.RAC10G1.inst
      1 ONLINE ONLINE raclinux1
      ora.RAC10G.RAC10G2.inst
      1 ONLINE ONLINE raclinux2
      ora.RAC10G.db
      1 ONLINE ONLINE raclinux1
      ora.cvu
      1 ONLINE ONLINE raclinux1
      ora.db11gr2.db
      1 OFFLINE OFFLINE Instance Shutdown
      ora.oc4j
      1 ONLINE ONLINE raclinux1
      ora.racdb.db
      1 ONLINE OFFLINE
      2 ONLINE ONLINE raclinux2 Open
      3 ONLINE ONLINE raclinux1 Open
      ora.raclinux1.vip
      1 ONLINE ONLINE raclinux1
      ora.raclinux2.vip
      1 ONLINE INTERMEDIATE raclinux1 FAILED OVER
      ora.raclinux3.vip
      1 ONLINE INTERMEDIATE raclinux1 FAILED OVER
      ora.rone.db
      1 ONLINE ONLINE raclinux1 Open
      ora.rone.ronesrvs.svc
      1 ONLINE ONLINE raclinux1
      ora.scan1.vip
      1 ONLINE ONLINE raclinux1
      [root@raclinux1 bin]# ifdown eth0
      [root@raclinux1 bin]# date
      Mon Feb 6 08:33:45 EET 2012
      [root@raclinux1 bin]# ./crsctl stat res -t
      ——————————————————————————–
      NAME TARGET STATE SERVER STATE_DETAILS
      ——————————————————————————–
      Local Resources
      ——————————————————————————–
      ora.DATA.dg
      ONLINE ONLINE raclinux1
      ONLINE ONLINE raclinux2
      ora.DATADG.dg
      ONLINE ONLINE raclinux1
      ONLINE ONLINE raclinux2
      ora.LISTENER.lsnr
      ONLINE OFFLINE raclinux1
      ONLINE ONLINE raclinux2
      ora.asm
      ONLINE ONLINE raclinux1 Started
      ONLINE ONLINE raclinux2 Started
      ora.gsd
      OFFLINE OFFLINE raclinux1
      OFFLINE OFFLINE raclinux2
      ora.net1.network
      ONLINE OFFLINE raclinux1
      ONLINE ONLINE raclinux2
      ora.ons
      ONLINE OFFLINE raclinux1
      ONLINE ONLINE raclinux2
      ora.registry.acfs
      ONLINE ONLINE raclinux1
      ONLINE ONLINE raclinux2
      ——————————————————————————–
      Cluster Resources
      ——————————————————————————–
      Listener
      1 ONLINE ONLINE raclinux2
      MyTest
      1 ONLINE UNKNOWN raclinux1 CHECK TIMED OUT
      MyTestVIP
      1 ONLINE ONLINE raclinux2
      MyTestVIP1
      1 ONLINE ONLINE raclinux2
      db11gr2
      1 ONLINE ONLINE raclinux2
      ora.LISTENER_SCAN1.lsnr
      1 ONLINE ONLINE raclinux2
      ora.RAC10G.RAC10G1.inst
      1 ONLINE ONLINE raclinux1
      ora.RAC10G.RAC10G2.inst
      1 ONLINE ONLINE raclinux2
      ora.RAC10G.db
      1 ONLINE ONLINE raclinux1
      ora.cvu
      1 ONLINE ONLINE raclinux2
      ora.db11gr2.db
      1 ONLINE ONLINE raclinux2 Open
      ora.oc4j
      1 ONLINE ONLINE raclinux1
      ora.racdb.db
      1 ONLINE OFFLINE
      2 ONLINE ONLINE raclinux2 Open
      3 ONLINE ONLINE raclinux1 Open
      ora.raclinux1.vip
      1 ONLINE INTERMEDIATE raclinux2 FAILED OVER
      ora.raclinux2.vip
      1 ONLINE ONLINE raclinux2
      ora.raclinux3.vip
      1 ONLINE INTERMEDIATE raclinux2 FAILED OVER
      ora.rone.db
      1 ONLINE ONLINE raclinux2 Open
      ora.rone.ronesrvs.svc
      1 ONLINE ONLINE raclinux2
      ora.scan1.vip
      1 ONLINE ONLINE raclinux2
      [root@raclinux1 bin]#

      [root@raclinux1 bin]# date
      Mon Feb 6 08:41:12 EET 2012
      [root@raclinux1 bin]#

      Comment by gjilevski | February 6, 2012 | Reply

  2. Good day.

    I should install oracle software for DB on each node of cluster ?

    Comment by oleg | April 24, 2012 | Reply

    • Hi,

      If you intend to run a database on a node you will need to install RDBMS software on that node.

      Regards,

      Comment by gjilevski | April 24, 2012 | Reply

  3. But if failover occurs , i need oracle sofware on another node (/bin directory) , to start my single instance database.

    Or I can install oracle software on network device whith ext3 fs (or ocfs2), and each node of cluster can restart oracle instance when First node will down.

    Comment by oleg | April 25, 2012 | Reply

    • Hi,

      If you intend to use a node to host a database ( pre or post failover) than install the RDBMS software on the node.
      If you intend to run a database on a node ( before or after a failover) you will need to install RDBMS software on that node.

      Regards,

      Comment by gjilevski | April 25, 2012 | Reply

  4. thank you

    Comment by oleg | April 25, 2012 | Reply

  5. I have performed following steps in my environment.

    - Installed GI(11.2.0.3) and Database (11.2.0.3) software on both node

    -created database on active node with DBCA. This will register ora.ora11g.db
    under CLUSTER_RESOURCES.
    - performed NODE FAILOVER test. When OS analyst had shutdown the active node
    +ASM instance and database had came on passive node and vice versa.

    - I haven’t created SCAN yet. Before requesting SCAN, I would like to perform
    Process kill test. If any of the oracle process got killed by any reason, Active
    node must be fail over to passive node.For that, I had used the action scripts
    mentioned in this artical and put that scripts on both node under $GRID_HOME/crs/public
    as root user. root own these scripts. Tested scripts, worked fine

    - Since DBCA already created database under cluster_resource, I had used following command to modify resource
    in order to change following configuration.
    HOSTING_MEMBERS, RESTART_ATTEMPTS, CARDINALITY, ACTION_SCRIPT.

    # ./crsctl modify resource ora.ora11g.db -attr ”
    AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent,
    ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/actionScript.pl,
    DESCRIPTION=, HOSTING_MEMBERS=host1 host2,
    DEGREE=1,
    ENABLED=1,
    AUTO_START=always,
    START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=30, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

    - it seems command ran successfully on active node but when i checked the update using following command,
    the modification is not visible.

    #./crsctl stat res ora.ora11g.db -v

    Hence, when i killed pmon , failover to passive node didn’t happen.

    QUESTION:

    1) How can i check the modification to resource ora.ora11g.db ?
    2) Even, I am not able to perform manual resource relocation between 2 node as per command listed in this artical such as

    ./crsctl relocate res db11gr2 -s raclinux2 -n raclinux3

    Thanks,

    Comment by John Parker | November 14, 2012 | Reply

    • Hi,

      I looked briefly at what you are suggesting and noticed that

      1. You should never, under any circumstance, modify ora. resource with crsctl. Use only srvctl to modify ora. resource
      2. Create your own resource
      3. Use the created resource to manage the database.

      You could modify the resource by

      1. Obtain the attributes of ora.db resource
      2. drop/dissable the ora.db resource
      3. Create a resource with modified original ora.db attributes
      4. Use the resource created in step 3 above from there onward

      Regards,

      Comment by gjilevski | November 14, 2012 | Reply

      • I have one database created using DBCA. Do I need to created another resource to manage failover between two node ? Or I can use Srvctl to modify that database .? I have tested actionscript against that database .

        Comment by John parker | November 14, 2012

      • Hi John,

        You need to create a new resource! You can use my article as a guideline. You should not modify ora.* resource using crsctl! Make sure that the database is not a cluster database, that is, a database handled by GI CRS. Remove altogether whatever ora. resource corresponds to the database and add it from scratch.

        Regards,

        Comment by gjilevski | November 14, 2012

      • Hi

        Can you paste result of ./crsctl stat res -t command of your configuration ?

        What happens when you tried to kill process ( Listener, ohasd, css,asm) ? Did active node failed over passive node when u killed these processes ?

        Thanks

        Comment by John parker | November 16, 2012

      • Hi,

        The output of ./crsctl stat res -t is in the article. Look also at the previous comments.

        I considered two cases

        1. HA with SCAN
        2. HA without SCAN

        I tested the following failures

        1. Interface down
        2. Node down
        3. DB down

        In the above failure cases the resource failed over to the other node.

        Oracle Clusterware failures result in either

        1. Restart of the Clusterware process
        2. Node restart

        For ASM failure add ASM in the list DEPENDENCIES
        Should look similar to as in the article
        START_DEPENDENCIES=’hard(ora.DATA.dg,ora.DATADG.dg,Listener) pullup(ora.DATA.dg,ora.DATADG.dg,Listener …..)’, STOP_DEPENDENCIES=’hard(intermediate:ora.DATA.dg,intermediate:ora.DATADG.dg,Listener …..)’,

        Regards,

        Comment by gjilevski | November 17, 2012

  6. - I am working/implementing HA with SCAN. oracle version 11.2.0.3
    - Node fail (restart) test went successfully. db & cluster resources are able to fail-over between active/passive node.
    - Stop/start “ohas” process between node went successfully.
    - Currently I am testing process kill test.I am using HA with SCAN artical. Performed step 1 to 6 successfully. I am able to perform step 1, create resource using root user. & step 2 (Set the resource permissions) . When i tried to test start/stop resource as per step 3, I am getting following error.

    CRS-2672: Attempting to start ‘test’ on ‘node1′
    CRS-2674: Start of ‘test’ on ‘node1′ failed
    CRS-2679: Attempting to clean ‘test’ on ‘node1′
    CRS-2678: ‘test’ on ‘node1′ has experienced an unrecoverable failure
    CRS-0267: Human intervention required to resume its availability.
    CRS-5802: Unable to start the agent process
    CRS-4000: Command Start failed, or completed with errors.

    ——————-
    Ohasd.log
    ——————-

    2012-11-16 15:44:47.548: [ AGFW][1127258432] {0:0:424} Created alert : (:CRSAGF00123:) : Failed to start the agent process: /grid/oracle/11.2.0.3/bin/scriptagent Category: -1 Operation: fail Loc: canexec2 OS error: 0 Other : no exe permission,
    file [/grid/oracle/11.2.0.3/bin/scriptagent]

    - Look like it is permission issue but The permission of scriptagent is set to default. I didn’t change it. I did research on metalink but haven;t found any artical describing this error/solution

    Do we need to change permission of scriptagent ? Did you received this error during implementation. ?

    Also, I had created actionscript using root user and oracle and root user has execute permission on them.

    Thank you!

    Comment by John Parker | November 18, 2012 | Reply

    • Hi,

      I did encounter similar error while geting GI control application resources. As far as I remember Oracle distinguishes between the user that creates the resource and the user that executes the resource on behalf of the original user.

      That is, if you create with GI owner grant permission to root and vice versa. Let me know how it goes.
      Did you pay attention to the permission part?

      2.Set the resource permissions

      [root@raclinux2 bin]# ./crsctl getperm resource db11gr2

      Name: db11gr2

      owner:root:rwx,pgrp:root:r-x,other::r–

      [root@raclinux2 bin]#

      [root@raclinux2 bin]# ./crsctl setperm resource db11gr2 -u user:oracle:rwx

      [root@raclinux2 bin]# ./crsctl setperm resource db11gr2 -u user:grid:rwx

      [root@raclinux2 bin]#

      [root@raclinux2 bin]# ./crsctl getperm resource db11gr2

      Name: db11gr2

      owner:root:rwx,pgrp:root:r-x,other::r–,user:oracle:rwx,user:grid:rwx

      [root@raclinux2 bin]#

      Comment by gjilevski | November 18, 2012 | Reply

      • Yes, I did step 2.Set the resource permissions. In my implementation, oracle user is the owner of GI and oracle database software. I had installed GI software and oracle database software using oracle user.

        # ./crsctl getperm resource test

        Name: test
        owner:root:rwx,pgrp:root:r-x,other::r–

        #./crsctl setperm resource test -u user:oracle:rwx

        # ./crsctl getperm resource test
        owner:root:rwx,pgrp:root:r-x,other::r–,user:oracle:rwx

        Comment by John | November 19, 2012

      • I am able to move forward now .

        Deleted test resource and recreated using oracle user. Resource test is online now on active node ‘node1′

        $ ./crsctl getperm res test
        Name: test
        owner:oracle:rwx,pgrp:dba:rwx,other::r–,user:root:rwx

        In step 3: test and start resource

        I am able to start resource .Once the resource is started, “ps” should show scriptagent.bin.

        ps -ef|grep scriptagent.bin
        oracle 10263 1 0 11:56 ? 00:00:00 /grid/oracle/11.2.0.3/bin/scriptagent.bin

        but when I tried to relocate resource test, it gave following error

        $ ./crsctl relocate res test -s node1 -n node2
        CRS-2546: Server ‘node2′ is not online
        CRS-4000: Command Relocate failed, or completed with errors.

        I have 2 node cluster. This error is obvious since node2 is passive node and there are no cluster and database resource runing.

        Did you face this error ?

        Further , to check whether ‘test’ is falling over to node2 or not, I had stopped OHasd process. All resources failed over to node2 but not resource “test ”

        Do we need to perform any other steps on passive node 2 in order to make resource visible and online all time when failover occurs?

        Que: what are the permission set on all Actionscripts ? I have root user ownership on them since i created scripts using root .

        -rwxr-xr-x 1 root root 1281 nov 10 14:00 actionscript.pl
        Thank you

        Comment by John parker | November 19, 2012

      • Hi John

        The I idea I POC was to have GI running on all nodes of the cluster ,that is Oracle Clusterware is running on all nodes. The Active-Passive categorization was from the stand point of the database, that is the database in case of a failure can failover to another node that s present but not servicing the database. Only one cluster node service the database at a time, but can be used for something else it is not servicing the database at the moment. The node is referred as a Passive since the database is not there, but the node is a part of the cluster.
        Having said that my comments are as follow:

        1. If you are referring to a node down or offline where you are attempting to failover to, I could expect the observed error. A node is available for a resource failover only if is ONLINE.
        2. There could be a lot of failures. In production systems it can take 1-2 weeks to test all possible failure scenarios. I did quite thorough test. In the article I mention few such as
        a. Kill an instance
        b. Interface failure
        c. Node down
        3. Once the node where it is supposed to failover is online Oracle Clusterware takes care of the failover.
        4. The scripts should be executable by root and GI installation owner

        Regards,

        Comment by gjilevski | November 20, 2012

      • It seems I did something wrong.

        1) as per mealink note, I had install GI software with option ” grid infrastructure for stand Alone server”. And I did separate installation on both node as per Metalink note.

        Can you conf

        Comment by John parker | November 20, 2012

      • Hi,

        You need a RAC GI install on all nodes.

        Regards,

        Comment by gjilevski | November 20, 2012

      • It seems I did something wrong

        (1) as per Metalink note, I had installed GI with option ” grid infrastructure for standalone server”

        Which option you had used for GI installation. ? ” GI for Cluster” or ” GI for standalone ” ?

        (2) I had created database using DBCA which register the database ” ora. Test.db” automatically protected by oracle restart technology available with GI. Hence it restarts automatically after any crash ( kill -9 pmon )

        Oracle restart is not cluster and cannot restart the database on any other nod of the cluster.
        Did you remove database before creating resource to manage database??

        I think thiese 2 question will help me to correct my implementation

        Thank you !!

        Comment by John parker | November 20, 2012

      • Hi,

        1. GI for Cluster on all nodes.
        2. Oracle RDBMS on all nodes – binary only no database.
        3. Create a NON-cluster database and protect it as in the article.

        Regards,

        Comment by gjilevski | November 20, 2012

  7. - Created resource . Resource status is

    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    TEST.DB
    1 OFFLINE OFFLINE

    While starting resource , I am getting following error.

    ./crsctl start res TEST.db
    CRS-2672: Attempting to start ‘TEST.db’ on ‘node1′
    CRS-2674: Start of ‘TEST.db’ on ‘node1′ failed
    CRS-2679: Attempting to clean ‘TEST.db’ on ‘node1′
    CRS-2681: Clean of ‘TEST.db’ on ‘node1′ succeeded
    CRS-2563: Attempt to start resource ‘TEST.db’ on ‘node1′ has failed. Will re-retry on ‘node2′ now.
    CRS-2672: Attempting to start ‘TEST.db’ on ‘node2′
    CRS-2674: Start of ‘TEST.db’ on ‘node2′ failed
    CRS-2679: Attempting to clean ‘TEST.db’ on ‘node2′
    CRS-2681: Clean of ‘TEST.db’ on ‘node2′ succeeded
    CRS-2632: There are no more servers to try to place resource ‘TEST.db’ on that would satisfy its placement policy
    CRS-4000: Command Start failed, or completed with errors.

    Resource status :

    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    TEST.DB
    1 ONLINE OFFLINE

    - All directory structure are same on both node.

    Any one had faced this issue ?

    Thanks,

    Comment by John | December 1, 2012 | Reply

    • Hi,

      1. How many nodes are in the cluster? Is the cluster running on all nodes?
      2. How did you create the resource? is it like?

      [root@raclinux2 bin]# ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=, HOSTING_MEMBERS=raclinux1 raclinux2 raclinux3,DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=30, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network), STOP_DEPENDENCIES=hard(ora.net1.network),CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

      or

      root@raclinux3 bin]# ./crsctl add resource db11gr2 -type cluster_resource -attr ” AGENT_FILENAME=”%”CRS_HOME”%”/bin/scriptagent, ACTION_SCRIPT=/u01/app/11.2.0.3/grid/crs/public/db11gr2_actionScript.pl, DESCRIPTION=dbha, DEGREE=1, ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h, CHECK_INTERVAL=60, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60, RESTART_ATTEMPTS=1, OFFLINE_CHECK_INTERVAL=0, START_DEPENDENCIES=’hard(ora.DATA.dg,intermediate:Listener,ora.DATADG.dg) pullup(ora.DATA.dg,intermediate:Listener,ora.DATADG.dg)’, STOP_DEPENDENCIES=’hard(intermediate:ora.DATA.dg,intermediate:ora.DATADG.dg)’,CARDINALITY=1, FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, HOSTING_MEMBERS=raclinux3 raclinux1 raclinux2, PLACEMENT=restricted, LOAD=1, ACTIVE_PLACEMENT=0″

      3. What does ./crsctl stat res db11gr2 -v or ./crsctl stat res db11gr2 shows

      Regards,

      Comment by gjilevski | December 2, 2012 | Reply

  8. 1. Two node cluster (node1, node2). Yes, Cluster is running on both node. ASM instances are up and running on both node. (asm1, asm2)

    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    TEST.db
    1 ONLINE OFFLINE
    ora.LISTENER_SCAN1.lsnr
    1 ONLINE ONLINE node2
    ora.LISTENER_SCAN2.lsnr
    1 ONLINE ONLINE node1
    ora.LISTENER_SCAN3.lsnr
    1 ONLINE ONLINE node1
    ora.cvu
    1 ONLINE ONLINE node1
    ora.oc4j
    1 ONLINE ONLINE node1
    ora.scan1.vip
    1 ONLINE ONLINE node2
    ora.scan2.vip
    1 ONLINE ONLINE node1
    ora.scan3.vip
    1 ONLINE ONLINE node1
    ora.node1.vip
    1 ONLINE ONLINE node1
    ora.node2.vip
    1 ONLINE ONLINE node2

    2. created resource using resource file made from database which was created using DBCA. and created new resource using oracle user. not root.

    $ ./crsctl add resource TEST.db -type cluster_resource -file /grid/oracle/11.2.0/crs/public/resource.txt

    $cat resource.txt

    NAME=TEST.db
    TYPE=cluster_resource
    ACL=owner:oracle:rwx,pgrp:dba:rwx,other::r–
    ACTION_SCRIPT=/grid/oracle/11.2.0/crs/public/activepassive.sh
    ACTIVE_PLACEMENT=0
    AUTO_START=always
    CARDINALITY=1
    CHECK_INTERVAL=30
    DEGREE=1
    DESCRIPTION=Oracle Database resource
    ENABLED=1
    FAILURE_INTERVAL=0
    FAILURE_THRESHOLD=0
    HOSTING_MEMBERS=node1 node2
    LOAD=1
    LOGGING_LEVEL=1
    OFFLINE_CHECK_INTERVAL=0
    PLACEMENT=restricted
    RESTART_ATTEMPTS=1
    SCRIPT_TIMEOUT=60
    START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLAS.dg) weak(type:ora.listener.type,uniform:ora.ons) pullup(ora.DATA.dg,ora.FLAS.dg)
    START_TIMEOUT=0
    STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.FLAS.dg)
    STOP_TIMEOUT=0
    UPTIME_THRESHOLD=1h
    SERVER_POOLS=*

    3.) $ ./crsctl stat res TEST.db -v
    NAME=TEST.db
    TYPE=cluster_resource
    LAST_SERVER=node2
    STATE=OFFLINE
    TARGET=ONLINE
    CARDINALITY_ID=1
    CREATION_SEED=79
    RESTART_COUNT=0
    FAILURE_COUNT=0
    FAILURE_HISTORY=
    ID=TEST.db 1 1
    INCARNATION=0
    LAST_RESTART=NEVER
    LAST_STATE_CHANGE=NEVER
    STATE_DETAILS=
    INTERNAL_STATE=STABLE

    - My action script:

    #!/bin/bash
    export ORACLE_HOME=/oracle/product/11.2.0
    export ORACLE_SID=TEST
    export PATH=$PATH:$ORACLE_HOME/bin

    case $1 in
    ‘start’)
    $ORACLE_HOME/bin/sqlplus /nolog <<EOF
    connect / as sysdba
    startup
    exit
    EOF
    RET=0
    ;;
    'stop')
    $ORACLE_HOME/bin/sqlplus /nolog <<EOF
    connect / as sysdba
    shutdown immediate
    exit
    EOF
    RET=0
    ;;
    'clean')
    $ORACLE_HOME/bin/sqlplus /nolog <<EOF
    connect / as sysdba
    shutdown abort
    exit
    EOF
    RET=0
    ;;
    'check')

    found=`ps -ef | grep smon | grep $ORACLE_SID | wc -l`
    if [ $found = 0 ]; then
    RET=1
    else
    RET=0
    fi
    ;;
    *)
    RET=0
    ;;
    esac
    if [ $RET -eq 0 ]; then
    exit 0
    else
    exit 1
    fi

    Thanks,

    Comment by John | December 2, 2012 | Reply

    • Hi,

      Pay attention that I have set HOSTING_MEMBERS=raclinux1 raclinux2 raclinux3
      You do not have it.

      Regards,

      Comment by gjilevski | December 2, 2012 | Reply

  9. My configuration i have 2 node ( active and passive) only. Hence I have mentioned HOSTING_MEMBERS=node1 node2.

    Comment by John | December 2, 2012 | Reply

    • Hi,

      Look at the logs. Also look for any ora.* resourse associated with the database, if any remove it. Ideally, you shoud have used the dbca and create a stand alone database only, that is not registered with the GI.

      Regards,

      Comment by gjilevski | December 2, 2012 | Reply

  10. There are no any ora..db resource in my cluster. I had created database using DBCA. and then droped it from OCR registry using srvctl. then , i have created new resource which is cluster type resource.

    log location I am using is $GRID_HOME/logs/crsd/crsd.log. I don’t see any error there . Where is the exact location of log which give detail error information.?

    Comment by John | December 3, 2012 | Reply

    • Hi,

      There might be many thinbgs that can cause this. Please check that

      1. Clusterware is running on all nodes
      2. Check the logs in a directory that in my case is on ALL NODES
      [root@raclinux1 raclinux1]# pwd
      /u01/app/11.2.0.3/grid/log/raclinux1
      [root@raclinux1 raclinux1]# ls
      acfs acfssec client cssd evmd mdnsd
      acfslog admin crflogd ctssd gipcd ohasd
      acfsrepl agent crfmond cvu gnsd racg
      acfsreplroot alertraclinux1.log crsd diskmon gpnpd srvm
      [root@raclinux1 raclinux1]#

      3. Repeat the test.

      Please let me know how it goes.

      Regards,

      Comment by gjilevski | December 4, 2012 | Reply

    • Hi,

      Also look at the output from olsnodes. Make sure that the nodes are listed in the HOSTING_MEMBERS parameter. Majke sure that GI is running on allnodes and that the network hostname are resolvable.

      Regards,

      Guenadi Jilevski

      Comment by gjilevski | December 5, 2012 | Reply

  11. Hi,

    Thanks for your post, it did help me a lot.
    But I have a problem, the resource when I restart the node 1, goes to node 2 but when node 1 is alive again, the resource doesn’t return to node 1.

    Is that the expected behaviour or not ?.

    Thanks.

    German.

    Comment by GPabon | January 29, 2013 | Reply

    • Hi,

      This is an expected and normal behavior. The GI performs the failover or restart but not the fall back. You will need to fail it over yourself.

      Best Regards,

      Comment by gjilevski | January 30, 2013 | Reply

  12. Hi,

    Yes, I know the command… What I mean is the Grid Infrastructure does it automatically when the “master node” becomes stable again.

    Thanks.

    Comment by GPabon | January 31, 2013 | Reply

    • Hi,

      GI does not restore/relocate automatically the resource for you when the original node become available and stable. You might want to do it manually or let it run until the node it is running on fails and resource is failed over.

      Best Regards

      Comment by gjilevski | January 31, 2013 | Reply

  13. Hi,
    for this scenario, we have to install clusterware as standalone or cluster.

    Comment by shahid | March 18, 2013 | Reply

    • Hi,

      Install the GI onthe cluster where you want for the databases to reside ( before or after the failover)

      Best Regards,

      Comment by gjilevski | March 18, 2013 | Reply

  14. Hi guenadi,

    My scenario and requirement is

    Have to implement active-passive failover configuration with 4 database running in seperate resource group.

    Please help me to decide the approach. Appreciate your time and help.

    I have a deadline to finsh this task by today.

    Comment by shahid | March 18, 2013 | Reply

    • Hi,

      Look at the article and the other available Oracle documentations and RAC One node feature and decide what best fits for the requirements that you have and the SLA you must meet.

      Best Regards,

      Comment by gjilevski | March 19, 2013 | Reply

  15. Amazing things here. I am very happy to see your post.
    Thank you a lot and I am having a look forward to touch you.
    Will you please drop me a e-mail?

    Comment by california | July 2, 2013 | Reply


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 771 other followers

%d bloggers like this: