DB2 WITH HACMP

DB2 WITH HACMP

HACMP™ Smart Assist for DB2® extends an existing HACMP configuration to include monitoring and recovery support for DB2 Universal Database™ (UDB) Enterprise Server Edition.

HACMP Smart Assist for DB2 allows you to automatically configure HACMP in an environment where DB2 is already configured to make non-partitioned DB2 instances highly available.

HACMP Smart Assist cannot be used to configure a cluster in a partitioned (DB2 UDB DPF) environment.

To make use of HACMP Smart Assist for DB2, the DB2 UDB software must be installed on all nodes that are going to be part of the cluster, and DB2 instances must be configured on some of the nodes.

The shared physical disks that support the DB2 instances must be configured on all cluster nodes that will serve as takeover nodes for the DB2 instances.

HACMP Smart Assist automatically discovers your DB2 configuration and lets you create either one of the following cluster configurations:

A hot standby two-node cluster

A mutual takeover two-node cluster

What is A mutual takeover two-node cluster ?

A cluster with more than two nodes and more than two instances where some of the nodes serve as takeover nodes for the configured DB2 instances.

How HACMP works with DB2 ?

Automatically monitors DB2 instances running on the nodes. If an application instance becomes unavailable, HACMP tries to restart the application three times on the node. If the application does not start after three attempts, HACMP starts the application on another HACMP cluster node. (This is the default behavior which you can change).

Planning:

• Make sure that DB2 installation path are same across all participating nodes in HACMP cluster.
• The DB2® instance should be installed in the DB2 instance owner's home directory. HACMP™ scripts (in particular, the discovery script) assume this.
• When you install the DB2 server, create a DB2 database instance only from the primary server.
• Make sure that there is an appropriate .rhosts file (as required by DB2), in the home directory of the database instance owner to start and stop the database instance.
• Specify the IP service label for the hostname in the db2nodes.cfg file and in the /db2home1/db2inst1/ .rhosts file in your environment. (Note, the directory path can be different, depending on your configuration. This path is provided here only as an example.)
• Ensure that the DB2 server on both the home and the standby nodes have the same configuration settings for:
  • User access
    You can configure user privileges through HACMP, and synchronize those user accounts among the nodes in a cluster. The DB2 instance users and groups should have the same ID on all nodes in the cluster.
  • Port number assignments
    DB2 adds the port numbers to the /etc/services file on the primary cluster node for the DB2 instance when the database instance is created. Copy these port numbers to the /etc/services file on the other cluster nodes that will be included in the resource group containing this DB2 instance.
    The following shows example entries in an /etc/services file:

    DB2_db2inst1 60000/tcp
    DB2_db2inst1_1 60001/tcp
    DB2_db2inst1_2 60002/tcp
    DB2_db2inst1_END 60003/tcp
    db2c_db2inst1 50000/tcp

  During HACMP cluster verification, the cluster verification utility checks for this condition and, in the absence of a valid configuration, automatically corrects this condition for you.
  In configurations that have more than one DB2 instance in the cluster, HACMP adds the port entries in such a way as to avoid port conflicts.

Turn off Automatic start of DB2 Fault monitor:

If the DB2 Fault Monitor Coordinator is set to automatically start, the /etc/inittab file contains the following entry:

fmc:2:respawn:<DB2DIR>/bin/db2fmcd #DB2 Fault Monitor Coordinator, 

   Where DB2DIR is the installed path for DB2.

To turn off automatic restart of the DB2 Fault Monitor Coordinator, enter the following command:

“# chitab "fmc:2:off:<DB2DIR>/bin/db2fmcd #DB2 Fault Monitor Coordinator, 

   Where DB2DIR is the installed path for DB2.

# chitab "ibmdir:2:off:/usr/ldap/sbin/rc.ibmdir > /dev/null 2>&1"

Turning off automatic start up of DB2 instance on NODE:

Ensure that the DB2® instance is not set to automatically restart on system startup.

To turn off this function, you can run the command: db2iauto -off InstanceName as the DB2 instance owner.

Preparing to configure DB2 in HACMP:

Make sure /usr has 20 MB free space.

Synchronize your existing HACMP configuration.

1. Enter smit hacmp
2. In SMIT, select Initialization and Standard Configuration > Verify and Synchronize HACMP Configuration and press Enter.

Create a cluster snapshot to capture your HACMP configuration before HACMP Smart Assist for DB2 makes changes to the HACMP configuration.

If cluster services are running, stop them before proceeding:

1. Enter the fastpath smit cl_admin
2. Select Manage HACMP Services > Stop Cluster Services and press Enter.
3. Select Stop now and press Enter.

Create a /db2home1/db2inst1/.rhosts file on each cluster node. HACMP Smart Assist for DB2 uses this file during setup of the HACMP resource configuration. You can delete this file after your configuration is in place.

Make sure that volume groups are varied on and file systems are mounted before you run the HACMP Smart Assist for DB2 application.

Make sure that the DB2 instances for which you are configuring HACMP are running to enable HACMP to discover all configured resources.

Creating HACMP configuration:

Check that the appropriate configuration entries are made in the AIX configuration files (in /etc/services and others), and add the DB2 service IP label to/etc/hosts on all nodes, or make the service IP label available via DNS.

Configure the volume groups that contain the DB2 instance shared home directory on a volume group that is attached to both nodes. Do this for each DB2 instance that is configured.

Turn off db2iauto. DB2 should not automatically restart on system restart.

Configuring HACMP cluster and nodes:

From the first node, enter smit hacmp.

In SMIT, select Initialization and Standard Configuration > Configure an HACMP Cluster and Nodes and press Enter.

Enter cluster name

Enter cluster nodes

Displays a list of the nodes that are currently configured.

If you skip this step and go directly to the HACMP Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available panel, you will be directed back to this step as part of the process.

Discovering and configuring DB2 resources:

After adding nodes, you can use the Initialization and Standard Configuration > Make Applications Available panel for the rest of the Smart Assist process. 

To use DB2 having V9.1 onwards, export a variable DSE_INSTALL_DIR to the installed path of DB2.

For example, if you use DB2 V9.1 and the installation path is /opt/IBM/db2/V9.1, then you need to export a variable called DSE_INSTALL_DIR as export DSE_INSTALL_DIR=/opt/IBM/db2/V9.1

In SMIT, select Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available > Add an Application to the HACMP Configuration and press Enter.

Select DB2 UDB non-DPF Smart Assistant.

SMIT displays a list of possible configurations:

• DB2® Mutual Takeover
• DB2 Single Instance

Select one of the configurations and press Enter.

DB2® Mutual Takeover - Is for having two or more instances in HACMP.

DB2 Single Instance - Is for monitoring single instances.

Verifying and starting HACMP cluster:

To verify and synchronize:

Enter smit hacmp

In SMIT, select Initialization and Standard Configuration > Verify and Synchronize HACMP Configuration and press Enter.

To start HACMP cluster:

Enter the fastpath smit cl_admin

Select Manage HACMP Services > Start Cluster Services and press Enter.

Testing the availability of DB2 applications:

In SMIT, select Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available >Test Your Application for Availability and press Enter.

Select the application you want to test and press Enter.

Press Enter to continue with the test of the selected application.

Understanding the configuration of HACMP start and stop scripts:

The following list describes the HACMP start and stop scripts:

Application	Start Script Description	Stop Script Description
DB2	Calls db2start using the specified DB2 instance name.	Calls db2stop using the specified DB2 instance name.

Changing or showing the configurations for DB2 applications:

In SMIT, select Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available > Change/Show an Application's HACMP Configuration and press Enter.

Select the application configured by DB2 Smart Assist to change or show and press Enter.

Make the changes as needed in the field(s) for the selected application and press Enter.

Changing the existing DB2 instance resource group:

You can change the following for a configured resource group:

• The name of the resource group
• The nodes in the list of participating nodes
• The inter-site management policy of a resource group
• The priority of participating nodes (by changing their position in the list of participating nodes).
• Attributes of the resource group, such as a fallback timer, attributes of volume groups and file systems in the resource group, tape resources and Workload Manager classes.
• The startup, fallover and fallback policies for resource groups .

NOTE:

You can change the resource group startup, fallover or fallback policy only for a resource group that has no resources in it, or after you remove all the resources from the resource group. To change the resource group's startup, fallover or fallback policies, remove the resources first. After you make the changes, add the resources manually to the resource group by selecting them from the pick lists.

You can change most of the attributes of a resource group in an active cluster without having to stop and then restart cluster services. However, to change thename of a resource group, you must stop and then restart the cluster to make the change part of the current cluster configuration.

To change attributes for a DB2 instance resource group:

1. Enter smit hacmp
2. In SMIT, select Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available > Change/Show the Resources Associated with your Application and press Enter.
3. Select a DB2 instance and its associated resource group from the list and press Enter.
   SMIT displays the Change/Show Resources and Attributes for a Resource Group for the application's resource group.

4.  Change field values as needed.

5.  Press Enter to change the resource group information stored in the HACMP Configuration Database.

6  .Return to previous SMIT panels to perform other configuration tasks.

7.  Verify and synchronize the changes you made. In SMIT, select the Verify and Synchronize HACMP Configuration option from the Initialization and Standard Configuration menu.

If the Cluster Manager is running on the local node, synchronizing cluster resources triggers a dynamic reconfiguration event.

Removing a node from DB2 instance resource group:

You can remove one or more nodes at a time from the DB2® instance resource group.

To remove a node from the DB2 instance resource group:

1. Enter smit hacmp
2. In SMIT, select Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available > Change/Show the Resources Associated with your Application and press Enter.
   A list of applications, including the DB2 instances and their associated resource groups appears.
3. Select a DB2 instance for which you want to remove a node and press Enter.
   SMIT displays the resource group for the chosen DB2 instance. You can remove one or more nodes at a time. If you remove all nodes from the DB2 instance resource group, the system removes the resource group from the cluster configuration.
4. Remove the node from the participating nodelist and press Enter.

Removing a DB2 smart assist application instance:

You can remove the DB2® Smart Assist data, including the HACMP resource group and associated application server and application monitors.

The DB2 instance remains on the node; it is no longer monitored and managed by HACMP™.

To remove Smart Assist DB2 instance data from the cluster:

1. Enter smit hacmp
2. In SMIT, select Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available > Remove an Application from the HACMP Configuration and press Enter.
   SMIT displays a list of available applications
3. Select DB2 from the list of applications.
   The list of DB2 instances and the associated resource groups appears.
4. Select the DB2 instance and resource group that you want to remove from the Smart Assist application and press Enter.
   The system removes the resource group, the application server and the application monitor associated with the particular DB2 instance that was included in this resource group. The DB2 instance itself is not removed from the configuration.

Testing DB2 instance resource group:

Quickly test fallover and fallback functions in your cluster with DB2® instances by using the automated HACMP™ cluster test tool.

To test your HACMP Smart Assist for DB2 configuration

1. In SMIT, select Initialization and Standard Configuration > Configuration Assistants > Make Applications Highly Available >Test Your Application for Availability and press Enter.
   SMIT displays a list of applications.
2. Select the DB2 instance and resource group you want to test and press Enter.
   You are prompted to confirm the action by pressing Enter again.
3. Press Enter to continue with the test of the selected application.
   The Cluster Test Tool runs and displays status messages to the screen and stores output from the tests in the file /var/hacmp/log/cl_testtool.log. Messages indicate when a test starts and finishes and provide additional status information. More detailed information, especially when verbose logging is enabled, is stored in the log file that appears on the screen. Information is also logged to the hacmp.out file.

Managing DB2 users and group:

Use the C-SPOC (Cluster Single Point of Control) function of HACMP™ to manage the DB2® instance users.

There are three different types of DB2 users:

• The instance owner. Often the user name is the same as the instance name, such as db2inst1.
• The owner or administrator of the UDF and stored procedures, such as db2fenc1 , and associated group, db2fgrp1.
• The administration server user, such as dasusr1 and the associated group dasadm1.

In general, DB2 relies upon several users whose group, user IDs and passwords should be synchronized across the nodes where the DB2 instance can reside.

Verifying a cluster with DB2 instances:

Initial verification:

HACMP Smart Assist verifies the following:

• DB2 instances have a home directory for the DB2 instance primary node that resides on a shared volume group.
• The instance home volume group contains only one DB2 instance. Multiple DB2 instances in the same volume group are not supported.
• The physical disks that belong to the home volume group are accessible on all takeover nodes.
• The volume group is imported on the primary node for the DB2 instance. The volume group does not have to be imported on the takeover nodes, an automatic corrective action of the verification process imports the volume group on the takeover nodes if necessary.
• User and group names for the instance owner user are defined on all primary and takeover nodes.
• The user ID and group ID are identical on all nodes for the instance owner user and group.
• Supported DB2 version filesets are installed on the participating nodes of an instance resource group.
• DB2 DBF partition instances are not supported and if the system detects a partition instance, this instance cannot be added to the cluster.

Verification checks for storage configuration:

During verification, HACMP™ verifies the certain aspects of your storage configuration.

These aspects include:

• The DB2® instance home directory resides on shared storage.
• Shared volume groups are accessible on all nodes where a particular DB2 instance might reside in the cluster. This requires that all physical disks (hdisks) are defined on all participating cluster nodes.
  If the DB2 instance volume groups do not exist on a takeover node, HACMP automatically imports those volume groups as part of the verification automatic corrective action. For automatic import to be successful, the takeover nodes must already share the same set of disks.

Verification checks for security and HACMP .rhosts file:

HACMP™ ensures that the .rhosts file security for DB2® instances is properly configured by running the cluster verification process. 

The automatic corrective action of the HACMP verification utility notifies you of security errors and lets you correct them at the time they are discovered.

In addition, verification ensures that the DB2 instance home directory /db2home1/db2inst1/.rhosts file contains an entry for the service IP aliased label used for DB2 instance communication to clients and other tiers for every DB2 instance that HACMP monitors. Typically, the preferred entry in .rhosts may appear as follows:

<cluster IP address> <DB2 instance owner user name>

Or as:

+ <DB2 instance owner user name>

In general, although the latter entry should only be used in trusted environments, the HACMP verification does not flag an error if the all hosts entry appears in the .rhosts file, but issues a warning indicating that all remote hosts with the user name DB2 Instance owner name are granted access to the local host.

An automatic corrective action of the verification process adds the appropriate entry if the field is missing from .rhosts.

Other verification checks:

HACMP™ Smart Assist verifies certain issues and provides an automatic corrective action to fix the errors.

These issues include:

• Each node that participates in a DB2® instance resource group has the Fault Monitor Coordinator (FMC) turned off. If the Fault Monitor coordinator is not disabled, an automatic corrective action prompts you to disable it.
• The port numbers are added to the /etc/services file on the primary cluster node for the DB2 instance. An automatic corrective action of the verification process adds the necessary port numbers. See Planning DB2 installation and configuration.
• The DB2 instances are not set to automatic start on system reboot. An automatic corrective action of the verification process prompts you to disable automatic startup of the DB2 instance managed by HACMP.

Changing start and stop scripts:

HACMP™ Smart Assist for DB2® supplies the start and stop scripts that HACMP uses to start and stop the HACMP application servers for a cluster node.

HACMP configuration summary:

Naming convention:

The names of the components created use a standard naming convention to make it easy to identify the various HACMP™ components.

The following list shows the naming conventions used:

db2	DB2® server
rg	Resource Group
as	Application Server

HACMP components created for DB2:

HACMP Component	Name
Resource group	InstanceName_ResourceGroup where InstanceName is the name of the instance of the DB2 database
HACMP application server	InstanceName_ApplicationServer where InstanceName is the name of the instance of the DB2 database.
HACMP custom monitor	InstanceName_SQLMonitor where InstanceName is the name of the instance of the DB2 database. This file is located in the directory /usr/es/sbin/cluster/sa/db2/sbin
Start script	cl_db2start This file is located in the directory /usr/es/sbin/cluster/sa/db2/sbin
Stop script	cl_db2stop This file is located in the directory/usr/es/sbin/cluster/sa/db2/sbin

Settings for application monitors:

Custom application monitoring checks the health of an application with a custom monitor method at user-specified polling intervals.

Name	InstanceName_SQLMonitor
Application Server(s) to Monitor	InstanceName_ApplicationServer
Method	/usr/es/sbin/cluster/sa/db2/sbin/cl_db2cmon -i <InstanceName> -A <ApplicationName> -d <DatabaseToMonitor>
Mode	Long-running monitoring
Interval	120 sec.
Hung Monitor Signal	9
Stabilization Interval	240 sec.
Restart Count	3
Restart Interval	1440 sec.
Action on Application Failure	Fallover
Cleanup Method	/usr/es/sbin/cluster/sa/db2/sbin/cl_db2stop InstanceName
Restart Method	/usr/es/sbin/cluster/sa/db2/sbin/cl_db2start InstanceName

Process monitor:

The process monitor for DB2® in an HACMP™ cluster determines whether the parent process for the DB2 instance, db2sysc, is still running for the DB2 instance. 

Should this process terminate, the DB2 instance attempts to run a cleanup script and to restart up to three times before the DB2 instance falls over to the next node. 

After three attempts, the application server associated with this monitor falls over to another node that participates in this resource group.

The following table lists the default settings for the process monitor:

Name	InstanceName_ProcessMonitor
Monitor Mode	Long-running monitoring
Process to Monitor	db2sysc
Process Owner	Instance Owner
Instance Count	1
Stabilization Interval	240 sec.
Restart Count	3
Restart Interval	1440 sec.
Action on Application Failure	Fallover
Cleanup Method	/usr/es/sbin/cluster/sa/db2/sbin/cl_db2stopInstanceName
Restart Method	/usr/es/sbin/cluster/sa/db2/sbin/cl_db2startInstanceName

Upgrade guide for DB2 HACMP servers:

Link:

http://www-01.ibm.com/support/docview.wss?uid=swg21461186

If your DB2 server is configured in a HACMP environment and you plan to upgrade the server, refer to this guide.

High level overview of the steps: 

1. Install the DB2 server product you are upgrading to on both nodes 

2. Stop HACMP on both active and standby nodes 

3. Upgrade your DB2 server on the active node only, as per the standard documentation found in the Information Center 

4. Install the sample HACMP scripts on both active and standby nodes 

5. Startup your HACMP environment 

6. Perform post-upgrade verification steps 

1) Stop HACMP on both nodes 

Stop the HACMP environment on both the active and standby server.: 

smitty hacmp → System Management → Manage HACMP services → Stop Cluster Services 

2) Install DB2 for LUW Version 9.7 

Install DB2 for LUW Version 9.7 (preferably the latest fix pack available) on both the active and standby nodes. Install to a different path than where your Version 8.2 product is installed. 

DB2 UDB Version 8.2 will be installed under: 

/usr/opt/db2_08_01 

Install DB2 for LUW Version 9.7 in a path such as : 

/opt/IBM/db2/V9.7 

Refer to the Information Center documentation for details on installing your new DB2 server, and ensure you meet all the pre-requisites as specified. 

3) Upgrade your DB2 Server 

Perform the DB2 server upgrade on your active node only. Refer to the supporting documentation of the server you're upgrading DB2 to. Follow all the necessary steps including: pre-upgrade steps, upgrading the DB2 server itself, and performing post-upgrade steps. 

4) Install sample HACMP scripts (Optional) 

Install the sample HACMP scripts provided in the DB2 instance you have just upgraded, on both the active and standby nodes.

As an example, installing sample HACMP scripts includes the following steps: 

1. Go to <INSTANCE_HOME>sqllib/samples/es path 
2. Locate the following three files:

• rc.db2pe.eee (DPF) or rc.db2pe.ee (non-DPF)
• db2_proc_restart.eee (DPF) or db2_proc_restart.ee (non-DPF)
• db2_inst_ha.local

3. Copy rc.db2pe.eee/ee to rc.db2pe 
4. Copy db2_proc_restart.eee/ee to db2_proc_restart 
5. Use db2_inst_ha.local to install the HACMP scripts (run once for each clustered database).

• db2_inst_ha.local <hostname> . <database_name>
  
  Where:
  hostname is the host of the node
  . is the current directory
  database_name is the database name

5) Start up your HACMP environment 

As root, run the following command on both active and standby nodes: 

Smitty hacmp → System Management → Manage HACMP services → Start Cluster Services 

6) Post-upgrade verification 

After the DB2 server environment has been upgraded, and the HACMP services have been restarted on both nodes, verify that the DB2 server and your HACMP services are running correctly.

Refer to the Information Center documentation for post-upgrade verification steps.

To verify your HACMP services, some basic steps include:

• Run 'clstat' on both active and standby nodes to verify the cluster is back online.
• Perform a failover test. As root run,

Smitty hacmp → System Management → Manage HACMP services → Stop Cluster Services (select action, "Move Resource Groups")

• Analyze /var/hacmp/logs/hacmp.out log file to make sure everything is setup as expected.

Refer to Information Center documentation on HACMP for further details on how you can best validate that your environment is ready.