Overview
This article describes how to set up and operate a manual Disaster Recovery (DR) environment for JetPatch Manager (till version 4.2.x). It covers initial setup, activating the standby environment upon failure, reverting to the active environment, and troubleshooting data replication issues.
Setting Up the DR Environment
First, set up PostgresSQL database replication.
Then do the following:
On Each JetPatch Manager Console (Active and Standby)
- Install the JetPatch Manager Console.
- Stop JetPatch Agent Manager services:
|
/etc/init.d/tomcat_watchdog stop /etc/init.d/tomcat stop /etc/init.d/nginx stop |
- Open the following file for editing:
| /usr/share/tomcat/default/conf/intigua.properties |
- Replace the following three keys with the correct values for each server's respective database:
- db.url
- db.username
- db.password
On the Active JetPatch Manager Console
- Start JetPatch Agent Manager services:
|
/etc/init.d/tomcat_watchdog start /etc/init.d/tomcat start /etc/init.d/nginx start |
- Install rsync:
| yum install -y rsync |
On the Standby Server
- Place the sync script received from JetPatch in:
| /usr/share/intigua |
- Install rsync:
| yum install -y rsync |
- Establish a secure, trusted connection to the active server (required for rsync):
|
ssh-keygen ssh-copy-id root@<active> |
Where <active> is the active server's resolvable hostname or IP address.
You will be prompted for the active server's root password.
- Create the following directory:
| /etc/intigua |
- Configure synchronization:
| ./sync.py --src=<active> --createconfig |
Where <active> is the active server's resolvable hostname or IP address.
- Copy intigua-sync to:
| /etc/init.d |
- Start synchronization:
| service intigua-sync start |
- Optionally, test synchronization:
| ./sync.py --src=<active> |
Check that the script is running and that its output shows data being synchronized.
Final Setup Steps
- Copy the following file from the active server to the standby server:
| /usr/share/tomcat/default/conf/intigua.secret |
- Log in to the active server's Console and configure the connector management services' Communication setting to the active JetPatch Manager Console's hostname (not IP address).
- Make sure this setting is deployed to all endpoints.
Activating the Standby Environment
Upon failure of the active environment, perform the following steps to activate the standby:
- On the standby server, verify that replication has been running:
| service intigua-sync status |
- If still possible, stop services on the active server:
|
/etc/init.d/tomcat_watchdog stop /etc/init.d/tomcat stop /etc/init.d/nginx stop |
- On the active database, stop PostgreSQL:
| /etc/init.d/postgresql-9.3 stop |
- On the standby database, create a failover trigger file:
| touch /var/lib/pgsql/9.3/data/failover.trigger |
- On the standby server, start services:
|
/etc/init.d/tomcat_watchdog start /etc/init.d/tomcat start /etc/init.d/nginx start |
- Update DNS records so that the URL previously pointing to the active server now points to the standby server.
Reverting to the Active Environment
After activating the standby environment, use the following procedure to bring the originally active environment back up. If necessary, rebuild the JetPatch Manager Console and/or database first.
| Warning: This procedure overwrites database contents in the original active environment, assuming the standby database is current. If this is not the case, following this procedure could result in permanent data loss. |
On the Standby Server — Stop Services
- Stop the standby server:
|
/etc/init.d/tomcat_watchdog stop /etc/init.d/tomcat stop /etc/init.d/nginx stop |
On the Originally Active Database Host
- Stop the database:
| /etc/init.d/postgresql-9.3 stop |
- Back up the data directory by renaming it:
| mv /var/lib/pgsql/9.3/data /var/lib/pgsql/9.3/data_old |
- Recover current data from the standby database:
|
su - postgres /usr/pgsql-9.3/bin/pg_basebackup -h <standbyDB> -U postgres -D /var/lib/pgsql/9.3/data -X stream -P |
Where <standbyDB> is the IP address or resolvable name of the standby database.
- Overwrite the copied configuration files with the backed-up originals:
|
cp /var/lib/pgsql/9.3/data_old/postgresql.conf /var/lib/pgsql/9.3/data cp /var/lib/pgsql/9.3/data_old/pg_hba.conf /var/lib/pgsql/9.3/data |
On the Standby Database Host
- Stop the database:
| /etc/init.d/postgresql-9.3 stop |
- Return the database to standby mode:
| mv /var/lib/pgsql/9.3/data/recovery.done /var/lib/pgsql/9.3/data/recovery.conf |
- Start the database:
| /etc/init.d/postgresql-9.3 start |
On the Active Database Host
- Start the database:
| /etc/init.d/postgresql-9.3 start |
- Verify that both databases are running and replicating by inspecting the following directory on each server:
| /var/lib/pgsql/9.3/data/pg_log |
Final Revert Steps
- Copy the following from the current standby server to the originally active server:
|
/usr/share/intigua/package /usr/share/tomcat/default/conf/intigua.secret |
- Start the active server:
|
/etc/init.d/tomcat_watchdog start /etc/init.d/tomcat start /etc/init.d/nginx start |
- Update DNS records to point back to the active server.
Troubleshooting Data Replication
In a DR deployment, the active database may get ahead of the standby database. This may appear in the log as:
| ERROR: requested starting point 0/3B000000 is ahead of the WAL flush position of this server 0/39044BD0 |
To manually replicate data from the active database to the standby, perform the following:
On the Active JetPatch Manager Console — Stop Services
- Stop the active server:
|
/etc/init.d/tomcat_watchdog stop /etc/init.d/tomcat stop /etc/init.d/nginx stop |
On the Standby Database Host
- Stop the database:
| /etc/init.d/postgresql-9.3 stop |
- Back up the data directory:
| mv /var/lib/pgsql/9.3/data /var/lib/pgsql/9.3/data_old |
- Recover current data from the active database:
|
su - postgres /usr/pgsql-9.3/bin/pg_basebackup -h <standbyDB> -U postgres -D /var/lib/pgsql/9.3/data -X stream -P |
Where <standbyDB> is the IP address or resolvable name of the standby database.
- Overwrite the copied configuration files with the backed-up originals:
|
cp /var/lib/pgsql/9.3/data_old/postgresql.conf /var/lib/pgsql/9.3/data cp /var/lib/pgsql/9.3/data_old/pg_hba.conf /var/lib/pgsql/9.3/data |
On the Active Database Host
- Stop the database:
| /etc/init.d/postgresql-9.3 stop |
- Return the database to standby mode:
| mv /var/lib/pgsql/9.3/data/recovery.done /var/lib/pgsql/9.3/data/recovery.conf |
- Start the database:
| /etc/init.d/postgresql-9.3 start |
On the Standby Database Host
- Start the database:
| /etc/init.d/postgresql-9.3 start |
Comments
0 comments
Please sign in to leave a comment.