Setting up Manual Disaster Recovery

First, set up PostgresSQL database replication.  Then do the following:

1. For each of the two JetPatch Manager Console:
   1. Install the JetPatch Manager Console.
   2. Stop JetPatch Agent Manager services:

      /etc/init.d/tomcat_watchdog stop

      /etc/init.d/tomcat stop

      /etc/init.d/nginx stop

   3. Open the following file for editing:

      /usr/share/tomcat/default/conf/intigua.properties

   4. Replace the following three keys' values with the correct values, on each server for its respective database:
      • db.url
      • db.username
      • db.password
2. On the active JetPatch Manager Console, start JetPatch Agent Manager services:

   /etc/init.d/tomcat_watchdog start

   /etc/init.d/tomcat start

   /etc/init.d/nginx start

3. On the active JetPatch Manager Console, install rsync by running:

   yum install -y rsync

4. On the standby server, do the following:
   1. Place the script you received from JetPatch Agent Manager in:

      /usr/share/intigua

   2. Install rsync by running:

      yum install -y rsync

   3. Establish a secure, trusted connection to the active server (to enable rsync to retrieve data) by running:

      ssh-keygen

      ssh-copy-id root@<active>

      where <active> is the active server's resolvable name or IP address.

      You will be prompted for the active server's root password.

   4. Create the following directory:

      /etc/intigua

   5. Configure synchronization by running:

      ./sync.py --src=<active> -–createconfig

      where <active> is the Active server's resolvable name or IP address.

   6. Copy intigua-sync to:

      /etc/init.d

   7. Start synchronization by running:

      service intigua-sync start

   8. Optionally, test synchronization by running:

      ./sync.py --src=<active>

      where <active> is the active server's resolvable name or IP address.

5. Check that the script is running, and that its output shows data being synchronized.
6. Copy the following file from the active server to the standby server:

   /usr/share/tomcat/default/conf/intigua.secret

7. Log into the active server's Console, and configure connector management services' Communication setting to the active JetPatch Manager Console's hostname (not IP address). Make sure this setting is deployed to all endpoints.

Activating the Standby Environment

If you have a Disaster Recovery deployment, then upon failure of the active environment, do the following to activate the standby environment:

1. On the standby JetPatch Manager Console, make sure that replication has been occurring, by checking the replication service status:

   service intigua-sync status

2. If still possible, stop services in the active location:
   • On the active server, run:

     /etc/init.d/tomcat_watchdog stop

     /etc/init.d/tomcat stop

     /etc/init.d/nginx stop

   • On the active database, run:

     /etc/init.d/postgresql-9.3 stop

3. On the standby database, create an activation trigger file by running:

   touch /var/lib/pgsql/9.3/data/failover.trigger

4. On the standby server, start services by running:

   /etc/init.d/tomcat_watchdog start

   /etc/init.d/tomcat start

   /etc/init.d/nginx start

5. Update DNS records so that the URL that until now pointed to the active server now points to the standby server.

Reverting to the Active Environment

After activating a standby environment, to bring the originally active environment back up, you can, if necessary, rebuild the JetPatch Manager Console and/or database as when setting it up.

Once the originally active environment is available, revert to it as follows:

This procedure overwrites database contents in the original environment, under the assumption that the standby database is current. If this is not the case, following this procedure could result in permanent data loss.

1. Stop the standby server by running:

   /etc/init.d/tomcat_watchdog stop

   /etc/init.d/tomcat stop

   /etc/init.d/nginx stop

2. On the originally active database host:
   1. Stop the database by running:

      /etc/init.d/postgresql-9.3 stop

   2. Back up the data directory by renaming it:

      mv /var/lib/pgsql/9.3/data /var/lib/pgsql/9.3/data_old

   3. Recover current data from the standby database by running:

      su - postgres

      /usr/pgsql-9.3/bin/pg_basebackup -h <standbyDB> -U postgres -D /var/lib/pgsql/9.3/data -X stream -P

      where <standbyDB> is the IP address or resolvable name of the standby database.

   4. Overwrite the copied configuration files with the backed up ones by running:

      cp /var/lib/pgsql/9.3/data_old/postgresql.conf /var/lib/pgsql/9.3/data

      cp /var/lib/pgsql/9.3/data_old/pg_hba.conf /var/lib/pgsql/9.3/data

3. On the standby database host:
   1. Stop the database by running:

      /etc/init.d/postgresql-9.3 stop

   2. Return the database to standby mode by running:

      mv /var/lib/pgsql/9.3/data/recovery.done /var/lib/pgsql/9.3/data/recovery.conf

   3. Start the database by running:

      /etc/init.d/postgresql-9.3 start

4. On the active database host, start the database by running:

   /etc/init.d/postgresql-9.3 start

5. Verify that the databases are running and replicating by inspecting the following directory on both database servers:

   /var/lib/pgsql/9.3/data/pg_log

6. Copy the following directory and file from the current, standby server to the originally active one:

   /usr/share/intigua/package

   /usr/share/tomcat/default/conf/intigua.secret

7. Start the active server by running:

   /etc/init.d/tomcat_watchdog start

   /etc/init.d/tomcat start

   /etc/init.d/nginx start

8. Update DNS records to point to the active server.

Troubleshooting Data Replication

In a Disaster Recovery deployment, it may happen that the active database is ahead of the standby database. An indication of this could be that the log file records something like:

CST >ERROR: requested starting point 0/3B000000 is ahead of the WAL flush position of this server 0/39044BD0

In these cases, manually replicate data from the active database to the standby database as follows:

1. Stop the active JetPetch Manager Console by running:

   /etc/init.d/tomcat_watchdog stop

   /etc/init.d/tomcat stop

   /etc/init.d/nginx stop

2. On the standby database host:
   1. Stop the database by running:

      /etc/init.d/postgresql-9.3 stop

   2. Back up the data directory by renaming it:

      mv /var/lib/pgsql/9.3/data /var/lib/pgsql/9.3/data_old

   3. Recover current data from the standby database by running:

      su - postgres

      /usr/pgsql-9.3/bin/pg_basebackup -h <standbyDB> -U postgres -D /var/lib/pgsql/9.3/data -X stream -P

      where <standbyDB> is the IP address or resolvable name of the standby database.

   4. Overwrite the copied configuration files with the backed up ones by running:

      cp /var/lib/pgsql/9.3/data_old/postgresql.conf /var/lib/pgsql/9.3/data

      cp /var/lib/pgsql/9.3/data_old/pg_hba.conf /var/lib/pgsql/9.3/data

3. On the active database host:
   1. Stop the database by running:

      /etc/init.d/postgresql-9.3 stop

   2. Return the database to standby mode by running:

      mv /var/lib/pgsql/9.3/data/recovery.done /var/lib/pgsql/9.3/data/recovery.conf

   3. Start the database by running:

      /etc/init.d/postgresql-9.3 start

4. On the standby database host, start the database by running:

   /etc/init.d/postgresql-9.3 start