Troubleshooting Data Replication

In a Disaster Recovery deployment, it may happen that the active database is ahead of the standby database. An indication of this could be that the log file records something like:

CST >ERROR: requested starting point 0/3B000000 is ahead of the WAL flush position of this server 0/39044BD0

In these cases, manually replicate data from the active database to the standby database as follows:

1. Stop the active JetPetch Manager Console by running:

   /etc/init.d/tomcat_watchdog stop

   /etc/init.d/tomcat stop

   /etc/init.d/nginx stop

2. On the standby database host:
   1. Stop the database by running:

      /etc/init.d/postgresql-9.3 stop

   2. Back up the data directory by renaming it:

      mv /var/lib/pgsql/9.3/data /var/lib/pgsql/9.3/data_old

   3. Recover current data from the standby database by running:

      su - postgres

      /usr/pgsql-9.3/bin/pg_basebackup -h <standbyDB> -U postgres -D /var/lib/pgsql/9.3/data -X stream -P

      where <standbyDB> is the IP address or resolvable name of the standby database.

   4. Overwrite the copied configuration files with the backed up ones by running:

      cp /var/lib/pgsql/9.3/data_old/postgresql.conf /var/lib/pgsql/9.3/data

      cp /var/lib/pgsql/9.3/data_old/pg_hba.conf /var/lib/pgsql/9.3/data

3. On the active database host:
   1. Stop the database by running:

      /etc/init.d/postgresql-9.3 stop

   2. Return the database to standby mode by running:

      mv /var/lib/pgsql/9.3/data/recovery.done /var/lib/pgsql/9.3/data/recovery.conf

   3. Start the database by running:

      /etc/init.d/postgresql-9.3 start

4. On the standby database host, start the database by running:

   /etc/init.d/postgresql-9.3 start