Restore Corrupt Data

Using PIT, a "point-in-time" restore capability provided by AWS, you can restore your database to a state ranging from the last database snapshot to a few minutes before the current time.

1. Shut down the Kubernetes cluster using the ../vof_xom_deployment/k8s-shutdown.sh [ENV namespace] script. This script dumps the number of replicas for each deployment into the [env-namespace]-[timestamp].replicas file in the current directory. Keep this file safe because it is required for restoring the cluster. As a result of this script execution, every deployment will be scaled to zero replicas and effectively stopped.

   The API GW should be updated to return Error Code 503 (Service Unavailable) for all API GW endpoints. JS should provide user with a proper error notification.

2. Log into the AWS console and switch to DeploymentRoleV2@[account_id you restore] role.
3. Navigate to RDS > Instances > [rds instance you want to restore].
4. On the instance page, note the latest restore time. You cannot restore to any time later than that time.
5. In the Instance actions menu on the right, click Restore to point in time.
6. In the dialogue, select the restore time and the name of the new restored instance.
7. Return to Instances and check when the restore instance is available.
8. In the meantime, run Kubernetes cleanup/backup jobs using .k8s_cleanup. Run the Kubernetes cleanup jobs to extract data and clean up the Kafka queues. Make k8s_cleanup ENV_NAMESPACE=namespace THOR_VERSION=cmt-build-version BACKUP_FROM_DATE=date BACKUP_FROM_TIME=time. For example, make k8s_cleanup ENV_NAMESPACE=testenv3 THOR_VERSION=cmt-104 2018-10-25 118 BACKUP_FROM_DATE=2019-01-02 BACKUP_FROM_TIME=16:40. The backed-up data is pushed to ENV--vlocity-xom-restore-backup S3 buck.

   This script creates the following job pods:

   • completer-cleanup: Drains the Completer Kafka queue:

     java -jar restore.jar --job_type=COMPLETER_CLEANUP

   • incoming-backup: Prepares the incoming orders from the time configured and pushes to the S3 bucket. For example:

     java -jar restore.jar --job_type=INCOMING_BACKUP --from_date=2018-05-30 --from_time=20:30

   • asset-backup: Prepares the asset creation data from the time configured and pushes to the S3 bucket. For example:

     java -jar restore.jar --job_type=ASSET_CHANGE --from_date=2018-05-30 --from_time=20:30

9. Once the restored instance is up, ensure that its Security group and Parameter group are the same as the original RDS. You can find these values in the RDS Instance Details page. Use the modify option to change it if necessary.
10. Navigate to Route 53 and select the zone for your database. Update CNAME for postgres to point to the Endpoint parameter of the newly created PSQL. Wait for 60 seconds.
11. Execute cat [k8s-namespace]-[timestamp].replicas | ../vof_xom_deployment/k8s_restore.sh [k8s-namespace]. This recreates the required secrets and restarts the Kubernetes cluster. Wait for all pods to start.
12. Delete the old database.
13. Restart the cluster.
14. Reconfigure API Gw to its normal configuration.
15. Re-synchronize Catalog.