ROLLBACK.md

Example Rollback

• Example Rollback
  • Phase 1 Rollback
    • Phase 1 Complete Rollback
  • Phase 2 Rollback
    • Phase 2 Stage Rollback
      • Scale down all Phase 2 services on GCP
    • Un-swap Phase 2 databases
    • Cleanup Phase 2 databases
    • Phase 2 Complete Rollback
  • Phase 3 Rollback
    • Phase 3 Stage Rollback
      • Scale down all Phase 3 services on GCP
    • Un-swap Phase 3 databases
    • Cleanup Phase 3 databases
    • Phase 3 Complete Rollback
  • Phase 4 Rollback
    • Phase 4 Stage Rollback
      • Scale down all Phase 4 services on GCP
    • Un-swap Phase 4 databases
    • Cleanup Phase 4 databases
    • Phase 4 Complete Rollback
  • Phase 5 Rollback
    • Phase 5 Stage Rollback
      • Scale down all Phase 5 services on GCP
    • Un-swap Phase 5 databases
    • Un-cutover Phase 5 databases
    • Phase 5 Complete Rollback
  • Complete Rollback
    • Scale Down All Services on GCP
    • Scale AWS Back to Original
    • Switch Kafka to Un-mirrored Cluster
    • Un-swap Databases
    • Cleanup Databases
    • Scale All Services on GCP to Targets

Phase 1 Rollback

These are the steps to perform a rollback of just Phase 1 to the previous phase.

Refer to Complete Rollback for complete rollback steps.

Phase 1 Complete Rollback

These are the steps to perform a rollback of Phase 1 back to a clean environment.

Refer to Complete Rollback for complete rollback steps.

Phase 2 Rollback

Phase 2 Stage Rollback

These are the steps to perform a rollback of just Phase 2 to the previous phase.

Scale down all Phase 2 services on GCP

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype App --action scaletargetdown --maxworkers 10 --force --phase 2

Un-swap Phase 2 databases

1. Edit node_interfaces/Database.py and add ctx.forward(unswap) to the top of the preswap method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action preswap --maxworkers 10 --phase 2

Cleanup Phase 2 databases

1. Edit node_interfaces/Database.py and add ctx.forward(cleanup) to the top of the precutover method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action precutover --maxworkers 10 --phase 2

Phase 2 Complete Rollback

These are the steps to perform a rollback of Phase 2 back to a clean environment.

Refer to Complete Rollback for complete rollback steps.

Phase 3 Rollback

Phase 3 Stage Rollback

These are the steps to perform a rollback of just Phase 3 to the previous phase.

Scale down all Phase 3 services on GCP

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype App --action scaletargetdown --maxworkers 10 --force --phase 3

Un-swap Phase 3 databases

1. Edit node_interfaces/Database.py and add ctx.forward(unswap) to the top of the preswap method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action preswap --maxworkers 10 --phase 3

Cleanup Phase 3 databases

1. Edit node_interfaces/Database.py and add ctx.forward(cleanup) to the top of the precutover method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action precutover --maxworkers 10 --phase 3

Phase 3 Complete Rollback

These are the steps to perform a rollback of Phase 3 back to a clean environment.

Refer to Complete Rollback for complete rollback steps.

Phase 4 Rollback

Phase 4 Stage Rollback

These are the steps to perform a rollback of just Phase 4 to the previous phase.

Scale down all Phase 4 services on GCP

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype App --action scaletargetdown --maxworkers 10 --force --phase 4

Un-swap Phase 4 databases

1. Edit node_interfaces/Database.py and add ctx.forward(unswap) to the top of the preswap method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action preswap --maxworkers 10 --phase 4

Cleanup Phase 4 databases

1. Edit node_interfaces/Database.py and add ctx.forward(cleanup) to the top of the precutover method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action precutover --maxworkers 10 --phase 4

Phase 4 Complete Rollback

These are the steps to perform a rollback of Phase 4 back to a clean environment.

Refer to Complete Rollback for complete rollback steps.

Phase 5 Rollback

Phase 5 Stage Rollback

These are the steps to perform a rollback of just Phase 5 to the previous phase.

Scale down all Phase 5 services on GCP

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype App --action scaletargetdown --maxworkers 10 --force --phase 5

Un-swap Phase 5 databases

1. Edit node_interfaces/Database.py and add ctx.forward(unswap) to the top of the preswap method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action preswap --maxworkers 10 --phase 5

Un-cutover Phase 5 databases

1. Edit node_interfaces/Database.py and add ctx.forward(cleanup) to the top of the precutover method.
2. Execute the following command to swap databases:

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action precutover --maxworkers 10 --phase 5

Phase 5 Complete Rollback

These are the steps to perform a rollback of Phase 5 back to a clean environment.

Refer to Complete Rollback for complete rollback steps.

Complete Rollback

Scale Down All Services on GCP

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype App --action scaletargetdown --maxworkers 10 --force

Scale AWS Back to Original

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype App --action rollbacksource --maxworkers 10 --force

Switch Kafka to Un-mirrored Cluster

Un-swap Databases

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --node data-key-service-db --action unswap-dks --maxworkers 10 --force

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action unswap --maxworkers 10 --force

Cleanup Databases

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype Database --action cleanup --maxworkers 10 --force

Scale All Services on GCP to Targets

./orchestrate.sh --environment ${MUDRA_ENVIRONMENT} --datafiles orchestration_datafiles --nodetype App --action scaletarget --maxworkers 10 --force