Document a "rollback to previous release" process

[dacook]

With a recent S1 bug introduced by a new release, we needed to quickly respond, and revert to the previous release for all servers.

The general method is to perform a deployment of the previous release, but there's a few things to consider. We should document those somewhere useful, like a wiki page.

[dacook]

To consider when deploying a previous release:

• Was data migrated by the new release, and does it need reversing? Generally it is probably find not to reverse it.

Note: the rollback.yml playbook is almost never a good idea, because this will also restore the last database backup, which will discard any new data.

Can we speed up the process? The deploy process will create a db backup and clean up transient data which takes a long time.
Maybe we could update the rollback.yml playbook to not restore the db by default, and only restore when a parameter is provided.
BTW that playbook is out of date, it only restarts puma. It should also restart sidekiq.

[mkllnk]

A lot has changed since the rollback playbook was written. I like your idea to remove the database restoration from that playbook. Restoring the previous version could also be as simple as changing the current symlink and restarting services.