Improve backups system [8knot db clone] #1
Description
Currently we are just calling CHECKPOINT on the augur DB and snapshotting its disk (therefore likely relying on postgres recovery to find a consistent db state).
This was done in the interest of speed, especially due to complications surrounding how to correctly call the start and stop backup queries from the same session while running other ansible tasks in between (without causing session loss and thus backup mode being disabled) (see; forum.ansible.com/t/how-to-properly-manage-pg-backup-start-across-ansible-tasks-for-consistent-snapshots/44081)
Doing this properly also requires copying WAL files around, which i presently dont know where to store them.
Potential puzzle pieces include:
- writing a custom ansible module to maintain one postgres session for the commands and run other tasks in between
- having a script on the server that does the start and stop backup (possibly reliant on timing though, creating a race condition and/or requiring IAM credentials to be placed on the server)
- multi-threaded ansible may be needed (sounds complicated)
This is an issue intended to document this temporary shortcut for later review, especially once focus turns to creating actual backups