Safe storage beyond Redunda

[tripleee]

The items we had in Redunda were lost. The instance is now back up but with a clean slate. Ideally, there should be a way to store the current state in multiple places (back up your backups!)

In particular, it would be nice to have a checkpoint feature to pull the current data out of a running client prior to rebooting its host and/or container before there is a crisis involving one or more of metasmoke, Redunda, and AWS.

[tripleee]

There is now a simple nightly cron job pulsebackup on the EC2 instance where Halflife runs which copies the dot directory from the running instance to a local Git repository in pulsedata. I'll see about mirroring this repo to a proper place.

[tripleee]

Just restored (most) notifications from yesterday from backups. The prestart script should be refactored to take a backup, stop the old instance, start a new instance in offline mode, copy in the backup, and only then restart the bot.

For what it's worth, I now have an offsite backup of the pulsedata Git repo, though the automation is probably still somewhat brittle.

[tripleee]

(Or , of course, I should figure out how to get Redunda to do exactly this!)

[tripleee]

Tangentially, reminder to self, how to get useful JSON diffs out of Git:

echo '*.json diff=json' >.gitattributes
git config diff.json.textconv 'jq .'

Adapted from https://t-a-w.blogspot.com/2016/05/sensible-git-diff-for-json-files.html

[tripleee]

I have refactored the restart script to copy in a fresh version from backup but it still looks like Redunda is rolling back the fresh data to an older version. Not really in a place where I can experiment, and Redunda seems to be down anyway ... /-: