Copies production-only backups code to main branch

[mouse-reeve]

Description

This moves some code that lived only on the production branch over to main. I actually think this is the last material difference between the two branches!

• Related Issue #
• Closes #

What type of Pull Request is this?

• Bug Fix
• Enhancement
• Plumbing / Internals / Dependencies
• Refactor

Does this PR change settings or dependencies, or break something?

• This PR changes or adds default settings, configuration, or .env values
• This PR changes or adds dependencies
• This PR introduces other breaking changes

Details of breaking or configuration changes (if any of above checked)

Documentation

• New or amended documentation will be required if this PR is merged
• I have created a matching pull request in the Documentation repository
• I intend to create a matching pull request in the Documentation repository after this PR is merged

Tests

• My changes do not need new tests
• All tests I have added are passing
• I have written tests but need help to make them pass
• I have not written tests and need help to write them

[ilkka-ollakka]

I think this is missing the use of that container in db-node in docker-compose.yml -file?

I might prefer to run this backup as separate container outside db, but that can be looked at later on if it is actually feasible.

[mouse-reeve]

You're right, I should have looked more closely at whether things were actually working in production. I'm going through the history and sussing it out now

[mouse-reeve]

@ilkka-ollakka I've got it to what looks like the right state, but I'm not seeing backups generated, and when I go into the container and run the backup script, I'm getting this error:

pg_dump: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?

I'm at an impasse on that for the moment.

[ilkka-ollakka]

impasse

Looking at the dockerfile, I think you should run that container instead of default postgresql, so it has cron running in same host.

If you run it as separate container, you need to provide hostname and password to pg_dump, you can use -h db to tell hostname and PGPASSWORD env-variable to tell password (PGPASSWORD=$POSTGRES_PASSWORD).

If you can show what docker-compose you had/how you tested it, I can retry to replicate the issue locally?

[mouse-reeve]

The current state of the code in this PR is how I was testing

[ilkka-ollakka]

ah, I'll take a look at this tomorrow to see most feasible flow to get it running.

[ilkka-ollakka]

I did this PR against this branch: #3829 making the backup node to log output better from cron and run backups separate from database-container.

[hughrun]

Where are we on this @mouse-reeve? Ready for more testing?

[mouse-reeve]

Yes! A third pair of eyes on this would be great

[hughrun]

This seems to work pretty well, albeit with my comment about DEBUG needing to be explicitly false, not merely "not true".

This is not a blocker, but it would be handy if admins could set the time for the daily backups by using an environment variable. Midnight UTC could be a peak usage time.

[hughrun]

I think this probably makes sense as an explicit check. However I'm just noting here that we should be very clear in the documentation that if DEBUG is not set at all (e.g. it's commented out), then backups won't run. We don't want anyone to find this out the hard way.

[ilkka-ollakka]

I think we could leave this check out and just modify docker-compose.dev.yml not to have this container running in dev env. I can create PR for that tonight as an example.