What is the best way to run a command in the offen/docker-volume-backup container before creating a backup?

[ottobolyos]

I am using using timescale/timescaledb-ha which runs PostgreSQL and all related commands as postgres user which has UID/GID set to 1000 (it is a non-root, non-system user). This causes an issue, as docker-volume-backup runs all commands are root , therefore everytime it creates a backup, the files are root-owned and pg_dumpall cannot access/overwrite the files in a subsequent backup.

Now, this can be easily worked around by running chown -R 1000:1000 /backup in the docker-volume-backup container before actually creating the backup. However, what is the best way to accomplish this?

Note that ATM I am using offen/docker-volume-backup:v2.25.1 (set up by a former colleague) and I use the backup label on the database container to defined the backup command:

"docker-volume-backup.exec-pre=/bin/sh -c 'pg_dumpall -U $PGUSER --clean --file /home/postgres/pgdata/backup/dumpall.sql'"

As that docker-volume-backup version still uses external cron, I could bind-mount /var/spool/cron/crontabs/root and change the ownership of /backup before running backup (i.e. 0 1 * * * chown -R 1000:1000 /backup && backup 2>&1), I’d like to have a kind of confirmation if this will work and is the best solution from your PoV.

Thank you in advance! 🙏

[m90]

Is there anything that keeps you from updating the Image version? Newer versions support running as non-root https://offen.github.io/docker-volume-backup/how-tos/use-as-non-root.html

[m90]

Please also note that if you have two labeled commands of the same type on two different containers, there is no guarantee about execution order as these are run in parallel. This is mentioned in the documentation: https://offen.github.io/docker-volume-backup/how-tos/run-custom-commands.html

If this is your problem, I'd advise the ugly but definitely working solution of prepending a sleep 5 && to the command you want to run last as chowning will likely be much faster than 5 seconds. If it's not, adjust the given timeout.

[ottobolyos]

I'm a bit lost in all that information.

All in all, I want to make sure the order of the chown and dump commands is defined by me. I want to make sure each time the commands are executed, they are executed in a specific order (it does not actually matter if chown is executed first or second, but it must be always executed either first or second).

If you want this issue to be adressed further, could you provide a compose file that allows me to reproduce the non-predictable execution of labeled commands you are surmising in a simple way?

I use the Docker compose from this comment of mine with docker-volume-backup.archive-pre: /bin/sh -c 'chown -R 1000:1000 /backup' added to the backup container.

I could create a minimal reproduction Docker Compose file if still needed.

Please also note that if you have two labeled commands of the same type on two different containers, there is no guarantee about execution order as these are run in parallel. This is mentioned in the documentation: https://offen.github.io/docker-volume-backup/how-tos/run-custom-commands.html

Yes, I know it is mentioned in the docs and that is exactly why I am afraid of using these two labels like this.

I don’t know if there is a way to make sure they are ordered in a specific order (using an env var or something), however, another option would be to create another label for the backed up container (in my case it is postgres), something like docker-volume-backup.chown-user-group set to 1000:1000 in my case and if it is defined (and passes ^[0-9]+:[0-9]+$), automatically execute the chown command on /backup in the backup container after creating the backup/dump (i.e. after execution of a command in the backed up container). That should be possible IMHO.

If this is your problem, I'd advise the ugly but definitely working solution of prepending a sleep 5 && to the command you want to run last as chowning will likely be much faster than 5 seconds. If it's not, adjust the given timeout.

Yeah, that is an option, but while it might minimise the race conditions, there are still chances that the backup command will run longer that expected.

[m90]

I don’t know if there is a way to make sure they are ordered in a specific order

There is no such option.

another option would be to create another label for the backed up container (in my case it is postgres), something like docker-volume-backup.chown-user-group

That sounds overly specific to me and I've never seen it requested, so I don't think this will be added.

Yeah, that is an option, but while it might minimise the race conditions, there are still chances that the backup command will run longer that expected.

True, but it's all I can offer at the moment.

[ottobolyos]

That sounds overly specific to me and I've never seen it requested, so I don't think this will be added.

It is requested now. 😉

I get it is quite specific, however, anyone who wants to back up a container which is run as a non-root user could get into this issue all the time. Therefore, I think it is a good idea to add a way to make sure it works as expected.

To make the option more general, I see two options:

1. let the user choose the order in which the containers are backed up:

   • a user would define an env var to a comma-separated list of container names;
   • the containers would be backed up in the order they are defined in that env var;
   • non-existent containers and containers without the required labels would be skipped (a warning/debug log could be output);

2. let the user run a command defined in a container (e.g. postgres) which will be executed in the backup container:

   • e.g. using a label like docker-volume-backup.backup.archive-pre (note the added .backup);
   • while the label is defined for the container to back up, the command will be executed in the backup container.

WDYT? 🤔

[m90]

i'll repeat what I said, I think it's too specific and am not interested in adding it, sorry.