BACKUP_STOP_CONTAINER_LABEL not stopping container when referencing an env variable

[cxhamilton]

I chose to externalize all of my docker-volume-backup configuration parameters to an external docker env file. My goal is to have a consistent approach to my backups, having only to change one file if I want to make a holistic change. I only need to set BACKUP_NAME (my made-up variable) for each specific container, and everything else is handled by reference.

This approach works perfectly, save for docker-volume-backup stopping the container. I've tried setting BACKUP_STOP_CONTAINER_LABEL to ${BACKUP_NAME} and $${BACKUP_NAME} to no avail. The backup happens and the backup files are rotated, but the container is never stopped (or restarted).

I'm honestly unsure if this is a bug, an unsupported feature, or my understanding docker.

Here are the important bits from my configuration files:

docker-compose.yaml

services:
    plex:
        ...
        labels:
            - docker-volume-backup.stop-during-backup=plex

    plex-backup:
        container_name: plex-backup
        image: offen/docker-volume-backup:latest
        environment:
            - BACKUP_NAME=plex
            - BACKUP_CRON_EXPRESSION=${BACKUP_CRON_EXPRESSION}
            - BACKUP_ARCHIVE=${BACKUP_ARCHIVE}
            - BACKUP_FILENAME_EXPAND=${BACKUP_FILENAME_EXPAND}
            - BACKUP_FILENAME=${BACKUP_FILENAME}
            - BACKUP_LATEST_SYMLINK=${BACKUP_LATEST_SYMLINK}
            - BACKUP_PRUNING_PREFIX=${BACKUP_PRUNING_PREFIX}
            - BACKUP_RETENTION_DAYS=${BACKUP_RETENTION_DAYS}
            - BACKUP_STOP_CONTAINER_LABEL=${BACKUP_STOP_CONTAINER_LABEL}
            - BACKUP_EXCLUDE_REGEXP=${BACKUP_EXCLUDE_REGEXP}
            - NOTIFICATION_URLS=${NOTIFICATION_URLS}
            - NOTIFICATION_LEVEL=${NOTIFICATION_LEVEL}
        volumes:
            ...
        depends_on:
            - plex
        restart: unless-stopped

external.env

BACKUP_CRON_EXPRESSION=0 2 * * *
BACKUP_ARCHIVE=/archive
BACKUP_FILENAME_EXPAND=true
BACKUP_FILENAME=$${BACKUP_NAME}_%Y%m%d%H%M%S.tar.gz
BACKUP_LATEST_SYMLINK=$${BACKUP_NAME}.latest.tar.gz
BACKUP_PRUNING_PREFIX=$${BACKUP_NAME}_
BACKUP_RETENTION_DAYS=30
BACKUP_STOP_CONTAINER_LABEL=$${BACKUP_NAME}
BACKUP_SOURCES=/backup
BACKUP_EXCLUDE_REGEXP=\.log$
NOTIFICATION_URLS=pushover://shoutrrr:<APP_KEY>@<USER_KEY>/?priority=1
NOTIFICATION_LEVEL=error

Any support or suggestions would be much appreciated.

[m90]

Do you mount the host's docker socket into the backup container? I.e. does your list of volumes contain the following line

- /var/run/docker.sock:/var/run/docker.sock:ro

[m90]

In your log output, there should be line sth like this

Stopping 1 container(s) labeled `docker-volume-backup.stop-during-backup=some-token` out of 20 running container(s).

What does it say when a. using the hardcoded value and b. using your $BACKUP_NAME setup?

[cxhamilton]

Explicitly using "plex":

12/31/2023 09:23:13 AM Installing cron.d entry with expression 0 2 * * *.
12/31/2023 09:23:13 AM Starting cron in foreground.
12/31/2023 09:23:13 AM crond: crond (busybox 1.36.1) started, log level 8
01/01/2024 02:00:00 AM crond: USER root pid  10 cmd backup 2>&1
01/01/2024 02:00:04 AM time=2024-01-01T02:00:04.768-05:00 level=INFO msg="Stopping 1 container(s) labeled `docker-volume-backup.stop-during-backup=plex` out of 27 running container(s)."
01/01/2024 02:53:16 AM time=2024-01-01T02:53:16.481-05:00 level=INFO msg="Created backup of `/backup` at `/tmp/plex_20240101020004.tar.gz`."
01/01/2024 02:53:20 AM time=2024-01-01T02:53:20.379-05:00 level=INFO msg="Restarted 1 container(s) and the matching service(s)."
01/01/2024 02:56:40 AM time=2024-01-01T02:56:40.289-05:00 level=INFO msg="Stored copy of backup `/tmp/plex_20240101020004.tar.gz` in `/archive`." storage=Local
01/01/2024 02:56:40 AM time=2024-01-01T02:56:40.485-05:00 level=INFO msg="Created/Updated symlink `plex.latest.tar.gz` for latest backup." storage=Local
01/01/2024 02:56:41 AM time=2024-01-01T02:56:41.443-05:00 level=INFO msg="Pruned 1 out of 22 backups as they were older than the given deadline of 2023-12-02T02:57:41.392383393-05:00." storage=Local
01/01/2024 02:56:41 AM time=2024-01-01T02:56:41.456-05:00 level=INFO msg="Removed tar file `/tmp/plex_20240101020004.tar.gz`."
01/01/2024 02:56:41 AM time=2024-01-01T02:56:41.456-05:00 level=INFO msg="Finished running backup tasks."

Using $BACKUP_NAME:

12/28/2023 12:24:01 PM Installing cron.d entry with expression 0 2 * * *.
12/28/2023 12:24:01 PM Starting cron in foreground.
12/28/2023 12:24:01 PM crond: crond (busybox 1.36.1) started, log level 8
12/29/2023 02:00:00 AM crond: USER root pid  10 cmd backup 2>&1
12/29/2023 02:51:23 AM time=2023-12-29T02:51:23.657-05:00 level=INFO msg="Created backup of `/backup` at `/tmp/plex_20231229020001.tar.gz`."
12/29/2023 02:53:58 AM time=2023-12-29T02:53:58.225-05:00 level=INFO msg="Stored copy of backup `/tmp/plex_20231229020001.tar.gz` in `/archive`." storage=Local
12/29/2023 02:53:58 AM time=2023-12-29T02:53:58.595-05:00 level=INFO msg="Created/Updated symlink `plex.latest.tar.gz` for latest backup." storage=Local
12/29/2023 02:53:59 AM time=2023-12-29T02:53:59.183-05:00 level=INFO msg="Pruned 2 out of 22 backups as they were older than the given deadline of 2023-11-29T02:54:59.170821944-05:00." storage=Local
12/29/2023 02:53:59 AM time=2023-12-29T02:53:59.187-05:00 level=INFO msg="Removed tar file `/tmp/plex_20231229020001.tar.gz`."
12/29/2023 02:53:59 AM time=2023-12-29T02:53:59.187-05:00 level=INFO msg="Finished running backup tasks."

[cxhamilton]

I think I understand my issue. I believe that I wrongly assumed that docker-volume-backup would expand all environment variables at runtime. Go isn't one of the languages I'm familiar with, but I looked at how newScript() handles the environment in script.go.

This caught my eye:

135     if s.c.BackupFilenameExpand {
136         s.file = os.ExpandEnv(s.file)
137         s.c.BackupLatestSymlink = os.ExpandEnv(s.c.BackupLatestSymlink)
138         s.c.BackupPruningPrefix = os.ExpandEnv(s.c.BackupPruningPrefix)

I re-read the configuration documentation and realized my assumption. The only environment variables expanded at runtime are specific and only expanded with BACKUP_FILENAME_EXPAND. Based on my approach, I need BACKUP_STOP_CONTAINER_LABEL expanded at runtime, which is not supported.

I'd appreciate confirmation that my analysis is correct. If so, would you consider adding environment expansion support to all environment variables?

Regardless, thank you for your patience and support. Happy New Year!

[m90]

Your assumptions are correct, yes. Even if expanding env vars by default would probably make sense, it's a breaking change I can't easily introduce here (see #80).

I think there should be a workaround for you though. When you use the conf.d feature and mount your configuration file into the container, its content will be sourced before the values are passed to the script. I think this should expand the values in your .env files and give you what you want, even if it's a bit pell-mell.

See

docker-volume-backup/entrypoint.sh

Lines 17 to 22 in 64daf7b

	for file in /etc/dockervolumebackup/conf.d/*; do
	source $file
	BACKUP_CRON_EXPRESSION="${BACKUP_CRON_EXPRESSION:-@daily}"
	echo "Appending cron.d entry with expression $BACKUP_CRON_EXPRESSION and configuration file $file"
	(crontab -l; echo "$BACKUP_CRON_EXPRESSION /bin/sh -c 'set -a; source $file; set +a && backup' 2>&1") | crontab -
	done

and https://offen.github.io/docker-volume-backup/how-tos/run-multiple-schedules.html

[cxhamilton]

Thanks for the discussion and the suggestions!