Why docker compose does not restart a container that "exited" 24h ago and has "restart: always" ?

[f18m]

Hi,
I know this is more like a question related to docker compose rather than doco-cd, but I think here there is enough know-how to answer this.

So I have a container (Immich server) whose composition is defined as:

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:v1.137.3
    volumes:
      - immich_upload_local:/usr/src/app/upload:rw
      - /etc/localtime:/etc/localtime:ro
      - photos_nas:/data/import/photos:ro
      - immich_backup_nas:/usr/src/app/upload/backups:rw
    ...
    restart: always

So I would expect docker-compose to restart this container always. But I see:

 docker ps -a  | grep immich_server
cf6202c28a83   ghcr.io/immich-app/immich-server:v1.137.3                          "tini -- /bin/bash -…"   2 weeks ago      Exited (255) 24 hours ago                                                                                                                                      immich_server

So the container looks like exited with a non-zero exit status 24h ago... and I cannot see any attempt by docker to restart it in the last 24 hours...

What happened 24 hours ago? I rebooted the host. And I also know why the container was not successfully started immediately after boot: it had to do with some temporary network issue and the fact that e.g. the "photos_nas" volume is actually a remote SMB mount.
Indeed in the "docker inspect immich_server" output I see:

[
    {
        "Id": "cf6202c28a837a4175efe4343b70e8a9df9e3b63d8d1df3a0fc9c73ef8dab130",
        "Created": "2025-08-06T13:02:48.695404451Z",
        "Path": "tini",
        "Args": [
            "--",
            "/bin/bash",
            "-c",
            "start.sh"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 255,
            "Error": "error while mounting volume '/var/lib/docker/volumes/immich_photos_nas/_data': failed to mount local volume: mount //192.168.1.5/photos/:/var/lib/docker/volumes/immich_photos_nas/_data, data: username=****,password=********,uid=1000,gid=1000: no route to host",
  ...

So I understand that last time (24h ago) docker tried to start this container, it was unable to mount the remote folder.
My question is: why doesn't docker keep trying to restart this container?

The network issue is gone now and if I do a "docker compose up -d" in the folder where the Immich docker composition is defined, the container comes up again (and stays up).

This question is also actually connected with #542 because whatever is the answer, I think I will need somehow to work around this docker issue and I would like not to bypass doco-cd and mess with the docker compositions it's supposed to own, but rather use some API exposed by doco-cd to work around docker issues.

[kimdre]

The restart policy only starts working when the container has started successfully before to prevent restart loops. Since the container could not start properly after you rebooted the host, the docker engine never tried to restart the container.
https://docs.docker.com/engine/containers/start-containers-automatically/#restart-policy-details

[f18m]

Ok thanks for the hint.
So that means that docker "retry" policies do not cover 100% of the cases.
I cannot even constrain the start of "docker" daemon to happen after all network dependencies (eg. the NAS with the photos) are up, because most of the docker containers do not have such dependencies...

I guess I will need to implement some "watchdog" on my own on top of docker/doco-cd... which brings me back to #542 :)

[kimdre]

You can change the docker systemd service to wait for your network or network-storage to be available.

[f18m]

You can change the docker systemd service to wait for your network or network-storage to be available.

Yeah my issue with that is that most of the docker containers I run do not have dependencies on the network storage. So placing a wait for the network storage to be available seems a suboptimal choice; e.g. what happens if network storage never goes back online? all docker containers will not be running... this is like propagating a single-point-of-failure to all containers :(

[f18m]

btw the version 0.33 allowed me to implement a basic watchdog on top of doco-cd that simply uses the /v1/api/project/{projectName}/start endpoint every few minutes to ensure that all my docker compositions are (completely) up and running.
Should a transient (network) error issue that prevents a docker container from being created, I think that my watchdog implemented on top of doco-cd will auto-heal that (once the transient issue is resolved and docker compose can successfully create the container).

Thanks!