Symlinks removed on a brief vhost container restart deadlock nginx-proxy permanently

[Perlover]

Summary

When a vhost container is stopped only momentarily — a stack being recreated by docker compose up, an image update, a restart — the companion deletes that vhost's certificate symlinks. If nginx-proxy happens to (re)start in that window with a config docker-gen wrote before the deletion, it exits:

[emerg] cannot load certificate "/etc/nginx/certs/<host>.crt":
        BIO_new_file() failed (No such file or directory)

From that point nothing recovers, because the two halves wait on each other:

• the companion refuses to do any work while nginx-proxy is down — Error: nginx-proxy container nginx isn't running, logged once per $ACME_HTTP_CHALLENGE_LOCATION cycle, indefinitely;
• nginx-proxy cannot start until the symlinks the companion would recreate are back.

The certificate itself is untouched and perfectly valid the whole time. Only the symlinks are gone.

I hit this on a production host; it stayed down for 40 hours until a human noticed, with every other container healthy.

Reproduction

1. nginx-proxy + companion + one vhost container, certificate already issued.

2. Recreate the stack (docker compose up -d after an image change) so the vhost container is briefly down while the companion is still running.

3. The companion regenerates its domain list from the containers running at that instant and finds none:

   Generated '/app/letsencrypt_service_data' from 7 containers
   

4. remove_all_symlinks treats every vhost as disabled and removes <host>.crt / <host>.key.

5. nginx-proxy starts with the config from a few seconds earlier and dies on the missing file.

6. Deadlock.

Why it is hard to diagnose

The removal is silent unless DEBUG=1:

https://github.com/nginx-proxy/acme-companion/blob/v2.8.2/app/letsencrypt_service#L150-L156

for extension in .crt .key .dhparam.pem .chain.pem; do
    file="${disabled_domain}${extension}"
    if [[ -n "${file// }" ]] && [[ -L "/etc/nginx/certs/${file}" ]]; then
        [[ "$DEBUG" == 1 ]] && echo "Removing /etc/nginx/certs/${file}"
        rm -f "/etc/nginx/certs/${file}"
    fi
done

So the log shows a healthy companion, then hours of "nginx-proxy container isn't running", and nothing that explains where the certificate went. The only trace is the mtime of /etc/nginx/certs.

This code is unchanged from at least 2.2.x through v2.8.2.

Suggestions

Either of these would break the loop:

1. Do not act on an empty enabled-domains list. A list that collapses to zero is far more likely to be a container restart than an operator removing every vhost at once, and skipping the cleanup in that case costs nothing — a genuinely removed vhost gets cleaned on the next run.
2. Do not skip symlink maintenance when nginx-proxy is down. Creating a symlink needs neither a running nginx nor an ACME challenge; only issuance and reload do. Restoring symlinks before the "isn't running" check would let nginx start on its own.

Logging the removal unconditionally (it is a rare, consequential event) would also have turned a 40-hour outage into a five-minute diagnosis.

Related but not the same: #653 is about a user mounting /etc/nginx/certs into their own container and wanting symlinks to persist. This report is about nginx-proxy itself being unable to start, and the companion then refusing to fix it.

Environment

• companion 2.2.9 (behaviour verified identical in v2.8.2)
• nginx-proxy with a separate docker-gen container
• one certificate covering two SANs, both vhosts on the same container

[buchdag]

@Perlover thanks for the bug report and sorry for the hard to diagnose issue.

I think the issue is rather on nginx-proxy's side, which should not start the nginx process before at least a first successful run of docker-gen (to guarantee an up-to date nginx configuration that don't point to no longer existing files).

I suspect this issue can't be replicated with the three containers version of the stack (nginx / docker-gen / acme-companion) because in this setup the proxy's docker-gen won't be killed by nginx failing to start and will eventually generate a correct configuration for nginx.

edit: agreed that this log line should not be debug only though.