Clarifying auto-update and rollback behaviour

[deuill]

I've been playing around with podman auto-update for io.containers.autoupdate=local, and am seeing some unexpected behaviour I was hoping to clarify here. Firstly, what I'm seeing, assuming an example podman run ... example.com/test:1.0 invocation:

• Podman will only auto-update if the actual 1.0 tag for the example.com/test image is updated -- tagging a newer version (e.g. 2.0) will have this be ignored. The mechanism here is essentially a digest match between the running and stored version for a given image name.
• Rollbacks will use the user-provided name, not the digest, and attempt to re-tag with that name. This is similar to doing podman tag <new-name> <old-name>.

The question I'm seeking clarification on is, how do rollbacks work if <new-name> is necessarily the same as <old-name>, since podman auto-update only does a digest match for one given name?

Since updating between tags for the same base name is likely wrought with perils (using "latest updated" or semantic versioning not likely to work for everyone), I would expect rollbacks to re-tag against the old digest, not the old (user-provided) name.

I'm probably missing something, though, since the functionality here has been available for a while; documentation does not help illuminate a real-world use-case. Having an example of one would definitely help.

[mheon]

@vrothberg PTAL

[vrothberg]

I've been playing around with podman auto-update for io.containers.autoupdate=local, and am seeing some unexpected behaviour I was hoping to clarify here. Firstly, what I'm seeing, assuming an example podman run ... example.com/test:1.0 invocation:

* Podman will only auto-update if the actual `1.0` tag for the `example.com/test` image is updated -- tagging a newer version (e.g. `2.0`) will have this be ignored. The mechanism here is essentially a digest match between the running and stored version for a given image name.

That is correct. There's an inherent ambiguity in using tags, so the identify of an image is the digest.

* Rollbacks will use the user-provided name, not the digest, and attempt to re-tag with that name. This is similar to doing `podman tag <new-name> <old-name>`.

What Podman does essentially is to podman tag $oldDigest $name. Notice the $oldDigest; the tag/name never changes.

The question I'm seeking clarification on is, how do rollbacks work if <new-name> is necessarily the same as <old-name>, since podman auto-update only does a digest match for one given name?

Technically old-name and new-name are the same. What changes is the digest. So Podman knows the digest of the "old image" and the one of "new image". During rollback, Podman does a podman tag $oldDigest $name and restarts the systemd unit.

I'm probably missing something, though, since the functionality here has been available for a while; documentation does not help illuminate a real-world use-case. Having an example of one would definitely help.

The "local" policy is very similar to the "registry" one - at least in theory. But instead of pulling down a new image from a registry (i.e., podman pull $name), Podman does a podman image inspect $name and compares the "new" digest to the "old" one. If it has changed, the image is considered to be updated/newer.

An example may be if an admin chooses to manually update images.

[deuill]

Thanks for clarifying here, after reading through this and working through test-cases added as part of the original commit, I've been able to pinpoint the source of my confusion -- indeed podman auto-update --rollback works well when Type=notify is used with --sdnotify=container.

However, Type=notify with --sdnotify=conmon, or Type=simple will not work -- the former is what I'm operating for the majority of services I run (since very few public services come with systemd_notify support and some notion of run state notifications is better than none), hence my confusion.

I realize that the documentation for podman auto-update calls out --sdnotify=container as having the best results, though I'm unsure if it's the only way of getting the correct behaviour out of auto-updates, or whether there's some --cidfile magic that helps podman auto-update realize the actual container process died if conmon doesn't get to it first.

[vrothberg]

[...] or whether there's some --cidfile magic that helps podman auto-update realize the actual container process died if conmon doesn't get to it first.

I find it difficult to come up with a metric of when a container can be considered to have failed? If it exists non-zero after n seconds? Especially the timeout is machine-bound - some are faster, some under load etc.

Using sdnotify is so far the most accurate solution, as (the developers of) the containers decide whether the workloads have started successfully by explicitly sending the READY message. In other words, I think it's best to let the containers signal when they're ready rather than "guessing".

But that if a good "guess" works for certain use cases, we should definitely consider it.

[deuill]

The way I understand it, the issue comes down to this: services with --sdnotify=conmon will return as having being started successfully, even the container entrypoint fails immediately -- conmon will send READY=1 once the container itself (but not the entrypoint) is initialized, and so the response to org.freedesktop.systemd1.Manager.RestartUnit will be one of success.

Solving this is understandably a tough issue; there have been some discussions about exposing a --sdnotify=healthcheck option to have ready state notifications be trigged on health-check pass, which would solve this for containers with health-checks defined, but it seems this may not be desirable at the moment. Alternatively, tying rollbacks to health-checks, as per rhatdan's comment below might also solve this.

My gut feeling is that most container failures happen close to initialization -- either arguments given are incorrect, configuration is malformed or missing, etc. -- and a janky-but-workable solution might be to perform all pending update tasks but defer rollbacks until the end, waiting for up to, say, a second, and fetching the collected status of all units from Systemd via something like ListUnitsByNames. As you say, a sleep-based solution is likely to be quite inaccurate, though perhaps still more accurate than what it currently is (at least, for containers which don't handle ready state notifications themselves)?

[vrothberg]

The way I understand it, the issue comes down to this: services with --sdnotify=conmon will return as having being started successfully, even the container entrypoint fails immediately -- conmon will send READY=1 once the container itself (but not the entrypoint) is initialized, and so the response to org.freedesktop.systemd1.Manager.RestartUnit will be one of success.

Yes. But even when the container has started without exiting immediately, it is hard to "guess" when a container can be considered to have started successfully.

My gut feeling is that most container failures happen close to initialization -- either arguments given are incorrect, configuration is malformed or missing, etc. -- and a janky-but-workable solution might be to perform all pending update tasks but defer rollbacks until the end, waiting for up to, say, a second, and fetching the collected status of all units from Systemd via something like ListUnitsByNames. As you say, a sleep-based solution is likely to be quite inaccurate, though perhaps still more accurate than what it currently is (at least, for containers which don't handle ready state notifications themselves)?

We can certainly add new --sdnotify=timeout=30s kind of option that waits for 30 seconds and would only send the READY message if the container is still running.

[rhatdan]

What about a healthcheck? When a container starts, do we know if it is healthy or not? Or do they start healthy and end up unhealthy?
If the state went Unknown->UnHealthy or Unknown->Healthy, then we could check the health status via a podman wait for healthy call.
Perhaps with a multi minute timer.

[vrothberg]

I am not sure about health checks and think they are two separate issues.

A health check is primarily used to figure if a container fails at some point in the future. Containers usually start healthy but may fail in the (distant) future. How I envision health checks in systemd is to keep the units robust in conjunction with the recently added on-failure actions (see aad29e7).

We have a number of pieces in that context:

• A (Podman) container running in a systemd unit
• The systemd unit's restart policy is "on-failure"
• The container has a health check configured with --health-on-failure=kill
• The container's auto-update policy is set to "registry"

The auto-update policy makes sure that the service pulls in updates and bug fixes and IMO fits into automation. The health-check stuff and the restart policy make sure that the service (in whatever version) is kept alive and fits into robustness; once the container turned unhealthy, the on-failure action kills the container and the container's systemd unit will restart it automatically.

Certainly a health-check could be used to in conjunction with --sdnotify but the logic is custom anyways and in that case I would keep the two concerns separate:

• The container indicating that it started successfully (--sdnotify)
• The container fails at some point in the future (--health-*)

[vrothberg]

In other words, if we repurpose health-checks for sdnotify things, we sacrifice the on-failure actions. So I have mixed feelings in promoting that.

[rhatdan]

The problem with sd-notify is that most containers do not know how to do it, and it becomes a more difficult programming problem. I think healthchecks are easier to code and understand. Can I curl the website and get back a proper response.

[vrothberg]

Let us cook this idea a bit. I am sure we can find a way to integrate the two without sacrificing the on-failure actions. --sdnotify=healthy,5 for instance could wait for 5 consecutive health checks and if the container's still healthy return READY.

[deuill]

I'd assume that, even without the the ability to wait on consecutive health-checks, both --sdnotify=healthy and --on-failure=kill are orthogonal to one another; even if health-checks pass once, then invariably fail the next time, it's probably the health-check itself (either timeout, start-period, retries, etc.) that's flaky and needs fixing, not the --sdnotify definition.

Otherwise, health-checks that fail immediately after start will presumably cause the RestartUnit call to return failed, and therefore have a rollback take effect, as expected.