-
|
Hey, I'm kinda confused, I have a docker container that declares a bunch of stage2 services in I'm setting the I don't understand how to prevent this behavior and terminate the container properly. I managed to stumble on these two issues: #599 and #601 which seem to mention a similar behavior to what I'm experiencing, and the solution being to set The thing is though, I don't want to enforce a strict time limit for my services to become ready, my container works in kind of a dynamic way, and the services running in there are user-defined. In most cases, it takes quite a while for them to become ready (dozens of seconds), I don't have any reasonable number that I could use for that value, the wait time for readiness should be 0 (infinite), or something that differs per-service. When I do set this env var to something non-zero though, the container does seem to terminate within that specified time, but until then, the failing service just keeps restarting in an infinite loop. That's not good enough for me, since that number would need to be at least 100s to be on the safe side (maybe even more), as it can contain a lot of services with various dependencies and they can take different amounts of time to become ready. But I don't want to have to wait 100s for my container to exit if I already know that one of the services exitted, the moment that happens, I already want to enter into the termination phase, and stop all of the other services, regardless of their readiness (they should still be stopped gracefully though). So, to put it simply, I don't want services not becoming ready within some time to be considered a problem (or if it should be a problem, it should be decided per-service through the Other things I triedOther than using the And so, I wasn't able to terminate the container from inside with any method that I attempted to use, and I'm left with no idea how to go about doing so. I don't know if this is a bug, or just a misunderstanding of the behavior on my side, if it's a bug, maybe move this from discussions to an issue, if not, what's going on? And what should I do to get the container to actually terminate at the moment any of the services fails, without also needlessly waiting? Should I use something different than the halt command here to achieve this? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 3 replies
-
|
The thing to understand is that s6, the process supervisor, was made to handle processes dying - which can happen for a variety of reasons, such as failing to acquire resources, the network being inaccessible, etc. - and that's why it restarts daemons by default. It considers these errors to be temporary, and in order to tell it that no, a process death is a serious issue and the whole container should shut down, you have to do some extra work. On top of that, since in most cases services can depend on one another, s6-overlay uses the s6-rc service manager, which was made to ensure that a service is only started once all its dependencies are ready. It is only useful at container boot time, but it is important - and here as well, by default a process death is not seen as a permanent failure; permanent failure only happens when the service hasn't managed to become ready after a certain time, despite being restarted if it dies. That is why you see all these configuration knobs with timeouts. Your use case is a little peculiar because you are saying "in normal circumstances my daemons never die; any death should trigger a full container shutdown", and also "I want infinite timeouts". It is possible to do that, it just requires some setup. First, if you want infinite timeouts, that's fine. Delete S6_CMD_WAIT_FOR_SERVICES_MAXTIME or any other variable that set timeouts, and s6-overlay won't time you out - it's just that you need to be careful to avoid cases where your container stalls forever. Second, the way to perform an action when a supervised process dies is to do the action you want in a exec /run/s6/basedir/bin/halt... except, not quite. That will work in cruise mode, when all the services have been launched and are just being supervised by s6. That will not work if a process dies while the services are still being brought up - because in that case, the s6-rc service manager is still running, and And the way to distinguish between both cases is to check whether s6-rc is currently running, which you can do by calling if s6-rc list ; then
exec /run/s6/basedir/bin/halt
else
exit 125
fiAdd this to all your services, and there you go. I agree that it's not intuitive at all, and that you need pretty esoteric knowledge of s6 to make it work, but it's the first time someone has your exact set of requirements. 😅 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the explanation and quick reply! Though while this solution does prevent the needless restart loop and does lead to container termination, it's still not exactly the solution I had in mind, since the termination phase only triggers once all of the other components pass readiness, not immediately after the first service fails, which introduces a lot of time waste, due to how slow my services are to start. This is my finish script (for every component, the only difference being in the name, the finish scripts are auto-generated during the container build phase, as mentioned, there's a bunch of such services, and each one of those is considered a primary service that should never fail): [root@86562b1c147e opt]# cat /etc/s6-overlay/s6-rc.d/idport-case-frontguard/finish
#!/bin/bash
name=idport-case-frontguard
if [ "$1" -eq 256 ]; then
e=$((128 + $2))
else
e="$1"
fi
echo "[aio-image] Component $name exited (code $e)" >&2
# Don't know if I should still be doing this with the below
#echo "$e" > /run/s6-linux-init-container-results/exitcode
if s6-rc list >/dev/null 2>&1; then
exec /run/s6/basedir/bin/halt
else
exit 125
fi
[root@86562b1c147e opt]#This is the log from stderr:As you can see from the log, the services kept on loading until all of them became ready, which took around 3 minutes, due to the various dependencies between them and it just taking a while for a single component to pass readiness in many cases. This is a huge time-waste for someone that would expect the container to just fail as soon as one of the services fail, and they aren't actively monitoring the logs, looking for failures to stop the container sooner manually. There's no reason for me to wait until the readiness of the other services if one already permanently failed, I already know that the container can be stopped at that point. I'd like to enter the termination phase immediately after the failure (sending sigterms to all of the services that are still starting up, and not continuing to start new ones as dependencies become met). Would this be possible? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I understand. The thing is, while the startup is being controlled by s6-rc, there isn't much you can do; the solution I suggested tells s6-rc that the transition failed, but s6-rc is still trying to bring up the other services in parallel and will only exit when it has a result (either success or failure) from everything; there's no way to tell it to abort. It's usually not a problem because most people operate with reasonable timeouts. If you want to quit immediately as soon as one service fails, you have to kill s6-rc, then call halt, rather than telling s6-rc that the service failed permanently. s6-rc isn't supervised, so there is no clean way of finding its pid, but in this case the hacky if ! s6-rc list >/dev/null 2>&1; then
kill `pgrep s6-rc`
fi
exec /run/s6/basedir/bin/haltThis is very much not how the whole system was intended to be used, but it's probably the easiest way to satisfy your requirements. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! That did the trick perfectly. I understand my use case is a bit unusual, but I do feel like understanding this behavior was quite unintuitive at first. I doubt I’d have figured it out without your explanation here. For that reason, I feel that it might be worth documenting this somewhere in the README (or at least mentioning it briefly and linking to this discussion) to clarify how the shutdown logic works and why readiness completion is required before halt can proceed. That could really help others who run into similar issues. It’s a bit unfortunate that this approach adds a dependency on That said, your solution works well and completely resolves my issue, thank you again for the detailed explanation and help! |
Beta Was this translation helpful? Give feedback.
-
|
It is indeed a flaw in the larger design, but it's the best I could come up with to offer parallel start while keeping compatibility with s6-overlay v2 that didn't inconvenience too many people. When I get the time, I'll try and find something that allows for immediate shutdown in a cleaner and easier way. |
Beta Was this translation helpful? Give feedback.
Yeah, I understand.
The thing is, while the startup is being controlled by s6-rc, there isn't much you can do; the solution I suggested tells s6-rc that the transition failed, but s6-rc is still trying to bring up the other services in parallel and will only exit when it has a result (either success or failure) from everything; there's no way to tell it to abort. It's usually not a problem because most people operate with reasonable timeouts.
If you want to quit immediately as soon as one service fails, you have to kill s6-rc, then call halt, rather than telling s6-rc that the service failed permanently. s6-rc isn't supervised, so there is no clean way of finding its pid, but in this case…