Notify workers of quick shutdown after graceful shutdown times out.

[adk-swisstopo]

As per #3385 this allows to distinguish between graceful and abnormal termination and gives a chance to workers to do something about it (e.g. to log a stack trace).

For backward compatibility, we keep the old 2 signals model if quick_shutdown_timeout is set to 0.

[pajod]

I like it, completely alleviates the concern that people might have depended on the previous behavior.

---

Note to reviewers: This patch carefully retains two questionable behaviors, which should probably be changed separately:

• the undocumented (and annoying, in CI-tests) minimum shutdown delay of 0.1 seconds
• the use of time.time() where time.monotonic() would be appropriate

However:

• sending SIGINT or SIGQUIT (though handled the same) might need to be retained as well

[zffocussss]

interesting try!

[pajod]

You requested a review, but I have nothing substantial to add. It appears to work as intended, and whatever could be improved can safely be done so in separate merges.

e.g. I would recommend eventually changing:

• remove the 0.1 minimum interval in sleep_until
• discard the 0.1 minimum shutdown interval from the worker, such that graceful shutdown does not typically trigger the entirely unnecessary Worker (pid:1337) was sent SIGKILL! Perhaps out of memory? hint when really nothing went wrong besides cfg.quick_shutdown_timeout being exceeded by 0.008 and thus SIGCHLD not having been processed on arbiter yet.

My naive approach at doing both introduces more ways to create unexpected behavior when actually using very small values for the 3 timeouts, so maybe that is something easier to get right after applying something like #3148.