Skip to content

Commit d7adc72

Browse files
SIGTERM is now graceful, the same as SIGINT.
1 parent 8ad497d commit d7adc72

File tree

12 files changed

+85
-61
lines changed

12 files changed

+85
-61
lines changed

context/kubernetes-integration.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,9 @@ spec:
3737
periodSeconds: 5
3838
failureThreshold: 12
3939
```
40+
41+
## Graceful Shutdown
42+
43+
Controllers handle `SIGTERM` gracefully (same as `SIGINT`). This ensures proper graceful shutdown when Kubernetes terminates pods during rolling updates, scaling down, or pod eviction.
44+
45+
**Note**: Kubernetes sends `SIGTERM` to containers when terminating pods. With graceful handling, your application will have time to clean up resources, finish in-flight requests, and shut down gracefully before Kubernetes sends `SIGKILL` (after the termination grace period).

context/systemd-integration.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,9 @@ WantedBy=multi-user.target
2020
```
2121

2222
Ensure `Type=notify` is set, so that the service can notify systemd when it is ready.
23+
24+
## Graceful Shutdown
25+
26+
Controllers handle `SIGTERM` gracefully (same as `SIGINT`). This ensures proper graceful shutdown when systemd stops the service.
27+
28+
**Note**: systemd sends `SIGTERM` to services when stopping them. With graceful handling, your application will have time to clean up resources, finish in-flight requests, and shut down gracefully before systemd escalates to `SIGKILL` (after the timeout specified in the service file).

fixtures/async/container/controllers/graceful.rb

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,6 @@
1111
class Graceful < Async::Container::Controller
1212
def setup(container)
1313
container.run(name: "graceful", count: 1, restart: true) do |instance|
14-
instance.ready!
15-
16-
# This is to avoid race conditions in the controller in test conditions.
17-
sleep 0.001
18-
1914
clock = Async::Clock.start
2015

2116
original_action = Signal.trap(:INT) do
@@ -26,6 +21,7 @@ def setup(container)
2621
end
2722

2823
$stdout.puts "Ready...", clock.total
24+
instance.ready!
2925

3026
sleep
3127
ensure

guides/kubernetes-integration/readme.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,9 @@ spec:
3737
periodSeconds: 5
3838
failureThreshold: 12
3939
```
40+
41+
## Graceful Shutdown
42+
43+
Controllers handle `SIGTERM` gracefully (same as `SIGINT`). This ensures proper graceful shutdown when Kubernetes terminates pods during rolling updates, scaling down, or pod eviction.
44+
45+
**Note**: Kubernetes sends `SIGTERM` to containers when terminating pods. With graceful handling, your application will have time to clean up resources, finish in-flight requests, and shut down gracefully before Kubernetes sends `SIGKILL` (after the termination grace period).

guides/systemd-integration/readme.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,9 @@ WantedBy=multi-user.target
2020
```
2121

2222
Ensure `Type=notify` is set, so that the service can notify systemd when it is ready.
23+
24+
## Graceful Shutdown
25+
26+
Controllers handle `SIGTERM` gracefully (same as `SIGINT`). This ensures proper graceful shutdown when systemd stops the service.
27+
28+
**Note**: systemd sends `SIGTERM` to services when stopping them. With graceful handling, your application will have time to clean up resources, finish in-flight requests, and shut down gracefully before systemd escalates to `SIGKILL` (after the timeout specified in the service file).

lib/async/container/controller.rb

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -134,9 +134,6 @@ def restart
134134
if container.failed?
135135
@notify&.error!("Container failed to start!")
136136

137-
Console.info(self, "Stopping failed container...")
138-
container.stop(false)
139-
140137
raise SetupError, container
141138
end
142139

@@ -152,8 +149,11 @@ def restart
152149

153150
@notify&.ready!(size: @container.size)
154151
ensure
155-
# If we are leaving this function with an exception, try to kill the container:
156-
container&.stop(false)
152+
# If we are leaving this function with an exception, kill the container:
153+
if container
154+
Console.info(self, "Stopping failed container...")
155+
container.stop(false)
156+
end
157157
end
158158

159159
# Reload the existing container. Children instances will be reloaded using `SIGHUP`.
@@ -222,9 +222,10 @@ def run
222222
::Thread.current.raise(Interrupt)
223223
end
224224

225+
# SIGTERM behaves the same as SIGINT by default.
225226
terminate_action = Signal.trap(:TERM) do
226-
# $stderr.puts "Received TERM signal, terminating...", caller
227-
::Thread.current.raise(Terminate)
227+
# $stderr.puts "Received TERM signal, interrupting...", caller
228+
::Thread.current.raise(Interrupt) # Same as SIGINT
228229
end
229230

230231
hangup_action = Signal.trap(:HUP) do

lib/async/container/forked.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ def self.fork(**options)
102102
::Process.fork do
103103
# We use `Thread.current.raise(...)` so that exceptions are filtered through `Thread.handle_interrupt` correctly.
104104
Signal.trap(:INT){::Thread.current.raise(Interrupt)}
105-
Signal.trap(:TERM){::Thread.current.raise(Terminate)}
105+
Signal.trap(:TERM){::Thread.current.raise(Interrupt)} # Same as SIGINT.
106106
Signal.trap(:HUP){::Thread.current.raise(Restart)}
107107

108108
# This could be a configuration option:

lib/async/container/group.rb

Lines changed: 29 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,20 @@
1010

1111
module Async
1212
module Container
13-
# The default timeout for interrupting processes, before escalating to terminating.
14-
INTERRUPT_TIMEOUT = ENV.fetch("ASYNC_CONTAINER_INTERRUPT_TIMEOUT", 10).to_f
15-
1613
# The default timeout for terminating processes, before escalating to killing.
17-
TERMINATE_TIMEOUT = ENV.fetch("ASYNC_CONTAINER_TERMINATE_TIMEOUT", 10).to_f
14+
GRACEFUL_TIMEOUT = ENV.fetch("ASYNC_CONTAINER_GRACEFUL_TIMEOUT", "true").then do |value|
15+
case value
16+
when "true"
17+
true # Default timeout for graceful termination.
18+
when "false"
19+
false # Immediately kill the processes.
20+
else
21+
value.to_f
22+
end
23+
end
24+
25+
# The default timeout for graceful termination.
26+
DEFAULT_GRACEFUL_TIMEOUT = 10.0
1827

1928
# Manages a group of running processes.
2029
class Group
@@ -155,49 +164,32 @@ def kill
155164
# Stop all child processes with a multi-phase shutdown sequence.
156165
#
157166
# A graceful shutdown performs the following sequence:
158-
# 1. Send SIGINT and wait up to `interrupt_timeout` seconds
159-
# 2. Send SIGTERM and wait up to `terminate_timeout` seconds
160-
# 3. Send SIGKILL and wait indefinitely for process cleanup
167+
# 1. Send SIGINT and wait up to `graceful` seconds if specified.
168+
# 2. Send SIGKILL and wait indefinitely for process cleanup.
161169
#
162-
# If `graceful` is false, skips the SIGINT phase and goes directly to SIGTERM → SIGKILL.
170+
# If `graceful` is true, default to `DEFAULT_GRACEFUL_TIMEOUT` (10 seconds).
171+
# If `graceful` is false, skip the SIGINT phase and go directly to SIGKILL.
163172
#
164-
# @parameter graceful [Boolean] Whether to send SIGINT first or skip directly to SIGTERM.
165-
# @parameter interrupt_timeout [Numeric | Nil] Time to wait after SIGINT before escalating to SIGTERM.
166-
# @parameter terminate_timeout [Numeric | Nil] Time to wait after SIGTERM before escalating to SIGKILL.
167-
def stop(graceful = true, interrupt_timeout: INTERRUPT_TIMEOUT, terminate_timeout: TERMINATE_TIMEOUT)
168-
case graceful
169-
when true
170-
# Use defaults.
171-
when false
172-
interrupt_timeout = nil
173-
when Numeric
174-
interrupt_timeout = graceful
175-
terminate_timeout = graceful
176-
end
177-
178-
Console.debug(self, "Stopping all processes...", interrupt_timeout: interrupt_timeout, terminate_timeout: terminate_timeout)
173+
# @parameter graceful [Boolean | Numeric] Whether to send SIGINT first or skip directly to SIGKILL.
174+
def stop(graceful = GRACEFUL_TIMEOUT)
175+
Console.info(self, "Stopping all processes...", graceful: graceful)
179176

180177
# If a timeout is specified, interrupt the children first:
181-
if interrupt_timeout
182-
clock = Async::Clock.start
183-
184-
# Interrupt the children:
178+
if graceful
179+
# Send SIGINT to the children:
185180
self.interrupt
186181

187-
# Wait for the children to exit:
188-
self.wait_for_exit(clock, interrupt_timeout)
189-
end
190-
191-
if terminate_timeout and self.any?
192-
clock = Async::Clock.start
182+
if graceful == true
183+
graceful = DEFAULT_GRACEFUL_TIMEOUT
184+
end
193185

194-
# If the children are still running, terminate them:
195-
self.terminate
186+
clock = Clock.start
196187

197188
# Wait for the children to exit:
198-
self.wait_for_exit(clock, terminate_timeout)
189+
self.wait_for_exit(clock, graceful)
199190
end
200-
191+
ensure
192+
# Do our best to clean up the children:
201193
if any?
202194
self.kill
203195
self.wait

lib/async/container/notify/pipe.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ def send(**message)
6969
@io.flush
7070
end
7171

72-
private
72+
private
7373

7474
def environment_for(arguments)
7575
# Insert or duplicate the environment hash which is the first argument:

readme.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,11 @@ Please see the [project documentation](https://socketry.github.io/async-containe
2626

2727
Please see the [project releases](https://socketry.github.io/async-container/releases/index) for all releases.
2828

29+
### Unreleased
30+
31+
- `SIGTERM` is now graceful, the same as `SIGINT`, for better compatibility with Kubernetes and systemd.
32+
- `ASYNC_CONTAINER_INTERRUPT_TIMEOUT` and `ASYNC_CONTAINER_TERMINATE_TIMEOUT` are removed and replaced by `ASYNC_CONTAINER_GRACEFUL_TIMEOUT`.
33+
2934
### v0.29.0
3035

3136
- Introduce `Client#healthy!` for sending health check messages.
@@ -65,10 +70,6 @@ Please see the [project releases](https://socketry.github.io/async-container/rel
6570

6671
- [Production Reliability Improvements](https://socketry.github.io/async-container/releases/index#production-reliability-improvements)
6772

68-
### v0.25.0
69-
70-
- Introduce `async:container:notify:log:ready?` task for detecting process readiness.
71-
7273
## Contributing
7374

7475
We welcome contributions to this project.

0 commit comments

Comments
 (0)