Skip to content

Windows: watchdog self-kill must os._exit, not no-op sys.exit from a bg thread#51

Open
danielhertz1999-bit wants to merge 2 commits into
CodeAbra:mainfrom
danielhertz1999-bit:fix/watchdog-self-kill-windows
Open

Windows: watchdog self-kill must os._exit, not no-op sys.exit from a bg thread#51
danielhertz1999-bit wants to merge 2 commits into
CodeAbra:mainfrom
danielhertz1999-bit:fix/watchdog-self-kill-windows

Conversation

@danielhertz1999-bit

Copy link
Copy Markdown
Contributor

Problem

The liveness watchdog can detect a wedged daemon but fails to actually kill it on Windows, so the daemon stays down until a manual taskkill.

The watchdog runs in a background daemon thread (name="iai-liveness-watchdog", daemon/__init__.py). When it decides to kill, _self_kill() in daemon/_watchdog.py branches on hasattr(signal, "SIGKILL"):

if hasattr(signal, "SIGKILL"):
    os.kill(os.getpid(), signal.SIGKILL)   # POSIX: signals the whole process
else:
    sys.exit(1)                            # Windows: NO-OP from a bg thread

On Windows there is no SIGKILL, so it falls to sys.exit(1) — which only raises SystemExit in the calling thread. From the background watchdog thread that unwinds the thread, not the process, so the wedged daemon keeps running. Task Scheduler's <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy> then treats the task as still "Running" and silently refuses to start a replacement, so the daemon stays down until someone kills the zombie by hand.

Observed in the wild: watchdog logs daemon_wedge_kill pid=<pid> (intent recorded) but the process with that pid is still alive minutes later.

Fix

Use os._exit(1) on the no-SIGKILL path. It terminates the whole process immediately from any thread and bypasses interpreter shutdown — which is what you want here, since normal cleanup could itself hang on the same wedge the watchdog is escaping. POSIX behavior (SIGKILL) is unchanged.

Test

The three existing _self_kill tests are all gated on @_REQUIRES_SIGKILL, so they skip on Windows — which is precisely why the no-op path shipped uncaught. The new test simulates the no-SIGKILL branch (via monkeypatch.delattr(signal, "SIGKILL")) so it runs on every platform, and asserts _self_kill calls os._exit and never sys.exit/os.kill. Verified both ways: passes on the fix, fails against the pre-fix source with Failed: sys.exit is a no-op from the watchdog thread.

Windows-only behavior change; POSIX paths are unaffected.

danielhertz1999-bit and others added 2 commits July 5, 2026 14:11
The liveness watchdog runs in a background daemon thread
(name="iai-liveness-watchdog"). When it decides to kill a wedged daemon,
_self_kill() branches on hasattr(signal, "SIGKILL"): POSIX sends SIGKILL
to its own pid (kills the whole process), but the Windows fallback called
sys.exit(1) -- which only raises SystemExit in the calling thread. From a
background thread that terminates the thread, not the process, so the
wedged daemon kept running. Task Scheduler's IgnoreNew instance policy
then treated the task as still "Running" and silently refused to start a
replacement, so the daemon stayed down until a manual taskkill.

Use os._exit(1) on the no-SIGKILL path: it terminates the whole process
immediately from any thread and bypasses interpreter shutdown, which could
otherwise hang on the same wedge the watchdog is trying to escape.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing _self_kill tests are all gated on @_REQUIRES_SIGKILL, so they
skip on Windows -- which is exactly how the no-op sys.exit path shipped
uncaught. The new test simulates the no-SIGKILL branch (so it runs on every
platform) and asserts _self_kill calls os._exit, never sys.exit. Verified
it fails against the pre-fix source.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant