Skip to content

SIGTERM to entire process group triggers wait for worker timeout #160

@AlanCoding

Description

@AlanCoding

Please confirm the following

  • I agree to follow this project's code of conduct.
  • I have checked the current issues for duplicates.
  • I understand that dispatcherd is open source software provided for free and that I might not receive a timely response.

Bug Summary

This is really an AWX issue, but tied into this.

The assumptions that dispatcherd signal handling currently make are that the main process gets a SIGTERM and that the child processes don't. From logs using AWX, and prior assumptions about supervisord, I am fairly sure that all processes receive a SIGTERM to shut down. This leads to the following logs:

tools_awx_1       | 2025-04-19 03:21:55,184 ERROR    [-] dispatcherd.service.pool Worker 4 pid=1868 failed to send exit message in 3 seconds
tools_awx_1       | 2025-04-19 03:21:55,185 ERROR    [-] dispatcherd.service.pool Worker 5 pid=2068 failed to send exit message in 3 seconds
tools_awx_1       | 2025-04-19 03:21:55,185 ERROR    [-] dispatcherd.service.pool Worker 6 pid=2704 failed to send exit message in 3 seconds
tools_awx_1       | 2025-04-19 03:21:55,185 ERROR    [-] dispatcherd.service.pool Worker 7 pid=2707 failed to send exit message in 3 seconds

Looking up the default behavior of SIGTERM in python, it is expected that execution just stops. It seems that's what we're seeing.

It may or may not be worth intercepting signals in the child processes, but the parent process should check the child exit code as a special case in the shutdown logic.

if proc.poll() is not None:
  proc.returncode  # do something with this info

Environment

N/A

Steps to reproduce

N/A

Actual results

a

Expected results

see above

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions