[8.19] (backport #10650) fix: zombie processes during restart

[mergify]

What does this PR do?

This PR fixes zombie/defunct processes that are left behind when Elastic Agent re-executes itself during restart. The fix involves:

1. Decreasing the EDOT collector shutdown timeout from 30 seconds to 3 seconds to accommodate the default 5-second timeout of the coordinator shutdown timeout
   • Adding a safety net that waits an additional second after killing a process to ensure Wait() is called
2. Improving graceful shutdown handling in the EDOT collector subprocess manager to ensure proper process cleanup
3. Adding debug logging throughout the shutdown process to better trace subprocess termination
4. Adding an integration test that verifies no zombie processes are left behind after agent restart

Why is it important?

Root Cause

When the Elastic Agent re-executes itself during restart, the following sequence occurs:

1. If a subprocess (particularly the EDOT collector or command components) takes longer than the coordinator's 5-second shutdown timeout, the agent proceeds to execve itself
2. During execve, all threads other than the calling thread are destroyed
3. This triggers the PDeathSig mechanism we enable for subprocesses
4. However, the parent process (pre-execve Elastic Agent) never reaps (waits for) the exit status of spawned subprocesses
5. Result: these subprocesses end up as defunct/zombie processes

Why This Affects EDOT More Than Beats

Beats subprocesses typically terminate almost immediately (within the 5-second window), so they don't become zombies. However, the EDOT collector's shutdown time seemed to be affected by:

• Number of pipeline workers
• Elasticsearch exporter configuration

Impact

• Resource leaks: Zombie processes consume PIDs and kernel memory
• Operational issues: Accumulation of zombies over multiple restarts
• Config update delays: EDOT subprocess restarts on every config change, and 20+ second shutdowns create significant latency

This fix ensures proper process cleanup regardless of shutdown duration while maintaining graceful termination when possible.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

Users may notice:

• Agent restarts take slightly longer (up to 35 seconds instead of 5 seconds in worst case)
• However, this ensures clean shutdowns and prevents zombie accumulation
• The tradeoff is worthwhile as zombie processes can cause operational issues over time

How to test this PR locally

Run TestMetricsMonitoringCorrectBinaries integration test

Related issues

• Elastic-Agent does not allow child process to clean up during shutdown #7756

  ---

  This is an automatic backport of pull request fix: zombie processes during restart #10650 done by Mergify.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 7a150fe

Failed CI Steps

• Win2022:sudo:default
• Win2022:sudo:default
• Win2025:sudo:default
• x86_64:sudo:rpm

History

cc @pkoutsovasilis