[beats receivers] Defunct `elastic-agent otel --supervised` process left behind when Elastic Agent re-executes itself

[cmacknz]

When the Elastic Agent reexecs itself we get a defunct collector sub-process. For comparison, we do not get defunct Beat sub-processes when this happens. Investigate and fix the difference in implementation that is causing this.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pkoutsovasilis]

Update

After investigating this issue, I've discovered that it's not limited to just the collector subprocess – it also affects command components.

Root Cause Analysis

I had a suspicion that collector shutdown time is affected by the number of pipeline workers in the Elasticsearch exporter configuration. The attached screenshots confirm this theory – the collector doesn't stop within the arbitrary 5-second wait time captured in this analysis, while the beats subprocesses stop almost immediately.

Here's what's happening:

1. When the collector takes longer than 5 seconds to shut down, Elastic Agent proceeds to execve itself
2. During execve, all threads other than the calling thread are destroyed
3. This triggers the PDeathSig mechanism we enable here
4. However, the parent process (pre-execve Elastic Agent) is not reaping (waiting for) the exit status of any spawned subprocesses from the previous Elastic Agent process
5. Result: these subprocesses end up as defunct/zombie processes

Why We Haven't Seen This With Beats

If I artificially slow down beats subprocesses to take longer than 5 seconds, the same thing happens to them. We've been "lucky" that beats typically terminate fast enough, or at least that's my working theory.

Testing Notes

I tried running the EDOT collector in K8s where pods were emitting logs, but running elastic-agent restart locally on my laptop didn't result in EDOT taking so much time to shutdown. Screenshots attached showing the defunct processes and timing behaviour.

Defunct edot subprocess

• Beats subprocesses stop almost immediately while the EDOT collector subprocess takes more than 5 sec

After my PR here

• Beats subprocesses stop almost immediately but agent waits for EDOT collector subprocess to stop which takes ~ 15 sec

Proposed Solution

Either way, I believe this needs to be handled appropriately within the given time constraints where a process is waited for to finish. In my PR, I've increased the coordinator timeout to accommodate subprocess timeouts. With this change:

• ✅ No defunct processes remain
• ⚠️ However, the EDOT subprocess now takes 20 seconds to stop (in certain environments)

Additional Concern

This is most probably worth taking a closer look at since the EDOT subprocess is restarted on each config change, and taking 20 seconds for each restart is a hefty price to pay for config updates.

[cmacknz]

Summarizing the discussions from the beats receivers meeting this morning:

• We keep the coordinator shutdown timeout at 5s, but make sure that the collector sub-process is sent SIGKILL before this time elapses. Sending SIGKILL after 3 or 4s and ensuring we can wait on the exited process should fix the defunct process.
  • We want to keep the collector shutdown timeout short (as in a few seconds) because we will wait for this amount of time for the re-exec during initial enrollment. Having this timeout get significantly longer (20-30s) will likely cause timeouts in CI systems and automation that installs+enrolls agents which will cause unnecessary disruption.
• Since this affects Beats, we could shorten the timeout there as well. However, since all Beats will convert to receivers they will eventually get this fix gradually which IMO is a safer way to make this change because of the feature flagging of beats receivers. If the change in shutdown timeout causes unexpected issues users can easily switch back to the old behavior, where as if we change the default for beats in code or the spec file they need to upgrade to get a fix if one turns out to be needed.
  • The safest things to do here is leave the beat shutdown timeout unchanged from what it is now.

Related, we also have a version of this problem in #2190 from the watcher that we explicitly want to live through the execve call, we just don't do anything to inherit the process file descriptor so we can wait on it in the new agent version.

[cmacknz]

#10786 as the follow up to make this configurable and have different values depending on where the restart is triggered from.