Log Loss During Logstash TCP Input Pipeline Reload – Request for Backup Mechanism to Preserve Unprocessed Events #237
Description
Logstash information:
Please include the following information:
- Logstash version (e.g.
bin/logstash --version): 9.0.3 - Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker): Built from Source
- How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes): Kubernetes container.
- How was the Logstash Plugin installed: Built from source as a gem
JVM (e.g. java -version): NA
OS version (uname -a if on a Unix-like system):
Description of the problem including expected versus actual behavior:
Log events are transmitted from a producer to Logstash via the TCP input interface. We’ve observed a scenario where log loss occurs during a Logstash pipeline reload. In this case, the log producer sends an event to the TCP stack, which acknowledges receipt back to the producer. However, due to the pipeline reload, Logstash does not read the event from the TCP socket, nor does it send any acknowledgment, resulting in the event being dropped.
Subsequent log events from the producer encounter errors such as broken pipe or connection reset by peer. If the producer implements a retry mechanism, it can resend those events. However, the initial event that was acknowledged by the TCP stack but not processed by Logstash is lost.
Request:
To mitigate this issue, we recommend implementing a mechanism in Logstash that temporarily stores incoming events during the termination of the TCP input receiver and resumes processing once the pipeline reload is complete and the receiver is active again.
Steps to reproduce:
Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
Provide logs (if relevant):