Duplicate of logs produced for HTTP input when logstash pipeline is reloaded

[sasikiranvaddi]

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version): 9.2.4
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker): Built from Source
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes): Kubernetes

Plugins installed: Not Applicable

JVM (e.g. java -version): JDK 21

Description of the problem including expected versus actual behavior:
We have a deployment where log producer uses HTTP as input interface towards Logstash to send logs. At times when certificates are renewed w.r.t logstash pipeline we observe log producer receives response "Remote end closed connection without response" which log producer considers the log event is failed to process and it re-sends log event again once the logstash pipeline is reloaded causing duplicate logs.

In logstash when pipeline is reloaded it initially calls shutdown of old pipeline1 and then it calls stop_inputs2 where all the incoming connections are stopped and then it proceeds with wait_for_shutdown to process the in-flight events. In this specific case if due to race condition if log event is received from log producer and before logstash acknowledges back if input connections are disconnected then this issue occur. As per current logstash design the closing of inputs connection seems not graceful.

Steps to reproduce: Rarely reproducible

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Send logs from Log producer continuously 1log/second
2. Ensure logstash pipeline is reloaded frequently maybe for every 15minutes
3. Check if any duplicate events are observed
4. Note: The issue occurrence is very rare

Provide logs (if relevant):

[sasikiranvaddi]

Hi,

Is there any proposal already to fix this problem in logstash? If yes, could you please share references if any.