Multiline received as last input will be dropped #57
Description
When sending data that includes a multiline at the very end, the multiline will be dropped by this codec.
For example, this affects every file being read with the stdin or file inputs (see below for mitigation when using file).
Example
stacktrace.log:
[2017-06-29] exception
stack trace line 1
stack trace line 2
logstash.javastacktrace.conf:
input {
stdin {
codec => multiline {
pattern => "^\s"
what => "previous"
}
}
}
filter {}
output { stdout { codec => rubydebug }}
Output:
% logstash-5.4.0/bin/logstash -f logstash.javastacktrace.conf < stacktrace.log
Sending Logstash's logs to /Users/cwurm/Products/Logstash/logstash-5.4.0/logs which is now configured via log4j2.properties
[2017-06-30T14:13:17,934][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-06-30T14:13:23,011][INFO ][logstash.pipeline ] Pipeline main started
[2017-06-30T14:13:23,116][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-06-30T14:13:26,041][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
Why?
The codec will buffer lines matching a multiline pattern, but not output them until it encounters a line not matching the multiline pattern.
While a flush method is implemented, it will never be called by Logstash: elastic/logstash#6523
Mitigation
Setting auto_flush_interval helps for inputs that don't exit (e.g. file, but not stdin).
However, it introduces a race condition - if the input or Logstash dies before the data is auto flushed it will still be lost.