Skip to content

Pipeline crashes with undefined method `each' for nil:NilClass error in event filter method #191

@erhudy

Description

@erhudy

Logstash information:

  1. Logstash version: 7.17.3
  2. Installation source: Elastic container image
  3. Being run: via Docker
  4. Included in container image (Grok 4.4.1)
  5. JVM version: whatever is in the container image (11?)
  6. OS: RHEL 7.7

Description of the problem including expected versus actual behavior: Our Logstash pipelines sometimes crash with a particular error in the Grok plugin:

{
    "level": "ERROR",
    "loggerName": "logstash.javapipeline",
    "timeMillis": 1719522558635,
    "thread": "[beats-plain]>worker11",
    "logEvent": {
        "message": "Pipeline worker error, the pipeline will be stopped",
        "pipeline_id": "beats-plain",
        "error": "(NoMethodError) undefined method `each' for nil:NilClass",
        "exception": {
            "metaClass": {
                "metaClass": {
                    "exception": "Java::OrgJrubyExceptions::NoMethodError",
                    "backtrace": [
                        "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_grok_minus_4_dot_4_dot_1.lib.logstash.filters.grok.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.1/lib/logstash/filters/grok.rb:300)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)",
                        "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1821)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)",
                        "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134)",
                        "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:299)"
                    ],
                    "thread": "#<Thread:0x4cc678c3@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:53 sleep>"
                }
            }
        }
    }
}

I suspect that there is a race condition of some sort based on the fact that we can have Logstash instances go weeks or even longer without suffering from this problem and yet relatively low-traffic instances will exhibit the pipeline crash and require a Logstash restart (not that this seems directly tied to low traffic, it happens on both low- and high-traffic instances). When it does happen, we will get a couple of these tracebacks on different threads in Logstash and then the pipeline grinds to a halt and we get an alert that it is no longer passing events and we have to come and restart Logstash.

Steps to reproduce: Unfortunately there is no known reproducer for the bug, it just happens randomly.

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it:

If I had one I would put it here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions