Logstash keeps going missing after restart due to heap size

[jtrodriguez]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

64

RAM

256GB

Storage for /

223GB

Storage for /nsm

65691GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I have been running 2.4.60 for about a week now and my Logstash started to go missing repeatedly. I can restart it and it will come back for about a minute.

The log has a couple different FATAL errors.

"[FATAL][org.logstash.Logstash ] uncaught error (in thread Ruby-0-Thread-140: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.16.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:216)"

"[FATAL][org.logstash.Logstash ] uncaught error (in thread logstash-pipeline-flush)
java.lang.OutOfMemoryError: Java heap space"

I have read that I need to increase the heap size, but I am unsure of where I would do that.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

The setting for that would be in SOC > Administration > Configuration > logstash > settings > lsheap. Also, see https://docs.securityonion.net/en/2.4/logstash.html#lsheap

[jtrodriguez]

Thank you! It looks like that did it. It's been running without fail for 20 hours now.

Didn't think to check the GUI. I am still getting use to the new easy features. I was digging through the filesystem looking for the proper configuration file to modify.