After system updates log ingest stopped

[alveare1]

Version

2.4.90

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8 to 16 depending on node

RAM

16 or 32gb

Storage for /

1tb

Storage for /nsm

8tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Ingest of events stopped after update and unable to find any issues as to why it stopped. Please let me know where would be a good place to look.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[rh-ops]

What is the output of the following command?

salt-call pillar.get logstash

[alveare1]

Everything is green. No issues

[rh-ops]

When you check your logstash log at /opt/so/log/logstash/logstash.log, are there any errors listed in the log related to memory? They will look like this:

java.lang.OutOfMemoryError: Cannot reserve 16777216 bytes of direct buffer memory (allocated: 512908265, limit: 524288000)

[alveare1]

Nothing, not even one line with error in it.

[rh-ops]

Check for OOM messages in rather dmesg or cat /var/log/messages
Check the elasticsearch logs cat /opt/so/log/elasticsearch/securityonion.log
And check disk space with df -h