It seems like I need to reboot elastic everyday "so-elastic-restart --force"

[TotieBash]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

airgap

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

30

Storage for /

200

Storage for /nsm

200

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

It seems like I need to reboot elastic everyday "so-elastic-restart --force". Otherwise SOC dashboard and Kibana are empty for the last 8-16 hours query. If I query last 24 hours it spits out data. When I reboot the docker by "so-elastic-restart --force" all the data in the past 16 hours are there. Which tells me that they are being ingested by Elasticsearch but for some reason SOC dashboard and Kibana is not spitting it out.

I also have 2 other SO 2.4.40 doing the same thing. Am the only one?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Are there any clues in the SOC log? /opt/so/log/soc/sensoroni-server.log

[TotieBash]

@cm-ops I don't see OOM warning in logstash anymore but it is still choking on data. The so-redis-count is still backed-up. And all 3 of them started choking data at the same time 3 hours ago. I am using Suricata for metadata engine to extract network flows. I am not using Zeek for all 3 systems, in fact docker ps does not show zeek. Below is what is on the logstash tail logs multiple times.

[logstash.outputs.elasticsearch] Retrying failed action ...........pipeline=>"suricata.common"
[logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request

[TotieBash]

Past 6hour

Past 24hours

log output

[TotieBash]

I change logstash batch_size again from 500 to 2048, I'll check tomorrow if it stays up.

[TotieBash]

Mar 21 still the same isssue with batch_size 2048

[cm-ops]

What was the actual error in the above Logstash log? The screenshot cuts it out.

[TotieBash]

illegal_state_exception. Here's full screenshot.

[cm-ops]

I have my test server running, but have not run into this issue yet. Will continue to monitor. Do you have any Elasticsearch/Logstash customizations configured?

[TotieBash]

No customization, I do use the Cisco IOS integration plugins but other than that it is vanilla base install... And the monitor port is only monitoring around 10Kbps average.

[cm-ops]

I added the Cisco IOS integration and still cannot replicate. But I took a closer look at your Logstash log screenshot of the error. Would you be able to check the suricata.common pipeline and see if there are errors in { "set": { "field": "event.dataset", "value": "{{ message2.event_type }}" } }, that processor? You can check it with so-elasticsearch-pipeline-stats suricata.common.

I see the event_type in your log is ike, that set processor should be making the event.dataset = suricata.ike, but if you look towards the end it is just suricata. So it is failing on { "pipeline": { "if": "ctx?.event?.dataset != null", "name": "suricata.{{event.dataset}}" } } hence the pipeline failure for suricata..

Also, if you can get that piece of the log out of the eve.json, I can test with it. I have not been able to replicate the issue so far.

[TotieBash]

@cm-ops I didn't see any error when I ran so-elasticsearch-pipeline-stats suricata.common.
Where can I email the snippet of the sanitized eve.json?

The SO running Zeek metadata is up for 7 days.

The SO running Suri is down since last Thursday when I so-logstash-restart

The second SO running Suri is down as well since Thursday.

[cm-ops]

If the log is sanitized are you able to add it to this thread?

[TotieBash]

@cm-ops here is the sanitize log.
ike-suricata-eve-log-sanitized.txt

[cm-ops]

Thanks, I was able to replicate, suspect it is something malformed in the json and it is failing to parse out the json. Will talk with the devs about my findings.

[TotieBash]

ok thank you so much... In the mean time I am using Zeek as my work around.... I still desire SURI to handle metadata and I guess the future direction is for SURI to also handle full-packet-capture as well. Since STENO has been archived by Google.

[TotieBash]

Chris just some amplifying info. This is how I extracted the log:
zcat eve-2024-04-02*.json.gz | grep 1583301747739198 | jq

Then I sanitized the following:
src_ip
dest_ip
vendor_ids
Then all the obvious REDACTED values

[cm-ops]

I gathered that from the logs, I had to undo the jq to get them to work. When the log was proper json, I could see it in the console, when it wasn't, I saw the error for the wrong pipeline. Some intial thoughts were to add an ignore failure to the pipeline or an intermediary drop if it cannot parse the json. But I have not had time to test recently.