Ingested syslogs are missing

[ben-sec]

Hello!

I used to ingest syslogs into Elasticsearch from my firewall via grok pattern configurations in /opt/so/saltstack/local/salt/elasticsearch/files/ingest/syslog (on my manager node). I recently updated to SO 2.3.270 and today I can't find any current or past firewall data regarding this ingest at all in Elasticsearch anymore. I checked Logstash and Elasticsearch logs, but couldn't find any clues.

Did you change something regarding the syslog ingest and/or did you delete specific indices that store that data?

Cheers, Ben

[weslambert]

TMK, nothing should have changed.

Here are the release notes for 2.3.270.
https://docs.securityonion.net/en/2.3/release-notes.html

Is the firewall still allowing connections on port 514?
Are there any clues in the Curator log to show when the older data was deleted?

[ben-sec]

The firewall and also an EDR are still sending logs. The EDR logs get parsed and ingested as usual. Also the firewall messages, for which I don't have matching grok patterns, can be found in Elasticsearch. But I can't find any other (parsed) firewall logs.

All syslogs are going into the SO-syslog indices (all green in ES) and I'm parsing a name (beside other details) for the field "event.module" (member1 and member2 for the firewall logs). A filter on event.module: member1 or event.module: member2 shows no results anymore. I looked 90 days back. Nothing. How can that happen? Beside parsed logs I tried to find (possibly) unparsed firewall logs (e.g. an IP address that I have accessed and that should appear in the firewall logs) in ES in order to find the data somewhere. Nothing.

I find a lot of Curator log entries like 2023-10-12 06:34:09,608 INFO HEAD https://search_node_ip:9200/_alias/so-syslog-2023.01.04 [status:404 duration:0.020s] (for all kinds of indices). Beside that, I have no clue what to look for.

Again, I did only an update to 2.3.270. I didn't touch Elasticsearch, my grok patterns or our firewall. Can somebody solve this riddle?

Cheers, Ben

[ben-sec]

No more ideas what might has happened?

[weslambert]

What version of Security Onion were you running before 2.3.270?

[ben-sec]

2.3.260

[ben-sec]

Any news for me?

[ben-sec]

Just in case that somebody has the same problems, I finally solved them by doing the following:

1. I dumped my custom syslog ingest configuration (saved only the grok patterns for later ...)
2. I took the latest default syslog ingest configuration that comes with SO 2.3.270 (which looked slightly different that the one I used before ...)
3. I added all my grok patterns again
4. Parsed syslogs are comming in again!