Syslog results not showing up with event.dataset:syslog

[kwwv]

I have had success in the past sending syslog to Security Onion. Specifically, by:

1. Setting my remote service to send to port 514 on my Security Onion manager.
2. Using so-allow to update the firewall rules
3. Updating /opt/so/saltstack/default/salt/elasticsearch/files/ingest/syslog to include a custom grok.
4. Restarting elasticsearch

Fwiw, before even updating the pipeline I was still able to see unparsed syslog traffic in Kibana with the search "event.dataset:syslog". This isn't working for a new service I am trying to add.

I properly updated the remote service and I can see the traffic being sent to the manager in tcpdump. I updated so-allow to include the source ip and when I search in Kibana I don't see any IPTables drops for that ip. However, I don't see any syslog messages parsed by ES for this service.

How can I troubleshoot this?

[cm-ops]

Have you checked so-elasticsearch-pipeline-stats syslog to see if there is failures in the pipeline?

[kwwv]

Thanks @cm-ops
I see 0 results for everything from pipeline-stats. However, when I search for metadata.pipeline:syslog in kibana I see results.

so-elasticsearch-pipeline-stats syslog | head -n 60
{
  "count": 0,
  "time_in_millis": 0,
  "current": 0,
  "failed": 0,
  "processors": [
    {
      "dissect": {
        "type": "dissect",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "remove": {
        "type": "remove",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },
    {
      "grok": {
        "type": "grok",
        "stats": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        }
      }
    },...

[cm-ops]

Some other checks for the pipeline if you see the traffic making it to the interface (in addition to the Elasticsearch pipeline command).

Check to see if events are flowing through a Logstash pipeline:
so-logstash-pipeline-stats $pipeline (manager and/or search)

Check Logstash log on search node(s) to see if there is an issue with indexing:
tail -f /opt/so/log/logstash/logstash.log

Verify your local syslog config defaults match the one in /opt/so/saltstack/default/salt/elasticsearch/files/ingest/syslog (minus you customizations)

[kwwv]

Logstash stats are 0 and and logstash.log has no errors. I also verified my syslog configuration.

Edit: Removed winlogbeat reference as not to cross topics.

[kwwv]

I think I am making some progress. I noticed that the ingest pipeline section of Kibana has older pipelines that have not been removed. For example, we are on Filebeat-8.4.3 but we have multiple pipelines still from Filebeat-7.X. I removed older pipelines and added an on_failure component to all syslog pipelines left. For example:

  "description" : "syslog pipeline",
 "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }}"
      }
    }
  ]
,
  "processors" : [...

I am starting to see more syslog entries now.
Does anyone know why there is a zeek.syslog and syslog ingest pipeline? How does ES decide which one to apply? Can I remove the zeek.syslog one?

[kwwv]

Where is the configuration setting that tells security onion to apply syslog to incoming traffic?

[cm-ops]

https://github.com/Security-Onion-Solutions/securityonion/blob/master/salt/logstash/pipelines/config/so/1001_preprocess_syslogng.conf This will determine what gets the syslog tag.

[kwwv]

Thank you @cm-ops.

I am still sooo confused. When I run tcpdump I see 100+ syslog messages being sent to Security Onion per second. When I check event.module:syslog I see less than 5% of those messages processed. The text from the missing messages can't be found with a free text search and there are no log errors I can find.

I even reverted back to the original syslog ingest pipeline and added a "catch-all" to my grok processors with the thought it could be a pattern issue, but that did not catch any additional messages:

      "grok": {
        "field": "message",
          "patterns": [
            "^<%{INT:syslog.priority:int}>%{TIMESTAMP_ISO8601:syslog.timestamp} +%{IPORHOST:syslog.host} +%{PROG:syslog.program}(?:\\[%{POSINT:syslog.pid:int}\\])?: %{GREEDYDATA:real_message}$",
            "^<%{INT:syslog.priority}>%{DATA:syslog.timestamp} %{WORD:source.application}(\\[%{DATA:pid}\\])?: %{GREEDYDATA:real_message}$",
            "^<%{INT:syslog.priority}>%{GREEDYDATA:real_message}$",    <--- this should catch anything that begins with <123>
            "^%{SYSLOGTIMESTAMP:syslog.timestamp} %{SYSLOGHOST:syslog.host} %{SYSLOGPROG:syslog.program}: CEF:0\\|%{DATA:vendor}\\|%{DATA:product}\\|%{GREEDYDATA:message2}$"
          ],

Even just a port that could catch this data and place it unparsed into ElasticSearch would be better than silently losing it. Is there some way to get the raw syslog data indexed?

[kwwv]

This is looking like a firewall issue. I added my source IP (192.168.1.3) to the allow list for syslog and I can see the rule in iptables:

iptables -L -v -n --line-numbers | grep 514 | grep -v 1514
34       0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.17          tcp dpt:514
35       0     0 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.17          udp dpt:514
79       0     0 ACCEPT     udp  --  *      *       192.168.1.0/24       0.0.0.0/0            udp dpt:514
80       0     0 ACCEPT     udp  --  *      *       192.168.1.0/24       0.0.0.0/0            udp dpt:514
81       0     0 ACCEPT     tcp  --  *      *       192.168.1.0/24       0.0.0.0/0            tcp dpt:514
82       0     0 ACCEPT     udp  --  *      *       192.168.1.3          0.0.0.0/0            udp dpt:514
83       0     0 ACCEPT     udp  --  *      *       192.168.1.3          0.0.0.0/0            udp dpt:514
84       0     0 ACCEPT     tcp  --  *      *       192.168.1.3          0.0.0.0/0            tcp dpt:514

However, there are blocks in /var/log/kern.log on port 514 for 192.168.1.3:

Mar 22 16:59:03 khq-so kernel: [ 3299.205928] IPTables-dropped: IN=ens160 OUT= MAC=XXX SRC=192.168.1.3 DST=10.83.83.95 LEN=139 TOS=0x00 PREC=0x00 TTL=64 ID=963 PROTO=UDP SPT=514 DPT=514 LEN=119
Mar 22 16:59:32 khq-so kernel: [ 3328.294540] IPTables-dropped: IN=ens160 OUT= MAC=XXX SRC=192.168.1.3 DST=10.83.83.95 LEN=169 TOS=0x00 PREC=0x00 TTL=64 ID=34285 PROTO=UDP SPT=514 DPT=514 LEN=149

Anywhere I should check for this filter rule to be coming from?

[kwwv]

What are the servers sending syslog? Could they be sending in a format that is not getting recognized as syslog?

Bingo. This was it. I changed the format of the sending servers from syslog RFC 3164 to RFC 5424 and it started getting through. Maybe this server is incorrectly implementing RFC 3164?

As I said above, I added an on_failure component to /opt/so/saltstack/default/salt/elasticsearch/files/ingest/syslog which I can see working for syslog servers sending JSON. It seems likely that the syslog dropped traffic is happening upstream in filebeat or logstash. Is there a similar on_failure in those components or someway that I could get to the root cause?

[cm-ops]

I do see some format checks in the FB code https://github.com/elastic/beats/tree/main/filebeat/input/syslog but there appears to be both RFC 3164 and RFC 5424 in there, haven't look too closely at Elastic's code however,

[kwwv]

I still don't know why RFC 3164 is getting dropped. The only idea I have is that the traffic looks close enough to syslog that it comes in through filebeat but is off enough that it gets dropped before reaching the ingest pipeline. Do you have any ideas for how to prove or disprove this theory?

[cm-ops]

You could check the Filebeat stats and see if there is something in there - curl localhost:5066/stats | jq

[kwwv]

@cm-ops I think I have it isolated. If a single IP sends "too much" syslog traffic, SO silently discards it. Just by slowing down the amount of traffic sent from some of my devices the syslog pipeline is working. I still have no idea where the discard event is occurring and its maddening.