Custom Elasticsearch pipeline not processing any logs

[launchdaemon]

I have a standalone install of Security Onion 2.3.50 installed on Ubuntu 18.04.

I am trying to send syslogs from a firewall that is not supported by the default syslog parser, /opt/so/conf/elasticsearch/ingest/syslog. After a lot of reading and banging my head against the keyboard I have got to a point where I have:

1. Copied the default syslog parser to /opt/so/saltstack/local/salt/elasticsearch/files/ingest/test.
2. Written new grok patterns to match the events I'm most interested in from the logs I'm sending.
3. Restarted Elasticsearch with sudo so-elasticsearch-restart.
4. Confirmed that the new pipeline is detected now with sudo so-elasticsearch-pipelines-list.

However, sudo so-elasticsearch-pipeline-stats test returns that the pipeline has not processed any entries.

I had not really changed anything from the default syslog parser apart from the name and the grok patterns.

What I'm struggling to find any information on is how Elasticsearch knows which pipeline to use when it first receives a log? I understand you can direct input to another pipeline however I grep'd the parser folder for syslog and the only results were the syslog parser and the zeek.syslog parser.

[launchdaemon]

I have seen #3547 where the syslog parser is modified directly but had avoided doing that as I assumed that was not best practice? Also in that instance it was dedicated to only processing that kind of logs whereas this is not the case for me.

I can see logs are definitely being received as I can see them in Kibana when filtering by syslog, they are just unparsed.

[weslambert]

The initial pipeline for events is defined in either filebeat.yml, or in the Logstash output for the particular type of event.

[jchenturbo]

Hi.. i have the same issue.. ive been searching for any clues and stumbled upon this thread.. did you ever figure this out?

[kwwv]

+1

[InfosecGoon]

If you're having a similar issue, I would suggest starting a new post with the details (version of SO, deployment type, what sort of data you're ingesting, etc.)

[kwwv]

I did: #9930

I think I have read every issue and all documentation associated with custom ingest pipelines and there isn't a definitive answer to ops question. Most responses (if solved) tend to result in add a grok pattern to the syslog ingest.

Put simply, if I add a custom ingest pipeline ... how do I associate it with incoming traffic?