How does elasticsearch ingest work in SO

[stijnpieters]

I've been struggling to create custom parsing for my firewall logs (non-pfsense), and I think I've found out how an ingest pipeline should be made.

However, what I don't yet understand is how elastic knows which ingest parser to use. All of my firewall logs currently come into the syslog event.module, and I don't know what I should change for them to actually get into my own event.module "sophos_utm".

Do I need to change any other config except for dropping the ingest parser in /opt/so/saltstack/local/salt/elasticsearch/files/ingest/, or is there still something wrong with my parser?

parser for reference:
sophos_utm_ingest.txt
example log message for reference:
sophos_log_redacted.txt

EDIT:
I might have found out that my grok filter is complete garbage, but I am still wondering how elastic knows which parsers to use.

[senatorhotchkiss]

I second this. It would really be nice to know how input is hitting ingest files and basic examples of processing. I actually had to redesign all my logging, so logfiles are written to windows eventlog and shipped using winlogbeat. I could not figure out, how to get a basic file beat working.

[weslambert]

With regard to flow, I'll see if I can put something together soon. I'll also have a look @stijnpieters when I get a chance to see if I can provide some greater insight.

[senatorhotchkiss]

Thanks Wes, that would be awesome.

[kwwv]

+1
@weslambert - Did you add this? I see https://docs.securityonion.net/en/2.3/ingest.html but that doesn't explain how an ingest is selected.