Parsing IIS logs from filebeat

[senatorhotchkiss]

So I got a little problem with my IIS logs not being parsed. Only the message field is generated containing the full text.
As I understand the processing is now done using elasticsearch and not logstash. I tried copying the old definition to the /opt/so/saltstack/local/salt/elasticsearch/files/ingest/ - after restarting elastic, the file is copied to /opt/so/conf/elasticsearch/ingest/

Unfortunately this does not generate any new fields.

Can somebody point me in the right direction?

filter {
if "IIS" in [tags] {
csv {
separator => " "
columns => ["event_data.date","event_data.time","event_data.s-ip","event_data.cs-method","event_data.cs-uri-stem","event_data.cs-uri-query","event_data.s-port","event_data.cs-username","source_ip","event_data.csu","event_data.csr","event_data.sc-status","event_data.sc-substatus","event_data.sc-win32-status","event_data.time-taken"]
}
}
}

[senatorhotchkiss]

I tried rewriting the entire thing using the dissect function as used here: https://github.com/Security-Onion-Solutions/securityonion/blob/master/salt/elasticsearch/files/ingest/filterlog

Unfortunately no dice.

I am not sure if I need to reference the new ingest file in some other global config?

[weslambert]

Can you described the rest of your pipeline file?

[senatorhotchkiss]

Hi Wes. Appreciate the reply. Actually I don't have more in my file. Reading your post, makes me figure, this is probably the issue.

I just need an example like a document (line) where message contains 1,2,3 tagged with TEST.
1,2,3 should then be split into values.

Then I can use that example to reflect my own data input.

I hope that makes sense.

[senatorhotchkiss]

On my 12th hour trying to grasp how a simple csv filebeat parsing can be done using elasticsearch ingest. An example would be highly appreciated.

Looking at the other pipelines/ingester files (syslog, filterlog, common etc.) does not really give me insight, on how the filtering and processesing is done.

Say you have a file being shipped from Windows using filebeat with this pattern:
1,2,3

How can I can split the message field like:
header1: 1
header2: 2
header3: 3

Thanks in advance

[dougburks]

Have you looked at the CSV ingest processor?
https://www.elastic.co/guide/en/elasticsearch/reference/master/csv-processor.html

[senatorhotchkiss]

Thanks Doug. I think I got the processor right:

{
"csv": {
"field": "message",
"target_fields": ["header1", "header2"]
}
}

It is probably something to do with required parameters in the file before or after above example. Not sure if it should be shipped to another processor?

I also need some way of filtering to what input this is applied. Like only beat.hostname or specific tags.

[senatorhotchkiss]

What am I doing wrong here? Does the custom filename have to be the same, as one of the original files? I got this impression in another post here: #2323

If yes, what file(s) would be correct for filebeat modifications.

[senatorhotchkiss]

The file I created using VI in "/opt/so/saltstack/local/salt/elasticsearch/files/ingest/" and observe copied successfully to the /opt/so/conf/elasticsearch/ingest/ directory.

{
"description" : "test",
"processors" : [
{ "csv": { "if": "agent.hostname == 'WINBEATHOST'", "field": "message", "target_fields": ["Header1", "Header2", "Header3"] } },
]
}

Still there is no new fields in my Kibana. :-(

In desperate need of a solution, to eliminate my legacy SO w. Logstash processing. Appreciate feedback.