local ingest

[ktroberson]

So I have a local nfs-share mounted in my security onion build. This nfs-share contains logs from an existing zeek and suricata sensor. What's is the best way to ingest that data? I was thinking of using a logstash input and output configuration file. But I would like to explore my options for completing this task.

version is 2.4
logs are already in JSON format

[cm-ops]

How much data are you looking to ingest? Logstash is a way, but you would want to make sure they get to the appropriate ingest pipeline in SO.

[ktroberson]

No i understand. I was thinking of sending through logstash unparsed and let it hit elasticsearch so it can do the parsing. But I am not sure if this is the best route

[ktroberson]

So just to be clear, I have an existing suricata and zeek docker runnung on ubuntu platform that is sending its *.log and eve.json to a nfs share and I want to pull in that data into my onion. I m looking for the best course of action. FYSA I don't have an option of rebuilding that sensor.

[cm-ops]

You could also edit the zeek-logs and suricata-logs custom log integration in so-grid-nodes_general and add the location of those logs for the EA to pick up. That would probably be easier and it would make sure they get to the right ingest node parser.

[ktroberson]

ok, i did this and I still dont see the logs. I have verified that the elastic agent is working, i have looked at so-elastic-agent-inspect and everything looks good. I have verified the logs are in the right directory. Do you have any other suggestions?

[ktroberson]

Ok, i made some adjustments and its working now. But if logs are being sent by elasticagent to logstashed unparsed, wouldnt it make more since to just use logstash to grab the logs and send them unparsed to elasticsearch