Data Lake

[pfendlek]

I am interested in guidance and/or recommendations for ingesting log data from a data lake using tools available through Security Onion. Ideally, my team will push all data to the data lake and then we will use Security Onion features for ingest, analysis, reporting, etc.

[reyesj2]

When you pull logs directly from the service using the Elastic Agent logs are automatically parsed, checkout the supported integrations here https://docs.securityonion.net/en/2.4/elastic-fleet.html#integrations

However, if pulling from a data lake you would need some custom ingest pipelines to parse the data for you correctly

[pfendlek]

Thank you for the feedback.

We did review the ingest/integration documentation provided by Security Onion, and we have not ruled that out.

However, we are interested in pushing all logs to a central location (data lake) for ingest, analysis, reporting, etc. using Security Onion.

If we understand correctly, we can setup lambda to perform preprocessing/reformatting of data, and then use the AWS services such as Glue (with Apache Spark), Step Functions, and Amazon Managed Workflows (with Apache Airflow) to push it to a database. We aren't sure (with Security Onion) which connector we need to use to push the data into its Elasticsearch.

Ideally, we would like to use OpenSearch, but we're not entirely sure if it's possible to connect Security Onion with OpenSearch.

Additional guidance is appreciated.

[pfendlek]

Related to this topic, we noticed upon navigating here: https://docs.securityonion.net/en/2.3/logstash.html#adding-new-logs, a section titled 'Adding New Logs'.

Since we are using Security Onion 2.4, we switched to this link: https://docs.securityonion.net/en/2.4/logstash.html#adding-new-logs, and it appears that the content is missing.

Guidance and recommendations for Security Onion 2.4 are appreciated.

[reyesj2]

2.4 logs are ingested via the elastic agent. You may be able to find an elastic agent integration that can pull logs from your upstream source. https://www.elastic.co/integrations/data-integrations
However, not all elastic integrations are currently supported. Support for more integrations get added almost every release as time allows

[pfendlek]

Thank you.

Related to this topic, we noticed upon navigating here: https://docs.securityonion.net/en/2.3/filebeat.html, a section titled 'Filebeat'.

Since we are using Security Onion 2.4, we switched to this link: https://docs.securityonion.net/en/2.4/filebeat.html, and it appears that the content is missing.

Recommendations for using Filebeat with Security Onion 2.4 are appreciated.