Netflow log ingestion, filling up logs?

[LeoSysAdminThatCodes]

Im about to have netflow logs pointed at my SO instance. How do I know that SO is equipt to deal with excess logs (more logs then it has disk space for). Does it delete them as needed as new logs come in?

[InfosecGoon]

Assuming that you're using the Netflow module from Filebeat and a recent version of SO, those logs should be put into their own so-netflow-* index; by default, those indices will be closed after 30 days and deleted after 365. Those lifecycle settings can be changed by modifying /opt/so/saltstack/local/pillar/global.sls and adding an so-netflow configuration stanza under elasticsearch:index-settings that matches the syntax of the ones that are already there.

If you start to run low on disk space, the indices will be deleted oldest-first to keep total usage under logsizelimit, as described here:

https://docs.securityonion.net/en/2.3/elasticsearch.html#deleting-indices

This is assuming that you're using a standalone or a basic Elasticsearch installation with cross-cluster search, it's a little more complicated with a true cluster.

[LeoSysAdminThatCodes]

Thanks! What's it like for a distributed architecture?

…

On Mon, Jul 3, 2023 at 2:57 PM Matthew Gracie ***@***.***> wrote: Assuming that you're using the Netflow module from Filebeat and a recent version of SO, those logs should be put into their own so-netflow-* index; by default, those indices will be closed after 30 days and deleted after 365. Those lifecycle settings can be changed by modifying /opt/so/saltstack/local/pillar/global.sls and adding an so-netflow configuration stanza under elasticsearch:index-settings that matches the syntax of the ones that are already there. If you start to run low on disk space, the indices will be deleted oldest-first to keep total usage under logsizelimit, as described here: https://docs.securityonion.net/en/2.3/elasticsearch.html#deleting-indices This is assuming that you're using a standalone or a basic Elasticsearch installation with cross-cluster search, it's a little more complicated with a true cluster. — Reply to this email directly, view it on GitHub <#10688 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYNC3A62VEEH66U33RYIHLLXOMIXXANCNFSM6AAAAAAZ2HXCTU> . You are receiving this because you authored the thread.Message ID: <Security-Onion-Solutions/securityonion/repo-discussions/10688/comments/6346762 @github.com>

[InfosecGoon]

The same thing applies to a distributed architecture that's using cross-cluster search, which is the default configuration in 2.3. If you're using a true Elasticsearch cluster, you might need to do a little more tuning because the logsizelimit directive doesn't apply any more like it does in a basic install.