Alerts stop working after a few days

[Jonny-db]

Hello,

We have been testing Security Onion for about a month now, however we keep running into the same issue with the alert system.

We are sending Syslogs from a Watchguard firewall and we are using the Windows Syslog integration in elastic to send Windows Security logs. We have created two playbooks, each for successful logins to the respective systems. This appears to work really well and we have used this as a baseline.

However, we keep encountering an issue where it appears that randomly, overnight or over the weekend, the alerts will stop appearing on the security onion page and we can see the logs that would have triggered an alert within the hunt tab. We originally though this to be a virtualisation issue and so have rebuilt the system on a separate server. This appeared to run a lot better but the issue has reared its head again. We are at a loss as to what is causing this as it works very well and then just stops even after we have not touched the system during this period.

We have found a similar issue previously raised, however it appears an unrelated fix seemed to sort the issue. (#5375)

Here is the required information:

Version: 2.4.60 (we also had this issue on 2.4.50).
We are running an on prem setup, having tried on a virtualised system and also bare metal. The host has an internet connection.
We have installed both versions from the official ISO.
We only have one node running.
The first virtualised setup had 8 cores, 32GB RAM, 500GB Storage
The second bare metal setup had 6 cores, 16GB RAM and 256GB Storage
We configured the node with the standard preset.
We have not noticed any issues with sniffing traffic, we can see all the data being correctly parsed.
All so-status nodes are running.
No errors were noted when running sudo so-status or sudo salt-call salt.highstate.
There are no errors within the grid.

Any information is greatly appreciated while we are still learning the system.

Thanks

[cm-ops]

What is the output of the following:

du -hd1 /nsm
so-elasticsearch-indices-list

[Jonny-db]

Hello,

The output for the commands are as follows;

Thanks

[cm-ops]

Looks like you have about 7 days of suricata events ingested. Are you able to see Alerts going back that long?

[Jonny-db]

So off the back of your message, we did a bit of a test and cleared out some of the logs via 'so-nsm-clear & so-elastic-clear' which seems to have fixed the issue.

I presume we are ingesting too many logs for the storage of the box it is on (256).

Thank you for your assistance!