SO stops logging when the nsm partition reaches 89%

[toushin-taishi]

Hello.

I'm new to securityonion so please bear with me if my problem is trivial.

My issue is securityonion stops logging when the nsm partition reaches 89%.

/dev/mapper/system-nsm 1.6T 1.4T 173G 89% /nsm

The way for securityonion to start logging again is by clearing nsm using the commands so-nsm-clear and so-elastic-clear.
I was under the impression that securityonion automatically manages disk space if it reaches 90% by deleting old data.
Is my understanding wrong?

How do I enable this disk management feature?

I am using securityonion version 2.3.260 in a standalone configuration.

I really appreciate any help you can provide.

[dougburks]

Yes, it should automatically delete old data once the partition reaches 90%.

What is the output of the following?

sudo du --max-depth=1 -h /nsm

Have you checked the logs in /opt/so/log/ to see if there are any clues as to why it's not automatically purging?

[toushin-taishi]

Hello. Thank you for getting back to me. Here is the output of the du command:

sudo du --max-depth=1 -h /nsm

12G /nsm/docker-registry
1.2G /nsm/repo
680K /nsm/kratos
623M /nsm/backup
514M /nsm/elasticsearch
1.2T /nsm/pcap
4.0K /nsm/pcaptmp
0 /nsm/pcapout
27G /nsm/pcapindex
1.1G /nsm/suricata
74G /nsm/zeek
0 /nsm/soc
276M /nsm/mysql
2.1G /nsm/influxdb
1.3M /nsm/grafana
24K /nsm/osquery
459M /nsm/wazuh
121G /nsm/strelka
0 /nsm/import
4.0K /nsm/logstash
1.4T /nsm

I don't know if this is the effect or the cause, but logstash is consuming a lot of CPU and is throwing OOM errors:

java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 1045894047, limit: 1048576000)

However, I still have some memory left unused, though I noticed that the swap space is all used up:

total used free shared buff/cache available
Mem: 65768648 35093748 12384332 662972 18290568 29475964
Swap: 8388604 8388384 220

[dougburks]

What are your hardware specs?

How much traffic are you monitoring?

[toushin-taishi]

My hardware specs are as follows:

CPU: 16 core Intel(R) Atom(TM) CPU C3958 @ 2.00GHz
RAM: 64G
HDD: 2TB SSD

For the traffic, around 300+ windows client using winlogbeat and sysmon and 1 pfsense firewall.

[dougburks]

CPU: 16 core Intel(R) Atom(TM) CPU C3958 @ 2.00GHz

Most folks avoid low power Intel Atom processors for CPU intensive tasks like this.

For the traffic, around 300+ windows client using winlogbeat and sysmon and 1 pfsense firewall.

I'm specifically interested in how much network traffic you're monitoring in terms of Gbps to see if you meet the minimum requirements here:
https://docs.securityonion.net/en/2.3/hardware.html#sensor-hardware-considerations

Full packet capture is using the vast majority of your disk space:

1.2T /nsm/pcap

So if you haven't already, I would at least start with tuning what gets written to disk:
https://docs.securityonion.net/en/2.3/bpf.html

If your hardware is underpowered, then you may need to disable steno altogether:
https://docs.securityonion.net/en/2.3/stenographer.html#disabling

[toushin-taishi]

I'm specifically interested in how much network traffic you're monitoring

in terms of Gbps to see if you meet the minimum requirements here: The average traffic is around 200Mbps. I am also curious why it is swapping out while there is 50% RAM left on the server. Also, considering that the hardware is underpowered, is it the cause of why data is not being purged when the nsm partition reaches 90%? Thank you for the pointers, I will check it out.

…

On Mon, Aug 21, 2023 at 9:21 PM Doug Burks ***@***.***> wrote: CPU: 16 core Intel(R) Atom(TM) CPU C3958 @ 2.00GHz Most folks avoid low power Intel Atom processors for CPU intensive tasks like this. For the traffic, around 300+ windows client using winlogbeat and sysmon and 1 pfsense firewall. I'm specifically interested in how much network traffic you're monitoring in terms of Gbps to see if you meet the minimum requirements here: https://docs.securityonion.net/en/2.3/hardware.html#sensor-hardware-considerations Full packet capture is using the vast majority of your disk space: 1.2T /nsm/pcap So if you haven't already, I would at least start with tuning what gets written to disk: https://docs.securityonion.net/en/2.3/bpf.html If your hardware is underpowered, then you may need to disable steno altogether: https://docs.securityonion.net/en/2.3/stenographer.html#disabling — Reply to this email directly, view it on GitHub <#11045 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIBHNI3QNZKQAJLUI6FDLV3XWNOELANCNFSM6AAAAAA3QOPPE4> . You are receiving this because you authored the thread.Message ID: <Security-Onion-Solutions/securityonion/repo-discussions/11045/comments/6780918 @github.com>

[dougburks]

I am also curious why it is swapping out while there is 50% RAM left on the server.

If you're confident that you have plenty of RAM and don't need swap, then you can disable swap altogether.

Also, considering that the hardware is underpowered, is it the cause of why
data is not being purged when the nsm partition reaches 90%?

It's possible that it's not able to purge fast enough to keep disk space under control. That's why I recommended filtering what's being written to disk or disabling steno altogether if necessary.