/nsm/zeek/logs fills up almost entire disk on forward node

[ejgh-oe]

Version

2.4.150

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

64

RAM

128GB

Storage for /

500GB

Storage for /nsm

3.5T

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hi,

In order to investigate an event that happened around 12 hours ago I tried to pull the corresponding pcap. However instead of the pcap I got the following message

After trying to pull pcaps for some other alerts and they all ended up with the exact same error message I started digging a little deeper:

It turned out that on my capture node data in /nsm/zeek/logs fills up almost the entire space on the /nsm partition.

To be specific: /nsm is 3.5TB and /nsm/zeek/logs alone uses 3.2TB. Took a look at the data in /nsm/zeek/logs - it goes back to early 2024 which was the time the node was set up initially - so basically zeek kept all logs starting with day one :-(

So my question: How can I get zeek to delete these old logs automatically??

Thanks much in advance for any clue...

PS: changing ILM-settings for the zeek indices doesn't help since data in /nsm/zeek/logs is no elastic index data.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[robbiemarshall]

One option would be to adjust your PCAP retention settings to allocate more space for PCAP. You can do this by going into SOC, then Administration –> Configuration –> pcap –> config –> diskfreepercentage. More specifics can be found here: https://docs.securityonion.net/en/2.4/stenographer.html#disk-free-percentage

[ejgh-oe]

Hi,

One option would be to adjust your PCAP retention settings to allocate more space for PCAP. You can do this by going into SOC, then Administration –> Configuration –> pcap –> config –> diskfreepercentage. More specifics can be found here: https://docs.securityonion.net/en/2.4/stenographer.html#disk-free-percentage

Thanks for the hint. Just for my understanding: Disk usage of /nsm is currently

2318776692      zeek
649016676       pcap
247482960       strelka
79839648        pcapindex
6549632 docker-registry
2706240 repo
1422748 elastic-fleet
36060   suricata
8       pcaptmp
0       suripcap
0       pcapout
0       import

and disk total

/dev/mapper/nsm-nsm  3.5T  3.2T  397G  89% /nsm

So pcaps are around 600GB which, given a total disk space of 3,5TB is around 11% which is already higher than the default value of 10 I've not got in Administration –> Configuration –> pcap –> config –> diskfreepercentage. Wouldn't bumping diskfreepercentage to say 30% as you mentioned above just increase the allowed disk space to be commited to pcaps - without taking care of old zeeks logs that are filling up a majority of disk space (total disk 3.5TB, Zeek logs around 2.5TB)? Put in another way: Does increasing diskfreepercentage as you mentioned make sure zeek logs are deleted accordingly so I don't up with a disk 100% full?

BTW, I tried changing the log expire interval for zeek zeek > config > zeekctl > LogExpireInterval from the default 0 (unlimited) to 180day, but that didn't change a thing - still got logs that go from the beginning (i.e. initial node installation till today) :-(

[robbiemarshall]

There is a so-sensor-clean script that handles deleting the zeek logs based on the crit_disk_usage function, which is 90%. Another option would be to utilize BPFs to filter out zeek logs that are not relevant to your environment. More info here: https://docs.securityonion.net/en/2.4/bpf.html#bpf

In the mean time, you could manually delete those older zeek logs to free up some space.

[ejgh-oe]

Thanks alot - I think that's the way to go for me:

• Manually delete older zeek logs
• adapt BPF filters