PCAPs Not getting cleaned up filling /nsm

[pgatty]

Version

2.4.0

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

16GB

Storage for /

163GB

Storage for /nsm

327GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I've been running Security Onion 2.4 for several months now and everything has been fine until a few weeks ago when whatever job manages the PCAP space stopped cleaning up old files. Prior to this issue the free space on /nsm partition would stay at 60%. Now PCAPs will completely fill up the disk space necessitating a manual purge of PCAPs. I changed the diskfreepercentage from 10 to 25 under Administration –> Configuration –> pcap –> config –> diskfreepercentage and synced the grid, but the free disk space will drop all the way to 0%. I've looked through the stenographer log and don't see any errors. I've tried looking through some of the other logs in /opt/so/log but nothing stands out as a problem. Currently /nsm is at 88% used,

Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 7.6G 32K 7.6G 1% /dev/shm
tmpfs 3.1G 323M 2.8G 11% /run
/dev/mapper/system-root 163G 26G 137G 16% /
/dev/sda2 507M 368M 140M 73% /boot
/dev/sda1 1022M 6.2M 1016M 1% /boot/efi
/dev/mapper/system-tmp 2.0G 149M 1.8G 8% /tmp
/dev/mapper/system-nsm 327G 286G 41G 88% /nsm
tmpfs 1.6G 0 1.6G 0% /run/user/939
tmpfs 1.6G 0 1.6G 0% /run/user/0
tmpfs 1.6G 0 1.6G 0% /run/user/1000

Here's the output of sudo du -csh * from /nsm
7.2G backup
6.2G docker-registry
2.8G elastic-fleet
114G elasticsearch
0 import
1.1G influxdb
600K kratos
4.0K logstash
270M mysql
140G pcap
2.7G pcapindex
0 pcapout
0 pcaptmp
216M redis
8.6G repo
65M rules
0 soc
28M strelka
284K suricata
496M zeek
283G total

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[petiepooo]

What is the latest contents of /opt/so/logs/so-sensor-clean, and does its timestamp match the current date/time? Cron should run that every minute for zeek, strelka, and suricata logs, but pcap (stenographer) and elasticsearch (curator) should be cleaning themselves separately.

[pgatty]

I've checked the contents of /opt/so/logs/sensor_clean.log and see the jobs cleaning up strelka, and suricata files then following repeated continuously:

Tue Jan 16 02:19:16 AM UTC 2024 - No old Zeek logs available to clean up in /nsm/zeek/logs/
Tue Jan 16 02:19:16 AM UTC 2024 - No old files available to clean up in /nsm/strelka/processed
Tue Jan 16 02:19:16 AM UTC 2024 - No old files available to clean up in /nsm/suricata
Tue Jan 16 02:19:16 AM UTC 2024 - No old files available to clean up in /nsm/pcapout

The odd thing is I only see entries in the sensor_clean.log for a very narrow time slice on the 16th between 02:19:02 AM UTC and 02:52:53 AM UTC. The logs for the 14th and 15th are empty.

[petiepooo]

I believe that's normal, as the so-sensor-clean script is silent if the disk does not need cleaning. The time range you see in those output lines would be during the time it is trying to free up space, right? The output just shows there's nothing to delete among the files it manages, as the space is used by something else, like pcap.

I can't contribute further, sorry. Perhaps the devs can add more.

[pgatty]

@petiepooo - Thanks for taking the time to respond and trying to help.

So I ended up rebooting my Security Onion instance this morning to see if that would fix anything and within a few minutes after the reboot, something kicked in and removed 85GB of PCAP files. So the problem is clearly with PCAP clean up and it appears that something happens that causes the stenographer clean up job to fail. Looking through the stenographer log at the time of the reboot, I can see it clean up all the pcap index files in /nsm/pcapindex for the PCAPs I manually deleted, but I don't see any log entries for the deletion of the 85GB of PCAP files in /nsm/pcap.

I will continue to monitor the situation, If this happens again, is there anything I can check within stenographer to see if I can locate the problem or check the health of stenographer beyond running so-status?

[petiepooo]

See Security-Onion-Solutions/securityonion-image#509

[pgatty]

I tried getting into the so-steno container and it doesn't work. If I run docker exec -ti so-steno bash as my normal user I get a permission denied message. If I run with sudo it works, but the file /usr/local/bin/so-steno.sh doesn't exist. Apologies I'm a Docker newb. Thanks again for your efforts to help be debug this.

[petiepooo]

Yes, docker would need to be run as root unless your user is a member of the docker group.

My mistake; the file to edit is at /usr/local/sbin/so-steno.sh.

[petiepooo]

And you can use nano instead of vi if you prefer, of course.

[pgatty]

I managed to edit the file, but the killall command doesn't exist in the container. Should I just kill all the stenographer processes running as root and stenographer using their PID's i.e. kill -9 PID then relaunch stenographer with verbose logging via /usr/local/bin/so-steno.sh &

[petiepooo]

Yes, but not with the -9. Find the pid of runuser and run kill -STOP <pid>. Then kill stenographer with just kill <pid>.

FWIW, the -9 option for kill tells the kernel to remove the process from its process table. It's the "nuke from high orbit" option, and I don't recommend it except as a last ditch effort. It's more polite to send a normal -TERM (-15, the default) signal to the process first and "ask" it to clean up and terminate itself rather than yank the rug out from under it with a -KILL (-9)...

The -STOP (-19) to runuser, BTW, tells the kernel to stop scheduling the process, which is why it hangs around in the ps output. If it were to run again, say by sending -CONT (-18), it would see its child the (defunct) shell has exited and would exit itself, eventually stopping the entire container. We don't want that in this case as we want to manually restart the processes in the existing container.

I was testing in the 2.3.280 so-steno image. Killall is there but must not be in your newer 2.4 image... oh well. Thanks for sticking with the debugging effort.

[pgatty]

I thought I'd post an update to this. I was never able to figure out what was causing the problem and ended up reinstalling Security Onion (2.4.50) rather than have /nsm fill up every 3 - 4 days. It's been running for over a week now and PCAP cleanup is working as expected. Sometimes it's just easier to nuke the site from orbit ;). I wanted to thank @petiepooo for all his help.