Stenographer not purging old data

[danielsamsCSG]

Good Morning all - Quick one for you. According to the Docs, Stenographer stores to /nsm/pcap and starts deleting old files when it hits 90% cap. I have a 2tb drive just for the /nsm/ partition for my enterprise distributed deployment and today Stenographer is giving an error with log info saying "Couldnt create temp directory: no space left on device". Looked at my /nsm/ partition and it is full with 1.8tb:100% used. Maybe I missed something in setup to get it to purge?

[weslambert]

This may not be just Stenographer, but could be other data.

You might try something like the following to get a better idea of what might be taking up space:

du -hs /nsm/* | sort -n

Other types of data under /nsm include Zeek logs, Elasticsearch data, Suricata eve.json.

Also, depending on how much traffic you are monitoring, you may be filling up the disk before you get back around to being able to purge it.

[sinnetll19]

how do you change the disk full percentage before it purges or the time to retain logs?

[weslambert]

For the so-sensor-clean script, it's handled by CRIT_DISK_USAGE, which defaults to 90%. You could copy that script from /opt/so/saltstack/default/salt/common/tools/sbin to /opt/so/saltstack/local/salt/common/tools/sbin and modify the value there, but you'd still want to be careful with what Steno's value is set to for it's own purging. We have an issue open to improve the overall purging and reduce conflict/contention b/w the various purging processes.

[sinnetll19]

thank you for the response on how to change the CRIT_DISK_USAGE. how do i currently change the steno value for purging?