How does Suricata PCAP retention work?

[alan-lafleur]

Version

2.4.211

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

20 physical, 80 logical

RAM

132 Gig

Storage for /

293G

Storage for /nsm

1.5T

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I'm trying to figure out how Suricata pcap retentions works (stenographer is disabled). I have 2 search nodes configured the same but they do collect different amounts of traffic on each. secionionfwd1 is holding steady at 90% NSM partition usage and secionionfwd 2 is at 71%. Config -> Suricata -> pcap -> maxsize is at 1164G for both, but that does not appear to be what is keeping the NSM volumes in-check. If I do a du -h --max-depth=1 from fwd1 /nsm I see the suripcap has 806G, which is less than 1164, so it seems some other mechanism is helping to keep the fwd1 /nsm to stay at 90%.
Can anyone provide some guidance on how it works?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[alan-lafleur]

Update: If I reduce maxsize the overall /nsm volume does drop so that does appear to be the correct mechanism, however, I'm still confused about the numbers. Perhaps the du -h --max-depth=1 command is missing some files that maxsize is including in the calculation?

[cm-ops]

There is also a script that runs on a cron - so-sensor-clean. That script will remove Zeek, Suricata, and Strelka logs on the sensor in /nsm when the disk reaches 90% utilization.

[alan-lafleur]

Thanks cm-ops!

[TOoSmOotH]

Suricata has its own cleanup mechanism based on the maxsize setting but it then does some math. The code looks like this:

{%   set maxfiles = (SURICATAMERGED.pcap.maxsize * 1000 / (SURICATAMERGED.pcap.filesize[:-2] | int) / SURICATAMERGED.config['af-packet'].threads | int) | round | int %}
{%   do SURICATAMERGED.config.outputs['pcap-log'].update({'max-files': maxfiles}) %}

Suricata cleans up based on file counts and not size. "max-files" is the number of pcap files per thread to keep. So if you have it set to 100GB in the pillar, pcap.filesize set to 5GB and then threads set to 5, your maxfiles would be set to 4. 5 threads x 4 files that are 5GB is 100GB.

It's not exact and can be tweaked. If you have a sensor that sees a lot of traffic then this number is pretty accurate. If you have a sensor that doesn't see a lot of traffic this could be less accurate since it might not fill the pcap file before roll over. It;s cleanup mechanism is just counting files and doesn't care about their sizes which can cause this to happen.

[alan-lafleur]

Thanks for the detailed info, that helps a lot.