No Suricata alerts when the disk is 90% full

[Lesterol]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

56

RAM

256

Storage for /

315 GB

Storage for /nsm

27 Tb

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi all.
I found some problem. When filling out the /nsm section on the sensor, I don't get any Suricata alerts. After clearing the pcap files in the /nsm/suripcap section, alerts appear again.
The configuration is as follows: from the very beginning, the Suricata engine was setted for pcap recording. Compression is set to maximum (16). The diskfreepercentage value defaults to 10%.
Latest entries in the log file /opt/so/suricata/suricata.log:

[100 - W#89-bond0] 2024-05-23 07:17:06 Info: log-pcap: Initializing PCAP ring buffer for /nsm/suripcap/%n/so-pcap.%t.
[100 - W#89-bond0] 2024-05-23 07:17:06 Notice: log-pcap: Ring buffer initialized with 700 files.
[101 - W#90-bond0] 2024-05-23 07:17:06 Info: log-pcap: Initializing PCAP ring buffer for /nsm/suripcap/%n/so-pcap.%t.
[101 - W#90-bond0] 2024-05-23 07:17:06 Notice: log-pcap: Ring buffer initialized with 678 files.
[11 - Suricata-Main] 2024-05-23 07:17:06 Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket'
[11 - Suricata-Main] 2024-05-23 07:17:13 Notice: threads: Threads created -> W: 90 FM: 1 FR: 1   Engine started.

The root partition is only 5% full, the condition of all containers is ok.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[capitangiaco]

You have to see here: sudo tail -f /opt/so/log/elasticsearch/securityonion.log
I think you are exceeding the disk watermark of elastic, you have to increase space and decrease the logs retention:

[Lesterol]

Hi. Thanks for the reply.
/opt/so/log/elasticsearch/security onion.log looks like this:

[2024-05-25T13:40:55,757][INFO ][org.elasticsearch.cluster.metadata.MetadataMappingService] [.ds-logs-zeek-so-2024.05.25-000032/ 1DJ9VPK_Q2usQF3zOOprJA] display update [_doc]
[2024-05-25T13:42:52,206][INFORMATION ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] the [analyst] role has been updated
[2024-05-25T13:42:52,294][INFORMATION ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] the [auditor] role has been updated
[2024-05-25T13:42:52,368][INFORMATION ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-analyst]
[2024-05-25T13:42:52,454][INFORMATION ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] the [limited auditor] role has been updated
[2024-05-25T13:46:56,183][INFORMATION ][org.elasticsearch.cluster.metadata.MetadataMappingService] [.ds-logs-zeek-so-2024.05.25-000032/ 1DJ9VPK_Q2usQF3zOOprJA] display update [_doc]
[2024-05-25T13:59:55,384][INFORMATION ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] the [analyst] role has been updated
[2024-05-25T13:59:55,470][INFORMATION ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] the [auditor] role has been updated
[2024-05-25T13:59:55.570][INFORMATION ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-analyst]
[2024-05-25T13:59:55.655][INFORMATION ][org.elasticsearch.xpack.security.action.role.Transport Management] updated role [limited auditor]

I think the point is different: I have a distributed architecture, and when I wrote about the 90% fullness /nsm of the disk partition, I meant that this happens on the sensor (forward). There are no elastic logs on the sensor, and the storage itself is only 10% full.

[Lesterol]

As soon as I clear the pcaps on the sensor, the alerts appear again. It seems that Suricata does not rotate, and at the same time there are no errors in the logs.
Probably because diskfreepercentage only works for Stenographer and cleans the /nsm/pcap/ folder, but not /nsm/suripcap/

[capitangiaco]

Out of curiosity, how much are:
the PCAP Retention value (on Grid ->Sensor node) ?
the Size in GB for total usage size of PCAP on disk ( Administration, Configuration, Suricata, Config, Pcap, Maxsize ?

[Lesterol]

@dougburks

[dougburks]

@Lesterol Please review the discussion guidelines at #1720 and specifically this:

Tagging Individuals
Please do not tag an individual in a discussion unless that individual has already volunteered to help you in that discussion.

Have you reviewed all of the documentation at https://docs.securityonion.net/en/2.4/suricata.html#pcap?

What is the output of the following?

sudo du --max-depth=1 -h /nsm

sudo du --max-depth=1 -h /nsm/suripcap/

df -h /nsm

[Lesterol]

I apologize for the tagging.
Yes, I read the documentation in the pcap subsection.
I completely cleared suripcap and now the disk is filling up again. If the problem persists, I will be able to answer your questions.
But from memory I remember that maxsize was set automatically to 19446 gb.
Also, at the time of the problem, I remember that the /suripcap folder was about 22TB. Anyway, I'll post later.

[Lesterol]

After completely cleaning /suripcap the rotation works, the problem is solved.