Update Suricata configuration.

[vaschthestampede]

In the Sensor of my distributed system I have spikes of packets dropped by Suricata even if the CPU is very underutilized, it rarely goes beyond 40%, searching I found that it could be useful to increase the buffer-size in the Suricata configuration file (/opt/so/conf/suricata/suricata.yaml).
So I modified the file, bringing the value from 32768 to 2097152, and restarting the docker so-suricata:2.4.130.
But when I try to do it the change to the file is overwritten after a short time.

Can you tell me how to make this change permanently?
Can you tell me the correct way to change these settings?

[dougburks]

There are other things you should tune first before needing to change buffer size:
https://docs.securityonion.net/en/2.4/suricata.html#performance

[vaschthestampede]

I tried to update the values;
suricata –> config –> max-pending-packets from 5000 to 5000000
suricata –> config –> af-packet –> threads from 15 to 32

But the sensor broke, so-suricata missing.

[dougburks]

If you revert your changes and wait 15 minutes, then so-suricata should start running again.

I'd recommend following the tuning order as shown at https://docs.securityonion.net/en/2.4/suricata.html#performance:

1. First, tune the NIDS ruleset. If you're running the default Emerging Threats ruleset, then there are likely many thousands of rules which are not applicable to your environment. Disabling those unneeded rules can improve Suricata performance and decrease packet loss.
2. Next, determine if there is traffic that you can exclude altogether via BPF. Ignoring that unneeded traffic can also improve Suricata performance decrease packet loss.
3. If steps 1 and 2 don't help, then you can increase max-pending-packets but start with a smaller value like 50000 and see if that helps.
4. Since you said above that the CPU rarely goes beyond 40%, increasing af-packet threads may not help much. You can try increasing it some but you most likely don't need 32.

[vaschthestampede]

Thank you for your reply and for the time you are dedicating to me. 🥇

1. I'm trying, even following the documentation but I just can't find the Suricata tuning page.
2. I want to monitor all traffic, so no exclusions.
3. Ok, I'll leave it at 15.
   The traffic I'm monitoring has peaks of up to 7Gb but currently packet losses of up to 1.25Gb. To be honest, packet losses from Suricata don't seem to depend too much on the amount of traffic.

Following the suricata documentation you indicated I set:

• suricata > config > max-pending-packets to 65000
• suricata > config > af-packet > ring-size to 100000

But it doesn't seem to have changed too much.

[dougburks]

I'm trying, even following the documentation but I just can't find the Suricata tuning page.

What exactly do you mean?

Are you saying that you can't find the Suricata tuning page in the documentation? Here are the relevant pages:
https://docs.securityonion.net/en/2.4/nids.html#nids
https://docs.securityonion.net/en/2.4/detections.html#detections
https://docs.securityonion.net/en/2.4/suricata.html#suricata

Or are you saying that you can't find the Suricata tuning page in the software?
The documentation pages above should tell you where to go in the software. Depending on what you're trying to do, you will either go SOC Detections or SOC Config.

I want to monitor all traffic, so no exclusions.

Almost all networks have some traffic that can be excluded. A common example would be backup traffic.

The traffic I'm monitoring has peaks of up to 7Gb but currently packet losses of up to 1.25Gb. To be honest, packet losses from Suricata don't seem to depend too much on the amount of traffic.

How many physical CPUs are in your sensor?

What is the brand and model of the CPU?

How many cores and threads?

Is this a physical machine or virtual machine?

Are you collecting traffic from a tap or span port?

[vaschthestampede]

Or are you saying that you can't find the Suricata tuning page in the software?

I can't find the Suricata tuning page in Security Onion.

Almost all networks have some traffic that can be excluded. A common example would be backup traffic.

Sure, but I do this filter as a switch.

Are you collecting traffic from a tap or span port?

From a span port.

What is the brand and model of the CPU?
How many cores and threads?
Is this a physical machine or virtual machine?

Intel Xeon Gold 6226R
16 core 32 threads
Right now it's physics.
I did the first test with a VM but I had the same Suricata drop problem, so I switched to physics, but it didn't bring any improvements.

[dougburks]

I can't find the Suricata tuning page in Security Onion.

As I mentioned in my previous reply, the documentation pages above should tell you where to go in the software. Depending on what you're trying to do, you will either go to SOC Detections or SOC Config.

SOC Detections:
https://docs.securityonion.net/en/2.4/detections.html#detections

SOC Config:
https://docs.securityonion.net/en/2.4/suricata.html#configuration

From a span port.

Have you checked to see if your span port is dropping packets? Looking at your SOC Grid screenshot above, Zeek Loss is zero but Capture Loss is non-zero which usually means upstream packet loss in the tap or span port:
https://docs.securityonion.net/en/2.4/zeek.html#packet-loss-and-capture-loss

How many physical CPUs are in your sensor?

I asked this in my previous reply and I don't see an explicit answer but I'm assuming you have one physical CPU. If you have multiple physical CPUs, please make sure you see the NUMA note at https://docs.securityonion.net/en/2.4/suricata.html#performance.

If you indeed have just one physical CPU, then I would go back to my previous suggestion to tune the NIDS ruleset. By default, the Emerging Threats ruleset has over 40K enabled rules many of which do not apply to your network. Disabling unneeded rules should make Suricata more efficient.

You could start with disabling entire categories of unneeded rules as shown at:
https://docs.securityonion.net/en/2.4/nids.html#enabling-and-disabling-with-regex

Once you're down to the bare minimum categories, then you can disable individual rules in SOC Detections as shown at:
https://docs.securityonion.net/en/2.4/nids.html#managing-existing-nids-rules

[vaschthestampede]

Have you checked to see if your span port is dropping packets? Looking at your SOC Grid screenshot above, Zeek Loss is zero but Capture Loss is non-zero which usually means upstream packet loss in the tap or span port

It doesn't seem so because the traffic is still below the 10Gb that the port is capable of but I want to look for the command to check it from the switch.

How many physical CPUs are in your sensor?

Sorry, I missed this question.
Right now there is one CPU, soon I will put another one to increase the number of cores.

I understand that excluding both traffic and rules to check can help but I would understand this if there was excessive CPU usage.
In this case not only do I not go above 40% but rarely above 20%.

I then understand that the indications are 2CPUs for every 200Mbps for Suricata worker or Zeek worker.
https://docs.securityonion.net/en/2.4/hardware.html#:~:text=200Mbps%20per%20Suricata%20worker%20or%20Zeek%20worker
So for the traffic levels I have I would need 80 cores or (I hope) threads but the issue of very low utilization remains.
Why do I need more cores if the ones I have are not being used.

[dougburks]

It doesn't seem so because the traffic is still below the 10Gb that the port is capable of but I want to look for the command to check it from the switch.

It's still quite possible that the span port is dropping packets even if it's not reaching the 10Gb capacity of the port. For example, many switches will stop copying traffic to the span port temporarily if CPU usage gets too high.

Right now there is one CPU, soon I will put another one to increase the number of cores.

Adding a second physical CPU will actually make it more difficult to tune due to NUMA considerations (see my previous reply). I'd recommend sticking with a single physical CPU to keep it simple. One option is to use a single processor with more cores. Another option is to have two physical sensors each with 1 physical CPU and both boxes being fed by a flow-based load balancer.

By the way, if you're in the market for new sensor hardware, you might want to consider our SOS hardware appliances at https://securityonion.com/hardware as they are specifically designed to handle certain traffic loads.

I then understand that the indications are 2CPUs for every 200Mbps for Suricata worker or Zeek worker.
https://docs.securityonion.net/en/2.4/hardware.html#:~:text=200Mbps%20per%20Suricata%20worker%20or%20Zeek%20worker
So for the traffic levels I have I would need 80 cores or (I hope) threads

That's a conservative number and it varies based on the size of your ruleset. If you decrease the number of enabled rules in your ruleset, then you can handle more traffic with fewer workers.

Even if tuning the ruleset doesn't specifically help Suricata performance, you still need to tune your ruleset so that you're not overwhelming yourself or your team with a bunch of false positives and nuisance alerts.

So I'd start with tuning your ruleset and seeing if that decreases the packet loss any.

You can also check the Suricata log to see if there any additional clues there:
https://docs.securityonion.net/en/2.4/suricata.html#diagnostic-logging

[vaschthestampede]

You can also check the Suricata log to see if there any additional clues there:
https://docs.securityonion.net/en/2.4/suricata.html#diagnostic-logging

sudo docker logs so-suricata
i: suricata: This is Suricata version 7.0.9 RELEASE running in SYSTEM mode
W: af-packet: bond0: AF_PACKET block-size is not large enough for max fragmented IP packet size (65561)
i: threads: Threads created -> W: 5 FM: 1 FR: 1 Engine started.

I tried setting suricata > config > max-pending-packets to 65561 but it breaks.
The highest value I was able to set is 65534.
The problem is that whatever setting you change seems to only increase the droped packets.

With the help of the assistance of the switch manufacturer they managed to verify that there are no lost packets in the mirror.

Adding a second physical CPU will actually make it more difficult to tune due to NUMA considerations (see my previous reply). I'd recommend sticking with a single physical CPU to keep it simple. One option is to use a single processor with more cores. Another option is to have two physical sensors each with 1 physical CPU and both boxes being fed by a flow-based load balancer.

I am aware of this but it is the only way I can work to add cores.
Also now I'm doing a test with a consumer PC with more cores and a higher frequency and the problems remain.

Even if tuning the ruleset doesn't specifically help Suricata performance, you still need to tune your ruleset so that you're not overwhelming yourself or your team with a bunch of false positives and nuisance alerts.

I would agree with you if the system is overwhelming but it is not, CPU is consistently below 40% and most of the time below 20%.
By the way I still haven't figured out how to set this in security onion.

[vaschthestampede]

I have tried with various configurations and the result is always the same.
It does not matter the number of cores, frequency, dual or single socket, I always have the same trend of packets dropped by suricata.

I think the problem is elsewhere and in fact my network has a particularity that I don't think everyone has.
I have a dual core switch setup that are connected to each other in MLAG and both act as routers with a virtual IP using the VRRP protocol.
Of course I forward the traffic from both switches to the probe.
Is this type of configuration natively supported or does some setting need to be changed?
For example, I can think of the possibility that a packet is received by one switch and forwarded to the other and this could cause a mismatch if the probe does not recognize this possibility.

[vaschthestampede]

Some time has passed and I have done other tests, in particular the last one is very interesting to me.
Currently I have the sensor on a very performing machine (Intel Core i9-14900KF) and set to use surricata only on the performing cores while the other services in the more efficient cores.
I only get packet losses when there are heavy samba transfers!!!
A CPU goes to 100% and you experience packet loss.
The quantity doesn't matter much, I have streams even over 8Gbits without packet loss if there are no samba transfers.

So I have no other ideas on how to proceed...

[dougburks]

I only get packet losses when there are heavy samba transfers!!!

Have you tried using a BPF to filter out the samba traffic? Or narrow it down to a subset to try to pinpoint an IP address or subnet that may be problematic?
https://docs.securityonion.net/en/2.4/bpf.html

You previously mentioned your dual core switch setup. Is it possible that it is causing the samba traffic to be duplicated from the perspective of the sensor?

What is the standard MTU on your network?

What is the output of the following?

ifconfig bond0

ifconfig $(cat /sys/class/net/bond0/bonding/slaves)