How to enable mmap-locked for af_packet (suricata)?

[AlexG14]

Hi,
according to this https://docs.suricata.io/en/suricata-7.0.4/performance/high-performance-config.html
I want to lock mmap for af_packet.
I tried to add 'mmap-locked: yes' to the minion's /opt/so/saltstack/local/pillar/minions/server_sensor.sls but it seems to be ignored.

The problem is 'suricata packet loss'. Could be 10% for 600-800Mbps peaks, or 30-35% for 2Gbit+ peak...
I've tried to implement all tuning guides (ones referred in SO docs and suricata docs). Tried different setups (two NICs from different vendors\ RSS\ No RSS \different affinity settings..). It helps somehow but does not really solve the problem.

Thank you.

[reyesj2]

Can you share some more information about your deployment?

• What version of Security Onion are you using?
• What are the hardware specs of your sensor(s)

Would you be able to share your suricata stats.log? Found in /opt/so/log/suricata/stats.log

Does Zeek show any loss?

[AlexG14]

Hi Jorge,

Thank you for your help!

1. Latest available I believe (2.4.60).
2. HPE Proliant DL380 Gen8, 2 x Intel(R) Xeon(R) CPU E5-2698 (80 total threads, HT is enabled), 128GB RAM, Intel Corporation 82599ES NIC for capture (also tried with NetXtreme BCM5719).
3. Yes, Zeek has 0-3% loss

stats log is attached.

[TOoSmOotH]

A couple of things I would try are disabling your custom rules and seeing if there is a change. The other is identifying why you are seeing such drastic spikes in traffic and identifying the host or hosts responsible and possibly filtering that out. It is possible that a single flow is overloading a worker thread trying to inspect such a large flow if that is the case. For the size of your box it should easily do over 2Gbit/s of traffic.

[AlexG14]

"For the size of your box it should easily do over 2Gbit/s of traffic."
That's puzzling me. Why such a big box can not handle such a trivial load?

Ok. I'll disable my ruleset and will try to identify large flows (I've tried to do that before but had no success).
For a large flows bypass is enabled it the config:

stream:
memcap: 128mb
checksum-validation: 'yes'
inline: auto
reassembly:
memcap: 512mb
depth: 1mb
toserver-chunk-size: 2560
toclient-chunk-size: 2560
randomize-chunk-size: 'yes'
bypass: 'yes'

Appreciate all your help and suggestions. Do not get me wrong ... But what about locking mmap? :) Suricata docs is recommending to do this, a few threads on suricata forums with similar symptoms are resolved with this setting... What do you think?

[TOoSmOotH]

mmap lock is not straight forward since it requires some addition docker permissions and some other config changes as well that are not part of the suricata config. I would not consider 2Gbps as "high performance" and you should easily be able to do that with the stock config assuming you have enough cores assigned to suricata. How many cores do you have assigned to suricata?

[TOoSmOotH]

#12637

You will be able to enable this in 2.4.70. Keep in mind that enabling this feature can cause your sensor to lock up.

[AlexG14]

"How many cores do you have assigned to suricata?"
Started with the default 50% (39 cores). Tried a few different configurations.
Now it is: 16 cores for workers (it's a max RSS queue count for ixgbe driver), 6 for management. All of them are isolated with isolcpu.
From all these experiments I'd say there is little to no difference between 39 and 16 cores (threads).

"can cause your sensor to lock up"
Can you please explain a bit more? :) What should I expect exactly?

Thank you so much for your time and expertise! Appreciate this.

[AlexG14]

Just for context. Another one pic:

so.. the actual amount of traffic seems irrelevant for losses, what is looking relevant is spikes itself.