Set NIC Channels combined to 1 for monitor interfaces: another case

[Lesterol]

Version

2.4.170

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

128

RAM

256

Storage for /

3.5 TB

Storage for /nsm

24 TB

Network Traffic Collection

span port

Network Traffic Speeds

more than 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi guys!

I always follow your updates closely. I would like to address my case of the topic with changing NIC Channels combined to 1 for monitor interfaces. This actually works on most of our sensors. But there is one server where applying this setting has different consequences. In general, there is a server with two INTEL XXV710 for 25GbE SFP28 cards. The following settings are already installed on them:

> Pre-set maximums:
> RX:                     4096
> RX Mini:                n/a
> RX Jumbo:               n/a
> TX:                     4096
> TX push buff len:       n/a
> Current hardware settings:
> RX:                     4096
> RX Mini:                n/a
> RX Jumbo:               n/a
> TX:                     4096
> RX Buf Len:             n/a
> CQE Size:               n/a
> TX Push:                n/a
> RX Push:                n/a
> TX push buff len:       n/a
> TCP data split:         n/a

With a dynamic value of combined 119 (out of 128), we get the following drops:

tail -15 /nsm/zeek/logs/current/capture_loss.log
{"ts":1757313325.119854,"ts_delta":300.0005638599396,"peer":"worker-1-2","gaps":19727,"acks":448621,"percent_lost":4.397252915044102}
{"ts":1757313325.361318,"ts_delta":300.0000030994415,"peer":"worker-1-53","gaps":10733,"acks":429484,"percent_lost":2.4990453660671874}
{"ts":1757313325.393993,"ts_delta":300.0000638961792,"peer":"worker-1-40","gaps":13886,"acks":341377,"percent_lost":4.067643690113862}
{"ts":1757313325.365775,"ts_delta":300.0000321865082,"peer":"worker-1-17","gaps":118645,"acks":738022,"percent_lost":16.076079032874357}
{"ts":1757313325.939524,"ts_delta":300.0002489089966,"peer":"worker-1-60","gaps":27980,"acks":471238,"percent_lost":5.937551725455078}
{"ts":1757313325.835932,"ts_delta":300.00002002716064,"peer":"worker-1-63","gaps":13639,"acks":334121,"percent_lost":4.082054106147174}
{"ts":1757313325.246767,"ts_delta":300.00003004074097,"peer":"worker-1-15","gaps":62629,"acks":530825,"percent_lost":11.798426976875618}
{"ts":1757313325.84109,"ts_delta":300.0001058578491,"peer":"worker-1-29","gaps":159978,"acks":729552,"percent_lost":21.928251858674912}
{"ts":1757313325.45357,"ts_delta":300.0002329349518,"peer":"worker-1-37","gaps":273809,"acks":4205135,"percent_lost":6.511301064056208}
{"ts":1757313325.518303,"ts_delta":300.0000970363617,"peer":"worker-1-61","gaps":156875,"acks":1854364,"percent_lost":8.459773809241335}
{"ts":1757313325.800351,"ts_delta":300.0000009536743,"peer":"worker-1-3","gaps":497028,"acks":3414823,"percent_lost":14.555015003705902}
{"ts":1757313325.402581,"ts_delta":300.00013494491577,"peer":"worker-1-26","gaps":272281,"acks":664438,"percent_lost":40.97914327597157}
{"ts":1757313325.420019,"ts_delta":300.00024700164795,"peer":"worker-1-50","gaps":17600,"acks":411367,"percent_lost":4.278418054924192}
{"ts":1757313325.452492,"ts_delta":300.00006890296936,"peer":"worker-1-58","gaps":274641,"acks":867058,"percent_lost":31.675043653365748}
{"ts":1757313325.602161,"ts_delta":300.00000500679016,"peer":"worker-1-4","gaps":317401,"acks":1024251,"percent_lost":30.988595568859587}

However, after setting both maps to combined 1, we get the following:

tail -15 /nsm/zeek/logs/current/capture_loss.log
{"ts":1757315819.411712,"ts_delta":300.0000569820404,"peer":"worker-1-16","gaps":41066,"acks":127805,"percent_lost":32.13176323305035}
{"ts":1757315819.413525,"ts_delta":300.00002002716064,"peer":"worker-1-9","gaps":32641,"acks":97264,"percent_lost":33.55917914130614}
{"ts":1757315819.413547,"ts_delta":300.0000231266022,"peer":"worker-1-38","gaps":44166,"acks":135050,"percent_lost":32.703443169196596}
{"ts":1757315819.411798,"ts_delta":300.0000400543213,"peer":"worker-1-42","gaps":54131,"acks":186510,"percent_lost":29.023108680499703}
{"ts":1757315819.413992,"ts_delta":300.0007109642029,"peer":"worker-1-2","gaps":39440,"acks":111863,"percent_lost":35.25741308564941}
{"ts":1757315819.459885,"ts_delta":300.00024580955505,"peer":"worker-1-30","gaps":30505,"acks":95686,"percent_lost":31.88031686976151}
{"ts":1757315819.461737,"ts_delta":300.0000259876251,"peer":"worker-1-51","gaps":95409,"acks":181648,"percent_lost":52.524112569364924}
{"ts":1757315819.571443,"ts_delta":300.0000660419464,"peer":"worker-1-55","gaps":79992,"acks":155622,"percent_lost":51.401472799475656}
{"ts":1757315819.462066,"ts_delta":300.00004601478577,"peer":"worker-1-46","gaps":43341,"acks":115760,"percent_lost":37.44039391845197}
{"ts":1757315819.681099,"ts_delta":300.0003309249878,"peer":"worker-1-17","gaps":48926,"acks":119897,"percent_lost":40.806692410986095}
{"ts":1757315819.633135,"ts_delta":300.00054001808167,"peer":"worker-1-23","gaps":58541,"acks":122197,"percent_lost":47.90706809496142}
{"ts":1757315819.693018,"ts_delta":300.00015592575073,"peer":"worker-1-62","gaps":35862,"acks":110264,"percent_lost":32.523761155046074}
{"ts":1757315819.660327,"ts_delta":300.0000510215759,"peer":"worker-1-29","gaps":31107,"acks":99649,"percent_lost":31.216570161266045}
{"ts":1757315819.692666,"ts_delta":300.0000579357147,"peer":"worker-1-14","gaps":57419,"acks":120689,"percent_lost":47.57600112686326}
{"ts":1757315819.752717,"ts_delta":300.00060510635376,"peer":"worker-1-50","gaps":54625,"acks":123600,"percent_lost":44.194983818770226}

Moreover, before changing the value, the incoming traffic on monitoring was 10-25 Gb/s, and after the change it was 4-5 Gb/s.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

In the upcoming version 2.4.180, the combined setting will default to 1 but you'll be able to set the value to whatever works best for your deployment.

Our testing so far has indicated that combined 1 should be an improvement for the vast majority of deployments out there. For your one server to have that much of a change in performance definitely seems like an anomaly and I have to wonder if there is something else going on.

Is this a physical machine or VM?

Is it one physical CPU or multiple?

How many Zeek workers?

How many Suricata workers?

Any other sniffing processes on the box?

In addition to the capture loss statistics share above, what was the Zeek packet loss during this time?

It's also worth noting that Zeek capture loss can be indicative of upstream loss in your SPAN port so it may not be an issue with your Security Onion machine or its configuration.

[Lesterol]

So, in order:

1. Physical server
2. Multiple, 2 sockets
3. 63 workers
4. 63 workers
5. No, there is a clean installation of the Security Onion sensor
6. Zeek losses are currently about 5%
   Yes, there may be minor losses on the SPAN itself, but the difference when switching combined 1 and 119 (as it was before) is visible to the naked eye (about 10% capture loss with combined 119 and ~50% loss with combined 1). Moreover, Inbound Monitor Traf with combined 1 is cut to 4-5 Gbps, while with the default value, traffic comes without restrictions - 10-35 Gbps

[Lesterol]

After changing the bond0 mode to Active Backup with combined 64, the performance became much better

[Lesterol]

I noticed that when changing the mode to Active Backup on all sensors, the loss rates drop significantly.

[dougburks]

Multiple, 2 sockets

If you have multiple physical CPUs, have you seen the NUMA references in the docs?
https://docs.securityonion.net/en/2.4/zeek.html#performance

After changing the bond0 mode to Active Backup with combined 64, the performance became much better

After changing the bond0 mode to Active Backup, are you sure that you are still seeing traffic from both NICs at the same time?