Monitor Interface Drops

[cvfdblaze624]

I just installed a new distributed deployment on my network, all on hardware (Dell Servers running Ununtu 20.04 LTS with SO installed after), have a manager, search, honeypot and forward node all setup and working. The only issue with the setup seems to be on the forward node, on the Grafana interface it is showing a high rate of drops on the monitor interface, none for steno/suricata/zeek and I got a few alerts from Zeek saying it expected at least 1 TCP ACK and got zero so I assume it is indeed dropping some traffic. I am getting Zeek and Suricata alerts on what traffic is being sniffed and everything reports OK on all boxes from running SO-status. The NIC is an Intel on a Dell server, traffic is sent via a 1G Shark TAP. I did have a few issues when I did the install on the forward node, kept telling me the monitor interface was unmanaged by NetworkManager and I had to follow a guide to disable the Ubuntu networking and switch to using NetworkManager only which fixed the install issue and allowed it to proceed without error. Below is the Ethtool output for the NIC and I am not seeing any potential issues there either. Not sure what to do next trouble shooting wise. Thanks for any help!!

ethtool -k eno2
Features for eno2:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

ethtool -g eno2
Ring parameters for eno2:
Pre-set maximums:
RX:             2047
RX Mini:        0
RX Jumbo:       0
TX:             511
Current hardware settings:
RX:             200
RX Mini:        0
RX Jumbo:       0
TX:             511

[dougburks]

Do you use jumbo frames on your network?

[cvfdblaze624]

I do not have any devices configured to use jumbo frames, and I know my perimeter firewall is not configured to allow them. I just tried installing a new Intel NIC and adding it to the bond0 interface with so-monitor-add but it seems to have the same issue with drops. I used this same shark TAP with my old SO install and drops were rare if ever

[dougburks]

Have you tried a different cable?

Are you able to temporarily try a different tap or perhaps a span port just to see if it makes any difference?

[cvfdblaze624]

So I just replaced all the cables with fresh brand new, no difference in drops. I don't have the ability to set up a SPAN port on my switch but I'll try replacing the TAP next. It was working fine with my old setup which was a single node, that's why my gut says it is something on the forward node not behaving correctly or an error that occured during install

[dougburks]

How much traffic are you monitoring?

How much loss are you seeing? I'm having trouble reading your Grafana screenshot.

[cvfdblaze624]

Sorry the screenshot is from a phone but you can see the max over a 24 hour time frame for both traffic and loss. I actually downloaded some large files to see if something was getting buffered and dropping but I didn't notice a change in loss with larger bandwidth usage

[dougburks]

Please run the following on your sensor and let us know what the output is:

ifconfig bond0

[cvfdblaze624]

Here is the info for the monitor interface as well as the bond0 interface

ifconfig

bond0: flags=5443<UP,BROADCAST,RUNNING,PROMISC,MASTER,MULTICAST>  mtu 1500
        ether 68:05:ca:f9:a8:04  txqueuelen 1000  (Ethernet)
        RX packets 58146248  bytes 41382634486 (41.3 GB)
        RX errors 0  dropped 44212  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp2s0: flags=2499<UP,BROADCAST,RUNNING,NOARP,PROMISC,SLAVE>  mtu 1500
        ether XXXX  txqueuelen 1000  (Ethernet)
        RX packets 58146248  bytes 41382634486 (41.3 GB)
        RX errors 0  dropped 208  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18  memory 0xda7c0000-da7e0000

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

Slave Interface: enp2s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: XXX
Slave queue ID: 0

[dougburks]

The actual drops from ifconfig look fairly minimal. Perhaps Grafana is reporting the wrong numbers?

[cvfdblaze624]

I am also getting Zeek errors about too little traffic and too much loss which makes me think the numbers are real, Zeek is saying 10% loss in the alerts on Kibana, however on Grafana it shows 0% loss for Zeek, Suricata, and Sten which I am not sure why there is a discrepancy there. I switched out my TAP for a new SharkTAP and I didn't see any change in the packet loss. Any other thoughts on things to try? I rarely saw any loss on my old setup which was a single node, otherwise I am loving the performance boost of having a distributed setup now.

[dougburks]

Are you able to share the actual Zeek messages? Please note there is a difference between packet loss and capture loss:
https://docs.securityonion.net/en/2.3/zeek.html#packet-loss-and-capture-loss

[cvfdblaze624]

Here is an example of the alert that keeps getting generated, currently the % ranges from 10-39% with 1800 alerts in the last 24 hours

{"ts":1686231690.220971,"note":"CaptureLoss::Too_Much_Loss","msg":"The capture loss script detected an estimated loss rate above 11.036%","peer_descr":"worker-1-7","actions":["Notice::ACTION_LOG"],"email_dest":[],"suppress_for":3600.0}

[dougburks]

on Grafana it shows 0% loss for Zeek

The capture loss script detected an estimated loss rate above 11.036%

If Grafana shows 0% Zeek packet loss and at the same time Zeek is reporting capture loss of 11.036%, then that usually means that the capture loss is upstream in the tap, cabling, or there's some other network issue like asynchronous routing where maybe you're only seeing one side of the traffic for certain connections.

One of your Grafana screenshots above shows your max traffic at 2.5Mb/s. Is that what you expect or would you expect your network traffic to be higher than that? It seems rather small and I wouldn't expect any packet loss at that low volume.

How many Zeek workers are you running? If your traffic really is only 2.5Mb/s, then 1 Zeek worker should easily handle it. If you have more than 1, then you could try decreasing that value to see if it makes any difference.