bond0 interface dropping RX packets

[Meftian]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

64

Storage for /

200

Storage for /nsm

600

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I use SecurityOnion to sniff a network via a TAP-Port (fiber optic) with a average network-load of 100Mbps. I have a 10Gbps NIC in place, which is able to receive the traffic. Unfortunately, the bond0 interface has a very high package loss, altough the direct interfaces on the NIC do not have high packet loss.

ens3f0 and ens3f1 represent the sniffing interfaces, that I added with so-monitor-add to the bond0-interface.

ifconfig output of the affected interfaces; you can see the high amount of dropped RX packets on bond0. ens3f0 and ens3f1 have nearly no dropped RX packets:

[soadmin@aio-psx-sopoc01 current]$ ifconfig ens3f0
ens3f0: flags=2499<UP,BROADCAST,RUNNING,NOARP,PROMISC,SLAVE>  mtu 9000
        ether d4:f5:ef:ab:2a:a0  txqueuelen 1000  (Ethernet)
        RX packets 24506239  bytes 11541232688 (10.7 GiB)
        RX errors 0  dropped 146  overruns 0  frame 0
        TX packets 2220  bytes 133200 (130.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[soadmin@aio-psx-sopoc01 current]$ ifconfig ens3f1
ens3f1: flags=2499<UP,BROADCAST,RUNNING,NOARP,PROMISC,SLAVE>  mtu 9000
        ether d4:f5:ef:ab:2a:a8  txqueuelen 1000  (Ethernet)
        RX packets 38581942  bytes 38958284690 (36.2 GiB)
        RX errors 0  dropped 517  overruns 0  frame 0
        TX packets 2211  bytes 132660 (129.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[soadmin@aio-psx-sopoc01 current]$ ifconfig  bond0
bond0: flags=5443<UP,BROADCAST,RUNNING,PROMISC,MASTER,MULTICAST>  mtu 9000
        ether d4:f5:ef:ab:2a:a0  txqueuelen 1000  (Ethernet)
        RX packets 63162838  bytes 50545619581 (47.0 GiB)
        RX errors 0  dropped 62768381  overruns 0  frame 0
        TX packets 4435  bytes 266100 (259.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Thank you very much for any help and best regards.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

Does the dropped byte count stay the same or continually increase?

Have you tried rebooting?

What kind of NIC?

Have you tried a different NIC?

Have you tried running tcpdump on the physical interfaces to see if the traffic looks correct there?

Have you tried running tcpdump on bond0 to see if the traffic looks correct there?

Have you double-checked your fiber optics and TAP?

[Meftian]

Good Morning, thanks for the help, here are my answers:

Does the dropped byte count stay the same or continually increase?
It continually increases, as it should (the traffic flows correctly to the NIC)

Have you tried rebooting?
Yes, then the counter on ifconfig starts with 0 again

What kind of NIC?
Intel X710 Dual Port 10GbE SFP+

Have you tried a different NIC?
No, currently not, as the traffic on the physical interfaces have nearly no loss at all (but I'll try this)

Have you tried running tcpdump on the physical interfaces to see if the traffic looks correct there?
Yes, traffic looks correct on the two interfaces ens3f0 and ens3f1

Have you tried running tcpdump on bond0 to see if the traffic looks correct there?
Yes, traffic looks correct on bond0

Have you double-checked your fiber optics and TAP?
Yes

Thank you very much

[dougburks]

Yes, traffic looks correct on bond0

If the traffic looks correct on bond0, then does that mean that it's not actually dropping packets after all?

[Meftian]

No, thats not the case. As you can see in the original post or here, the optical physical interfaces (ens3f0/1) do not really drop packets, but bond0 does drop nearly everything (ens3f0/1 are added as sniffing interfaces to bond0):

ens3f0: flags=2499<UP,BROADCAST,RUNNING,NOARP,PROMISC,SLAVE>  mtu 9000
        ether d4:f5:ef:ab:2a:a0  txqueuelen 1000  (Ethernet)
        RX packets 5270947294  bytes 2700711903876 (2.4 TiB)
        RX errors 0  dropped 225329  overruns 0  frame 0
        TX packets 681373  bytes 40882380 (38.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
ens3f1: flags=2499<UP,BROADCAST,RUNNING,NOARP,PROMISC,SLAVE>  mtu 9000
        ether d4:f5:ef:ab:2a:a8  txqueuelen 1000  (Ethernet)
        RX packets 8312100902  bytes 8867567975518 (8.0 TiB)
        RX errors 0  dropped 2597  overruns 0  frame 0
        TX packets 681371  bytes 40882260 (38.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bond0: flags=5443<UP,BROADCAST,RUNNING,PROMISC,MASTER,MULTICAST>  mtu 9000
        ether d4:f5:ef:ab:2a:a0  txqueuelen 1000  (Ethernet)
        RX packets 13583048184  bytes 11568279874224 (10.5 TiB)
        RX errors 0  dropped 13526756124  overruns 0  frame 0
        TX packets 1362744  bytes 81764640 (77.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[dougburks]

If it's dropping nearly everything, then I wouldn't expect the traffic to look correct when running tcpdump.

Is it possible that bond0 is incorrectly reporting drops when it's not actually dropping?

Are you getting the NIDS alerts that you expect?

Are you getting the protocol metadata that you expect?

Are you getting the full packet capture that you expect?

You state above that you manually added the sniffing interfaces to the bond using so-monitor-add. Is there any reason why you didn't add them when you originally ran setup?

Have you tried doing a fresh installation and choosing the sniffing interfaces during setup to see if that makes a difference?