[Bug] Cannot run with `--xdp`: Packet sending exited with error

[duskmoon314]

Describe the bug

When adding --xdp, no packet is sent and Packet sending exited with error occurs

To Reproduce
Steps to reproduce the behavior:

1. Download 4.5.2-beta1, and build following the doc
2. `sudo tcpreplay --xdp -i eth1 some-pkts.pcap
3. See the error

Expected behavior
Tcpreplay works like without --xdp, sending packets.

Screenshots

System (please complete the following information):

• OS: Ubuntu 24.04 in vagrant-libvirt's vm (kvm qemu)
• OS version
  • 24.04.2 LTS (Noble Numbat)
  • 6.8.0-64-generic
• Tcpreplay Version: 4.5.2-beta1
• NIC: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

❯ sudo tcpreplay -V
tcpreplay version: 4.5.2-beta1 (build git:v4.5.2-beta1) (debug)
Copyright 2013-2024 by Fred Klassen <tcpreplay at appneta dot com> - AppNeta
Copyright 2000-2012 by Aaron Turner <aturner at synfin dot net>
The entire Tcpreplay Suite is licensed under the GPLv3
Cache file supported: 04
Not compiled with libdnet.
Compiled against libpcap: 1.10.4
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Packet editing: disabled
Fragroute engine: disabled
Injection method: PF_PACKET send()
Not compiled with netmap
Optional injection method: AF_XDP

Additional context
This seems to be a duplicated of #898, but with different NIC. But no error message seen in kern.log.

❯ tail /var/log/kern.log
2025-07-17T09:55:44.974345+00:00 vm-sec-4 kernel: ipmi_si: Unable to find any System Interface(s)
2025-07-17T09:55:45.062374+00:00 vm-sec-4 kernel: Bluetooth: Core ver 2.22
2025-07-17T09:55:45.062400+00:00 vm-sec-4 kernel: NET: Registered PF_BLUETOOTH protocol family
2025-07-17T09:55:45.062403+00:00 vm-sec-4 kernel: Bluetooth: HCI device and connection manager initialized
2025-07-17T09:55:45.062405+00:00 vm-sec-4 kernel: Bluetooth: HCI socket layer initialized
2025-07-17T09:55:45.062407+00:00 vm-sec-4 kernel: Bluetooth: L2CAP socket layer initialized
2025-07-17T09:55:45.062408+00:00 vm-sec-4 kernel: Bluetooth: SCO socket layer initialized
2025-07-17T09:55:45.720678+00:00 vm-sec-4 kernel: virtio_net virtio2 eth0: entered promiscuous mode
2025-07-17T10:57:42.115553+00:00 vm-sec-4 kernel: i40e 0000:03:00.0 eth1: entered promiscuous mode
2025-07-17T10:57:42.128777+00:00 vm-sec-4 kernel: i40e 0000:03:00.0 eth1: left promiscuous mode

I wonder where can I ger more debug info for this error.

[fklassen]

Thanks for the detailed report. I have had issues reproducing XDP errors, but I should be able to reproduce this with a bit more information.

• output ofuname -a.
• the kernel config file for the kernel version that is running (found in /boot/ directory
• run the test on a veth interface as described in [Bug] AF_XDP memory leak #935

I suspect the bug is in the virtio_net.ko driver. It is only used in KVM, and appears to be causing the issue.

[duskmoon314]

Here is the info:

❯ uname -a
Linux vm-sec-4 6.8.0-64-generic #67-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 15 20:23:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

The config is too long so I create a gits: https://gist.github.com/duskmoon314/39de931dab1833858c5d29f15aca2586

I also tried using XDP on the host machine (Debian 12), same error. I wonder whether it is an error of virtio_net.

Output of sending packets on veth:

❯ ./configure --enable-debug --enable-asan
❯ make -j
❯ sudo make install

❯ sudo ip link add type veth
❯ sudo ip link set dev veth0 up
❯ sudo ip link set dev veth1 up
❯ sudo tcpreplay --xdp --no-flow-stats -W -i veth0 ~/data/self-gen/carpet/syn-rand-20-t1-i5-n16-r10.pcap
Actual: 9941424 packets (636251136 bytes) sent in 107.70 seconds
Rated: 5907386.5 Bps, 47.25 Mbps, 92302.91 pps
Statistics for network device: veth0
        Successful packets:        9941424
        Failed packets:            0
        Truncated packets:         0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0

In addition, I can send packets with XDP on eth0 (The management virtual nic connecting vm and host)

[duskmoon314]

Here is the info when I tried XDP on the host machine:

$ uname -a
Linux pc4 6.1.0-37-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.140-1 (2025-05-22) x86_64 GNU/Linux

$ sudo tcpreplay --xdp --no-flow-stats -W -i veth0 ./syn-rand-20-t1-i5-n16-r10.pcap
Actual: 9941424 packets (636251136 bytes) sent in 95.29 seconds
Rated: 6676324.3 Bps, 53.41 Mbps, 104317.56 pps
Statistics for network device: veth0
        Successful packets:        9941424
        Failed packets:            0
        Truncated packets:         0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0

# sending on XL710
$ sudo tcpreplay --xdp --no-flow-stats -W -i ens5f0 -d 5 ./syn-rand-20-t1-i5-n16-r10.pcap
DEBUG5 in utils.c:our_safe_malloc() line 51: Malloc'd 112 bytes in sendpacket.c:create_umem_area() line 1482
DEBUG5 in utils.c:our_safe_malloc() line 51: Malloc'd 20 bytes in sendpacket.c:create_xsk_socket() line 1506
DEBUG5 in utils.c:our_safe_malloc() line 51: Malloc'd 344 bytes in sendpacket.c:xsk_configure_socket() line 1410
DEBUG5 in utils.c:our_safe_malloc() line 51: Malloc'd 1296 bytes in sendpacket.c:sendpacket_open_xsk() line 1451
DEBUG2 in send_packets.c:send_packets() line 419: packet 1 caplen 64
DEBUG4 in send_packets.c:send_packets() line 464: This packet time: 1735543952s 946901000nsec
DEBUG2 in send_packets.c:prepare_remaining_elements_of_batch() line 1347: Sending packets with LIBXDP in batch, packet numbers from 1 to 1

DEBUG2 in send_packets.c:send_packets() line 545: Sending packet #1
Packet sending exited with error!

I didn't see any error in journalctl -f's output.

[fklassen]

Thanks. I'll have a look at it and see if I can fix it for 4.5.2.

[duskmoon314]

I added some info to the error message:

// src/common/sendpacket.h
static inline void
kick_tx(struct xsk_socket_info *xsk)
{
    int ret = sendto(xsk_socket__fd(xsk->xsk), NULL, 0, MSG_DONTWAIT, NULL, 0);
    if (ret >= 0 || errno == ENOBUFS || errno == EAGAIN || errno == EBUSY || errno == ENETDOWN) {
        return;
    }
-    printf("%s\n", "Packet sending exited with error!");
+    printf("%s %d %s\n", "Packet sending exited with error!", ret, strerror(errno));
+    printf("xsk socket fd %d\n", xsk_socket__fd(xsk->xsk));
    exit (1);
}

Then I tried XDP on the host machine (Debian):

$ sudo tcpreplay --xdp --no-flow-stats -W -i ens5f0 ~/syn-rand-20-t1-i5-n16-r10.pcap
Packet sending exited with error! -1 Invalid argument
xsk socket fd 3

[fklassen]

Cool. Thanks for the information. I wonder why it works on some platforms.

…

On Thu, Jul 17, 2025 at 7:33 PM Campbell He ***@***.***> wrote: *duskmoon314* left a comment (appneta/tcpreplay#956) <#956 (comment)> I added some info to the error message: // src/common/sendpacket.h static inline void kick_tx(struct xsk_socket_info *xsk) { int ret = sendto(xsk_socket__fd(xsk->xsk), NULL, 0, MSG_DONTWAIT, NULL, 0); if (ret >= 0 || errno == ENOBUFS || errno == EAGAIN || errno == EBUSY || errno == ENETDOWN) { return; }- printf("%s\n", "Packet sending exited with error!");+ printf("%s %d %s\n", "Packet sending exited with error!", ret, strerror(errno));+ printf("xsk socket fd %d\n", xsk_socket__fd(xsk->xsk)); exit (1); } Then I tried XDP on the host machine (Debian): $ sudo tcpreplay --xdp --no-flow-stats -W -i ens5f0 ~/syn-rand-20-t1-i5-n16-r10.pcapPacket sending exited with error! -1 Invalid argumentxsk socket fd 3 — Reply to this email directly, view it on GitHub <#956 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOPWP2NB44TZOFWEV55FCD3JBMJFAVCNFSM6AAAAACBXRGNEOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAOBWGQ3DKMRQHE> . You are receiving this because you were assigned.Message ID: ***@***.***>

[fklassen]

I notice that with this recent commit in the virtio-net driver the xdp code has been touched. Maybe you need to upgrade to the latest kernel patches (I am using 6.1.146).

Mon Jul 7 04:14:54 2025 -0400 6aca3dad2145 virtio-net: ensure the received length does not exceed allocated size  [Bui Quang Minh]

In any case, it would be worthwhile to have Tcpreplay fail gracefully when the network driver bocks.

[fklassen]

@duskmoon314 please try out 4.5.2-beta2 for updated error messages. I also would be interested to know if the latest kernel will fix your issue.

[duskmoon314]

Sorry for responding late.

Here is the output using 4.5.2-beta2 in the VM:

❯ sudo tcpreplay --version
tcpreplay version: 4.5.2-beta2 (build git:v4.5.2-beta2) (debug)
Copyright 2013-2025 by Fred Klassen <tcpreplay at appneta dot com> - AppNeta
Copyright 2000-2012 by Aaron Turner <aturner at synfin dot net>
The entire Tcpreplay Suite is licensed under the GPLv3
Cache file supported: 04
Not compiled with libdnet.
Compiled against libpcap: 1.10.4
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Packet editing: disabled
Fragroute engine: disabled
Injection method: PF_PACKET send()
Not compiled with netmap
Optional injection method: AF_XDP

❯ sudo tcpreplay --xdp --no-flow-stats -W -i eth1 ~/data/self-gen/carpet/syn-rand-20-t1-i5-n16-r10.pcap
Send error: XDP is either not supported by this underlying network driver, or it has a bug.
Try upgrading to a newer kernel version and/or network driver and try again. (null)

Currently, I need this environment for some experiments, so I would prefer to test with a newer kernel several days later.

[fklassen]

Thanks. I have set up a similar system with kernel version 6.1.146, which includes the latest virtio_net fixes. I'll have time in the coming days to try it out, in case you don't get to it.

[Takaklas]

I am too facing issues with my intel x520 card, using --xdp flag.

Managed to get it working by loading the first demo of "xdp-tutorial" repo:

user@debian:~/Downloads/xdp-tutorial/basic01-xdp-pass$ sudo tcpreplay -i enp1s0f0 --xdp ~/Documents/ether0.pcap 
Send error: XDP is either not supported by this underlying network driver, or it has a bug.
Segmentation fault
user@debian:~/Downloads/xdp-tutorial/basic01-xdp-pass$ sudo xdp-loader load -m native enp1s0f0 xdp_pass_kern.o
user@debian:~/Downloads/xdp-tutorial/basic01-xdp-pass$ sudo xdp-loader status
CURRENT XDP PROGRAM STATUS:

Interface        Prio  Program name      Mode     ID   Tag               Chain actions
--------------------------------------------------------------------------------------
lo                     <No XDP program loaded!>
enp3s0                 <No XDP program loaded!>
enp1s0f0               xdp_dispatcher    native   456  63e8febd672ce2c6 
 =>              50     xdp_prog_simple           465  3b185187f1855c4c  XDP_PASS
enp1s0f1               <No XDP program loaded!>

user@debian:~/Downloads/xdp-tutorial/basic01-xdp-pass$ sudo tcpreplay -i enp1s0f0 --xdp ~/Documents/ether0.pcap 
Actual: 1 packets (60 bytes) sent in 0.000003 seconds
Rated: 20000000.0 Bps, 160.00 Mbps, 333333.33 pps
Statistics for network device: enp1s0f0
	Successful packets:        1
	Failed packets:            0
	Truncated packets:         0
	Retried packets (ENOBUFS): 0
	Retried packets (EAGAIN):  0
user@debian:~/Downloads/xdp-tutorial/basic01-xdp-pass$ 

the loading of the demo changes the following in dmesg:
[ 1.073364] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
to:
[12614.987448] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 4

Not sure about the performance though, since (ether0.pcap contains 1 packet of 60 bytes):
sudo tcpreplay -i enp1s0f0 -K -l 100000 ~/Documents/ether0.pcap: 390.40 Mbps
sudo tcpreplay -i enp1s0f0 -K -l 100000 --xdp ~/Documents/ether0.pcap: 46.87 Mbps