dnsdist xsk NIC_queue_id more than 16

[top-secrett]

Hello
It is my config and dnsdist works fine, but if I want to use more than 16 NIC_queue_id I get error
Server has 64 CPU

dnsdist 0.0.30552.0.master.g86a5d8a96a comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2
Fatal Lua error: [string "chunk"]:2: Caught exception: Error creating a xsk socket of if_name ens1f0: Unknown error -16
stack traceback:
        [C]: in function 'newXsk'
        [string "chunk"]:2: in function 'addListeners'
        [string "chunk"]:15: in main chunk

function addListeners(i)
  xsk = newXsk({ifName="ens1f0", NIC_queue_id=i-1, frameNums=65536, xskMapPath="/sys/fs/bpf/dnsdist/xskmap"})
  addLocal("IP:53", {xskSocket=xsk, reusePort=true})
end

function addServers()
  newServer({address="IP:53", name="ns1-node2", pool="current_dc", checkClass=DNSClass.CHAOS, checkName="version.bind", checkType="TXT", checkTimeout=2000, maxCheckFailures=3, checkInterval=3, mustResolve=true, useClientSubnet=true, rise=2,})
end

addCapabilitiesToRetain("CAP_SYS_ADMIN")
addCapabilitiesToRetain("CAP_NET_ADMIN")

for i = 1,16 do
    addListeners(i)
    addServers()
end

[rgacogne]

Would you be able to strace the dnsdist process during startup up to the error? I would like to know exactly which syscall fails. -16 would in theory be EBUSY (Device or resource busy) which is quite unexpected.

[top-secrett]

@rgacogne
The same error I got when I set only one NIC_queue_id with value more than 16. Maybe I need to rebuild binary with some specific attribute
like this

for i = 17,18 do
    addListeners(i)
    addServers()
end

bind(168, {sa_family=AF_XDP, sxdp_flags=XDP_USE_NEED_WAKEUP, sxdp_ifindex=if_nametoindex("ens1f0"), sxdp_queue_id=16}, 16) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x775b81600000
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/dnsdist/xskmap", bpf_fd=0, file_flags=0, path_fd=0}, 20) = 169
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=169, key=0x7ffd6f01796c, value=0x7ffd6f017978, flags=BPF_ANY}, 32) = -1 E2BIG (Argument list too long)
futex(0x77642d77e230, FUTEX_WAKE_PRIVATE, 2147483647) = 0
close(169)                              = 0
munmap(0x775b82000000, 2147483648)      = 0
getsockopt(168, SOL_XDP, XDP_MMAP_OFFSETS, "\0\0\0\0\0\0\0\0\200\0\0\0\0\0\0\0@\1\0\0\0\0\0\0\304\0\0\0\0\0\0\0"..., [128]) = 0
munmap(0x77609901a000, 65856)           = 0
munmap(0x776099009000, 65856)           = 0
getsockopt(168, SOL_XDP, XDP_MMAP_OFFSETS, "\0\0\0\0\0\0\0\0\200\0\0\0\0\0\0\0@\1\0\0\0\0\0\0\304\0\0\0\0\0\0\0"..., [128]) = 0
munmap(0x77609903c000, 65856)           = 0
munmap(0x77609902b000, 65856)           = 0
munmap(0x775b81600000, 8392704)         = 0
close(7)                                = 0
getpid()                                = 172594
sendto(3, "<27>Feb  4 15:02:08 dnsdist[1725"..., 281, MSG_NOSIGNAL, NULL, 0) = 281
write(1, "Fatal Lua error: [string \"chunk\""..., 209Fatal Lua error: [string "chunk"]:2: Caught exception: Error inserting into xsk_map '/sys/fs/bpf/dnsdist/xskmap': -7
stack traceback:
        [C]: in function 'newXsk'
        [string "chunk"]:2: in function 'addListeners'
) = 209
write(1, "\t[string \"chunk\"]:16: in main ch"..., 36   [string "chunk"]:16: in main chunk
) = 36
exit_group(1)                           = ?
+++ exited with 1 +++

[rgacogne]

bpf(BPF_MAP_UPDATE_ELEM, {map_fd=169, key=0x7ffd6f01796c, value=0x7ffd6f017978, flags=BPF_ANY}, 32) = -1 E2BIG (Argument list too long)

Thanks, this is very helpful! This means we are trying to insert an element out of bounds of the eBPF socket array. Ah, I see what's happening. The eBPF code in contrib/xdp-filter.ebpf.src has a hard-coded limit to 16 entries on line BPF_XSKMAP_PIN(xsk_map, 16, "/sys/fs/bpf/dnsdist/xskmap");. You should be able to change this to 64 and restart python xdp.py. I guess we need to make this configurable and perhaps sets a higher default value. I'll look into it.

[rgacogne]

Crap, these maps are persistent so just restarting won't work. Let me check.

[rgacogne]

It turns out that you can simply do rm /sys/fs/bpf/dnsdist/xskmap and restart.

[rgacogne]

#15120 will improve the situation significantly.

[top-secrett]

@rgacogne copied changed files. It works now with NIC_queue_id = 60

[rgacogne]

Great, thanks a lot for testing and reporting back, much appreciated!