DPDK does not work

[inss-lab]

Hello,

System Information:

• DU: Intel i7-14900 (see CPU file)
• RU: Benetel RAN550, 100 MHz, 4×2 MIMO
• NIC on DU: Intel E810 with GNSS
• Connection: DU NIC is connected directly to the RU without a network switch. The DU NIC is configured as the PTP grandmaster.

I am working with srsRAN and a Benetel RU to connect with COTS phones. Without DPDK, the phone can successfully connect to the gNB and access the Internet. However, the throughput is very low (about 1 Mbit/s) which I suspect is due to CPU processing limitations. This is why I wanted to try using DPDK.

I followed the instructions here: https://docs.srsran.com/projects/project/en/latest/tutorials/source/dpdk/source/index.html. The gNB starts successfully (see output below), but when using DPDK, the RU does not receive any data packets from the DU (monitored using kpi.sh). Phone cannot see the network. When I disable DPDK, kpi.sh on the RU kpi.sh shows normal RX and TX packet counts, and phone can access the network.

I have included detailed logs and configuration files below and in the attachments.

Any suggestions or guidance would be greatly appreciated.

--== srsRAN gNB (commit d90cd4e) ==--

EAL: Detected CPU lcores: 24
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:01:09.0 (socket -1)
iavf_execute_vf_cmd(): Cmd 26 not supported
iavf_set_hena(): Failed to execute command of OP_SET_RSS_HENA
iavf_default_rss_disable(): fail to disable default RSS,lack PF support
TELEMETRY: No legacy callbacks, legacy socket not created
iavf_dev_init_vlan(): Failed to update vlan offload
iavf_dev_configure(): configure VLAN failed: -95
iavf_set_rx_function(): request RXDID[1] in Queue[0] is legacy, set rx_pkt_burst as legacy for all queues
Initializing the Open Fronthaul Interface for sector#0: ul_compr=[BFP,9], dl_compr=[BFP,9], prach_compr=[BFP,9], prach_cp_enabled=true
Cell pci=1, bw=100 MHz, 4T2R, dl_arfcn=637212 (n78), dl_freq=3558.18 MHz, dl_ssb_arfcn=634464, ul_freq=3558.18 MHz

N2: Connection to AMF on 127.0.1.100:38412 completed
==== gNB started ===
Type to view help

/dpdk-23.11/usertools$ sudo ./dpdk-devbind.py -s

Network devices using DPDK-compatible driver
0000:01:09.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf

Network devices using kernel driver
0000:00:14.3 'Device 7af0' if=wlp0s20f3 drv=iwlwifi unused=vfio-pci Active
0000:00:1f.6 'Ethernet Connection (17) I219-LM 1a1c' if=enp0s31f6 drv=e1000e unused=vfio-pci
0000:01:00.0 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f0 drv=ice unused=vfio-pci
0000:01:00.1 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f1 drv=ice unused=vfio-pci Active
0000:01:00.2 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f2 drv=ice unused=vfio-pci
0000:01:00.3 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f3 drv=ice unused=vfio-pci

Linux inss-OptiPlex-Tower-Plus-7020 5.15.0-1033-realtime #36-Ubuntu SMP PREEMPT_RT Tue Feb 7 18:41:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

mount | grep huge
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=1024M)

Grub
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.15.0-1033-realtime"
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=lsb_release -i -s 2> /dev/null || echo Debian
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

GRUB_CMDLINE_LINUX="default_hugepagesz=1GB hugepagesz=1G hugepages=20 skew_tick=1 intel_iommu=on intel_iommu=pt quiet splash vt.handoff=7"

cpuinfo_meminfo.txt

gnb.log

[ninjab3s]

Hi @inss-lab!

I see two potential issues in your setup. First of all the CPU you are using has Intel's big little architecture. We have seen in many deployments that these CPUs are not suited for realtime applications like our gNB. We strongly advise to use a different platform, best one that supports the regular AVX512 instructions. You can in theory run a RAN with this CPU but dont expect to run stable and I also dont think you wont get much throughput with it.

Second I think the NIC's firmware and drivers could be updated if you havent done so yet. Follow the steps below to upgrade the drivers and firmware:

Ice driver update:

1. Download the latest driver here: https://www.intel.com/content/www/us/en/download/19630/intel-network-adapter-driver-for-e810-series-devices-under-linux.html
2. unpack and navigate to the directory ice-2.2.9/src
3. In the src directory run the following commands:

make -j$(nproc)
sudo make -j$(nproc) install

iavf driver update:

1. Download the latest version here: https://www.intel.com/content/www/us/en/download/18159/intel-network-adapter-linux-virtual-function-driver-for-intel-ethernet-controller-700-and-e810-series.html
2. unpack and navigate to the directory iavf-4.13.16/src
3. In the src directory run the following commands:

make -j$(nproc)
sudo make -j$(nproc) install

Firmware upgrade:

1. Download the latest firmware here: https://www.intel.com/content/www/us/en/download/19626/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapters-e810-series-linux.html
2. Unpack and navigate to the directory E810/Linux_x64
3. In that directory run the following command:

sudo ./nvmupdate64e

The installer will confirm that the correct NIC is installed and ask you if you want to proceed with the upgrade. During the upgrade its recommended to make a backup of the current NVME image (you will be prompted with a question). Agree to all. Once finished please send me the output of the FW upgrade to confirm everything worked as expected.

Once you have done all these steps please reboot the server. To confirm everything worked out please send me the outputs of the following commands when you get a chance! Thanks!

ethtool -i enp1s0f0np0
modinfo iavf
modinfo ice

[inss-lab]

Hi @ninjab3s Thank you so much for the detailed instructions. The problem has been resolved by updating the drivers. We appreciate your help!

[paulscalise1]

Hi @ninjab3s,

I have a similar issue to @inss-lab.
Without DPDK, my COTS UE can connect to my setup (srsRAN, Benetel RAN550, and free5GC), albeit with lots of network instability.

When i attempt to run the DPDK build of srsRAN, the gNB program seems to freeze at the same log stamp each time. Most of the time i can kill the process, and other times it will freeze the entire machine.

I have included as much configuration and log files as possible to help shed some context on this issue.

• gNB: Intel® Xeon(R) Gold 6254 CPU @ 3.10GHz × 36 (More info in cpuinfo_meminfo.txt file), 192 GB RAM
• RU: Benetel RAN550, 100 MHz, 4×2 MIMO, firmware version: RAN550-1v1.4.1-NM-25fa970, config: ru_config.txt
• NIC: Intel E810 [8086:1593] (port enp59s0f2 split (PF + VF) for ptp grandmaster and OFH)
• Kernel: 5.15.0-1084-intel-iot-realtime
• Network: The gNB is connected to a Ufispace M3000-14XC in pass-through mode.

I updated and validated my /etc/default/grub with the following:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=50 cgroup_memory=1 cgroup_enable=memory intel_pstate=disable mce=off idle=poll processor.max_cstate=1 intel_idle.max_cstate=0 nohz=on skew_tick=1 nosoftlockup tsc=nowatchdog kthread_cpus=0,1,25,27,29,31,33,35 irqaffinity=0,1,25,27,29,31,33,35 nohz_full=2-24,26,28,30,32,34 rcu_nocbs=2-24,26,28,30,32,34 isolcpus=managed_irq,domain,2-24,26,28,30,32,34"

Prior to running the gNB, I run the following script, mostly to create the PF + VF split:

RU_ETH=enp59s0f2

sudo modprobe vfio-pci

# Route to the Core
sudo route add -net 10.100.200.0 netmask 255.255.255.0 gw 192.168.0.177
sleep 1

# Create a VF on Fronthaul + PTP port
echo 1 | sudo tee /sys/class/net/$RU_ETH/device/sriov_numvfs
sleep 2

# Create a new mac address for the OFH port
sudo ip link set dev $RU_ETH vf 0 mac 50:7c:6f:00:00:00
sleep 2

sudo ip link set $RU_ETH vf 0 spoofchk off trust on
sleep 2

RU_FH_VF=${RU_ETH}v0

sudo ip link set mtu 9600 dev $RU_FH_VF
sleep 2
sudo ip link set up dev $RU_FH_VF
sleep 2

sudo ip link set dev $RU_ETH mtu 9600
sleep 2
sudo ip link set dev $RU_ETH up
sleep 2

# bind RU_FH_VF to DPDK
RU_FH_VF_BUS_ID=$(sudo ethtool -i enp59s0f2v0 | awk '/bus-info/ {print $2}')
sudo ip link set dev $RU_FH_VF down
sleep 1
cd /home/tel-5g-gnb/dpdk-23.11/usertools/
sudo ./dpdk-devbind.py -b vfio-pci $RU_FH_VF_BUS_ID

I then run ptp and phc:

sudo ptp4l -i enp59s0f2 -H -m -f configs/__new__G.8275.1.cfg -l 7 --tx_timestamp_timeout=50

sudo phc2sys -s enp59s0f2 -w -m -R 8 -f configs/__new__G.8275.1.cfg

PTP/PHC config found here:

 [global]
 dataset_comparison               G.8275.x
 G.8275.defaultDS.localPriority   128
 G.8275.portDS.localPriority      128
 maxStepsRemoved                  255
 logAnnounceInterval              -3
 logSyncInterval                  -4
 logMinDelayReqInterval           -4

 network_transport                L2
 delay_mechanism                  E2E
 ptp_dst_mac                      01:1B:19:00:00:00


 domainNumber                     24
 clockClass                       6
 slaveOnly                        0

Then I reboot the benetel radio, wait for the PTP to sync and for the radio to initialize.

Lastly I start the gNB using my gnb_ru_ran550_tdd_n78_100mhz_4x2.txt config.

The gNB program never gets past initial setup (amf query, etc), as it stalls at the same log each time, shown here:
srsRAN_terminal_log.txt
My gnb.log has no errors, but also lacks the normal setup procedure information.

Thank you to anyone in advance for any help.

In the mean time, I have updated all of my drivers as you recommended above:

ethtool -i enp59s0f2
driver: ice
version: 2.3.10
firmware-version: 4.90 0x80020ef5 1.3863.0
expansion-rom-version: 
bus-info: 0000:3b:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


modinfo iavf
filename:       /lib/modules/5.15.0-1084-intel-iot-realtime/updates/drivers/net/ethernet/intel/iavf/iavf.ko
version:        4.13.16
license:        GPL
description:    Intel(R) Ethernet Adaptive Virtual Function Driver
author:         Intel Corporation, <[email protected]>
alias:          i40evf
srcversion:     4BBBF2D49F97597CD7878BE
alias:          pci:v00008086d00000DD5sv*sd*bc*sc*i*
alias:          pci:v00008086d00001889sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CDsv*sd*bc*sc*i*
alias:          pci:v00008086d00001571sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Csv*sd*bc*sc*i*
depends:        
retpoline:      Y
name:           iavf
vermagic:       5.15.0-1084-intel-iot-realtime SMP preempt_rt mod_unload modversions 


modinfo ice
filename:       /lib/modules/5.15.0-1084-intel-iot-realtime/updates/drivers/net/ethernet/intel/ice/ice.ko
firmware:       updates/intel/ice/ddp/ice.pkg
version:        2.3.10
license:        GPL v2
description:    Intel(R) Ethernet Connection E800 Series Linux Driver
author:         Intel Corporation, <[email protected]>
srcversion:     03F0BAA2C99BA44A720DFF5
alias:          pci:v00008086d00001888sv*sd*bc*sc*i*
alias:          pci:v00008086d000012DEsv*sd*bc*sc*i*
alias:          pci:v00008086d000012DAsv*sd*bc*sc*i*
alias:          pci:v00008086d000012DDsv*sd*bc*sc*i*
alias:          pci:v00008086d000012D8sv*sd*bc*sc*i*
alias:          pci:v00008086d000012DCsv*sd*bc*sc*i*
alias:          pci:v00008086d000012D5sv*sd*bc*sc*i*
alias:          pci:v00008086d000012D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000012D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000012D1sv*sd*bc*sc*i*
alias:          pci:v00008086d0000579Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000579Esv*sd*bc*sc*i*
alias:          pci:v00008086d0000579Dsv*sd*bc*sc*i*
alias:          pci:v00008086d0000579Csv*sd*bc*sc*i*
alias:          pci:v00008086d0000151Dsv*sd*bc*sc*i*
alias:          pci:v00008086d0000124Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000124Esv*sd*bc*sc*i*
alias:          pci:v00008086d0000124Dsv*sd*bc*sc*i*
alias:          pci:v00008086d0000124Csv*sd*bc*sc*i*
alias:          pci:v00008086d0000189Asv*sd*bc*sc*i*
alias:          pci:v00008086d00001899sv*sd*bc*sc*i*
alias:          pci:v00008086d00001898sv*sd*bc*sc*i*
alias:          pci:v00008086d00001897sv*sd*bc*sc*i*
alias:          pci:v00008086d00001894sv*sd*bc*sc*i*
alias:          pci:v00008086d00001893sv*sd*bc*sc*i*
alias:          pci:v00008086d00001892sv*sd*bc*sc*i*
alias:          pci:v00008086d00001891sv*sd*bc*sc*i*
alias:          pci:v00008086d00001890sv*sd*bc*sc*i*
alias:          pci:v00008086d0000188Esv*sd*bc*sc*i*
alias:          pci:v00008086d0000188Dsv*sd*bc*sc*i*
alias:          pci:v00008086d0000188Csv*sd*bc*sc*i*
alias:          pci:v00008086d0000188Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000188Asv*sd*bc*sc*i*
alias:          pci:v00008086d0000159Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000159Asv*sd*bc*sc*i*
alias:          pci:v00008086d00001599sv*sd*bc*sc*i*
alias:          pci:v00008086d00001593sv*sd*bc*sc*i*
alias:          pci:v00008086d00001592sv*sd*bc*sc*i*
alias:          pci:v00008086d00001591sv*sd*bc*sc*i*
depends:        gnss,kvm,mdev
retpoline:      Y
name:           ice
vermagic:       5.15.0-1084-intel-iot-realtime SMP preempt_rt mod_unload modversions 
parm:           debug:netif level (0=none,...,16=all) (int)

Here is some more system info:

End of /etc/fstab:

nodev /mnt/huge hugetlbfs pagesize=1G 0 0

DPDK Binding:

sudo ./dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:3b:11.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:19:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=vfio-pci 
0000:19:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=vfio-pci 
0000:19:00.2 'I350 Gigabit Network Connection 1521' if=eno3 drv=igb unused=vfio-pci 
0000:19:00.3 'I350 Gigabit Network Connection 1521' if=eno4 drv=igb unused=vfio-pci *Active*
0000:3b:00.0 'Ethernet Controller E810-C for SFP 1593' if=enp59s0f0 drv=ice unused=vfio-pci 
0000:3b:00.1 'Ethernet Controller E810-C for SFP 1593' if=enp59s0f1 drv=ice unused=vfio-pci 
0000:3b:00.2 'Ethernet Controller E810-C for SFP 1593' if=enp59s0f2 drv=ice unused=vfio-pci *Active*
0000:3b:00.3 'Ethernet Controller E810-C for SFP 1593' if=enp59s0f3 drv=ice unused=vfio-pci *Active*