DPDK is needed for both comparison testing against XDP and for having a faster packet generator.
Tariq reports that using DPDK based T-rex, he can send with 125Mpps (multi-TXq + CPU cores).
Fedora comes with DPDK packaged:
dnf install dpdk dpdk-tools dnf install dpdk-devel dpdk-doc dnf install python-pyelftools
First run had issues with dependencies:
$ sudo dpdk-pmdinfo Traceback (most recent call last): File "/bin/dpdk-pmdinfo", line 14, in <module> from elftools.common.exceptions import ELFError ImportError: No module named elftools.common.exceptions
Missing deps on python-pyelftools :
sudo dnf install python-pyelftools
The advantage with T-rex is that is contains its own copy of DPDK. Thus, you don’t need to install DPDK (in the correct version) on your system.
It’s possible to download binary version of trex, and install that from a tar.gz file, see instructions here:
https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_download_and_installation
Generate config:
sudo ./dpdk_setup_ports.py -c 09:00.0 09:00.1 -o trex_cfg02.yaml sudo ./dpdk_setup_ports.py -i
It turns out that out-of-tree kernel drivers might not be needed. It seems that Mellanox uses the infiniband drivers and userspace libraries for their DPDK driver.
Load kernel modules ib_uverbs and mlx5_ib:
sudo modprobe ib_uverbs sudo modprobe mlx5_ib
To get all the different userspace tools and library dependencies right, the easiest/only option seems to be via Mellanox OpenFabrics Enterprise Distribution (MLNX_OFED).
Get MLNX_OFED here: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
When installing it is possible to skip the kernel modules via parameter –user-space-only.
Installing:
sudo ./mlnxofedinstall --with-mft --with-mstflint --dpdk --upstream-libs --without-dkms --user-space-only
Download for Fedora 27: http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-4.3-1.0.1.0&mname=MLNX_OFED_LINUX-4.3-1.0.1.0-fc27-x86_64.tgz
The recent F27 kernel is too new.
$ sudo ./mlnxofedinstall Logs dir: /tmp/MLNX_OFED_LINUX.1498.logs General log file: /tmp/MLNX_OFED_LINUX.1498.logs/general.log The 4.16.13-200.fc27.x86_64 kernel is installed, MLNX_OFED_LINUX does not have drivers available for this kernel. You can run mlnx_add_kernel_support.sh in order to to generate an MLNX_OFED_LINUX package with drivers for this kernel. Or, you can provide '--add-kernel-support' flag to generate an MLNX_OFED_LINUX package and automatically start the installation.
Not much success with ./mlnx_add_kernel_support.sh script. Note that create_mlnx_ofed_installers.pl IS located in same directory.
$ sudo ./mlnx_add_kernel_support.sh --kernel `uname -r` WARNING: create_mlnx_ofed_installers.pl is missing, cannot build a repository. ERROR: Path to MLNX_OFED/mlnx-en directory is not defined.
Install RPM tool dependencies:
sudo dnf install createrepo sudo dnf install python2-devel redhat-rpm-config rpm-build sudo dnf install tk
Try using mlnxofedinstall –add-kernel-support option:
$ sudo ./mlnxofedinstall --add-kernel-support Note: This program will create MLNX_OFED_LINUX TGZ for fc27 under /tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64 directory. See log file /tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64/mlnx_ofed_iso.5306.log Checking if all needed packages are installed... Building MLNX_OFED_LINUX RPMS . Please wait... ERROR: Failed executing "MLNX_OFED_SRC-4.3-1.0.1.0/install.pl --tmpdir /tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64 --kernel-only --kernel 4.16.13-200.fc27.x86_64 --kernel-sources /lib/modules/4.16.13-200.fc27.x86_64/build --builddir /tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64/mlnx_iso.5306 --disable-kmp --build-only --distro fc27" ERROR: See /tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64/mlnx_ofed_iso.5306.log Failed to build MLNX_OFED_LINUX for 4.16.13-200.fc27.x86_64
Digging throug the RPM build errors, this compile error is the cause:
CC [M] /tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64/mlnx_iso.5306/OFED_topdir/BUILD/mlnx-ofa_kernel-4.3/obj/default/drivers/net/ethernet/mellanox/mlx5/core/ipo
ib/ipoib_vlan.o
/tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64/mlnx_iso.5306/OFED_topdir/BUILD/mlnx-ofa_kernel-4.3/obj/default/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c: In f
unction 'parse_tc_fdb_actions':
/tmp/MLNX_OFED_LINUX-4.3-1.0.1.0-4.16.13-200.fc27.x86_64/mlnx_iso.5306/OFED_topdir/BUILD/mlnx-ofa_kernel-4.3/obj/default/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c:2173:
18: error: implicit declaration of function 'tcf_mirred_ifindex'; did you mean 'tcf_mirred_dev'? [-Werror=implicit-function-declaration]
int ifindex = tcf_mirred_ifindex(a);
^~~~~~~~~~~~~~~~~~
tcf_mirred_dev
Install package ‘tk’ due to dependency (weird).
Try with Toke’s command:
$ sudo ./mlnxofedinstall --with-mft --with-mstflint --dpdk --upstream-libs --without-dkms --user-space-only Logs dir: /tmp/MLNX_OFED_LINUX.11926.logs/general.log General log file: /tmp/MLNX_OFED_LINUX.10238.logs/general.log Unsupported package: dkms This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.
It auto updates firmware:
Device Type: ConnectX5
Part Number: MCX556A-EDA_Ax
Description: ConnectX-5 Ex VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe4.0 x16; tall bracket; ROHS R6
PSID: MT_0000000009
PCI Device Name: 01:00.0
Base MAC: ec0d9adb11bc
Versions: Current Available
FW 16.20.1010 16.22.1002
PXE 3.5.0210 3.5.0403
UEFI N/A 14.15.0019
[...]
Device Type: ConnectX3Pro
Part Number: MCX354A-FCC_Ax
Description: ConnectX-3 Pro VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE;PCIe3.0 x8 8GT/s;RoHS R6
PSID: MT_1090111019
PCI Device Name: 02:00.0
Port1 MAC: f452147a58f1
Port2 GUID: f4521403007a58f2
Versions: Current Available
FW 2.36.5000 2.42.5000
PXE 3.4.0718 3.4.0752
From: Andrew Theurer <atheurer@redhat.com> :
We are using TRex, with our own scripts for the binary-search. All of our scripts are here: https://github.com/atheurer/trafficgen First, install trex: install-trex.sh Next, configure/launch trex: launch-trex.sh Finally, run a test: binary-search.py --traffic-generator=trex-txrx We recommend using Intel XL710 or XVV710 adapters to drive traffic. Intel Niantic will work, but it does not filter incoming packets reliably, so we can't get per-stream stats.
It is nice they have a script for installing trex in opt.
Some binary search commands I played with:
./binary-search.py --traffic-generator=trex-txrx --one-shot 1 --rate=0.1 --validation-runtime=20
./binary-search.py --traffic-generator=trex-txrx --run-bidirec 0 --search-runtime=10
./binary-search.py --traffic-generator=trex-txrx --run-bidirec 0 --search-runtime=10 --max-loss-pct 0.1
Setup steps from Andrew Theurer:
1) confirm BIOS has support for VT-d (PCI device assignment) 2) add kernel parameters: iommu=pt intel_iommu-on 3) use CPU isolation, tuned "cpu-partitioning" profile recommended, but also include "isolcpus=<cpulist>" boot paramter 3b) choose at least 8 cores from the same NUMA node as the PCI device used for traffic generator 4) reboot host 5) unbind the kernel module from the 2 PCI devices 6) load vfio and vfio-pci kernel modules 7) bind both PCI devices to vfio-pci (via sysfs, driverctl, or dpdk-devbind) 8) confirm isolcpus is working (sysfs) 9) run launch-trex.sh, confirm with screen -x that trex server is running 10) run a quick sanity check test: binary-search.py with at least --devices=<your PCI devices> --traffic-generator=trex-txrx --one-shot 1 --rate=0.1 --validation-runtime=30 11) run a binary search test (rerun and remove the --one-shot, --rate, and --validation-runtime options)
Manually created file for trex and could start it with:
sudo ./launch-trex.sh –yaml-file=/etc/trex\_cfg02-ixgbe.yaml
Given a correctly configured T-rex with a config file in /etc/trex-cfg.yaml, start the t-rex daemon:
sudo ./t-rex-64 -c 6 -i
where the -c argument is the number of threads to run (max is num_cores -2,
as t-rex uses two threads to do other work).
Then, launch the t-rex console in a separate window:
./trex-console
and from within the console run the benchmark script as follows:
start -f stl/udp_for_benchmarks.py -t packet_len=64,stream_count=12 --port 0 -m 100mpps
The packet_len and stream_count parameters are passed to the python script,
the -m switch sets the rate T-rex will (attempt to) run at (various units
available, see the online console help).
The other window will show the status as the test is running. To stop, simple
run stop in the console.
The udp_for_benchmarks.py script can run multiple streams, but the version
included with t-rex will run all streams with the same packets (i.e., port
numbers), which is not what we want if we are going to test multiple cores. The
modified version below will run each stream on a separate port (starting from
dstport 12); also set the dst IP appropriately according to the testbed setup.
from trex_stl_lib.api import *
# Tunable example
#
#trex>profile -f stl/udp_for_benchmarks.py
#
#Profile Information:
#
#
#General Information:
#Filename: stl/udp_for_benchmarks.py
#Stream count: 1
#
#Specific Information:
#Type: Python Module
#Tunables: ['stream_count = 1', 'direction = 0', 'packet_len = 64']
#
#trex>start -f stl/udp_for_benchmarks.py -t packet_len=128 --port 0
#
class STLS1(object):
'''
Generalization of udp_1pkt_simple, can specify number of streams and packet length
'''
def create_stream (self, packet_len, stream_count):
packets = []
for i in range(stream_count):
base_pkt = Ether()/IP(src="16.0.0.1",dst="10.70.2.2")/UDP(dport=12+i,sport=1025)
base_pkt_len = len(base_pkt)
base_pkt /= 'x' * max(0, packet_len - base_pkt_len)
packets.append(STLStream(
packet = STLPktBuilder(pkt = base_pkt),
mode = STLTXCont()
))
return packets
def get_streams (self, direction = 0, packet_len = 64, stream_count = 1, **kwargs):
# create 1 stream
return self.create_stream(packet_len - 4, stream_count)
# dynamic load - used for trex console or simulator
def register():
return STLS1()To avoid issues with hash collisions in hardware RX queue steering, this script can be used to setup manual flow rules for a range of UDP ports:
#!/bin/sh
IFACE=ens3f1
START_PORT=12
NUM_RINGS=$(ethtool -n $IFACE| egrep '[0-9]+ RX rings available' | cut -f 1 -d ' ')
for ring in $(seq 0 $(($NUM_RINGS - 1))); do
port=$((START_PORT + $ring))
ethtool -N $IFACE flow-type udp4 dst-port $port action $ring
doneRef: http://dpdk.org/doc/guides/linux_gsg
To get the upstream DPDK 18.05 to work on Ubuntu 16.04, first a newer version of rdma-core needs to be installed. The latest 18.0 source package from https://launchpad.net/ubuntu/+source/rdma-core/18.0-1 can be installed, then built using dpkg-buildpackage -uc -us. Installing all the .debs generated by this makes it possible to build dpdk with the mlx5 driver.
To enable the driver, edit config/common_base and enable the driver:
CONFIG_RTE_LIBRTE_MLX5_PMD=y CONFIG_RTE_LIBRTE_MLX5_DEBUG=y
Then configure the build:
make config T=x86_64-native-linuxapp-gcc
And finally build:
make -j24
Before running, enable huge pages:
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages mkdir /mnt/huge mount -t hugetlbfs nodev /mnt/huge
(Also possible to use 1gig pages, see http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html)
To see if everything works:
sudo ./build/app/testpmd -- -i
Starts in interactive mode. To run basic rxonly test:
set fwd rxonly start
To see current stats:
show port stats <id>
The F27 rdma-core looks too old.
The F27 packaged DPDK version 17.08 is likely also too old. Tried to run testpmd and it didn’t detect any interfaces.
Give DPDK is already packages, we can easily install the build dependencies:
sudo dnf builddep dpdk
And package kernel-devel is also needed for compiling DPDK.
The mlx driver(s) need some infiniband libraries. But the F27 package libibverbs-devel via rdma-core-devel is too old. The easiest work-around is to install the MLNX_OFED software tar.gz from Mellanox’es homepage.
If trying to install rdma-core-devel manually, you also need build dep:
sudo dnf install libnl3-devel
Get DPDK git source:
git clone git://dpdk.org/dpdk
Follow config+compile instructions from above Ubuntu guide. Remember to modify/enable the MLX5 defines in file config/common_base.
As documented in: http://dpdk.org/doc/guides/linux_gsg/linux_drivers.html
The Mellanox drivers uses a Bifurcated mode: https://dpdksummit.com/Archive/pdf/2016Userspace/Day02-Session04-RonyEfraim-Userspace2016.pdf
By using Infiniband verbs, which requires some “ib” kernel drivers and userspace libraries for infiniband “libibverbs” (which can be difficult to get installed in the correct version)
Thus, loading some extra infiniband kernel driver are needed:
modprobe ib_uverbs mlx5_ib