Skip to content

Commit 5a86354

Browse files
JamesJBarnettJamesJBarnett
authored
Merge pull request #247054 from mcgov/mcgov/mana
Create linux-mana-dpdk-pmd
2 parents 09987fe + dc1c90d commit 5a86354

File tree

3 files changed

+160
-9
lines changed

3 files changed

+160
-9
lines changed

‎articles/virtual-network/accelerated-networking-mana-overview.md

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -45,15 +45,7 @@ Several [Azure Marketplace](https://learn.microsoft.com/marketplace/azure-market
4545
We recommend using an operating system with support for MANA to maximize performance. In instances where the operating system doesn't or can't support MANA, network connectivity is provided through the hypervisor’s virtual switch. The virtual switch is also used during some infrastructure servicing events where the Virtual Function (VF) is revoked.
4646

4747
### Using DPDK
48-
Utilizing DPDK on MANA hardware requires the Linux kernel 6.2 or later or a backport of the Ethernet and InfiniBand drivers from the latest Linux kernel. It also requires specific versions of DPDK and user-space drivers.
49-
50-
DPDK requires the following set of drivers:
51-
1. [Linux kernel Ethernet driver](https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/microsoft/mana) (5.15 kernel and later)
52-
1. [Linux kernel InfiniBand driver](https://github.com/torvalds/linux/tree/master/drivers/infiniband/hw/mana) (6.2 kernel and later)
53-
1. [DPDK MANA poll-mode driver](https://github.com/DPDK/dpdk/tree/main/drivers/net/mana) (DPDK 22.11 and later)
54-
1. [Libmana user-space drivers](https://github.com/linux-rdma/rdma-core/tree/master/providers/mana) (rdma-core v44 and later)
55-
56-
DPDK only functions on Linux VMs.
48+
For information about DPDK on MANA hardware, see [Microsoft Azure Network Adapter (MANA) and DPDK on Linux](setup-dpdk-mana.md)
5749

5850
## Evaluating performance
5951
Differences in VM SKUs, operating systems, applications, and tuning parameters can all affect network performance on Azure. For this reason, we recommend that you benchmark and test your workloads to ensure you achieve the expected network performance.

‎articles/virtual-network/setup-dpdk-mana.md

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
---
2+
title: Microsoft Azure Network Adapter (MANA) and DPDK on Linux
3+
description: Learn about MANA and DPDK for Linux Azure VMs.
4+
author: mcgov
5+
ms.service: virtual-network
6+
ms.topic: how-to
7+
ms.date: 07/10/2023
8+
ms.author: mamcgove
9+
---
10+
11+
# Microsoft Azure Network Adapter (MANA) and DPDK on Linux
12+
13+
The Microsoft Azure Network Adapter (MANA) is new hardware for Azure virtual machines to enables higher throughput and reliability.
14+
To make use of MANA, users must modify their DPDK initialization routines. MANA requires two changes compared to legacy hardware:
15+
- [MANA EAL arguments](#mana-dpdk-eal-arguments) for the poll-mode driver (PMD) differ from previous hardware.
16+
- The Linux kernel must release control of the MANA network interfaces before DPDK initialization begins.
17+
18+
The setup procedure for MANA DPDK is outlined in the [example code.](#example-testpmd-setup-and-netvsc-test).
19+
20+
## Introduction
21+
22+
Legacy Azure Linux VMs rely on the mlx4 or mlx5 drivers and the accompanying hardware for accelerated networking. Azure DPDK users would select specific interfaces to include or exclude by passing bus addresses to the DPDK EAL. The setup procedure for MANA DPDK differs slightly, since the assumption of one bus address per Accelerated Networking interface no longer holds true. Rather than using a PCI bus address, the MANA PMD uses the MAC address to determine which interface it should bind to.
23+
24+
## MANA DPDK EAL Arguments
25+
The MANA PMD probes all devices and ports on the system when no `--vdev` argument is present; the `--vdev` argument is not mandatory. In testing environments it's often desirable to leave one (primary) interface available for servicing the SSH connection to the VM. To use DPDK with a subset of the available VFs, users should pass both the bus address of the MANA device and the MAC address of the interfaces in the `--vdev` argument. For more detail, example code is available to demonstrate [DPDK EAL initialization on MANA](#example-testpmd-setup-and-netvsc-test).
26+
27+
For general information about the DPDK Environment Abstraction Layer (EAL):
28+
- [DPDK EAL Arguments for Linux](https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#eal-in-a-linux-userland-execution-environment)
29+
- [DPDK EAL Overview](https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html)
30+
31+
## DPDK requirements for MANA
32+
33+
Utilizing DPDK on MANA hardware requires the Linux kernel 6.2 or later or a backport of the Ethernet and InfiniBand drivers from the latest Linux kernel. It also requires specific versions of DPDK and user-space drivers.
34+
35+
MANA DPDK requires the following set of drivers:
36+
1. [Linux kernel Ethernet driver](https://github.com/torvalds/linux/tree/master/drivers/net/ethernet/microsoft/mana) (5.15 kernel and later)
37+
1. [Linux kernel InfiniBand driver](https://github.com/torvalds/linux/tree/master/drivers/infiniband/hw/mana) (6.2 kernel and later)
38+
1. [DPDK MANA poll-mode driver](https://github.com/DPDK/dpdk/tree/main/drivers/net/mana) (DPDK 22.11 and later)
39+
1. [Libmana user-space drivers](https://github.com/linux-rdma/rdma-core/tree/master/providers/mana) (rdma-core v44 and later)
40+
41+
>[!NOTE]
42+
>MANA DPDK is not available for Windows; it will only work on Linux VMs.
43+
44+
## Example: Check for MANA
45+
46+
>[!NOTE]
47+
>This article assumes the pciutils package containing the lspci command is installed on the system.
48+
49+
```bash
50+
# check for pci devices with ID:
51+
# vendor: Microsoft Corporation (1414)
52+
# class: Ethernet Controller (0200)
53+
# device: Microsft Azure Network Adapter VF (00ba)
54+
if [[ -n `lspci -d 1414:00ba:0200` ]]; then
55+
echo "MANA device is available."
56+
else
57+
echo "MANA was not detected."
58+
fi
59+
60+
```
61+
62+
## Example: DPDK installation (Ubuntu 22.04)
63+
64+
>[!NOTE]
65+
>This article assumes compatible kernel and rdma-core are installed on the system.
66+
67+
```bash
68+
DEBIAN_FRONTEND=noninteractive sudo apt-get install -q -y build-essential libudev-dev libnl-3-dev libnl-route-3-dev ninja-build libssl-dev libelf-dev python3-pip meson libnuma-dev
69+
70+
pip3 install pyelftools
71+
72+
# Try latest LTS DPDK, example uses DPDK tag v23.07-rc3
73+
git clone https://github.com/DPDK/dpdk.git -b v23.07-rc3 --depth 1
74+
pushd dpdk
75+
meson build
76+
cd build
77+
ninja
78+
sudo ninja install
79+
popd
80+
```
81+
82+
## Example: Testpmd setup and netvsc test
83+
84+
Note the following example code for running DPDK with MANA. The direct-to-vf 'netvsc' configuration on Azure is recommended for maximum performance with MANA.
85+
86+
>[!NOTE]
87+
>DPDK requires either 2MB or 1GB hugepages to be enabled
88+
89+
```bash
90+
# Enable 2MB hugepages.
91+
echo 1024 | tee /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages
92+
93+
# Assuming use of eth1 for DPDK in this demo
94+
PRIMARY="eth1"
95+
96+
# $ ip -br link show master eth1
97+
# > enP30832p0s0 UP f0:0d:3a:ec:b4:0a <... # truncated
98+
# grab interface name for device bound to primary
99+
SECONDARY="`ip -br link show master $PRIMARY | awk '{ print $1 }'`"
100+
# Get mac address for MANA interface (should match primary)
101+
MANA_MAC="`ip -br link show master $PRIMARY | awk '{ print $3 }'`"
102+
103+
104+
# $ ethtool -i enP30832p0s0 | grep bus-info
105+
# > bus-info: 7870:00:00.0
106+
# get MANA device bus info to pass to DPDK
107+
BUS_INFO="`ethtool -i $SECONDARY | grep bus-info | awk '{ print $2 }'`"
108+
109+
# Set MANA interfaces DOWN before starting DPDK
110+
ip link set $PRIMARY down
111+
ip link set $SECONDARY down
112+
113+
114+
## Move synthetic channel to user mode and allow it to be used by NETVSC PMD in DPDK
115+
DEV_UUID=$(basename $(readlink /sys/class/net/$PRIMARY/device))
116+
NET_UUID="f8615163-df3e-46c5-913f-f2d2f965ed0e"
117+
modprobe uio_hv_generic
118+
echo $NET_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/new_id
119+
echo $DEV_UUID > /sys/bus/vmbus/drivers/hv_netvsc/unbind
120+
echo $DEV_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/bind
121+
122+
# MANA single queue test
123+
dpdk-testpmd -l 1-3 --vdev="$BUS_INFO,mac=$MANA_MAC" -- --forward-mode=txonly --auto-start --txd=128 --rxd=128 --stats 2
124+
125+
# MANA multiple queue test (example assumes > 9 cores)
126+
dpdk-testpmd -l 1-9 --vdev="$BUS_INFO,mac=$MANA_MAC" -- --forward-mode=txonly --auto-start --nb-cores=8 --txd=128 --rxd=128 --txq=8 --rxq=8 --stats 2
127+
128+
```
129+
130+
## Troubleshooting
131+
132+
### Fail to set interface down.
133+
Failure to set the MANA bound device to DOWN can result in low or zero packet throughput.
134+
The failure to release the device can result the EAL error message related to transmit queues.
135+
```
136+
mana_start_tx_queues(): Failed to create qp queue index 0
137+
mana_dev_start(): failed to start tx queues -19
138+
```
139+
140+
### Failure to enable huge pages.
141+
142+
Try enabling huge pages and ensuring the information is visible in meminfo.
143+
```
144+
EAL: No free 2048 kB hugepages reported on node 0
145+
EAL: FATAL: Cannot get hugepage information.
146+
EAL: Cannot get hugepage information.
147+
EAL: Error - exiting with code: 1
148+
Cause: Cannot init EAL: Permission denied
149+
```
150+
151+
### Low throughput with use of --vdev="net_vdev_netvsc0,iface=eth1"
152+
153+
Failover configuration of either the `net_failsafe` or `net_vdev_netvsc` poll-mode-drivers isn't recommended for high performance on Azure. The netvsc configuration with DPDK version 20.11 or higher may give better results. For optimal performance, ensure your Linux kernel, rdma-core, and DPDK packages meet the listed requirements for DPDK and MANA.

‎articles/virtual-network/setup-dpdk.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ DPDK consists of sets of user-space libraries that provide access to lower-level
2121

2222
DPDK can run on Azure virtual machines that are supporting multiple operating system distributions. DPDK provides key performance differentiation in driving network function virtualization implementations. These implementations can take the form of network virtual appliances (NVAs), such as virtual routers, firewalls, VPNs, load balancers, evolved packet cores, and denial-of-service (DDoS) applications.
2323

24+
A list of setup instructions for DPDK on MANA VMs is available here: [Microsoft Azure Network Adapter (MANA) and DPDK on Linux](setup-dpdk-mana.md)
25+
2426
## Benefit
2527

2628
**Higher packets per second (PPS)**: Bypassing the kernel and taking control of packets in the user space reduces the cycle count by eliminating context switches. It also improves the rate of packets that are processed per second in Azure Linux virtual machines.
@@ -39,6 +41,8 @@ The following distributions from the Azure Marketplace are supported:
3941

4042
The noted versions are the minimum requirements. Newer versions are supported too.
4143

44+
A list of requirements for DPDK on MANA VMs is available here: [Microsoft Azure Network Adapter (MANA) and DPDK on Linux](setup-dpdk-mana.md)
45+
4246
**Custom kernel support**
4347

4448
For any Linux kernel version that's not listed, see [Patches for building an Azure-tuned Linux kernel](https://github.com/microsoft/azure-linux-kernel). For more information, you can also contact [[email protected]](mailto:[email protected]).
@@ -57,6 +61,8 @@ In addition, DPDK uses RDMA verbs to create data queues on the Network Adapter.
5761

5862
## Install DPDK manually (recommended)
5963

64+
DPDK installation instructions for MANA VMs are available here: [Microsoft Azure Network Adapter (MANA) and DPDK on Linux](setup-dpdk-mana.md)
65+
6066
### Install build dependencies
6167

6268
# [RHEL, CentOS](#tab/redhat)

0 commit comments

Comments
 (0)