[PERFSCALE-4024] [RoCE] ib_write_bw support #199

josecastillolema · 2025-09-07T12:18:24Z

Type of change

Description

New ib_write_bw driver:

$ ./k8s-netperf -h| g ib
      --ib-write-bw               Use ib_write_bw as load driver (requires --privileged and --hostNet)

$ cat config.yaml
---
tests:
  - UDPStream:
    parallelism: 1
    profile: "UDP_STREAM"
    duration: 5
    samples: 1
    messagesize: 1024

$ ./k8s-netperf --ib-write-bw --privileged --hostNet --config=config.yaml
INFO[2025-09-07 12:13:25] Starting k8s-netperf (roce2@67cff32854ba9f343e3dadf87bbbbe1b27b34119)
INFO[2025-09-07 12:13:25] 📒 Reading config.yaml file.
INFO[2025-09-07 12:13:25] 📒 Reading config.yaml file - using ConfigV2 Method.
INFO[2025-09-07 12:13:25] 🔬 prometheus discovered at openshift-monitoring
INFO[2025-09-07 12:13:25] 🔨 Creating namespace: netperf
INFO[2025-09-07 12:13:25] 🔨 Creating service account: netperf
WARN[2025-09-07 12:13:25] ⚠️  No zone label
WARN[2025-09-07 12:13:25] ⚠️  Single node per zone and/or no zone labels
INFO[2025-09-07 12:13:25] 🚀 Starting Deployment for: client-host in namespace: netperf
INFO[2025-09-07 12:13:25] ⏰ Checking for client-host Pods to become ready...
INFO[2025-09-07 12:13:26] Looking for pods with label role=host-client
INFO[2025-09-07 12:13:26] 🚀 Starting Deployment for: server-host in namespace: netperf
INFO[2025-09-07 12:13:26] ⏰ Checking for server-host Pods to become ready...
INFO[2025-09-07 12:13:27] Looking for pods with label role=host-server
INFO[2025-09-07 12:13:32] 🗒️  Running ib_write_bw UDP_STREAM (service false) for 5s
+-------------------+-------------+------------+-------------+--------------+---------+-----------------+----------+-------------+--------------+-------+-----------+----------+---------+--------------------+--------------------------+
|    RESULT TYPE    |   DRIVER    |  SCENARIO  | PARALLELISM | HOST NETWORK | SERVICE | EXTERNAL SERVER | UDN INFO | BRIDGE INFO | MESSAGE SIZE | BURST | SAME NODE | DURATION | SAMPLES |     AVG VALUE      | 95% CONFIDENCE INTERVAL  |
+-------------------+-------------+------------+-------------+--------------+---------+-----------------+----------+-------------+--------------+-------+-----------+----------+---------+--------------------+--------------------------+
| 📊 Stream Results | ib_write_bw | UDP_STREAM | 1           | true         | false   | false           |          |             | 1024         | 0     | false     | 5        | 1       | 2708.930000 (Mb/s) | 0.000000-0.000000 (Mb/s) |
+-------------------+-------------+------------+-------------+--------------+---------+-----------------+----------+-------------+--------------+-------+-----------+----------+---------+--------------------+--------------------------+
+------------------+-------------+------------+-------------+--------------+---------+-----------------+----------+-------------+--------------+-------+-----------+----------+---------+-----------+
|       TYPE       |   DRIVER    |  SCENARIO  | PARALLELISM | HOST NETWORK | SERVICE | EXTERNAL SERVER | UDN INFO | BRIDGE INFO | MESSAGE SIZE | BURST | SAME NODE | DURATION | SAMPLES | AVG VALUE |
+------------------+-------------+------------+-------------+--------------+---------+-----------------+----------+-------------+--------------+-------+-----------+----------+---------+-----------+
| UDP Loss Percent | ib_write_bw | UDP_STREAM | 1           | true         | false   | false           |          |             | 1024         | 0     | false     | 5        | 1       | 0.000000  |
+------------------+-------------+------------+-------------+--------------+---------+-----------------+----------+-------------+--------------+-------+-----------+----------+---------+-----------+
INFO[2025-09-07 12:13:38] Cleaning resources created by k8s-netperf
INFO[2025-09-07 12:13:38] ⏰ Waiting for netperf Namespace to be deleted...

It only supports UDP_STREAM and it does not support services.
It also adds some logic to be more selective in the containers that runs in the server pod (vs the old model where all the drivers were run independently of the test). This is because the ib_write_bw needs specific hw so it makes no sense to run in unconditionally.

jtaleric · 2025-09-22T15:53:11Z

pkg/drivers/ibwritebw.go

+	config.Show(nc, i.driverName)
+
+	// ib_write_bw client command: "ib_write_bw -d mlx5_0 -x 3 -F $server_ip"
+	cmd := []string{"stdbuf", "-oL", "-eL", "ib_write_bw", "-d", "mlx5_0", "-x", "3", "-F", serverIP}


should we take the -d out? or have the user provide that? Looking at the docs, we could simply remove it and it pick the first ib device.... I just worry about hard coding specifics...

Done!
This is the new format --ib-write-bw=mlx5_0:3

jtaleric · 2025-10-13T16:25:58Z

@josecastillolema checking in here

josecastillolema · 2025-10-14T07:51:33Z

Sorry @jtaleric I was prioritizing other stuff.
I will parametrize the name of the interface and the GID.

rsevilla87 · 2025-10-17T09:57:14Z

@josecastillolema rebase pls

rsevilla87 · 2025-10-17T09:58:26Z

| 📊 Stream Results | ib_write_bw | UDP_STREAM | 1 | true | false | false | | | 1024 | 0 | false | 5 | 1 | 2708.930000 (Mb/s) | 0.000000-0.000000 (Mb/s) |

throughtput doesn't seem too high

josecastillolema · 2025-11-28T13:10:34Z

| 📊 Stream Results | ib_write_bw | UDP_STREAM | 1 | true | false | false | | | 1024 | 0 | false | 5 | 1 | 2708.930000 (Mb/s) | 0.000000-0.000000 (Mb/s) |

throughtput doesn't seem too high

@rsevilla87 I have double checked and for this particular test and packet size the data is correct, this comes from the output of ib_write_bw:

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MiB/sec]    BW average[MiB/sec]   MsgRate[Mpps]
 65536      130047           0.00               2709.33              0.043349
---------------------------------------------------------------------------------------

josecastillolema · 2025-11-28T13:18:06Z

Don't know what is going on with the linter, is complaining about files this PR does not modify, i.e.:
Error: pkg/k8s/kubevirt.go:16:2: ST1019: package "k8s.io/api/core/v1" is being imported more than once (staticcheck)

josecastillolema · 2025-11-28T13:30:18Z

@rsevilla87 @jtaleric can I have some eyes here?

rsevilla87

The PR has now many more content than initially, i.e: all the changes in kubernetes.go, can you move those code optimization changes to a new PR?

josecastillolema · 2025-11-28T14:11:14Z

The PR has now many more content than initially, i.e: all the changes in kubernetes.go, can you move those code optimization changes to a new PR?

The changes of kubernetes.go were always in this PR. Before this PR, you could not selectevly choose the backends (i.e.: I want to use netperf but I do not want uperf), all backends were always running. This model did not work for ib_write_bw, and thus the change.

rsevilla87 · 2025-12-04T11:12:01Z

cmd/k8s-netperf/k8s-netperf.go

 			log.Fatal("flags --udnl2 and --udnl3 are mutually exclusive; please set only one")
 		}
+		if ibWriteBwEnabled && (!privileged || !hostNetOnly) {
+			log.Fatalf("😭 ib_write_bw driver requires both --privileged and --hostNet flags")


Rather than aborting, why don't you set these flags when ib_write_bw is enabled?

I think aborting and requiring the proper command from the user is the proper way to go.
Otherwise we would have to deal with multiple invalid combinations that are valid without the privileged and hostNetwork flags, i.e.: --vm, --udn, etc.

rsevilla87 · 2025-12-04T11:13:22Z

cmd/k8s-netperf/k8s-netperf.go

+
+		// If a specific driver is explicitly requested, disable the default netperf driver
+		if (iperf3 || uperf || ibWriteBwEnabled) && !cmd.Flags().Changed("netperf") {
+			netperf = false


We should be able to support combining netperf with other drivers, the logic starting in line 270 has that intention

rsevilla87 · 2025-12-04T11:17:52Z

cmd/k8s-netperf/k8s-netperf.go

 	rootCmd.Flags().BoolVar(&netperf, "netperf", true, "Use netperf as load driver")
 	rootCmd.Flags().BoolVar(&iperf3, "iperf", false, "Use iperf3 as load driver")
 	rootCmd.Flags().BoolVar(&uperf, "uperf", false, "Use uperf as load driver")
+	rootCmd.Flags().StringVar(&ibWriteBw, "ib-write-bw", "", "Use ib_write_bw as load driver, requires nic:gid format (e.g., mlx5_0:0, requires --privileged and --hostNet)")


I wonder if at this point is better to use a StringSlice flag

rootCmd.Flags().StringSliceVar(&RequestedDrivers, "drivers", []string{"netperf"}, fmt.Sprintf("Comma separated list of metrics drivers to use, supported values: %s", supportedDrivers))

so we can run k8s-netperf with

k8s-netperf run --drivers netperf,uperf,ib_write_bw --ib-write-bw=mlx5_0:0 --config bla.yml

Using this logic would save some lines.

This change will break local CI, Prow, etc.
Are we sure?

ok, we can address that in a different PR

pkg/drivers/ibwritebw.go

rsevilla87 · 2025-12-04T11:32:41Z

pkg/drivers/ibwritebw.go

+			return stdout, err
+		}
+	} else {
+		retry := 3


Remove the hardcoded retry logic

It is exactly the same approach (different number, iperf uses 10), as iperf implements: https://github.com/cloud-bulldozer/k8s-netperf/blob/main/pkg/drivers/iperf.go#L129

pkg/k8s/kubernetes.go

Signed-off-by: Jose Castillo Lema <josecastillolema@gmail.com>

Updated ibwritebw driver to match the new ExtractUdnIp function signature that requires a network name parameter. Also added support for Cudn networks. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Signed-off-by: Jose Castillo Lema <josecastillolema@gmail.com>

josecastillolema · 2026-01-21T11:21:16Z

@rsevilla87 ready for review, I have addressed most of the comments

jtaleric

nice work @josecastillolema !!!

josecastillolema requested review from jtaleric and rsevilla87 September 7, 2025 12:18

jtaleric reviewed Sep 22, 2025

View reviewed changes

josecastillolema force-pushed the roce2 branch from 45bea8e to d7e7a8a Compare November 28, 2025 13:01

josecastillolema requested a review from jtaleric November 28, 2025 13:18

rsevilla87 requested changes Nov 28, 2025

View reviewed changes

josecastillolema requested a review from rsevilla87 November 28, 2025 14:12