Add EFA support to rust-ibverbs with working example. by e-ngo · Pull Request #81 · jonhoo/rust-ibverbs

e-ngo · 2025-11-05T23:58:43Z

Adding EFA Support to `rust-ibverbs`

Tested on: AWS instances with EFA NICs

Key Parity Functions Added

build_efa()
Mirrors build(), but uses the extended libibverbs API to create an SRD QP.
handshake_efa()
Unlike RC QP handshakes, EFA is connectionless. This function:
- Transitions QP state: INIT → RTR → RTS
- Creates and binds the required Address Handle (AH) to authorize sends to the remote peer.
post_write_efa()
Uses extended API to build a work request.
ud_data.addr must point to the target AH.

These functions are all hidden behind a prop passed down from the QueuePairBuilder (QueuePairBuilder.enable_efa(true)). By enabling this, the proper efa version of the method will be called.

This keeps the user's interfaces nearly the exact same.

…nable_efa(true)).

xmakro · 2025-11-06T04:13:32Z

ibverbs/src/lib.rs

+        wr_id: u64,
+        imm_data: Option<u32>,
+    ) -> io::Result<()> {
+        let qp_ex = unsafe { ffi::ibv_qp_to_qp_ex(self.qp) };


check that the result is not nullptr, then dereference once and use below, then you likely dont need the unsafe after this line

hmm, not able to remove the full unsafe blocks., Since inner functions like wr_rdma_write_imm is unsafe as well.

xmakro · 2025-11-06T04:13:53Z

ibverbs/src/lib.rs

+        _imm_data: Option<u32>,
+    ) -> io::Result<()> {
+
+        let qp_ex = unsafe { ffi::ibv_qp_to_qp_ex(self.qp) };


same here, check this succeeded, then dereference

ibverbs/src/lib.rs

xmakro · 2025-11-06T04:18:09Z

ibverbs/src/lib.rs

+            } else {
+                self.qp.ah
+            };
+            self.qp.ah = ah;


seems a bit roundabout to store this with if else in a variable, why not just set
qp.ah = test_ah; in the inf branch, and delete the else branch and let statement?

Gotcha. Yea i was trying to be too clever. Simplified it. Handshakes should only be called once per qp is the assumption i will make here. i think it's fair to assume?

xmakro · 2025-11-06T04:19:04Z

ibverbs-sys/build.rs

    println!("cargo:rustc-link-search=native={manifest_dir}/vendor/rdma-core/build/lib");
    println!("cargo:rustc-link-lib=ibverbs");
+    // Only link EFA if the library exists (i.e., on EFA-capable hosts)
+    if Path::new("/opt/amazon/efa/lib/").exists() {


instead of relying on a path, maybe the right thing is to feature gate this?

Cool, how does this look?

xmakro · 2025-11-06T04:19:25Z

ibverbs/src/lib.rs

 use std::time::Duration;

 const PORT_NUM: u8 = 1;
+const UD_QKEY: u32 = 0x12345678;


what is this ?

QKEY is used to validate communication between two QPs, specifically for UD QPs. In this case I am using a random u32.

ibverbs/src/lib.rs

…evice open time.

…a test.

xmakro · 2025-11-07T02:56:27Z

ibverbs-sys/build.rs

    println!("cargo:rustc-link-search=native={manifest_dir}/vendor/rdma-core/build/lib");
    println!("cargo:rustc-link-lib=ibverbs");

+    if env::var("CARGO_FEATURE_EFA").is_ok() && Path::new("/opt/amazon/efa/lib/").exists() {


since this is now feature gated, i would remove:
Path::new("/opt/amazon/efa/lib/").exists()

xmakro · 2025-11-07T04:44:13Z

ibverbs/src/lib.rs

+                qp_type: self.qp_type,
+                sq_sig_all: 0,
+            };
+            unsafe { ffi::ibv_create_qp(self.pd.pd, &mut attr as *mut _) }


I think these codepaths can be further unified by using Ibv_create_qp_ex in the non-efa case, then you can use the same ibv_qp_init_attr_ex for both cases

xmakro · 2025-11-07T04:45:54Z

ibverbs-sys/build.rs

    eprintln!("run bindgen");
    let bindings = bindgen::Builder::default()
        .header("vendor/rdma-core/libibverbs/verbs.h")
+        .header("vendor/rdma-core/providers/efa/efadv.h")


maybe the bindings for the efa headers and efa specific methods should only be created if the efa feature is enabled

xmakro · 2025-11-07T05:00:33Z

ibverbs/src/lib.rs

+    /// Remote RDMA write with EFA.
+    /// immediate data can be used to signal the completion of the write operation
+    /// the other side uses post_recv on a dummy buffer and get the imm data from the work completion
+    pub fn post_write_efa(


I read more about the ibv ex api documentation, this is actually not EFA specific. Is using the wr-api with function calls instead of the post-api a requirement for EFA? We might be able to unify the codepaths for EFA and non EFA to both use the wr-api since it seems to be the more efficient and powerful API. Is it supported on all normal hardware these days?

Is using the wr-api with function calls instead of the post-api a requirement for EFA
I believe so, primarily we need to set the ud_addr in the wr_id which seems to only be available in the extended API.

Is it supported on all normal hardware these days?
I would assume so, the ibv_wr api seems to date back 7 years https://github.com/linux-rdma/rdma-core/blob/0d977c57c486bc13e3a6be0f17c29eace5657633/libibverbs/man/ibv_wr_post.3.md?plain=1#L29. I think that api should have backwards compatibility, as long as we try to make sure we're not setting too many other fields.

I can try to merge it. Do you mind if I do it in a follow up PR?

ud_addr is actually part of ibv_send_wr (it's just called "ud" there), so we can use the existing ibv_post_send API in the implementation of post_{send,write,read}:

struct ibv_send_wr { uint64_t wr_id; ... union { ... struct { struct ibv_ah *ah; uint32_t remote_qpn; uint32_t remote_qkey; } ud; } wr;

I will give it a try

xmakro · 2025-11-07T05:35:55Z

ibverbs/src/lib.rs

+            (*qp_ex).wr_set_sge_list.unwrap()(qp_ex, local.len(), local.as_ptr() as *mut ffi::ibv_sge);
+            // TODO(Eric): Consider creating and caching AH for each remote_endpoint...
+            // This way a single QP can be shared for multiple remote endpoints.
+            (*qp_ex).wr_set_ud_addr.unwrap()(qp_ex, self.ah, remote_endpoint.num, UD_QKEY);


Since UD QueuePair works with any other UD QP, the handshake API does not make much sense for this. I'm wondering how we should change the API. Should we distinguish between ConnectedQueuePair and DatagramQueuePair?

That's a good point. My initial reaction was that I wanted to merge the API points as much as possible so that there's less branching off in user code. What APIs would you reckon that would be different here or that we would want to distinguish?

Add EFA support to rust-ibverbs with working example.

65d3ba9

e-ngo force-pushed the erngo/add-efa-support branch from 57a1b57 to 65d3ba9 Compare November 5, 2025 23:59

e-ngo added 4 commits November 5, 2025 16:53

Hide efa specific API behind a simple setting in QueuePairBuilder (.e…

1e91180

…nable_efa(true)).

Remove the use of tokio.

fff18f0

Merge handshake() and handshake_efa since they are so similar.

f0eb993

Merge build & build_efa.

2a0aba2

xmakro reviewed Nov 6, 2025

View reviewed changes

e-ngo added 4 commits November 6, 2025 00:04

Simply passying of enabled_efa to just set it in the context during d…

48ce668

…evice open time.

Clean up some code based off of first feedback. Also add post_read_ef…

62e563d

…a test.

Feature gate efa lib linking.

359a022

Add conditional compilation configuration flags.

44aae0b

xmakro reviewed Nov 7, 2025

View reviewed changes

e-ngo added 3 commits November 8, 2025 00:49

Add lazy loading feature to efa dependent verbs...

6a9c320

Remove unused code.

8701d8f

Readd missing import

23e8433

Conversation

e-ngo commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Adding EFA Support to rust-ibverbs

Key Parity Functions Added

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xmakro Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

e-ngo commented Nov 5, 2025 •

edited

Loading

Adding EFA Support to `rust-ibverbs`

xmakro Nov 7, 2025 •

edited

Loading