RDMA Write Operation Fails When Fetching Model Weights

[zeocax]

🐛 Describe the bug

python -m apps.grpo.main --config apps/grpo/qwen3_8b.yaml

Error Log

...
[TitanTrainer-0/2] 2025-11-14 20:00:59 INFO Pushing weights for policy version 1
[TitanTrainer-1/2] 2025-11-14 20:00:59 INFO Pushing weights for policy version 1
[ReferenceModel-0/1] 2025-11-14 20:01:02 INFO [GC] Performing periodic GC collection took 0.00 seconds
[ReferenceModel-0/1] 2025-11-14 20:01:10 INFO [GC] Performing periodic GC collection took 0.00 seconds
[TitanTrainer-0/2] 2025-11-14 20:01:10 INFO Completed weights push in 11.85 seconds
[TitanTrainer-1/2] 2025-11-14 20:01:11 INFO Completed weights push in 12.03 seconds
[Generator-0/1] 2025-11-14 20:01:11 INFO [Generator] Fetching weights for v1 to shared memory 
CRITICAL:root:Unhandled exception in actor endpoint
Traceback (most recent call last):
  File "/mnt/shared-storage-user/[USERNAME]/zeocax/envs/torchforge/lib/python3.12/site-packages/monarch/_src/actor/actor_mesh.py", line 935, in handle
    result = await the_method(*args, **kwargs) 
  File "/mnt/shared-storage-user/[USERNAME]/zeocax/envs/torchforge/lib/python3.12/site-packages/torchstore/storage_volume.py", line 65, in get
    return await self.store.get(key, transport_buffer, request)
  File "/mnt/shared-storage-user/[USERNAME]/zeocax/envs/torchforge/lib/python3.12/site-packages/torchstore/storage_volume.py", line 257, in get
    await transport_buffer.write_from(extracted_tensor)
  File "/mnt/shared-storage-user/[USERNAME]/zeocax/envs/torchforge/lib/python3.12/site-packages/torchstore/transport/buffers.py", line 219, in write_from
    await self.rdma_buffers[idx].write_from(chunk)
  File "/mnt/shared-storage-user/[USERNAME]/zeocax/envs/torchforge/lib/python3.12/site-packages/monarch/_src/actor/future.py", line 138, in mark_complete
    func, value = fut.set_result, await coro
  File "/mnt/shared-storage-user/[USERNAME]/zeocax/envs/torchforge/lib/python3.12/site-packages/monarch/_src/rdma/rdma.py", line 333, in write_from_nonblocking
    res = await self._buffer.write_from(
Exception: failed to write from buffer: RDMA polling completion failed: Send work completion failed with status: 9, vendor error: 138, wr_id: 0, send_cq_idx: 0 [lkey=2546176, rkey=2546176, addr=0x7fe166e7f040, size=622329856]

Maybe monarch's issue?

Versions

torchmonarch 0.1.2
torchshow 0.5.2
torchstore 0.1.2
torchtitan 0.2.0

[allenwang28]

Hi @zeocax can you describe what type of machine/Cloud you're currently running on? It is indeed a Monarch RDMA issue, likely where it's inappropriately using the wrong NIC.

If you're able to run

$ rdma link show

could help reveal more info

[zeocax]

Hi @allenwang28 thanks for looking into this! Here's the rdma link show output:

➜  ~ rdma link show

link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev brainpf0 
link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev brainpf1 
link mlx5_2/1 state ACTIVE physical_state LINK_UP netdev brainpf2 
link mlx5_3/1 state ACTIVE physical_state LINK_UP netdev brainpf3 
link mlx5_4/1 state ACTIVE physical_state LINK_UP netdev brainpf4 
link mlx5_5/1 state ACTIVE physical_state LINK_UP netdev brainpf5 
link mlx5_6/1 state ACTIVE physical_state LINK_UP netdev brainpf6 
link mlx5_data_0/1 state ACTIVE physical_state LINK_UP netdev enp213s0np0 
link mlx5_7/1 state ACTIVE physical_state LINK_UP netdev brainpf7 
link mlx5_bond_0/1 state ACTIVE physical_state LINK_UP netdev enp86s0f0np0 
link mlx5_20/1 state ACTIVE physical_state LINK_UP netdev brainvf0 
link mlx5_21/1 state ACTIVE physical_state LINK_UP netdev brainvf1 
link mlx5_22/1 state ACTIVE physical_state LINK_UP netdev brainvf2 
link mlx5_23/1 state ACTIVE physical_state LINK_UP netdev brainvf3 
link mlx5_24/1 state ACTIVE physical_state LINK_UP netdev brainvf4 
link mlx5_25/1 state ACTIVE physical_state LINK_UP netdev brainvf5 
link mlx5_26/1 state ACTIVE physical_state LINK_UP netdev brainvf6 
link mlx5_27/1 state ACTIVE physical_state LINK_UP netdev brainvf7

And please note after some tests, I found that this issue occurs intermittently, so I personally don't think it's a device/hardware issue. (unless there's some device-related randomness involved)

After further testing, I found that:

It only happens during the first weight fetching
Once the first step completes successfully, all subsequent steps work without this issue

This pattern suggests there might be some RDMA initialization or state-related randomness causing this.
Do you have any suggestions for debugging this or potential workarounds?

[allenwang28]

yeah so I was debugging something similar to this before. For context, Monarch tries to auto assign the NIC based on the PCI affinity. It works according to the H100 machines we've tried so far, but it doesn't make a distinction between the NIC's capability. So e.g. it's trying to mix front-end + back-end which leads to this error.

This PR fixed it for me: meta-pytorch/monarch#1832

If that helps you too please let me know, can check it in!