[Bug]: Worker: Process failed for slice (): transport retry counter exceeded

[zx-ai]

Bug Report

Worker: Process failed for slice (opcode: 1, source_addr: 0x14c97801f040, length: 65536, dest_addr: 0x1474af020000, local_nic: mlx5_7, peer_nic: 172.16.37.132:14127@mlx5_4, dest_rkey: 2359727, retry_cnt: 0): transport retry counter exceeded. This type of error occurs intermittently.

I try export MC_MS_AUTO_DISC=0， this help solve this type of error . But I’m curious why turning off auto_discover_ avoids this issue？

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

[00fish0]

I’ve run into a similar issue before. My guess is that it’s caused by the auto‑discovery NIC selection.
When MC_MS_AUTO_DISC=1, it enumerates all RDMA devices and chooses NICs in a random/round‑robin way. If the host has NICs on different fabrics, unstable links, or mismatched GID/MTU settings, it can intermittently pick a “bad path,” leading to transport retry counter exceeded.
With MC_MS_AUTO_DISC=0, only the explicitly specified NICs are used, so the path is fixed and the issue goes away.

[alogfans]

It seems that local:mlx5_7 -> remote:mlx5_4 are not physically connected. Turning off auto_discover_ avoids this path.