Multi-Node NVLink failure due to memory reference error in compute-domain-daemon

[Wennn]

We are using the 25.8.1 version and when MNNVL is not working in a workload deployed as a Deployment or StatefulSet, I noticed the following errors in compute-domain-daemon:

[ERROR] Failed to get new memory reference with rc: 87
[ERROR] Failed to allocate memory reference.
[ERROR] Failed to get export ref for UUID: <a UUID> for request from node id 0
[ERROR] Received import response from node id 0 with failure status 3

If the workload has 2 pods, deleting the recreating these 2 pods can mitigate, and during this process we don't need to recreate the ComputeDomain resource.

These multi-node NVLink workloads are running on GB200 and GB300 capacities using driver version 580.105.08

NCCL test results, and the fact that only restarting all workloads working suggest that it's unlikely to be related to hardware failure.

Is it possible to be a leaking of resource or process created by the ComputeDomain stack?

Please let me know if you need any additional information

Thanks!

[guptaNswati]

we may need to check for any stale connections.

can you share the complete compute-domain-daemon container logs with output of nvidia-imex-ctl -q from inside the pod before and after the restart.

And did you make any changes to the the default cleanup setting

[shivamerla]

If the workload has 2 pods, deleting the recreating these 2 pods can mitigate, and during this process we don't need to recreate the ComputeDomain resource.

@jackfrancis did you verify that the daemon pod is indeed ready before the workload pods could run? Please check the Restoring IMEX daemon ensemble barrier behavior section here. Since you mention that restart of the workload pods resolve the issue, most likely the daemon is not yet ready during the first workload run.