`ucx-cuda` DEB package `Recommends` pulls in mismatched NVIDIA driver libraries, breaking existing GPU environments

[amahussein]

Describe the bug

Starting with UCX 1.20.0, the ucx-cuda DEB package declares Recommends: libnvidia-compute | libnvidia-ml1. Since apt installs Recommends by default, this causes NVIDIA driver userspace libraries to be pulled in automatically when installing UCX — even in environments that already have a working GPU driver.

When the version of the recommended libnvidia-compute package (resolved from the apt repository) does not match the kernel driver already installed on the host, this results in a driver/library version mismatch that breaks GPU functionality:

$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 595.45

In UCX 1.19.x and earlier, the ucx-cuda package had no Recommends field, so installing UCX was harmless to the system's existing driver setup.

Steps to Reproduce

1. Start with a system or container that has a working NVIDIA GPU driver (e.g., kernel driver 590.44.01)
2. Install UCX 1.20.0 DEB packages:

   wget https://github.com/openucx/ucx/releases/download/v1.20.0/ucx-1.20.0-ubuntu22.04-mofed5-cuda12-x86_64.tar.bz2
   tar -xvf ucx-1.20.0-ubuntu22.04-mofed5-cuda12-x86_64.tar.bz2
   apt install -y *.deb

3. Observe that apt automatically installs additional NVIDIA packages as recommended dependencies:

   The following additional packages will be installed:
     libnvidia-cfg1 libnvidia-common libnvidia-compute libnvidia-decode
     libnvidia-gpucomp nvidia-persistenced
   

4. Run nvidia-smi — it fails with Driver/library version mismatch

Expected behavior

UCX should not pull in driver packages, even as soft dependencies. UCX uses the CUDA Driver API via forward-compatible libcuda.so, which is designed to work across driver versions. The driver is a system-level component managed independently of UCX.

Setup and versions

• UCX version: 1.20.0
• OS: Ubuntu 22.04
• Package: ucx-1.20.0-ubuntu22.04-mofed5-cuda12-x86_64.tar.bz2
• Host driver: 590.44.01
• Pulled driver: 595.45.04 (from NVIDIA CUDA apt repository)

Additional information

• Downstream issue: [FEA] Upgrade to UCX 1.20 NVIDIA/spark-rapids#14055
• Downstream PR: Upgrade ucx to 1.20 NVIDIA/spark-rapids#14383

[tvegas1]

As per #10711, --no-install-recommends should be the way to avoid the breakage in a container.

[amahussein]

As per #10711, --no-install-recommends should be the way to avoid the breakage in a container.

Thanks for the pointer to --no-install-recommends and to #10711 for the background.

I agree that --no-install-recommends is a workable downstream mitigation. My concern is more about the package's default behavior. Requiring every downstream consumer — every Dockerfile, CI pipeline, and deployment script — to remember a non-obvious flag is fragile. UCX 1.19 and earlier required no such workaround.

Since apt installs Recommends by default, ucx-cuda can pull in NVIDIA userspace driver libraries automatically. In environments where the driver stack is already managed by the host or base image, that is risky: the package manager may install a userspace library version that does not match the running kernel driver, which can break an already working GPU setup.

That is why I think Suggests is a better fit than Recommends here.

Per Debian Policy, Recommends is for packages that should be installed in almost all normal cases, while Suggests is for packages that may be useful but are not needed for a reasonable installation. For UCX-CUDA, preexisting driver-managed environments seem like the normal case, especially in containers and cluster deployments. In that context, pulling driver libraries by default feels too strong.

So I agree --no-install-recommends avoids the immediate issue, but I do not think it answers the packaging question. My suggestion would still be to change this relationship from Recommends to Suggests, so the dependency remains visible without making it part of the default install path.

[amahussein]

Hi @yosefe, could you clarify the intended role of these libraries?

If downstream users install ucx-cuda with --no-install-recommends, what functionality is expected to be missing or broken?

More specifically:

• What use case motivated adding these packages as Recommends?
• What behavior does UCX rely on from them?
• In which environments would skipping them be incorrect?

I’m trying to understand whether these packages are actually needed for normal UCX-CUDA operation, or whether they are only helpful in a narrower set of setups.

[amahussein]

After further investigation, we found that the issue is more severe with the CUDA 13 variant of the UCX 1.20 packages.

CUDA 13 variant (ucx-1.20.0-ubuntu22.04-mofed5-cuda13-x86_64.tar.bz2):

Package: ucx-cuda
Version: 1.20.4b7a6ca
Depends: libc6 (>= 2.34), libnvidia-compute-580 (>= 580.65.06), ucx
Recommends: libnvidia-compute | libnvidia-ml1

Here, libnvidia-compute-580 is a hard Depends, not a Recommends. The --no-install-recommends workaround does NOT prevent this package from being installed. This means:

• apt install --no-install-recommends ucx-cuda-1.20.0.deb still installs libnvidia-compute-580 and its transitive dependencies (libnvidia-cfg1-580, libnvidia-common-580, libnvidia-decode-580, libnvidia-gpucomp-580, nvidia-persistenced)
• These packages overwrite libcuda.so.1 and libnvidia-ml.so.1 symlinks, causing the same driver/library version mismatch described in the original report

[tvegas1]

Here, libnvidia-compute-580 is a hard Depends, not a Recommends. The --no-install-recommends workaround does NOT prevent this package from being installed. This means:

Please note that this specific Depends issue has been fixed with: #11141.

[rakhmets]

We have moved the libraries from the Depends section to the Recommends section so that users can avoid installing these libraries if they use the container environment.
We have not moved them to the Suggests section, so as not to break the existing behavior. E.g., for users using the native host environment.

Perhaps we can plan to move them to the Suggests section, as usage in a container environment is more common at the moment. @yosefe WDYT?