[BUG]: libcugraph 26.04 nightly wheels fail to load with older `nvidia-nvjitlink` installed (CUDA 13 only)

[jameslamb]

Version

26.04

Which installation method(s) does this occur on?

Pip

Describe the bug.

The latest libcugraph-cu13 wheels fail to load with errors like this:

libcugraph.so: undefined symbol: __nvJitLinkGetErrorLog_13_1, version libnvJitLink.so.13

If the libnvJitLink.so.13 found by the loader is older than the one libcugraph was built against.

I found this while working on rapidsai/cugraph-gnn#413 because PyTorch packages are == pinned to older nvidia-nvjitlink packages.

This behavior makes the latest cugraph / libcugraph / pylibcugraph packages uninstallable with any CUDA variants of torch<=2.10, which will affect cugraph-pyg.

Minimum reproducible example

Setting up the environment:

docker run \
    --rm \
    --gpus all \
    -it rapidsai/citestwheel:26.04-cuda13.1.1-rockylinux8-py3.11 \
    bash

# torch==2.10.0+cu130 requires 'nvidia-nvjitlink==13.0.88', just directly
# specifying that here to make the example smaller
pip install \
    --prefer-binary \
    'nvidia-nvjitlink==13.*,<13.1' \
    'pylibcugraph-cu13==26.4.*,>=26.4.0a32'

Import pylibcugraph fails like this:

$ python -c "import pylibcugraph"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/pyenv/versions/3.11.14/lib/python3.11/site-packages/pylibcugraph/__init__.py", line 15, in <module>
    import pylibcugraph.comms
  File "/pyenv/versions/3.11.14/lib/python3.11/site-packages/pylibcugraph/comms/__init__.py", line 4, in <module>
    from .comms_wrapper import init_subcomms
ImportError: libcugraph.so: cannot open shared object file: No such file or directory

But note that's misleading (will be fixed in rapidsai/build-planning#119) ... the real error looks like this:

$ python -c "import libcugraph; import libraft; libraft.load_library(); print(libcugraph.load._load_wheel_installation('libcugraph.so'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/pyenv/versions/3.11.14/lib/python3.11/site-packages/libcugraph/load.py", line 30, in _load_wheel_installation
    return ctypes.CDLL(lib, PREFERRED_LOAD_FLAG)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.11.14/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /pyenv/versions/3.11.14/lib/python3.11/site-packages/libcugraph/lib64/libcugraph.so: undefined symbol: __nvJitLinkGetErrorLog_13_1, version libnvJitLink.so.13

Upgrading to a newer nvidia-nvjitlink resolves this.

$ pip install 'nvidia-nvjitlink==13.*,>=13.1'
$ python -c "import pylibcugraph; print(pylibcugraph.__version__)"
26.04.00a32

Relevant log output

N/A

Environment details

pip list (click me)

$ pip list
Package                          Version
-------------------------------- ---------
attrs                            25.4.0
certifi                          2026.1.4
cuda-bindings                    13.1.1
cuda-pathfinder                  1.3.5
cuda-python                      13.1.1
cuda-toolkit                     13.0.2
cupy-cuda13x                     14.0.1
jsonschema                       4.26.0
jsonschema-specifications        2025.9.1
libcugraph-cu13                  26.4.0a32
libraft-cu13                     26.4.0a35
librmm-cu13                      26.4.0a47
numpy                            2.4.2
nvidia-cublas                    13.1.0.3
nvidia-curand                    10.4.0.35
nvidia-cusolver                  12.0.4.66
nvidia-cusparse                  12.6.3.3
nvidia-nccl-cu13                 2.29.3
nvidia-nvjitlink                 13.0.88
pip                              26.0.1
pylibcugraph-cu13                26.4.0a32
pylibraft-cu13                   26.4.0a35
PyYAML                           6.0.3
rapids-dependency-file-generator 1.20.2
rapids-logger                    0.2.3
referencing                      0.37.0
rmm-cu13                         26.4.0a47
rpds-py                          0.30.0
setuptools                       79.0.1
tomlkit                          0.14.0
typing_extensions                4.15.0

Other/Misc.

I suspect the root cause is somewhere around this change:

• Temporarily disables tests on RTX Pro 6000 for nightly runs #5434 (comment)

And that libcugraph wheel metadata just needs to be updated like this:

• Add nvidia-nvjitlink to pip dependencies for packages that depend on nvidia-cusparse build-planning#253

Other related issues / conversations:

• [BUG] Unable to install via pip in uv environment for 26.2 cuml#7796
• Unpin nvjitlink from ctk version NVIDIA/numba-cuda#809
• Ensuring CTK minor version compatibility for cccl.c.parallel NVIDIA/cccl#4851
• Use cuda-toolkit for multiple linux cuda packages pytorch/pytorch#174390

Code of Conduct

• I agree to follow cuGraph's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[jameslamb]

I think maybe libcugraph needs a treatment similar to rapidsai/raft#2548, to load these libraries from wheels instead of the system-installed ones?

I don't recall why cugraph wasn't included in rapidsai/build-planning#137, but it does look to me like it's dynamically linking to some of those libraries and so probably should be using wheels for them:

$ ldd /pyenv/versions/3.11.14/lib/python3.11/site-packages/libcugraph/lib64/libcugraph.so
        linux-vdso.so.1 (0x00007c61b0b05000)
        libraft.so => not found
        libcusolver.so.12 => /usr/local/cuda/lib64/libcusolver.so.12 (0x00007c616aba7000)
        libcublas.so.13 => /usr/local/cuda/lib64/libcublas.so.13 (0x00007c61675d5000)
        libcublasLt.so.13 => /usr/local/cuda/lib64/libcublasLt.so.13 (0x00007c6143c83000)
        libcusparse.so.12 => /usr/local/cuda/lib64/libcusparse.so.12 (0x00007c613987c000)
        libcurand.so.10 => /usr/local/cuda/lib64/libcurand.so.10 (0x00007c6131161000)
        libgomp-855c301a.so.1.0.0 => /pyenv/versions/3.11.14/lib/python3.11/site-packages/libcugraph/lib64/../../libcugraph_cu13.libs/libgomp-855c301a.so.1.0.0 (0x00007c6130f25000)
        libnvJitLink.so.13 => /usr/local/cuda/lib64/libnvJitLink.so.13 (0x00007c612af5f000)
        librmm.so => not found
        librt.so.1 => /lib64/librt.so.1 (0x00007c612ad57000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007c612ab53000)
        librapids_logger.so => not found
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007c612a7be000)
        libm.so.6 => /lib64/libm.so.6 (0x00007c612a43c000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007c612a224000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007c612a004000)
        libc.so.6 => /lib64/libc.so.6 (0x00007c6129c2e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007c61b0800000)

[jameslamb]

In an offline discussion, we (@divyegala @bdice and I) found that this is a bigger issue than just cuGraph... it will affect cuVS, cuGraph, cuML, and maybe others.

It's also not specific to wheels, and can be reproduced with conda packages:

docker run \
    --rm \
    --gpus all \
    -it rapidsai/ci-conda:26.04-cuda13.1.1-rockylinux8-py3.11 \
    bash

conda install \
    --yes \
    -c rapidsai-nightly \
    -c conda-forge \
    'cuvs = 26.04' \
    'libcuvs =26.04' \
    'libnvjitlink >13.0,<13.1' \
    python=3.11

python -c "import cuvs.distance"
# ImportError: /opt/conda/bin/../lib/libcuvs.so: undefined symbol: __nvJitLinkGetErrorLog_13_1

CCCL faced similar issues in their wheels last year, and implemented a shim that hides the versioned nvJitLink symbols (deferring them to a runtime lookup): NVIDIA/cccl#4851 (cc @leofang @oleksandr-pavlyk)

We may want to try something similar in RAPIDS.

[jameslamb]

This is an all-of-RAPIDS issue. The next thing we're going to try (as a short-term fix) is to build all of RAPIDS CUDA 13 wheels with CTK 13.0: rapidsai/build-planning#257