wheels: use cuda-toolkit runtime wheels

[jameslamb]

Contributes to #5443

Related to rapidsai/build-planning#143

libcugraph.so dynamically links to several CUDA Toolkit libraries

$ ldd /pyenv/versions/3.11.14/lib/python3.11/site-packages/libcugraph/lib64/libcugraph.so
        ...
        libcusolver.so.12 => /usr/local/cuda/lib64/libcusolver.so.12 (0x00007c616aba7000)
        libcublas.so.13 => /usr/local/cuda/lib64/libcublas.so.13 (0x00007c61675d5000)
        libcublasLt.so.13 => /usr/local/cuda/lib64/libcublasLt.so.13 (0x00007c6143c83000)
        libcusparse.so.12 => /usr/local/cuda/lib64/libcusparse.so.12 (0x00007c613987c000)
        libcurand.so.10 => /usr/local/cuda/lib64/libcurand.so.10 (0x00007c6131161000)
        ...
        libnvJitLink.so.13 => /usr/local/cuda/lib64/libnvJitLink.so.13 (0x00007c612af5f000)
        ...

This proposes getting them from cuda-toolkit wheels, instead of system installations.

Notes for Reviewers

Benefits of this change

• reduces the risk of multiple copies of the same library being loaded
• allows the use of Python package versioning to manage compatibility
• consistency with other RAPIDS libraries (see Use CUDA wheels to avoid statically linking CUDA components in our wheels build-planning#35)
• reduces the risk of runtime issues with other libraries that use CTK wheels, like torch (wheels CI: stricter torch index selection, test oldest versions of dependencies cugraph-gnn#413 (comment))

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[jameslamb]

/ok to test

[jameslamb]

Just picking a point on the diff so we can have a threaded conversation.

The changes in this PR are useful and should be merged, but I don't think they fix the core problem.

I tested using wheels from this PR's CI and a modified version of the cugraph-pyg wheel-testing script and found the same library-loading issues described in #5443

details (click me)

docker run \
    --rm \
    --gpus all \
    --env GH_TOKEN=$(gh auth token) \
    -v $(pwd):/opt/work \
    -w /opt/work \
    -it rapidsai/citestwheel:26.04-cuda13.1.1-rockylinux8-py3.11 \
    bash

source rapids-init-pip

package_name="cugraph-pyg"

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# Download the libwholegraph, pylibwholegraph, and cugraph-pyg built in the previous step
COMMIT_ID=333d00bac743194617545ae1f3289be2afa0d835
LIBWHOLEGRAPH_WHEELHOUSE=$(
  RAPIDS_PY_WHEEL_NAME="libwholegraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-artifact cugraph-gnn 413 cpp wheel "${COMMIT_ID}"
)
PYLIBWHOLEGRAPH_WHEELHOUSE=$(
  rapids-get-pr-artifact cugraph-gnn 413 python wheel --pkg_name pylibwholegraph --stable "${COMMIT_ID}"
)
CUGRAPH_PYG_WHEELHOUSE=$(
    RAPIDS_PY_WHEEL_NAME="cugraph-pyg_cu13" RAPIDS_PY_WHEEL_PURE="1" rapids-get-pr-artifact cugraph-gnn 413 python wheel "${COMMIT_ID}"
)

# download libcugraph wheels
COMMIT_ID="d75f9d2f0d7f899ad5d37d3aa3886e77613c404e"
LIBCUGRAPH_WHEELHOUSE=$(
  RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-artifact cugraph 5444 cpp wheel "${COMMIT_ID}"
)
PYLIBCUGRAPH_WHEELHOUSE=$(
  rapids-get-pr-artifact cugraph 5444 python wheel --pkg_name pylibcugraph --stable "${COMMIT_ID}"
)

# generate constraints, accounting for 'oldest' and 'latest' dependencies
rapids-dependency-file-generator \
    --output requirements \
    --file-key "test_cugraph_pyg" \
    --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};dependencies=${RAPIDS_DEPENDENCIES};include_torch_extra_index=false" \
| tee "${PIP_CONSTRAINT}"

# ensure a CUDA variant of 'torch' is used
TORCH_WHEEL_DIR="$(mktemp -d)"
./ci/download-torch-wheels.sh "${TORCH_WHEEL_DIR}"

# notes:
#
#   * echo to expand wildcard before adding `[extra]` requires for pip
#   * '--extra-index-url pypi.nvidia.com' can be removed when 'cugraph' and
#     its dependencies are available from pypi.org
#
rapids-pip-retry install \
    --prefer-binary \
    --constraint "${PIP_CONSTRAINT}" \
    --extra-index-url 'https://pypi.nvidia.com' \
    "${LIBWHOLEGRAPH_WHEELHOUSE}"/*.whl \
    "$(echo "${PYLIBWHOLEGRAPH_WHEELHOUSE}"/pylibwholegraph_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)" \
    "$(echo "${CUGRAPH_PYG_WHEELHOUSE}"/cugraph_pyg_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)[test]" \
    "${LIBCUGRAPH_WHEELHOUSE}"/*.whl \
    "${PYLIBCUGRAPH_WHEELHOUSE}"/*.whl \
    "${TORCH_WHEEL_DIR}"/torch-*.whl

python -c "import pylibcugraph"
# ImportError: libcugraph.so: cannot open shared object file: No such file or directory

I think the core problem here is that libcugraph.so is linking in symbols from nvJitLink that only exist in nvJitLink>=3.1

$ LIBCUGRAPH=/pyenv/versions/3.11.14/lib/python3.11/site-packages/libcugraph/lib64/libcugraph.so
$ objdump -T "${LIBCUGRAPH}" | grep -i nvjit
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkGetErrorLog_13_1
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkGetLinkedCubin_13_1
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkCreate_13_1
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkGetLinkedCubinSize_13_1
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkAddData_13_1
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkDestroy_13_1
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkGetErrorLogSize_13_1
0000000000000000      DF *UND*  0000000000000000  libnvJitLink.so.13 __nvJitLinkComplete_13_1
000000000ab4dec0 g    DF .text  0000000000000031  Base        _ZNK19FatbinFragmentEntry6add_toERP9nvJitLink
000000000ab4dff0 g    DF .text  0000000000000e49  Base        _Z22check_nvjitlink_resultP9nvJitLink15nvJitLinkResult

Compare that to libraft.so in the same environment.

$ LIBRAFT=/pyenv/versions/3.11.14/lib/python3.11/site-packages/libraft/lib64/libraft.so
$ objdump -T "${LIBRAFT}" | grep -i nvjit
# (empty)

The root cause here might be that however cuGraph links nvJitLink may need to change.

[jameslamb]

Confirmed in an offline discussion that there's a bigger root cause here. Continuing that in rapidsai/cuvs#1855

But these changes in cuGraph are still useful on their own and could be merged any time.

[jameslamb]

/merge