Skip to content

cuda: Add explicit check for the use of Cuda Toolkit's >= 13 with ccs < 7.5#486

Merged
Treece-Burgess merged 1 commit intoicl-utk-edu:masterfrom
Treece-Burgess:09-28-2025-cuda-toolkit-13-check
Oct 12, 2025
Merged

cuda: Add explicit check for the use of Cuda Toolkit's >= 13 with ccs < 7.5#486
Treece-Burgess merged 1 commit intoicl-utk-edu:masterfrom
Treece-Burgess:09-28-2025-cuda-toolkit-13-check

Conversation

@Treece-Burgess
Copy link
Contributor

@Treece-Burgess Treece-Burgess commented Sep 28, 2025

Pull Request Description

In Cuda Toolkit 13, support for offline compilation for architectures prior to cc's < 7.5 have been removed.

Currently the cuda component is able to be compiled using Cuda Toolkit 13 on a machine with cc's < 7.5. However, if a user ran ./papi_component_avail they would be met with the cuda component being disabled with a message of:

Name:   cuda                    CUDA profiling via NVIDIA CuPTI interfaces
   \-> Disabled: Unable to initialize CUPTI profiler libraries.

This PR introduces a conditional check to make sure that a Cuda Toolkit version >= 13 is not being used on a machine with devices that have cc's < 7.5. If this conditional check is met then a more apt error message is now provided:

Name:   cuda                    CUDA profiling via NVIDIA CuPTI interfaces
   \-> Disabled: Cuda Toolkit's >= 13 do not support offline compilation for architectures prior to cc's < 7.5. On mixed cc machines utilize 'export CUDA_VISIBLE_DEVICES'.

Testing

  1. Using Cuda Toolkit 13 on a machine with mixed cc's (1 * V100 and 1 * H100) result in the proper error message being shown:
Name:   cuda                    CUDA profiling via NVIDIA CuPTI interfaces
   \-> Disabled: Cuda Toolkit's >= 13 do not support offline compilation for architectures prior to cc's < 7.5. On mixed cc machines utilize 'export CUDA_VISIBLE_DEVICES'
  1. Using Cuda Toolkit 13 on a machine with mixed cc's (1 * V100 and 1 * H100) and using export CUDA_VISIBLE_DEVICES=0 which corresponds to the H100 only being seen results in:

    • The cuda component being active
    • papi_component_avail, papi_native_avail, and papi_command_line all successfully working
    • The cuda component tests all passing
  2. Using Cuda Toolkit 12.6 on a machine with mixed cc's (1 * V100 and 1 * H100) results in:

    • The cuda component being active
    • papi_component_avail, papi_native_avail, and papi_command_line all successfully working
    • The cuda component tests all passing

Author Checklist

  • Description
    Why this PR exists. Reference all relevant information, including background, issues, test failures, etc
  • Commits
    Commits are self contained and only do one thing
    Commits have a header of the form: module: short description
    Commits have a body (whenever relevant) containing a detailed description of the addressed problem and its solution
  • Tests
    The PR needs to pass all the tests

@Treece-Burgess Treece-Burgess added status-ready-for-review PR is ready to be reviewed type-maintenance Update code to keep it compatible, secure, modern. component-cuda PRs and Issues related to the cuda component labels Sep 28, 2025
@tokey-tahmid
Copy link

I am testing this PR.

Copy link

@tokey-tahmid tokey-tahmid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested the PR on a machine with 1 * V100 and 1 * H100 with CUDA Toolkit 13.0. With export CUDA_VISIBLE_DEVICES, PAPI Utilities, and CUDA component tests function as expected. And the updated disable message is seen when CUDA_VISIBLE_DEVICES variable is not set.

@Treece-Burgess Treece-Burgess force-pushed the 09-28-2025-cuda-toolkit-13-check branch from da127a5 to e4dbac6 Compare October 12, 2025 23:16
@Treece-Burgess Treece-Burgess merged commit c4227a2 into icl-utk-edu:master Oct 12, 2025
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component-cuda PRs and Issues related to the cuda component status-ready-for-review PR is ready to be reviewed type-maintenance Update code to keep it compatible, secure, modern.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants