Skip to content

[CUDA] Enable CUDA Graph on CUDA Toolkit < 12.x #12394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 17, 2025
Merged

[CUDA] Enable CUDA Graph on CUDA Toolkit < 12.x #12394

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fc7f195
Enable CUDA Graph on CTK < 12.x
gaugarg-nv Mar 14, 2025
db400a6
Fix compilation errors with MUSA
gaugarg-nv Mar 17, 2025
a05f2b8
Disable CUDA Graph for MUSA
gaugarg-nv Mar 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ggml/src/ggml-cuda/common.cuh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -678,7 +678,7 @@ struct ggml_tensor_extra_gpu {
};


#if ((CUDART_VERSION >= 12000) && defined(GGML_CUDA_USE_GRAPHS)) || defined(GGML_HIP_GRAPHS)
#if (defined(GGML_CUDA_USE_GRAPHS)) || defined(GGML_HIP_GRAPHS)
#define USE_CUDA_GRAPH
#endif

Expand Down
12 changes: 7 additions & 5 deletions ggml/src/ggml-cuda/ggml-cuda.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2610,13 +2610,15 @@ static bool is_cuda_graph_update_required(ggml_backend_cuda_context * cuda_ctx,

static void update_cuda_graph_executable(ggml_backend_cuda_context * cuda_ctx) {

cudaGraphExecUpdateResultInfo result_info;
#ifdef __HIP_PLATFORM_AMD__
hipGraphNode_t errorNode;
hipError_t stat = hipGraphExecUpdate(cuda_ctx->cuda_graph->instance, cuda_ctx->cuda_graph->graph, &errorNode, &result_info);
#if (CUDART_VERSION < 12000 || defined(__HIP_PLATFORM_AMD__))
cudaGraphNode_t errorNode;
cudaGraphExecUpdateResult result_info;
cudaError_t stat = cudaGraphExecUpdate(cuda_ctx->cuda_graph->instance, cuda_ctx->cuda_graph->graph, &errorNode, &result_info);
#else
cudaGraphExecUpdateResultInfo result_info;
Copy link
Member

@ggerganov ggerganov Mar 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this correctly cudaGraphExecUpdateResultInfo? Or it should be cudaGraphExecUpdateResult?

As it is, cudaGraphExecUpdateResultInfo is only defined in vendors/musa.h:

llama.cpp/ggml/src/ggml-cuda/vendors/musa.h

Line 122 in db400a6

#define cudaGraphExecUpdateResultInfo musaGraphExecUpdateResult

Copy link
Contributor Author

@gaugarg-nv gaugarg-nv Mar 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct. cudaGraphExecUpdateResultInfo is declared in headers of CTK >= 12.x. CTK < 12.x declares cudaGraphExecUpdateResult.

Looked into MUSA headers and it seems musaGraphExecUpdate itself is commented out in its headers.
I realized earlier CUDA graph was disabled on MUSA platform and hence that part of code was not even getting compiled on MUSA. But, my change had enabled it on MUSA when I removed CUDART_VERSION >= 12000 check. I have again disabled CUDA graph on MUSA platform now. I have also updated musa.h to use right macros but this doesn't matter as code is not getting compiled on MUSA.

cudaError_t stat = cudaGraphExecUpdate(cuda_ctx->cuda_graph->instance, cuda_ctx->cuda_graph->graph, &result_info);
#endif
#endif // (CUDART_VERSION < 12000 || defined(__HIP_PLATFORM_AMD__))

if (stat == cudaErrorGraphExecUpdateFailure) {
#ifndef NDEBUG
GGML_LOG_DEBUG("%s: CUDA graph update failed\n", __func__);
Expand Down
2 changes: 1 addition & 1 deletion ggml/src/ggml-cuda/vendors/hip.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@
#define cudaGraphExecDestroy hipGraphExecDestroy
#define cudaGraphLaunch hipGraphLaunch
#define cudaErrorGraphExecUpdateFailure hipErrorGraphExecUpdateFailure
#define cudaGraphExecUpdateResultInfo hipGraphExecUpdateResult
#define cudaGraphExecUpdateResult hipGraphExecUpdateResult
#define cudaGraphNodeType hipGraphNodeType
#define cudaGraphNodeTypeKernel hipGraphNodeTypeKernel
#define cudaGraphInstantiate hipGraphInstantiate
Expand Down
Loading