Skip to content

CUDA: add a flag "GGML_CUDA_JETSON_DEVICE" for optimization #13861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

CUDA: add a flag "GGML_CUDA_JETSON_DEVICE" for optimization #13861

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
81044d3
CUDA: add a flag "GGML_CUDA_JETSON_DEVICE" for optimization(#13856)
Yangxiaoz May 28, 2025
8f1c07e
CUDA: Added illustrate for the "GGML_CUDA_JETSON_DEVICE" flag in buil…
Yangxiaoz May 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/build.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,8 @@ If your GPU is not officially supported you can use the environment variable [`H

On Linux it is possible to use unified memory architecture (UMA) to share main memory between the CPU and integrated GPU by setting environment variable `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1`. However, this hurts performance for non-integrated GPUs (but enables working with integrated GPUs).

When using NVIDIA Jetson series devices which have integrated GPU, setting environment variable `GGML_CUDA_JETSON_DEVICE=1` to optimize for the Unified Memory Architecture

## Vulkan

**Windows**
Expand Down
9 changes: 8 additions & 1 deletion ggml/src/ggml-cuda/ggml-cuda.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1064,6 +1064,9 @@ static const char * ggml_backend_cuda_host_buffer_type_name(ggml_backend_buffer_

GGML_UNUSED(buft);
}
static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) {
return buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name;
}

static void ggml_backend_cuda_host_buffer_free_buffer(ggml_backend_buffer_t buffer) {
CUDA_CHECK(cudaFreeHost(buffer->context));
Expand Down Expand Up @@ -3263,7 +3266,11 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
}

static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
return (ggml_backend_buft_is_cuda(buft) || ggml_backend_buft_is_cuda_split(buft)) && buft->device == dev;
if (getenv("GGML_CUDA_JETSON_DEVICE") != nullptr){
return (ggml_backend_buft_is_cuda(buft) || ggml_backend_buft_is_cuda_split(buft)||ggml_backend_buft_is_cuda_host(buft)) && buft->device == dev;
}else{
return (ggml_backend_buft_is_cuda(buft) || ggml_backend_buft_is_cuda_split(buft)) && buft->device == dev;
}
}

static int64_t get_op_batch_size(const ggml_tensor * op) {
Expand Down