Skip to content

Mtmd: add a way to select device for vision encoder #14236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 22, 2025
Merged

Mtmd: add a way to select device for vision encoder #14236

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6617f3c
Mtmd: add a way to select device for vision encoder
stduhpf Jun 17, 2025
f9fb321
simplify
stduhpf Jun 17, 2025
ffd0bd8
format
stduhpf Jun 26, 2025
baf7e3e
Warn user if manual device selection failed
stduhpf Jul 22, 2025
01e448c
initialize backend to nullptr
stduhpf Jul 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions tools/mtmd/clip.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -367,8 +367,8 @@ struct clip_ctx {
std::vector<ggml_backend_t> backend_ptrs;
std::vector<ggml_backend_buffer_type_t> backend_buft;

ggml_backend_t backend;
ggml_backend_t backend_cpu;
ggml_backend_t backend = nullptr;
ggml_backend_t backend_cpu = nullptr;
ggml_backend_buffer_ptr buf;

int max_nodes = 8192;
Expand All @@ -384,9 +384,18 @@ struct clip_ctx {
if (!backend_cpu) {
throw std::runtime_error("failed to initialize CPU backend");
}
backend = ctx_params.use_gpu
? ggml_backend_init_by_type(GGML_BACKEND_DEVICE_TYPE_GPU, nullptr)
: nullptr;
if (ctx_params.use_gpu) {
auto backend_name = std::getenv("MTMD_BACKEND_DEVICE");
if (backend_name != nullptr) {
backend = ggml_backend_init_by_name(backend_name, nullptr);
if (!backend) {
LOG_WRN("%s: Warning: Failed to initialize \"%s\" backend, falling back to default GPU backend\n", __func__, backend_name);
}
}
if (!backend) {
backend = ggml_backend_init_by_type(GGML_BACKEND_DEVICE_TYPE_GPU, nullptr);
}
}

if (backend) {
LOG_INF("%s: CLIP using %s backend\n", __func__, ggml_backend_name(backend));
Expand Down
Loading