Skip to content

fix(cuda): MTP + sm_89 compatibility for GCC 12 host compiler - #150

Merged
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
altifilmperisi:feature/mtp-sm89-compat
Jun 4, 2026
Merged

fix(cuda): MTP + sm_89 compatibility for GCC 12 host compiler#150
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
altifilmperisi:feature/mtp-sm89-compat

Conversation

@altifilmperisi

Copy link
Copy Markdown

Summary

  • Disable -compress-mode flag — incompatible with GCC 12 host compiler (passed to GCC instead of nvcc)
  • Guard FP8 __nv_fp8_e4m3 conversion behind __CUDA_ARCH__ >= 900 — FP8 PTX instructions require sm_90+, causing build failure on RTX 4060 Ti / Ada Lovelace (sm_89)

Why

Without these fixes, building with -DGGML_CUDA=ON fails on sm_89 GPUs (RTX 4060 Ti, 4070, 4080, 4090) when using GCC 12 as the CUDA host compiler:

  1. gcc-12: error: unrecognized command-line option '-compress-mode=...'
  2. ptxas: Feature 'cvt with .e4m3x2/.e5m2x2' requires .target sm_90 or higher

These changes allow llama-server to build successfully with CUDA support on Ada Lovelace GPUs while retaining full MTP (Multi-Token Prediction) support from upstream.

- Disable -compress-mode flag (incompatible with GCC 12 host compiler)
- Guard FP8 __nv_fp8_e4m3 conversion behind __CUDA_ARCH__ >= 900
  (FP8 PTX instructions require sm_90+, causing build failure on RTX 4060 Ti / sm_89)

Allows llama-server to build with CUDA on Ada Lovelace GPUs while
retaining MTP (Multi-Token Prediction) support from upstream.
@TheTom
TheTom merged commit 02f9af6 into TheTom:feature/turboquant-kv-cache Jun 4, 2026
1 check passed
@TheTom

TheTom commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Thanks. The __CUDA_ARCH__ >= 900 guard on the fp8 e4m3 path is the right call, sm_89 has to take the fallback. Builds clean on sm_121 here with no numeric change. Merged. Minor note for later: disabling -compress-mode globally fixes gcc12 but drops binary-size compression on every config, so a host-compiler version guard would be tidier down the line. Not blocking. Appreciate the fix.

KGardevoir pushed a commit to KGardevoir/llama-cpp-turboquant that referenced this pull request Jun 16, 2026
…heTom#150)

- Disable -compress-mode flag (incompatible with GCC 12 host compiler)
- Guard FP8 __nv_fp8_e4m3 conversion behind __CUDA_ARCH__ >= 900
  (FP8 PTX instructions require sm_90+, causing build failure on RTX 4060 Ti / sm_89)

Allows llama-server to build with CUDA on Ada Lovelace GPUs while
retaining MTP (Multi-Token Prediction) support from upstream.

Co-authored-by: savasuyar <savasuyar@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants