Force FP32 compute in GLM4 FFN Down (#13101)

city96 · JohannesGaessler · web-flow · commit 558a76471346 · 2025-04-25T14:38:34.000+02:00
* Force FP32 compute in cuBLAS GEMM * Revert "Force FP32 compute in cuBLAS GEMM" This reverts commit 6efd872. * Force F32 compute in GLM4 ffn down * Edit comment to clarify issue Co-authored-by: Johannes Gäßler <johannesg@5d6.de> --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
@@ -803,6 +803,10 @@ ggml_tensor * llm_graph_context::build_ffn(
 
     if (down) {
         cur = build_lora_mm(down, cur);
+        if (arch == LLM_ARCH_GLM4) {
+            // GLM4 seems to have numerical issues with half-precision accumulators
+            ggml_mul_mat_set_prec(cur, GGML_PREC_F32);
+        }
     }
 
     if (down_b) {

Original file line number	Diff line number	Diff line change
`@@ -803,6 +803,10 @@ ggml_tensor * llm_graph_context::build_ffn(`
`803`	`803`
`804`	`804`	`if (down) {`
`805`	`805`	`cur = build_lora_mm(down, cur);`
	`806`	`+ if (arch == LLM_ARCH_GLM4) {`
	`807`	`+ // GLM4 seems to have numerical issues with half-precision accumulators`
	`808`	`+ ggml_mul_mat_set_prec(cur, GGML_PREC_F32);`
	`809`	`+ }`
`806`	`810`	`}`
`807`	`811`
`808`	`812`	`if (down_b) {`