Skip to content

Commit c0dfae7

Browse files
authored
add missing norm topk bias
1 parent ee09828 commit c0dfae7

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

src/llama-graph.cpp

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -981,6 +981,9 @@ ggml_tensor * llm_graph_context::build_moe_ffn(
981981
ggml_tensor * weights_sum = ggml_sum_rows(ctx0, weights); // [1, n_tokens]
982982
cb(weights_sum, "ffn_moe_weights_sum", il);
983983

984+
weights_sum = ggml_scale_bias(ctx0, weights_sum, 1.0, 1e-20);
985+
cb(weights_sum, "ffn_moe_weights_sum_biased", il);
986+
984987
weights = ggml_div(ctx0, weights, weights_sum); // [n_expert_used, n_tokens]
985988
cb(weights, "ffn_moe_weights_norm", il);
986989

0 commit comments

Comments
 (0)