Should probably re-normalize after "truncating"...

MaggotHATE · MaggotHATE · commit b3dce587c128 · 2024-08-19T16:36:01.000+05:00
diff --git a/base/llama-addon.cpp b/base/llama-addon.cpp
@@ -52,6 +52,11 @@ void llama_sample_xtc_addon(struct llama_context * ctx, llama_token_data_array *
         }
     }
 
+    candidates->sorted = false;
+
+    // Re-normalize probabilities
+    llama_sample_softmax(ctx, candidates);
+
     llama_set_time(ctx, t_start_sample_us);
 }
 

Original file line number	Diff line number	Diff line change
`@@ -52,6 +52,11 @@ void llama_sample_xtc_addon(struct llama_context * ctx, llama_token_data_array *`
`52`	`52`	`}`
`53`	`53`	`}`
`54`	`54`
	`55`	`+ candidates->sorted = false;`
	`56`	`+`
	`57`	`+ // Re-normalize probabilities`
	`58`	`+ llama_sample_softmax(ctx, candidates);`
	`59`	`+`
`55`	`60`	`llama_set_time(ctx, t_start_sample_us);`
`56`	`61`	`}`
`57`	`62`