ok, I fixed a bug and ran both vanilla LLama3.1-1B-Instruct and the sampler with settings of this branch on 200 samples of GSM8k.

Here are the results. @xjdr-alt

Without sampler, took ~5min

With entropix sampler, took ~10min

These are preliminary results of course.
I will run the full benchmark on the main branch soon.

I ran full GSM8k.

Without sampler, took ~30mins:

With entropix sampler, took more than 4 hours.

After applying the chat template, the results differ significantly. I benchmarked the 3B-Instruct model and I could reproduce Meta's original result from their blog post without the sampler.
https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

Without sampler, took ~30 min, batch size 1:

Entropix sampler, took ~10 hours:

Both benched on one 4090.

Is there anything wrong with this branch? @xjdr-alt

I ran the vanilla model with:

lm-eval --model hf --model_args pretrained=meta-llama/Llama-3.2-3B-Instruct --tasks gsm8k_cot_llama --batch_size 1 --log_samples --output_path logs/ --apply_chat_template --fewshot_as_multiturn

And the entropix sampler with:

python entropix/eval_main.py

Both with lm-eval-harness version

lm_eval==0.4.5

Here are the up to date results for 1B-Instruct.

Without sampler, took ~15min

With sampler, took ~5 hours