Introduce accuracy testing for LLMs in tt-forge.

[dgolubovicTT]

Closes tenstorrent/tt-xla#3355

Overview: LLM Accuracy Testing with TOP1/TOP5 Metrics

This PR introduces token-level accuracy testing for LLM benchmarks, measuring how well TT device predictions match CPU reference outputs.

Accuracy Metrics:

• TOP1 Accuracy: Percentage of tokens where the device's prediction matches the CPU reference model's top prediction (argmax)
• TOP5 Accuracy: Percentage of tokens where the device's prediction appears in the CPU reference model's top-5 predictions

Test Corpus:
We use "A Tale of Two Cities" by Charles Dickens as our reference text. The text is split into two parts:

• Prefill phase (first half): Input context fed to the model
• Decode phase (second half): Ground truth tokens for validation with teacher forcing

Critical Requirement - Sequence Length Matching:
The CPU reference outputs (.refpt files) must be generated with the same total_length as the input_sequence_length used during device testing. Mismatched sequence lengths cause accuracy degradation even with teacher forcing, because CPU model and compiled model need to have same input sequence size (context). If you change
input_sequence_length in tests, you must regenerate all reference outputs.

NOTE: We currently use almost identical approach to accuracy testing in tt-metal for 1-1 comparison. Same text corpus (Tale of Two Cities) and same top1/top5 metrics. However, since we test accuracy on llms only on n150, we had to use total sequence length 128 (for memory constraints). That means 64 tokens in prefil, and 64 for decode.

Introduce accuracy testing to test_llms.py

• Add --accuracy-testing argument to test_llms.py tests that we track in benchmark testing.
• Add --batch-size argument to accuracy tests in test_llms.py because we can't fit default batch 32 on device
  due to larger input sequence length required in accuracy testing.
  With batch size 32 and input sequence length, 7B and 8B models failed with OOM issues.
• Run accuracy tests in separate job called: run-n150-accuracy-benchmarks of the perf-benchmark-experimental workflow

Generating ground truth .refpt files (generate_reference_outputs.py)

Add generate_reference_outputs.py script that loads Huggingface model, runs it on "Tale of Two Cities" text corpus, and generates a .refpt file containing reference tokens and top-5 predictions for each position.
Generate reference top1/top5 token predictions for LLM accuracy benchmarking:

• Loads HuggingFace models on CPU
• Processes "Tale of Two Cities" text corpus with teacher forcing
• Outputs .refpt files containing reference tokens and top-k predictions
• Used by TokenAccuracy class to validate TOP1/TOP5 accuracy during benchmarks
• Added directory with reference .refpt files - /reference_outputs with README that explains how reference files are created and used
• Added generate_all_reference_outputs.sh that runs generate_reference_outputs.py for a list of models and dumps reference outputs to benchmark/tt-xla/reference_outputs

Ensures reproducibility through eval mode, disabled dropout, greedy decoding,
and StaticCache matching the benchmark environment. Reference files must be
regenerated if input_sequence_length changes.

Usage:
python3 scripts/generate_reference_outputs.py
--model "meta-llama/Llama-3.2-1B-Instruct"
--output_file "reference_outputs/Llama-3.2-1B-Instruct.refpt"
--total_length 128

Adding shared utility for decode (decode_utils.py)

Centralize LLM decode operations used by reference output generation and accuracy testing:

• Teacher forcing generation with ground truth tokens
• Reference top-k prediction generation for .refpt files
• Static cache and accuracy testing initialization
• Logits extraction and top-k token utilities

Prevents implementation drift between reference generation and benchmark paths
by sharing the same decode logic, tokenization, and cache semantics.

TokenAccuracy class for validating LLM inference quality(token_accuracy.py)

• Loads precomputed reference data from .refpt files (tokens, top1/top5 predictions)
• Validates torch/transformers versions match reference file for reproducibility
• Splits reference tokens into prefill (input) and decode (ground truth) windows
• Computes TOP1/TOP5 accuracy by comparing model predictions against reference
• Provides teacher forcing tokens for deterministic decode loops

Slight refactoring

• Simplify static cache initialization using init_static_cache helper
• Remove unused variables is_multichip and mesh from generate_and_benchmark function.

[odjuricicTT]

Looks great! A few minor changes requested.

[odjuricicTT]

Can we reduce this to a one model size per architecture? (e.g. 1 llama, 1 qwen etc). This will be run on perf runners by default which are overloaded currently.

Going forward we might want to have this as a seperate job and run it on shared runners by default.

[dgolubovicTT]

For start, I really want to track Accuracy on all models that we want to improve and do mixed precision on. I don't think we should jump to reducing reasonable tests before trying to solve it other way.

[vkovacevicTT]

Agreed @odjuricicTT
We should make it a seperate run like p150 and llmbox and default it to shared runners.

[odjuricicTT]

That makes sense. I propose the following:

• Remove all of these additional tests.
• Implement accuracy testing as a parameter to all existing tests (e.g. like num_layers)
• Add a new CI job that will call all tests and pass the e.g. --accuracy param
• Make sure that this job is run on the n150 shared runners

[dgolubovicTT]

Done, with slight corrections from @vvukomanTT and @vkovacevicTT

[odjuricicTT]

There are some refactors for naming saved files related to models in this PR: https://github.com/tenstorrent/tt-forge/pull/847/changes

Please take a look and try to reuse the same logic / variables for naming.

[dgolubovicTT]

Talked with @mvasiljevicTT offline. Her model name comes from pytest test ID. My model name is real Huggingface model name. function that generates these files generate_reference_outputs has huggingface model name as attribute and loads it and creates .refpt file with the same name. So it is completely independent on what test name is... Therefore, it would be hacky to force display_name (from Marina's PR).

[odjuricicTT]

This seems redundant, i'd rather recompute this outside of this function using the logits, would that work for you case?

The reasons for this are that it's not ok to assume batch here and we want to minimize what is measured into e2e time if it is not needed.

[dgolubovicTT]

Added support for batch_size != 1 so I think we are good now

[odjuricicTT]

That makes sense. I propose the following:

• Remove all of these additional tests.
• Implement accuracy testing as a parameter to all existing tests (e.g. like num_layers)
• Add a new CI job that will call all tests and pass the e.g. --accuracy param
• Make sure that this job is run on the n150 shared runners

[vkovacevicTT]

Looks good now!
Here is the run with latest changes: https://github.com/tenstorrent/tt-forge/actions/runs/21955566618