Improve benchmarks PCC computation

[vkovacevicTT]

Problem description

In Performance benchmark PCC for LLMs is calculated based on the first decode token.
We should take into consideration all tokens.

What's changed

PCC for LLMs is now computed for all tokens, final PCC is min PCC of all tokens.
Simplified compute_pcc function.

Checklist

• Single LLM test
• Single Vision model test
• Single Encoder test
• Run all LLMs with teacher forcing

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.12%. Comparing base (c666faf) to head (3bf1de7).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3306   +/-   ##
=======================================
  Coverage   28.12%   28.12%           
=======================================
  Files          33       33           
  Lines        4132     4132           
=======================================
  Hits         1162     1162           
  Misses       2970     2970           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bmalesevicTT]

Shouldn't the comment and the PR desc say "use min pcc"?

[vladimirjovanovicTT]

We need slight improvements to this approach to reflect causal nature of LLM decode process:

• I propose to check PCC for output tensors sequentially - we check PCC for each decode step, and assert on the first output that fails. When one token differs between CPU and TT, we can expect the following tokens to diverge, so PCC information for following tokens is of limited use.
• We should enable teacher forcing behavior as described in tenstorrent/tt-forge#859 (regenerating golden outputs).

Mid-term plan (IMO not needed for this PR), as discussed in https://docs.google.com/document/d/1nmFd002Ycv8wadpyOs1ZMMkV6zV6v0pWCl-LZsap1J8/edit?tab=t.0, "4. correctness" section, is to have a better metric than PCC for checking LLM generation correctness.

FYI/thoughts? @odjuricicTT @umalesTT @pglusacTT

[umalesTT]

We need slight improvements to this approach to reflect causal nature of LLM decode process:

• I propose to check PCC for output tensors sequentially - we check PCC for each decode step, and assert on the first output that fails. When one token differs between CPU and TT, we can expect the following tokens to diverge, so PCC information for following tokens is of limited use.
• We should enable teacher forcing behavior as described in tenstorrent/tt-forge#859 (regenerating golden outputs).

Mid-term plan (IMO not needed for this PR), as discussed in https://docs.google.com/document/d/1nmFd002Ycv8wadpyOs1ZMMkV6zV6v0pWCl-LZsap1J8/edit?tab=t.0, "4. correctness" section, is to have a better metric than PCC for checking LLM generation correctness.

FYI/thoughts? @odjuricicTT @umalesTT @pglusacTT

+1
We should also test tokens all the way up to full sequence length, we shouldn't make number of tokens for which we check pcc/other metrics configurable.