Skip to content

Add top1/top5 metrics tests in benchmark LLMs #3355

@dgolubovicTT

Description

@dgolubovicTT

Measuring only PCC can't reveal fine accuracy problems in LLMs. We need to test our accuracy on actual dataset.
My suggestion is to use same dataset and metrics as in tt-metal models, just to be able to easily compare both perf and accuracy. They are using top1 and top5 percentages over some text corpus.

Metadata

Metadata

Assignees

Labels

benchmarkIssues related to performance benchmarks

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions