P17/scripts/benchmark execution#519

Draft

gaurav-chawla-2 wants to merge 62 commits intostagingfrom

p17/scripts/benchmark-execution

Commits on Feb 1, 2026

initial skeleton of the script

Nitin Vig
authored and
Nitin Vig
committed
Pipeline Script

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 3, 2026

Added scripts to evaluate following metrics for a benchmark:
sardug10
committed
updated folder structure

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 4, 2026

Fixed precommit errors and changed folder name
sardug10
committed
Added HumanEval script

Nitin Vig
authored and
Nitin Vig
committed
Merge branch 'p17/scripts/benchmark-execution' of https://github.com/The-School-of-AI/LLM into p17/scripts/benchmarks-scripts/indic_glue
sardug10
committed
Merge branch 'main' into p17/scripts/benchmark-execution
nitin-vig
authored
Merge branch 'p17/scripts/benchmark-execution' of https://github.com/The-School-of-AI/LLM into p17/scripts/benchmarks-scripts/indic_glue
sardug10
committed
Fixed precommit check
sardug10
committed
Added scripts to evaluate following metrics for Indic Glue benchmark
sardug10
authored
Merge branch 'main' into p17/scripts/benchmark-execution
nitin-vig
authored

Commits on Feb 6, 2026

added benchmark scripts for IndiGlue

Nitin Vig
authored and
Nitin Vig
committed
Merge branch 'main' into p17/scripts/benchmark-execution
nitin-vig
authored

Commits on Feb 7, 2026

Added OLMES pipeline

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 8, 2026

fixed colab exection

Nitin Vig
authored and
Nitin Vig
committed
fixed colab exection

Nitin Vig
authored and
Nitin Vig
committed
fixed colab exection

Nitin Vig
authored and
Nitin Vig
committed
fixing dependency conflicts

Nitin Vig
authored and
Nitin Vig
committed
fixing dependency conflicts

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 9, 2026

fixing dependency conflicts

Nitin Vig
authored and
Nitin Vig
committed
fixing dependency conflicts

Nitin Vig
authored and
Nitin Vig
committed
Added colab specific checks

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 10, 2026

Added colab specific code

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 11, 2026

Added Benhcmark plan

Nitin Vig
authored and
Nitin Vig
committed
added benchmark config

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 12, 2026

removed unused code

Nitin Vig
authored and
Nitin Vig
committed
updated pipelines for reporting and fast feedback

Nitin Vig
authored and
Nitin Vig
committed
updated pipelines for reporting and fast feedback

Nitin Vig
authored and
Nitin Vig
committed

Commits on Feb 15, 2026

removed code execution benhcmarks

Nitin Vig
authored and
Nitin Vig
committed
Merge remote changes and apply GPQA mirror fix
Nitin Vig
committed
updated config
Nitin Vig
committed

Commits on Feb 17, 2026

Added batch size as arg and updated eval-runner to use it
gaurav-chawla-2
committed
Merge pull request #494 from The-School-of-AI/p17/scripts/benchmark-execution-requirements
nitin-vig
authored
added worker support

Nitin Vig
authored and
Nitin Vig
committed
added multi worker support

Nitin Vig
authored and
Nitin Vig
committed
removed vllm
nitin-vig
committed
added batch capability
nitin-vig
committed
Fix the README and running instructions
gaurav-chawla-2
committed
Merge pull request #495 from The-School-of-AI/p17/scripts/benchmark-eecution-readme
nitin-vig
authored

Commits on Feb 21, 2026

Commits on Feb 25, 2026

Merge branch 'staging' into p17/scripts/benchmark-execution
nitin-vig
authored