Support more repos in benchmarks

[codegen-sh]

Implements multi-repository support for the grainchain benchmarks as requested in CG-18528.

Changes Made

Repository Support

• TypeScript: outline (existing)
• Python: Added requests and fastapi repositories

New Files

• benchmarks/configs/outline.json - TypeScript repository configuration
• benchmarks/configs/requests.json - Python HTTP library configuration
• benchmarks/configs/fastapi.json - Python web framework configuration
• benchmarks/scripts/multi_repo_benchmark_runner.py - Docker-based multi-repo runner
• benchmarks/scripts/grainchain_multi_repo_benchmark.py - Grainchain-based multi-repo runner (recommended)
• test_multi_repo_dockerfile.py - Multi-repo testing script

Features

✅ Per-repository output generation - Each repo gets its own results file
✅ Snapshot testing - Trivial changes and snapshots tested for each repo
✅ Multiple languages - TypeScript and Python support
✅ Flexible configuration - JSON configs for each repository
✅ Two implementations - Docker and grainchain-based runners

Usage

Run all repositories:

source .venv/bin/activate
python benchmarks/scripts/grainchain_multi_repo_benchmark.py

Run specific repository:

source .venv/bin/activate  
python benchmarks/scripts/grainchain_multi_repo_benchmark.py --repo requests

Test setup:

source .venv/bin/activate
python test_multi_repo_dockerfile.py

Results Structure

benchmarks/results/
├── grainchain_multi_benchmark_TIMESTAMP.json  # Overall results
├── requests_grainchain_benchmark_TIMESTAMP.json  # Per-repo results
├── fastapi_grainchain_benchmark_TIMESTAMP.json
└── outline_grainchain_benchmark_TIMESTAMP.json

Addresses: CG-18511

---

💻 View my work • About Codegen

[linear]

CG-18528 Support more repos in benchmarks

CG-18511

• In the grainchain repo (see project context)
• Right now, we just try snapshotting and cloning with the outline repo
• Add a few more well-known repos in there, not too huge though
  • outline is TS
  • Try 1-2 with python, maybe airflow? or something that has pure python deps.
  • Run the benchmarks again!
  • Make sure that the output is generated on a per-repo basis
  • Make sure the snapshots get tested!
    • see test_dockerfile_simple for example