Skip to content

README.md

Latest commit

 

History

History
103 lines (75 loc) · 4.12 KB

README.md

File metadata and controls

103 lines (75 loc) · 4.12 KB

LLM Rust Python Compiler (Sonnet Run)

This workspace is a SWE-AF generated Rust/RustPython compiler focused on production-style Python execution: structured output, safety controls, and strong CLI performance.

Performance Headline

Steady-state performance: 88x to 602x faster than CPython subprocess (253.8x geometric mean across 5 snippets). Cold subprocess startup is 2.27x faster on average; warm-path reaches 31.8k ops/s.

  • 88x to 602x steady-state range
  • 253.8x geometric mean
  • 31.8k ops/s peak throughput

For repeated snippet execution, Rust warm median latency is 0.032 to 0.235 ms across the benchmark set.

What This Means

  • 253.8x is computed as CPython cold mean / Rust warm median.
  • This is the right comparison for repeated agent/LLM workloads, where process startup is amortized and execution-loop latency dominates.
  • This artifact was produced through autonomous multi-agent planning, implementation, review, merge, and verification with traceable logs under .artifacts*.

Full Benchmark Table (All 5 Snippets)

Method:

  • CPython cold: python3 -c <snippet> subprocess timing (30 runs, local machine).
  • Rust cold: target/release/llm-pyexec-cli subprocess timing (30 runs, local machine).
  • Rust warm: Criterion warm_throughput median from target/criterion.
  • Steady-state X: CPython cold mean / Rust warm median.

Cold speedup is process startup comparison; steady-state X is the execution-loop impact.

Snippet CPython cold mean (ms) Rust CLI cold mean (ms) Cold speedup (X) Rust warm median (ms) Steady-state X Rust warm throughput (ops/s)
bench_01 19.429 8.714 2.18x 0.144 135.0x 6,936
bench_02 18.975 8.442 2.26x 0.032 602.3x 31,807
bench_03 19.079 8.494 2.26x 0.060 315.5x 16,604
bench_04 19.041 8.534 2.23x 0.041 465.0x 24,408
bench_05 20.717 8.706 2.30x 0.235 88.3x 4,250

Footprint / Efficiency

  • Native release CLI binary size: 10.79 MiB (target/release/llm-pyexec-cli).
  • Measured max RSS during cold runs (8-sample mean across snippets):
    • Rust CLI: ~22.53 MB
    • CPython subprocess: ~16.32 MB

The result is a compact binary with predictable memory footprint and strong latency/throughput behavior for repeated LLM-style snippet execution.

Cost (From Build Logs)

Every agent harness runs Claude- Sonnet. From .artifacts-opt-100x-cli-20260217-162350/logs/*.jsonl, summing cost_usd where event == "end":

  • Total cost: $53.482703
  • Counted end events: 173
  • Cost reflects full autonomous build + review pipeline, not just one benchmark run.

Why This Looks Production-Grade

  • Multi-agent implementation + review + merge flow with artifact trail under .artifacts*.
  • Rust API returns typed execution results (stdout/stderr/return/error/duration).
  • Safety controls are implemented (module allowlist, timeout path, output limits).
  • Benchmark suite and correctness integration tests are included in-repo.

Build Request Used

curl -X POST http://localhost:8080/api/v1/execute/async/swe-planner.build \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "repo_path": "/workspaces/llm-rust-python-compiler-sonnet-20260217-133731",
      "goal": "Build end-to-end rust based python compiler CLI so execution is dramatically faster than CPython for short Python snippets, while preserving standard-library support and correctness.",
      "config": {
        "runtime": "claude_code",
        "models": {
          "default": "sonnet"
        }
      }
    }
  }'

Track:

curl http://localhost:8080/api/v1/executions/exec_20260217_105351_240cdrrj

Reproduce

# Build native CLI
cargo build --release -p llm-pyexec-cli

# Warm-path criterion benchmarks
cargo bench --bench pyexec_bench -- warm_throughput

# Optional: cold-start criterion (requires explicit CLI path)
LLMPYEXEC_CLI_PATH="$PWD/target/release/llm-pyexec-cli" cargo bench --bench pyexec_bench -- cold_start
LLMPYEXEC_CLI_PATH="$PWD/target/release/llm-pyexec-cli" cargo bench --bench pyexec_bench -- cpython_cold_start