Skip to content

Conversation

@cte
Copy link
Collaborator

@cte cte commented Jan 31, 2025

Description

Explainer video: https://www.youtube.com/watch?v=iidPH5KsNog
Exercises + web-ui: https://github.com/cte/Roo-Code-Benchmark
Inspiration: https://aider.chat/2024/12/21/polyglot.html#the-polyglot-benchmark

Additional context

Related Issues

Reviewers


Important

Introduces Aider-inspired polyglot benchmarks with Docker setup, CLI, and environment configuration, supporting multiple programming languages.

  • Benchmark System:
    • Introduces Aider-inspired polyglot benchmarks supporting C++, Go, Java, JavaScript, Python, and Rust.
    • Adds benchmark directory with setup scripts, prompts, and configuration files.
    • Implements CLI in benchmark/src/cli.ts for running benchmarks.
  • Docker and Environment:
    • Adds Dockerfile and .dockerignore for setting up a Docker environment.
    • Includes .env.local.sample for environment variable configuration.
  • Scripts and Configuration:
    • Adds package.json in benchmark for managing dependencies and scripts.
    • Updates root package.json to include benchmark-related scripts.
    • Modifies knip.json and .vscodeignore to include benchmark paths.
  • API and Core Changes:
    • Updates src/core/Cline.ts and src/exports/api.ts to handle new benchmark tasks.
    • Adds token usage tracking in src/exports/roo-code.d.ts and src/shared/getApiMetrics.ts.

This description was created by Ellipsis for f108dfa. It will automatically update as commits are pushed.

@changeset-bot
Copy link

changeset-bot bot commented Jan 31, 2025

🦋 Changeset detected

Latest commit: e2bdce0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
roo-cline Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@hannesrudolph hannesrudolph moved this to To triage in Roo Code Roadmap Mar 5, 2025
@hannesrudolph hannesrudolph moved this from To triage to PR - Needs Approval in Roo Code Roadmap Mar 6, 2025
@cte cte changed the title Start building out Aider-style benchmark infra Aider-inspired polyglot benchmarks Mar 7, 2025
@cte cte force-pushed the cte/benchmarks branch from 4ff2a0f to 1f50036 Compare March 7, 2025 21:20
@mrubens mrubens moved this from PR [Unverified] to PR [Greenlit] in Roo Code Roadmap Mar 10, 2025
@cte cte force-pushed the cte/benchmarks branch 5 times, most recently from 8a36522 to f8a2828 Compare March 14, 2025 09:31
@cte cte marked this pull request as ready for review March 14, 2025 09:33
@cte cte requested a review from mrubens as a code owner March 14, 2025 09:33
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. enhancement New feature or request labels Mar 14, 2025
@ellipsis-dev
Copy link
Contributor

ellipsis-dev bot commented Mar 14, 2025

This pull request is quite large, with 27 files changed and over 3000 lines added. It includes a variety of changes such as new features, documentation updates, and refactoring across different areas like Dockerfiles, environment variables, CLI scripts, and coding exercise instructions.

To improve the review process and maintainability, it might be beneficial to split this pull request into smaller, more focused ones. Here are some suggestions on how the changes could be grouped:

  1. Environment Setup and Configuration: This could include changes related to Dockerfiles, environment variables, and any setup scripts.
  2. Documentation: All changes related to documentation, such as README updates and coding exercise instructions, could be grouped together.
  3. CLI and Automation Scripts: Changes related to CLI scripts and automation tasks could form another pull request.
  4. Refactoring and Code Improvements: Any refactoring or code improvements that are not directly related to new features could be separated.

Please consider splitting the pull request accordingly to facilitate a more efficient review process. Thank you!

@KJ7LNW
Copy link
Contributor

KJ7LNW commented Mar 14, 2025

neat! see also: #532 #1614

@cte cte force-pushed the cte/benchmarks branch from 9435771 to bf1938d Compare March 16, 2025 18:36
@cte cte force-pushed the cte/benchmarks branch from ec152bb to 68591c7 Compare March 17, 2025 17:02
@cte cte force-pushed the cte/benchmarks branch from c84d6ab to ecda6ce Compare March 17, 2025 17:05
@cte cte force-pushed the cte/benchmarks branch from 427ca04 to 4ceb1eb Compare March 17, 2025 17:08
@cte cte force-pushed the cte/benchmarks branch from 4ceb1eb to f108dfa Compare March 17, 2025 17:09
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Mar 17, 2025
@cte cte merged commit 6d591fa into main Mar 17, 2025
12 checks passed
@cte cte deleted the cte/benchmarks branch March 17, 2025 17:17
@github-project-automation github-project-automation bot moved this from PR [Greenlit] to Done in Roo Code Roadmap Mar 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request lgtm This PR has been approved by a maintainer size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants