Skip to content

Conversation

@canrobins13
Copy link
Contributor

@canrobins13 canrobins13 commented May 29, 2025

Description

Allows for headless execution of evals in a Docker container.
Note that headless evals can only run without concurrency for now; attempting to run with concurrency introduces errors.
Future work will include integrating headless evals with a cloud runner and sharding across jobs


Important

Adds Docker support for headless evals, updates CLI for Docker-specific conditions, and includes Docker scripts in package.json.

  • Docker Support:
    • Adds Dockerfile for setting up a headless eval environment with necessary dependencies.
    • Introduces .dockerignore to exclude unnecessary files from Docker build.
  • CLI Changes:
    • Updates runExercise in index.ts to handle Docker environment, including concurrency checks and command adjustments.
    • Adds error handling for concurrency greater than 1 in Docker.
  • Scripts:
    • Updates package.json with Docker-related scripts: docker:build, docker:run, docker:start, docker:shell, docker:stop, docker:rm, and docker:clean.

This description was created by Ellipsis for a302414. You can customize this summary. It will automatically update as commits are pushed.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label May 29, 2025
@canrobins13 canrobins13 marked this pull request as ready for review May 29, 2025 22:18
@canrobins13 canrobins13 requested a review from mrubens as a code owner May 29, 2025 22:18
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels May 29, 2025
@canrobins13 canrobins13 changed the title [Evals] Try to resurrect headless evals with Docker [Evals] Headless evals with Docker May 29, 2025
@canrobins13 canrobins13 merged commit 6762b57 into main May 29, 2025
9 checks passed
@canrobins13 canrobins13 deleted the canyon/headless-evals branch May 29, 2025 22:36
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap May 29, 2025
@github-project-automation github-project-automation bot moved this from Triage to Done in Roo Code Roadmap May 29, 2025
SmartManoj pushed a commit to SmartManoj/Raa-Code that referenced this pull request Jun 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants