Evals improvements

[cte]

Context

• Improve the cli logging; in particular reduce the noise and surface only the most important things.
• Fix the hanging unit-test process issue by killing the entire process tree (using ps-tree) instead of relying on the execa timeout.
• Stop broadcasting all task messages to the web app; this was a half-baked feature and was only going to be useful for Docker-based eval runs.
• Update the task metrics as soon as they are broadcast.
• Update the PID list for the eval run every 30s (Hannes found it confusing when only a single PID was listed).
• Sync types.
• Install the language server VSCode extensions in the eval setup script.

Implementation

Screenshots

before	after

How to Test

Get in Touch

---

Important

Enhance evals by improving CLI logging, fixing test process handling, updating task metrics, and refining setup script.

• CLI Improvements:
  • Reduce logging noise in index.ts by removing unnecessary console.log statements.
  • Use ps-tree to kill entire process tree for unit tests in index.ts.
  • Adjust task start delay and timeouts in index.ts.
• Web App Enhancements:
  • Update task metrics immediately upon broadcast in run.tsx.
  • Remove task message broadcasting to web app in run.tsx.
  • Update PID list every 30s in use-process-tree.ts.
• Type and Schema Updates:
  • Sync types in roo-code.ts and schema.ts.
  • Update RooCodeSettings and GlobalSettings schemas in roo-code.ts.
• Setup Script:
  • Install VSCode extensions in setup.sh.
  • Sync database and validate OpenRouter API key in setup.sh.

^{This description was created by for b4ba4cb. It will automatically update as commits are pushed.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: b4ba4cb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ellipsis-dev]

Hello @cte,

Thank you for your contributions to the project! After reviewing the pull request, I noticed that it includes a variety of changes across different areas, such as CLI enhancements, web component refactoring, and settings updates. While these changes are all valuable, it might be beneficial to split them into smaller, more focused pull requests. This can help with easier review and testing.

Here are some suggestions on how the changes could be split:

1. CLI Enhancements: Include changes related to the CLI execution and process management, such as the addition of the ps-tree dependency and updates in index.ts.

2. Web Component Refactoring: Focus on changes in the web components, like the Run and TaskStatus components, and the integration of token usage metrics.

3. Settings and Configuration Updates: Group changes related to settings updates, such as those in schema.ts, roo-code-defaults.ts, and roo-code.ts.

4. Logging and Error Handling Improvements: Include changes that enhance logging and error handling, such as those in ipc/client.ts and ipc/server.ts.

5. Script and Setup Enhancements: Address changes in the setup script, including the VSCode CLI check and extension setup.

By splitting the pull request into these categories, it will be easier for reviewers to focus on specific areas and ensure thorough testing. Let me know if you have any questions or need further assistance!

Best regards,
Ellipsis