Skip to content

Commit 39d7f55

Browse files
authored
Update evals README.md (RooCodeInc#4420)
* Update README.md * Update README.md
1 parent f61b4e4 commit 39d7f55

File tree

1 file changed

+22
-0
lines changed

1 file changed

+22
-0
lines changed

packages/evals/README.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,28 @@ After clicking "Launch" you should find that a "controller" container has spawne
4747
The web app's UI should update in realtime with the results of the eval run:
4848
<img width="1053" alt="Screenshot 2025-06-05 at 12 14 52 PM" src="https://github.com/user-attachments/assets/6fe3b651-0898-4f14-a231-3cc8d66f0e1f" />
4949

50+
## Resource Usage
51+
52+
If you want to run evals with high parallelism by increasing the concurrency you need to be mindful of your Docker resource limits.
53+
54+
We've found the following formula to be helpful in practice:
55+
56+
```
57+
Memory Limit = 3GB * concurrency
58+
CPU Limit = 2 * concurrency
59+
```
60+
61+
The memory and CPU limits can be set from the "Resources" section of the Docker Desktop settings:
62+
63+
<img width="996" alt="Screenshot 2025-06-06 at 8 54 24 AM" src="https://github.com/user-attachments/assets/a1cbb27d-b09c-450c-9fa8-b662c0537d48" />
64+
65+
## Stopping
66+
67+
To stop an evals run early you can simply stop the "controller" container using Docker Desktop. This will prevent any new task containers from being spawned. You can optionally stop any existing task containers immediately or let them finish their current tasks at which point they will exit.
68+
69+
<img width="1302" alt="Screenshot 2025-06-06 at 9 00 41 AM" src="https://github.com/user-attachments/assets/a9d4725b-730c-441a-ba24-ac99f9599ced" />
70+
71+
5072
## Advanced Usage / Debugging
5173

5274
The evals system runs VS Code headlessly in Docker containers for consistent, reproducible environments. While this design ensures reliability, it can make debugging more challenging. For debugging purposes, you can run the system locally on macOS, though this approach is less reliable due to hardware and environment variability.

0 commit comments

Comments
 (0)