Skip to content

Commit e220bda

Browse files
docs: document gh workflow run for integration tests in ADDINGMODEL.md (#2783)
Co-authored-by: openhands <openhands@all-hands.dev>
1 parent d2a77ab commit e220bda

File tree

1 file changed

+29
-13
lines changed

1 file changed

+29
-13
lines changed

.github/run-eval/ADDINGMODEL.md

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -240,29 +240,44 @@ cd .github/run-eval
240240
MODEL_IDS="your-model-id" GITHUB_OUTPUT=/tmp/output.txt python resolve_model_config.py
241241
```
242242

243-
## Step 6: Run Integration Tests (Required Before PR)
243+
## Step 6: Create Draft PR
244244

245-
**Mandatory**: Integration tests must pass before creating PR.
245+
Push your branch and create a draft PR. Note the PR number returned - you'll need it for the integration tests.
246246

247-
### Via GitHub Actions
247+
## Step 7: Run Integration Tests
248248

249-
1. Push branch: `git push origin your-branch-name`
250-
2. Navigate to: https://github.com/OpenHands/software-agent-sdk/actions/workflows/integration-runner.yml
251-
3. Click "Run workflow"
252-
4. Configure:
253-
- **Branch**: Select your branch
254-
- **model_ids**: `your-model-id`
255-
- **Reason**: "Testing model-id"
256-
5. Wait for completion
257-
6. **Save run URL** - required for PR description
249+
Trigger integration tests on your PR branch:
250+
251+
```bash
252+
gh workflow run integration-runner.yml \
253+
-f model_ids=your-model-id \
254+
-f reason="Testing new model from PR #<pr-number>" \
255+
-f issue_number=<pr-number> \
256+
--ref your-branch-name
257+
```
258+
259+
Results will be posted back to the PR as a comment.
258260

259261
### Expected Results
260262

261263
- Success rate: 100% (or 87.5% if vision test skipped)
262264
- Duration: 5-10 minutes per model
263265
- Tests: 8 total (basic commands, file ops, code editing, reasoning, errors, tools, context, vision)
264266

265-
## Step 7: Create PR
267+
## Step 8: Fix Issues and Rerun (if needed)
268+
269+
If tests fail, see [Common Issues](#common-issues) below. After fixing:
270+
271+
1. Push the fix: `git add . && git commit && git push`
272+
2. Rerun integration tests with the same command from Step 7 (using the same PR number)
273+
274+
## Step 9: Mark PR Ready
275+
276+
When tests pass, mark the PR as ready for review:
277+
278+
```bash
279+
gh pr ready <pr-number>
280+
```
266281

267282
### Required in PR Description
268283

@@ -379,3 +394,4 @@ Fixes #[issue-number]
379394
- Recent model additions: #2102, #2153, #2207, #2233, #2269
380395
- Common issues: #2147 (hangs), #2137 (parameters), #2110 (vision), #2233 (variants), #2193 (preflight)
381396
- Integration test workflow: `.github/workflows/integration-runner.yml`
397+
- Integration tests can be triggered via: `gh workflow run integration-runner.yml --ref <branch>`

0 commit comments

Comments
 (0)