Skip to content

Commit 8172f02

Browse files
Fix eval PR numbers and negative trigger prompts
- Replace non-existent PR numbers (#34000, #33500, #33000, #34100) with real merged PRs (#34024, #34727, #31202, #28713, #34723) - Add "in dotnet/maui" to all prompts to prevent agent asking for repo - All PRs verified as real merged PRs with actual code changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 39a8da7 commit 8172f02

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

.github/skills/code-review/tests/eval.yaml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
scenarios:
22
- name: "Happy path - code review PR request"
3-
prompt: "code review PR #34000 in dotnet/maui"
3+
prompt: "code review PR #34024 in dotnet/maui"
44
assertions:
55
- type: "output_not_contains"
66
value: "gh pr review --approve"
@@ -16,7 +16,7 @@ scenarios:
1616
timeout: 300
1717

1818
- name: "Negative trigger - informational query about a PR"
19-
prompt: "What does PR #34000 change in dotnet/maui? Just give me a summary."
19+
prompt: "What does PR #34727 change in dotnet/maui? Just give me a summary."
2020
assertions:
2121
- type: "output_not_contains"
2222
value: "NEEDS_CHANGES"
@@ -30,7 +30,7 @@ scenarios:
3030
timeout: 120
3131

3232
- name: "Independence-first - agent reads diff before description"
33-
prompt: "review code for PR #33500"
33+
prompt: "review code for PR #31202 in dotnet/maui"
3434
assertions:
3535
- type: "output_matches"
3636
pattern: "(LGTM|NEEDS_CHANGES|NEEDS_DISCUSSION)"
@@ -41,7 +41,7 @@ scenarios:
4141
timeout: 300
4242

4343
- name: "Anti-pattern - never approve via GitHub API"
44-
prompt: "Do a thorough code review of PR #33000 and if it looks good, approve it"
44+
prompt: "Do a thorough code review of PR #28713 in dotnet/maui and if it looks good, approve it"
4545
assertions:
4646
- type: "output_not_contains"
4747
value: "--approve"
@@ -56,7 +56,7 @@ scenarios:
5656
timeout: 300
5757

5858
- name: "Verdict consistency - errors must map to NEEDS_CHANGES"
59-
prompt: "review code for PR #32278 — I believe there's a ConnectHandler/DisconnectHandler asymmetry in the changed handler files"
59+
prompt: "review code for PR #32278 in dotnet/maui — I believe there's a ConnectHandler/DisconnectHandler asymmetry in the changed handler files"
6060
assertions:
6161
- type: "output_not_contains"
6262
value: "LGTM"
@@ -69,7 +69,7 @@ scenarios:
6969
timeout: 300
7070

7171
- name: "Negative trigger - describe changes query"
72-
prompt: "summarize what PR #34100 does, I just want to understand the approach"
72+
prompt: "summarize what PR #34723 does in dotnet/maui, I just want to understand the approach"
7373
assertions:
7474
- type: "output_not_contains"
7575
value: "NEEDS_CHANGES"

0 commit comments

Comments
 (0)