Improve evaluate-pr-tests workflow: fork support, access gating, dry-run by PureWeen · Pull Request #34678 · dotnet/maui

PureWeen · 2026-03-26T16:46:38Z

Description

Overhauls the copilot-evaluate-tests gh-aw workflow to properly support fork PRs, add access gating, and improve the agent experience.

Problem

Fork PRs triggered via pull_request do not have access to repository secrets (COPILOT_GITHUB_TOKEN). This is a GitHub Actions platform limitation — secrets are scrubbed for fork pull_request events to prevent exfiltration. The forks: ["*"] compiler flag only removes the activation gate but does not provision secrets.

Solution

Switch to pull_request_target (runs in base repo context with secrets), gate on author access level, and add dry-run/noop flows.

Trigger Behavior

Trigger	When it fires	Who can use it
`pull_request_target`	PR opened/updated touching `src//tests/`	Auto for write-access authors; skipped for external contributors
`issue_comment`	`/evaluate-tests` comment on a PR	Maintainers only (admin/maintain/write)
`workflow_dispatch`	Manual trigger from Actions tab	Maintainers only; rejects fork PRs

Access Matrix

Permission Level	`pull_request_target`	`/evaluate-tests` comment	`workflow_dispatch`
admin / maintain / write	✅ Auto-runs	✅ Can trigger	✅ Can trigger
triage / read	❌ `pre_activation` blocks	❌ `pre_activation` blocks	❌ `pre_activation` blocks
external / fork	❌ `pre_activation` blocks	❌ `pre_activation` blocks	❌ PS1 rejects fork

External contributors' PRs require a maintainer to comment /evaluate-tests to trigger evaluation.

Fork Handling by Flow

Flow	Handling
`pull_request_target`	gh-aw `pre_activation` gates on write access; platform checks out PR in sandboxed container
`issue_comment`	`pre_activation` gates on commenter's write access; platform handles checkout in sandbox
`workflow_dispatch`	PS1 script rejects fork PRs (`isCrossRepository` check) + verifies author write access; restores agent infrastructure from base branch

Changes

File	Change	Reason
`.md`	`pull_request` → `pull_request_target`	Secrets available for fork PRs
`.md`	Removed `forks: ["*"]`	Not needed with `pull_request_target` (no fork gate to remove)
`.md`	Removed `ready_for_review` trigger type	Not supported by `pull_request_target`; draft PRs are already filtered by the `draft == false` condition
`.md`	Added `suppress_comment` input	Dry-run mode for testing without posting comments
`.md`	Added noop guidance in agent prompt	Agent calls `noop` with reason instead of silently exiting
`.md`	Updated fork fallback message	Removed stale `workflow_dispatch` recommendation for fork users
`.ps1`	Added fork + write-access checks	Defense-in-depth for `workflow_dispatch` path
`.ps1`	Updated comments and logging	Clarified security model and added diagnostic output

Security Model

pull_request_target runs the workflow from the base branch — fork PRs cannot alter the workflow file
The gh-aw agent runs in a sandboxed container with all credentials scrubbed
pre_activation gates all triggers on admin/maintainer/write roles
workflow_dispatch has additional defense-in-depth: PS1 rejects forks + checks author permissions
Safe outputs limited to 1 comment per run
Agent infrastructure (skills, instructions) always restored from base branch

Validation

Tested on PureWeen/maui with all three trigger paths:

Test	Run ID	Result
Same-repo PR (#22)	23603776593	✅ All green
Fork PR via `workflow_dispatch` (#23)	23605610535	✅ All green
Fork PR via `pull_request_target` (#23)	23606033617	✅ All green

- Change trigger from pull_request to pull_request_target so fork PRs have access to secrets (COPILOT_GITHUB_TOKEN) - Add roles: all to allow fork contributors (who have read permission) to trigger the workflow - Remove forks: ["*"] (not needed with pull_request_target) - Remove ready_for_review type (not supported by gh-aw for pull_request_target) - Update if condition and gate step to reference pull_request_target Validated on PureWeen/maui: - Same-repo PR: all green (run 23603776593) - Fork PR via workflow_dispatch: all green (run 23605610535) - Fork PR via pull_request_target: all green (run 23606033617) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-26T16:46:49Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34678

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34678"

Copilot

Pull request overview

Updates the gh-aw Evaluate PR Tests workflow triggers so fork PRs can be evaluated (by switching from pull_request to pull_request_target) while keeping the workflow’s gating/conditions aligned with the new event.

Changes:

Switched workflow trigger from pull_request to pull_request_target and updated related if: conditions.
Updated the gate step condition to run under pull_request_target.
Updated the compiled .lock.yml to reflect the new trigger configuration.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`.github/workflows/copilot-evaluate-tests.md`	Moves evaluation to `pull_request_target` and updates workflow conditions/gate accordingly.
`.github/workflows/copilot-evaluate-tests.lock.yml`	Regenerated compiled workflow reflecting the trigger/condition changes.

Copilot · 2026-03-26T16:50:16Z

.github/workflows/copilot-evaluate-tests.md

+  roles: all



roles: all is indented under on: (same level as workflow_dispatch), so the gh-aw compiler treats it as part of the event config instead of a top-level workflow setting. In the compiled lock file this ends up commented out and the membership/role gate is effectively removed, meaning anyone (including arbitrary /evaluate-tests issue comments) can trigger a pull_request_target run that has access to repository secrets. Unindent roles: all to the top level (aligned with on:/if:) and re-run gh aw compile so the intended role gating is applied.

Suggested change

roles: all

roles: all

This suggestion is incorrect. The gh aw compile command explicitly requires roles under on: — placing it at the top level produces:

error: Unknown property: roles. Did you mean: bots, jobs, tools. 'roles' belongs under 'on'

The lock file comment # roles: all # Roles processed as role check in pre-activation job is the compiler's standard format. With roles: all, the pre_activation role check is intentionally omitted (everyone is allowed). This was validated on PureWeen/maui where a fork contributor with read permission successfully triggered the workflow (run 23606033617).

@copilot thoughts?

@PureWeen's explanation is correct. The reviewer's suggestion has two issues:

roles: all placement: The gh-aw compiler requires roles under on: — placing it at the top level produces a compile error. The lock file line # roles: all # Roles processed as role check in pre-activation job is the compiler's standard format confirming the setting was processed. With roles: all, the compiler intentionally skips the pre-activation role gate (because everyone is allowed), which is exactly the intended behavior for fork contributors.

The fork guard suggestion (head.repo.fork == false in the activation if:): This would completely defeat the purpose of this PR, which is specifically to allow fork PRs to be evaluated.

The security model is sound:

pull_request_target runs the workflow from the base branch — forks cannot alter it

The agent container has all credentials scrubbed

safe-outputs limits the agent to posting 1 comment

The workflow is read-only (evaluate tests, post a comment)

No changes needed here.

Copilot · 2026-03-26T16:50:17Z

.github/workflows/copilot-evaluate-tests.lock.yml

+  pull_request_target:
    paths:
    - src/**/tests/**
    - src/**/test/**
    types:
    - opened
    - synchronize
    - reopened
-    - ready_for_review
+  # roles: all # Roles processed as role check in pre-activation job
  workflow_dispatch:


The compiled lock file shows roles: all commented out under the on: block, which indicates the roles setting was not applied. This also coincides with the removal of the pre_activation/membership check job, so the workflow no longer gates who can run it. Fix by moving roles: to the correct top-level location in the .md frontmatter and re-compiling so the lock file includes the intended role check behavior.

Copilot · 2026-03-26T16:50:17Z

.github/workflows/copilot-evaluate-tests.lock.yml

-    needs: pre_activation
    if: >
-      (needs.pre_activation.outputs.activated == 'true') && ((github.event_name == 'pull_request' && github.event.pull_request.draft == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&
+      (github.event_name == 'pull_request_target' && github.event.pull_request.draft == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&


jobs.activation no longer depends on a role/membership gate (needs: pre_activation and the needs.pre_activation.outputs.activated check are gone). With pull_request_target, this means the workflow can run with secrets for any matching PR/comment, which is a significant security/cost exposure. After fixing the roles placement in the .md, ensure the compiled lock restores the gating (or add an explicit guard) before allowing activation to proceed.

Suggested change

(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&

(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&

The workflow_dispatch step runs with GITHUB_TOKEN and checks out PR code. Restrict it to only process PRs from authors with write/maintain/admin access, preventing checkout of untrusted fork code in a privileged context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move the PR author permission check from inline workflow bash into the shared Checkout-GhAwPr.ps1 script. Any gh-aw workflow using this script now automatically gates on the PR author having write/maintain/admin access before checking out code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fork PRs are handled by pull_request_target (platform checkout in sandboxed container). The workflow_dispatch path should only process same-repo PRs from authors with write access. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restoring only skills/, instructions/, and copilot-instructions.md left other .github/ subdirs (pr-review/, scripts/, workflows/) from the PR branch. Restore the entire .github/ directory for complete coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Instead of deleting .github/ and restoring from main, merge the base branch into the PR branch after checkout. This produces the same state as a pull_request merge commit: PR changes + latest main. If the PR modifies a skill, the PR version wins; otherwise main's version is used. This lets contributors iterate on skills via workflow_dispatch while keeping everything else current. On merge conflict, falls back to the PR branch as-is with a warning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- pull_request_target: only auto-runs for OWNER/MEMBER/COLLABORATOR - issue_comment: /evaluate-tests only accepted from OWNER/MEMBER/COLLABORATOR - workflow_dispatch: unchanged - External PRs require maintainer /evaluate-tests comment to trigger Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Revert merge strategy to targeted git checkout (works in shallow clones) - Remove roles:all, restore gh-aw pre_activation with write-level checks - Remove author_association from if: (gh-aw handles access gating) - Update fork fallback message to remove stale workflow_dispatch advice Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add suppress_comment input for workflow_dispatch dry-run (evaluate without posting comment) - Add explicit noop guidance so the agent uses it instead of silently exiting - Update posting results section to respect dry-run mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Prevents silent fork check bypass when gh returns empty/malformed JSON — $null.isFork evaluates to $false in PowerShell, which would let the fork check pass incorrectly. Note: ready_for_review cannot be added to pull_request_target types yet — gh-aw compiler doesn't include it in the allowed type list. Filed as a known gap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 26, 2026 16:46

Copilot started reviewing on behalf of PureWeen March 26, 2026 16:47 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

PureWeen marked this pull request as draft March 26, 2026 17:07

github-actions bot and others added 5 commits March 26, 2026 12:59

PureWeen marked this pull request as ready for review March 26, 2026 20:29

Copilot started work on behalf of PureWeen March 26, 2026 20:41 View session

dotnet deleted a comment from github-actions bot Mar 26, 2026

Copilot finished work on behalf of PureWeen March 26, 2026 20:49

github-actions bot mentioned this pull request Mar 27, 2026

[repo-status] Daily Repo Status - March 27, 2026 #34694

Closed

github-actions bot and others added 3 commits March 27, 2026 09:26

PureWeen changed the title ~~Switch evaluate-pr-tests to pull_request_target for fork PR support~~ Improve evaluate-pr-tests workflow: fork support, access gating, dry-run Mar 30, 2026

kubaflo previously approved these changes Mar 30, 2026

View reviewed changes

PureWeen marked this pull request as draft March 30, 2026 23:15

PureWeen marked this pull request as ready for review April 2, 2026 13:58

PureWeen dismissed kubaflo’s stale review via 17c0f27 April 2, 2026 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve evaluate-pr-tests workflow: fork support, access gating, dry-run#34678

Improve evaluate-pr-tests workflow: fork support, access gating, dry-run#34678
PureWeen wants to merge 10 commits intomainfrom
fix/evaluate-tests-fork-support

PureWeen commented Mar 26, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

PureWeen Mar 26, 2026

Uh oh!

PureWeen Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false) \|\| github.event_name == 'workflow_dispatch' \|\| (github.event_name == 'issue_comment' &&
	(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false) \|\| github.event_name == 'workflow_dispatch' \|\| (github.event_name == 'issue_comment' &&

Conversation

PureWeen commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Problem

Solution

Trigger Behavior

Access Matrix

Fork Handling by Flow

Changes

Security Model

Validation

Uh oh!

github-actions bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

PureWeen Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

PureWeen Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

PureWeen commented Mar 26, 2026 •

edited

Loading

github-actions bot commented Mar 26, 2026 •

edited

Loading