Skip to content

Add Opt-In Expensive Tests Triggered by PR Comment #766

@bobzhang

Description

@bobzhang

Problem

Some tests in our CI are expensive (long-running, resource-heavy, or
require special secrets).
Running them for every PR is inefficient and unnecessary.

Goal

Enable opt‑in expensive test runs by commenting a slash-command on a
PR, for example:

/run-expensive-tests

This should:

  • Trigger a GitHub Actions workflow via issue_comment
  • Detect that the comment is on a PR
  • Verify the comment text starts with /run-expensive-tests
  • Check out the PR merge ref
  • Run the expensive test suite only on demand

Proposed Workflow (GitHub Actions)

Create .github/workflows/expensive-tests-on-comment.yml:

name: Expensive Tests (On Comment)

on:
  issue_comment:
    types: [created]

jobs:
  expensive-tests:
    # Must be a PR comment AND contain the trigger command
    if: >
      github.event.issue.pull_request &&
      startsWith(github.event.comment.body, '/run-expensive-tests')
    runs-on: ubuntu-latest

    steps:
      - name: Extract PR number
        run: echo "PR_NUMBER=${{ github.event.issue.number }}" >> $GITHUB_ENV

      - name: Checkout PR merge commit
        uses: actions/checkout@v4
        with:
          ref: refs/pull/${{ env.PR_NUMBER }}/merge

      - name: Run expensive tests
        run: ./scripts/run_expensive_tests.sh

Notes / Best Practices

  • issue_comment does fire for PR comments (PRs are issues
    internally)
  • github.event.issue.pull_request filters out non‑PR comments
  • This avoids running the expensive suite on every PR automatically
  • For repos needing extra safety:
    • restrict execution to team members:

      if: github.event.comment.author_association == 'MEMBER'
  • If the PR is from a fork:
    • secrets will not be available (GitHub restriction)
    • but compute-only tests still run fine

Acceptance Criteria

  • Team members can trigger expensive CI by commenting
    /run-expensive-tests
  • The workflow runs only when explicitly invoked
  • Regular fast CI remains unchanged
  • Documentation added to CONTRIBUTING.md

Next Steps

  • Add the workflow file
  • Add label/documentation for contributors
  • (Optional) Add bot reaction or status comment to PR

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions