Skip to content

Conversation

@justin808
Copy link
Member

@justin808 justin808 commented Nov 12, 2025

Summary

Fixes the /run-skipped-ci command which was broken because:

  1. It triggered ALL workflows including Pro tests
  2. The detect-changes job would run on the PR branch and see no Pro file changes
  3. Pro workflows would be skipped, defeating the purpose of the command

Changes

1. Run Only Minimum Dependency Tests

The command now only triggers workflows with minimum dependency configurations that are normally skipped on PRs:

  • main.yml: Ruby 3.2, Node 20, minimum dependencies
  • examples.yml: Ruby 3.2, minimum dependencies

It skips Pro workflows because they already run on all PRs (no need to re-run them).

2. Added Workflow Inputs

Added run_minimum_tests input to main.yml and examples.yml:

workflow_dispatch:
  inputs:
    run_minimum_tests:
      description: 'Run minimum dependency matrix (Ruby 3.2, Node 20)'
      required: false
      type: boolean
      default: false

3. Smart Matrix Exclusion

Updated matrix exclusion logic to respect the input:

  • When run_minimum_tests=true: Exclude latest matrix (run only minimum)
  • When run_minimum_tests=false: Exclude minimum on PRs (existing behavior)

Uses fromJSON() to avoid the empty string issue with conditional excludes:

exclude:
  # When run_minimum_tests is true, skip latest (run only minimum)
  - ${{ inputs.run_minimum_tests && fromJSON('{"ruby-version": "3.4", "node-version": "22", "dependency-level": "latest"}') || fromJSON('{}') }}
  # When run_minimum_tests is false, skip minimum on regular PRs
  - ${{ !inputs.run_minimum_tests && github.event_name == 'pull_request' && github.ref != 'refs/heads/master' && fromJSON('{"ruby-version": "3.2", "node-version": "20", "dependency-level": "minimum"}') || fromJSON('{}') }}

4. Improved PR Comments

The PR comment now clearly shows:

  • Which workflows are running (minimum dependency tests)
  • Which workflows are skipped (Pro tests, latest tests)
  • Why they're skipped (already run on all PRs)

Example:

🚀 **Skipped CI Tests Triggered**

✅ **Successfully triggered skipped CI tests**

**Running minimum dependency tests:**
- ✅ Main Tests
- ✅ Generator Tests

**Skipped (already run on PRs):**
- ⏭️  Pro Integration Tests
- ⏭️  Pro Package Tests

**What's running:**
- ✅ Minimum dependency versions (Ruby 3.2, Node 20)
- ✅ Generator tests with minimum dependencies

**Note:** Pro package tests and latest dependency tests are skipped because they already run on all PRs.

Testing

  • Test the workflow by commenting /run-skipped-ci on a PR
  • Verify only minimum dependency matrix runs
  • Check PR comment shows correct information

Related Issues

Closes #1999 (uses better matrix exclusion approach)

🤖 Generated with Claude Code


This change is Reviewable

justin808 and others added 7 commits November 10, 2025 19:03
This adds a `/run-full-ci` command that can be used in PR comments to run
the complete CI suite including tests normally skipped on PRs.

**New workflows:**
- `run-skipped-ci.yml` - Triggers on `/run-full-ci` comment, dispatches all CI workflows
- `pr-welcome-comment.yml` - Auto-comments on new PRs explaining the command

**Features:**
- Runs full test matrix (all Ruby/Node versions)
- Runs example generator tests
- Runs Pro package integration and unit tests
- Posts acknowledgment comment with workflow links
- Adds rocket reaction to trigger comment

**Documentation:**
- Added `.github/README.md` with comprehensive docs
- Explains why subset CI exists (fast feedback)
- Documents testing limitations (must be on master branch)
- Lists all available workflows and their purposes

This improves the PR workflow by letting contributors easily run full CI
on-demand without waiting for merge to master.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
This addresses critical security and reliability issues identified in the
initial implementation of the /run-full-ci comment trigger.

**Security Fixes:**
- Add permission check to restrict command to users with write access
- Prevents resource abuse from external contributors
- Posts informative message to unauthorized users

**Reliability Improvements:**
- Consolidate workflow triggers with better error handling
- Post detailed results comment listing succeeded/failed workflows
- Fail job if any workflows fail to trigger
- Add concurrency controls to prevent duplicate runs per PR
- Improve comment matching to require command at line start

**Workflow Updates:**
- Update main.yml, examples.yml to run minimum deps on workflow_dispatch
- Update Pro workflows to honor workflow_dispatch events
- Skip welcome comment for bot PRs (dependabot, renovate)

**Documentation:**
- Document security controls and access restrictions
- Document concurrency protection
- Explain workflow_dispatch behavior

These changes ensure the feature is secure, reliable, and provides
clear feedback when workflows fail to trigger.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
The previous approach using matrix.exclude with matrix context expressions
caused actionlint errors because the matrix context is not available in the
exclude section.

**Fix:**
- Move conditional logic to job-level if statements
- Check both event type and dependency level in single expression
- For regular PRs: only run latest versions
- For master/workflow_dispatch: run all versions including minimum

This properly filters matrix jobs without using unsupported contexts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Changes the PR comment command to match the workflow filename for consistency.

**Changes:**
- Update workflow trigger to listen for `/run-skipped-ci`
- Update welcome comment to show `/run-skipped-ci`
- Update README documentation to reference `/run-skipped-ci`
- Update workflow filename reference in docs

The command name now clearly indicates it runs the normally-skipped CI jobs
(minimum versions, examples, Pro tests).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
The previous approach of using matrix context in job-level if statements
caused actionlint errors. Fixed by using environment variables and step-level
conditionals instead.

**Solution:**
- Set SKIP_MINIMUM env var at job level using matrix context
- Add conditional `if: env.SKIP_MINIMUM != 'true'` to all steps
- First step exits early with helpful message on regular PRs
- Minimum versions only run on master or workflow_dispatch

This approach is actionlint-compliant and properly filters matrix jobs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Remove duplicate if conditions in examples.yml and main.yml that caused knip YAML parsing errors
- Replace SKIP_MINIMUM_ON_PR env var pattern with cleaner matrix-level exclude approach
- Add workflow verification in run-skipped-ci.yml to confirm workflows actually start
- Improve error handling with detailed status reporting showing verified/pending/failed workflows
- Fix actionlint validation errors

This eliminates inefficient step-level conditionals and ensures workflows are properly filtered at the job level, making them more efficient and easier to maintain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
The /run-skipped-ci command was triggering ALL workflows, which caused
detect-changes to skip them (since they run on the PR branch with no
Pro file changes). Now it correctly:

1. Only triggers workflows with minimum dependency matrix
   - main.yml: Ruby 3.2, Node 20
   - examples.yml: Ruby 3.2

2. Skips workflows that already run on all PRs
   - pro-integration-tests.yml (always runs)
   - pro-package-tests.yml (always runs)

3. Uses workflow_dispatch inputs to control matrix
   - Added run_minimum_tests input to main.yml and examples.yml
   - When true, excludes latest matrix (3.4/22)
   - When false, excludes minimum matrix on PRs (existing behavior)

4. Updates PR comment to show what ran and what was skipped
   - Clear explanation of which tests are running
   - Note about why Pro tests are skipped

This fixes the issue where detect-changes would stop the workflow
because it detected no Pro file changes on the PR branch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Nov 12, 2025

Warning

Rate limit exceeded

@justin808 has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 7 minutes and 6 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 8d2f1dd and bf1c628.

📒 Files selected for processing (7)
  • .github/README.md (1 hunks)
  • .github/workflows/examples.yml (2 hunks)
  • .github/workflows/main.yml (4 hunks)
  • .github/workflows/pr-welcome-comment.yml (1 hunks)
  • .github/workflows/pro-integration-tests.yml (3 hunks)
  • .github/workflows/pro-package-tests.yml (3 hunks)
  • .github/workflows/run-skipped-ci.yml (1 hunks)
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch justin808/run-skipped-ci-command

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@justin808
Copy link
Member Author

Closing in favor of PR with clean branch (no merge conflicts)

@justin808 justin808 closed this Nov 12, 2025
@justin808 justin808 deleted the justin808/run-skipped-ci-command branch November 12, 2025 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Fix workflow matrix exclusion logic with empty strings

2 participants