Skip to content

Conversation

Copilot
Copy link

@Copilot Copilot AI commented Jul 16, 2025

The test runner was incorrectly running all samples that start with a given prefix instead of just the exact sample that changed. For example, when nextjs-static was added in PR #417, the test runner ran tests for all samples starting with nextjs including nextjs, nextjs-blog, nextjs-claude, nextjs-cv, nextjs-documentation, and nextjs-postgres.

Root Cause

The issue was in the GitHub Actions workflow deploy-changed-samples.yml where sample names from changed_samples.txt were passed directly to the loadtest tool as regex patterns. Since the loadtest tool treats the -s parameter as regular expressions, unescaped sample names like nextjs-static would match any sample containing the substring nextjs.

Solution

Modified the workflow to:

  1. Escape regex special characters in sample names using awk '{gsub(/[.*+?^${}()|[\]\\]/, "\\\\&")}'
  2. Anchor patterns with ^ and $ to ensure exact matching: ^sample-name$

This ensures that when a sample like nextjs-static changes, only that exact sample is tested, not all samples with nextjs in their name.

Testing

Verified the fix works correctly:

  • Before: Pattern nextjs matched 6 samples (nextjs, nextjs-blog, nextjs-claude, nextjs-cv, nextjs-documentation, nextjs-postgres)
  • After: Pattern ^nextjs$ matches only 1 sample (nextjs)
  • ✅ Multiple samples work correctly when multiple files change
  • ✅ Intentional regex patterns still work (e.g., python.* for all python samples)

Fixes #418.


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

@Copilot Copilot AI changed the title [WIP] Test runner runs too many tests Fix test runner running too many tests due to unescaped regex patterns Jul 16, 2025
@Copilot Copilot AI requested a review from lionello July 16, 2025 05:42
Copilot finished work on behalf of lionello July 16, 2025 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test runner runs too many tests
2 participants