Add test splitting infrastructure with auto-detection support and GitHub Actions integration

.claude/skills/quarantine-test/SKILL.md

@@ -0,0 +1,228 @@
+---
+name: Quarantine Test
+description: Quarantine flaky or failing tests by adding the QuarantinedTest attribute using the QuarantineTools. Use when tests are failing intermittently or need to be excluded from regular test runs.
+allowed-tools: Read, Grep, Bash
+---
+
+# Quarantine Test
+
+## Purpose
+
+This skill quarantines flaky or failing tests by using the QuarantineTools utility to add the `[QuarantinedTest]` attribute. Quarantined tests are excluded from regular test runs and instead run in a separate quarantine workflow.
+
+## When to Use
+
+Invoke this skill when:
+- A test is failing intermittently (flaky)
+- A test doesn't fail deterministically
+- User requests to "quarantine a test"
+- A test needs to be temporarily excluded from CI while being fixed
+- You need to mark a test as quarantined with an associated GitHub issue
+
+## Important Context
+
+- Quarantined tests are marked with `[QuarantinedTest("issue-url")]` attribute
+- They are NOT run in the regular `tests.yml` workflow
+- They ARE run in the separate `tests-quarantine.yml` workflow every 6 hours
+- A GitHub issue URL is REQUIRED when quarantining tests
+- The QuarantineTools utility handles adding the attribute and managing using directives automatically
+
+## Instructions
+
+### Step 1: Identify the Test to Quarantine
+
+1. Get the fully-qualified test method name in format: `Namespace.ClassName.TestMethodName`
+2. If user provides partial name, use Grep to find the complete qualified name:
+   ```bash
+   grep -rn "void TestMethodName" tests/
+   ```
+3. Extract the namespace and class name from the file to build the fully-qualified name
+
+Example: `Aspire.Hosting.Tests.DistributedApplicationTests.TestMethodName`
+
+### Step 2: Get or Create GitHub Issue
+
+1. Check if the user provided a GitHub issue URL
+2. If not, ask: "What is the GitHub issue URL for tracking this flaky test?"
+3. The URL must be a valid http/https URL (e.g., `https://github.com/dotnet/aspire/issues/1234`)
+4. If no issue exists, suggest creating one first to track the test failure
+
+### Step 3: Run QuarantineTools
+
+Execute the QuarantineTools with the quarantine flag, test name(s), and issue URL:
+
+```bash
+dotnet run --project tools/QuarantineTools/QuarantineTools.csproj -- --quarantine "Namespace.ClassName.TestMethodName" --url "https://github.com/org/repo/issues/1234"
+```
+
+**Multiple tests** can be quarantined at once:
+```bash
+dotnet run --project tools/QuarantineTools/QuarantineTools.csproj -- --quarantine "Namespace.Class.Test1" "Namespace.Class.Test2" --url "https://github.com/org/repo/issues/1234"
+```
+
+**Command line flags:**
+- `-q` or `--quarantine`: Quarantine mode (add attribute)
+- `-i` or `--url`: GitHub issue URL (required for quarantine)
+- Tests: Fully-qualified method names (space-separated)
+
+### Step 4: Verify the Changes
+
+1. The tool will output which files were modified:
+   ```
+   Updated 1 file(s):
+    - tests/ProjectName.Tests/TestFile.cs
+   ```
+
+2. Read the modified file to confirm the attribute was added correctly:
+   ```bash
+   grep -A 2 -B 2 "QuarantinedTest" tests/ProjectName.Tests/TestFile.cs
+   ```
+
+3. Verify that:
+   - The `[QuarantinedTest("issue-url")]` attribute appears above the test method
+   - The `using Aspire.TestUtilities;` directive was added to the file (if not already present)
+
+### Step 5: Build and Run Tests to Confirm
+
+1. Build the test project to ensure no compilation errors:
+   ```bash
+   dotnet build tests/ProjectName.Tests/ProjectName.Tests.csproj
+   ```
+
+2. Run the test project with quarantine filter to verify the test is now quarantined:
+   ```bash
+   dotnet test tests/ProjectName.Tests/ProjectName.Tests.csproj --no-build -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+   ```
+
+3. Confirm the quarantined test is NOT executed in the regular test run
+
+4. Optionally, verify the test CAN be run with the quarantine filter:
+   ```bash
+   dotnet test tests/ProjectName.Tests/ProjectName.Tests.csproj --no-build -- --filter-trait "quarantined=true"
+   ```
+
+### Step 6: Report Results
+
+Provide a clear summary:
+- Which test(s) were quarantined
+- The GitHub issue URL used
+- Which file(s) were modified
+- Confirmation that the test builds and is properly excluded from regular runs
+- Remind the user to commit the changes
+
+## Examples
+
+### Example 1: Quarantine a single flaky test
+
+User: "Quarantine the TestDistributedApplicationLifecycle test, it's flaky. Issue: https://github.com/dotnet/aspire/issues/5678"
+
+Actions:
+1. Find the fully-qualified name using Grep:
+   ```bash
+   grep -rn "void TestDistributedApplicationLifecycle" tests/
+   ```
+   Result: `tests/Aspire.Hosting.Tests/DistributedApplicationTests.cs`
+
+2. Determine qualified name: `Aspire.Hosting.Tests.DistributedApplicationTests.TestDistributedApplicationLifecycle`
+
+3. Run QuarantineTools:
+   ```bash
+   dotnet run --project tools/QuarantineTools/QuarantineTools.csproj -- -q "Aspire.Hosting.Tests.DistributedApplicationTests.TestDistributedApplicationLifecycle" -i "https://github.com/dotnet/aspire/issues/5678"
+   ```
+
+4. Verify output shows file was updated
+
+5. Build and test:
+   ```bash
+   dotnet build tests/Aspire.Hosting.Tests/Aspire.Hosting.Tests.csproj
+   dotnet test tests/Aspire.Hosting.Tests/Aspire.Hosting.Tests.csproj --no-build -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+   ```
+
+6. Report: "Quarantined test `TestDistributedApplicationLifecycle` with issue #5678. The test is now excluded from regular CI runs and will run in the quarantine workflow."
+
+### Example 2: Quarantine multiple related tests
+
+User: "These three tests in RedisTests are all flaky, quarantine them: TestRedisConnection, TestRedisCache, TestRedisCommands. Issue: https://github.com/dotnet/aspire/issues/9999"
+
+Actions:
+1. Find fully-qualified names (assuming they're in `Aspire.Components.Tests.RedisTests`)
+
+2. Run QuarantineTools with multiple tests:
+   ```bash
+   dotnet run --project tools/QuarantineTools/QuarantineTools.csproj -- -q "Aspire.Components.Tests.RedisTests.TestRedisConnection" "Aspire.Components.Tests.RedisTests.TestRedisCache" "Aspire.Components.Tests.RedisTests.TestRedisCommands" -i "https://github.com/dotnet/aspire/issues/9999"
+   ```
+
+3. Verify all three were modified
+
+4. Build and test the project
+
+5. Report: "Quarantined 3 tests from RedisTests with issue #9999."
+
+### Example 3: User provides short test name
+
+User: "Quarantine CanStartDashboard - it keeps timing out"
+
+Actions:
+1. Ask for GitHub issue: "What is the GitHub issue URL for tracking this flaky test?"
+
+2. User provides: "https://github.com/dotnet/aspire/issues/4321"
+
+3. Find the test:
+   ```bash
+   grep -rn "void CanStartDashboard" tests/
+   ```
+   Found in: `tests/Aspire.Dashboard.Tests/DashboardTests.cs`
+
+4. Read the file to determine namespace: `Aspire.Dashboard.Tests`
+
+5. Build fully-qualified name: `Aspire.Dashboard.Tests.DashboardTests.CanStartDashboard`
+
+6. Run QuarantineTools:
+   ```bash
+   dotnet run --project tools/QuarantineTools/QuarantineTools.csproj -- -q "Aspire.Dashboard.Tests.DashboardTests.CanStartDashboard" -i "https://github.com/dotnet/aspire/issues/4321"
+   ```
+
+7. Build and verify
+
+8. Report success
+
+## Common Issues and Troubleshooting
+
+### Issue: "No method found matching"
+- **Cause**: The fully-qualified name is incorrect
+- **Solution**: Use Grep to find the exact namespace, class name, and method name
+
+### Issue: "The test is already quarantined"
+- **Cause**: The attribute already exists on the method
+- **Solution**: Verify by reading the test file; no action needed
+
+### Issue: Tool reports "Quarantining requires a valid http(s) URL"
+- **Cause**: The issue URL is missing or malformed
+- **Solution**: Ensure the URL starts with `http://` or `https://`
+
+### Issue: Build fails after quarantining
+- **Cause**: The QuarantineTools may have encountered a syntax issue (rare)
+- **Solution**: Read the modified file and check for syntax errors; the tool should handle this correctly
+
+## Important Notes
+
+1. **Always build after quarantining** to ensure the changes are valid
+2. **Run tests to confirm** the quarantined test is properly excluded
+3. **Don't forget to commit** the modified test files
+4. **Track with GitHub issues** - every quarantined test should have an associated issue
+5. **The QuarantineTools handles**:
+   - Adding the `[QuarantinedTest("url")]` attribute
+   - Adding `using Aspire.TestUtilities;` if needed
+   - Preserving file formatting and indentation
+   - Supporting nested classes and various namespace styles
+
+## Unquarantining Tests
+
+To unquarantine a test (remove the attribute), use:
+```bash
+dotnet run --project tools/QuarantineTools/QuarantineTools.csproj -- -u "Namespace.ClassName.TestMethodName"
+```
+
+The tool will:
+- Remove the `[QuarantinedTest]` attribute
+- Remove the `using Aspire.TestUtilities;` directive if no other tests in the file use it

.claude/skills/test-runner/SKILL.md

@@ -0,0 +1,166 @@
+---
+name: Aspire Test Runner
+description: Run tests for the Aspire project correctly, excluding quarantined and outerloop tests, with proper build verification. Use when running tests, debugging test failures, or validating changes.
+allowed-tools: Read, Grep, Glob, Bash
+---
+
+# Aspire Test Runner
+
+## Purpose
+
+This skill ensures tests are run correctly in the Aspire repository, following the project's specific requirements for test execution, including proper exclusion of quarantined and outerloop tests.
+
+## When to Use
+
+Invoke this skill when:
+- Running tests for a specific project or test class
+- Debugging test failures
+- Validating code changes
+- Verifying builds after modifications
+- User requests to "run tests" or "test my changes"
+
+## Critical Requirements
+
+**ALWAYS exclude quarantined and outerloop tests** in automated environments:
+- Quarantined tests are marked with `[QuarantinedTest]` and are known to be flaky
+- Outerloop tests are marked with `[OuterloopTest]` and are long-running or resource-intensive
+- These tests run separately in dedicated CI workflows
+
+## Instructions
+
+### Step 1: Identify Test Target
+
+1. If the user specifies a test project, use that path
+2. If the user mentions specific test methods or classes, identify the containing test project
+3. Use Glob to find test projects if needed:
+   ```bash
+   find tests -name "*.Tests.csproj" -type f
+   ```
+
+### Step 2: Build Verification (if needed)
+
+**Important**: Only build if:
+- There have been code changes since the last build
+- The user hasn't just run a successful build
+- You're unsure if the code is up to date
+
+If building is needed:
+```bash
+# Quick build with skip native (saves 1-2 minutes)
+./build.sh --build /p:SkipNativeBuild=true
+```
+
+### Step 3: Run Tests with Proper Filters
+
+**Default test run** (all tests in a project):
+```bash
+dotnet test tests/{ProjectName}.Tests/{ProjectName}.Tests.csproj --no-build -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+```
+
+**Specific test method**:
+```bash
+dotnet test tests/{ProjectName}.Tests/{ProjectName}.Tests.csproj --no-build -- --filter-method "*.{MethodName}" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+```
+
+**Specific test class**:
+```bash
+dotnet test tests/{ProjectName}.Tests/{ProjectName}.Tests.csproj --no-build -- --filter-class "*.{ClassName}" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+```
+
+**Multiple test methods**:
+```bash
+dotnet test tests/{ProjectName}.Tests/{ProjectName}.Tests.csproj --no-build -- --filter-method "*.Method1" --filter-method "*.Method2" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+```
+
+### Step 4: Handle Test Results
+
+**If tests pass**:
+- Report success to the user
+- Mention the number of tests that passed
+
+**If tests fail**:
+1. Analyze the failure output
+2. Identify which tests failed and why
+3. Check if failures are related to recent code changes
+4. Suggest fixes or next steps
+5. Do NOT mark the task as complete if tests are failing
+
+**If snapshot tests fail**:
+1. Tests using Verify library will show snapshot differences
+2. After verifying the new output is correct, run:
+   ```bash
+   dotnet verify accept -y
+   ```
+3. Re-run the tests to confirm they pass
+
+### Step 5: Report Results
+
+Provide a clear summary:
+- Number of tests run
+- Pass/fail status
+- Any warnings or issues
+- Next steps if failures occurred
+
+## Examples
+
+### Example 1: Run all tests for a specific project
+
+User: "Run tests for Aspire.Hosting.Testing"
+
+Actions:
+1. Identify test project: `tests/Aspire.Hosting.Testing.Tests/Aspire.Hosting.Testing.Tests.csproj`
+2. Run with proper filters:
+   ```bash
+   dotnet test tests/Aspire.Hosting.Testing.Tests/Aspire.Hosting.Testing.Tests.csproj --no-build -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+   ```
+3. Report results
+
+### Example 2: Run a specific test method
+
+User: "Run the TestingBuilderHasAllPropertiesFromRealBuilder test"
+
+Actions:
+1. Find the test using Grep:
+   ```bash
+   grep -r "TestingBuilderHasAllPropertiesFromRealBuilder" tests/
+   ```
+2. Identify project: `Aspire.Hosting.Testing.Tests`
+3. Run specific test:
+   ```bash
+   dotnet test tests/Aspire.Hosting.Testing.Tests/Aspire.Hosting.Testing.Tests.csproj --no-build -- --filter-method "*.TestingBuilderHasAllPropertiesFromRealBuilder" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+   ```
+4. Report results
+
+### Example 3: Run tests after making changes
+
+User: "I just modified the hosting code, run tests to verify"
+
+Actions:
+1. Identify affected test projects (e.g., `Aspire.Hosting.Tests`)
+2. Build first since code was modified:
+   ```bash
+   ./build.sh --build /p:SkipNativeBuild=true
+   ```
+3. Run tests:
+   ```bash
+   dotnet test tests/Aspire.Hosting.Tests/Aspire.Hosting.Tests.csproj -- --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
+   ```
+   Note: No `--no-build` flag since we need to pick up the changes
+4. Report results
+
+## Common Pitfalls to Avoid
+
+1. **Never omit the quarantine and outerloop filters** - this will run flaky tests
+2. **Don't use `--no-build`** if code has changed - the changes won't be tested
+3. **Don't run the full test suite** - it takes 30+ minutes, use targeted testing
+4. **Don't ignore snapshot test failures** - they indicate output changes that need review
+5. **Don't forget the `--` separator** before filter arguments
+
+## Valid Test Filter Switches
+
+- `--filter-class` / `--filter-not-class`: Filter by class name
+- `--filter-method` / `--filter-not-method`: Filter by method name
+- `--filter-namespace` / `--filter-not-namespace`: Filter by namespace
+- `--filter-trait` / `--filter-not-trait`: Filter by trait (category, platform, etc.)
+
+Switches can be repeated to filter multiple values. Class and method filters expect fully qualified names, unless using a prefix like `*.ClassName`.