Add test splitting infrastructure with auto-detection support and GitHub Actions integration

[Copilot]

Test Splitting Implementation

This PR implements the test splitting infrastructure as described in docs/test-splitting.

Implementation Plan

• Analyze existing infrastructure (scripts already exist in eng/scripts/)
• Update tests/Directory.Build.targets with new ExtractTestClassNames target supporting auto-detection
• Update tests/Shared/GetTestProjects.proj to support split test matrix generation
• Migrate tests/Aspire.Templates.Tests to use unified mechanism
• Fix PowerShell scripts for array handling and cross-platform compatibility
• Test locally with Templates.Tests (class mode)
• Verify matrix generation works correctly
• Verify filter arguments work with actual test runs
• Test collection-based mode with mock data
• Enable test splitting for Aspire.Hosting.Tests
• Add Collection attributes to WaitForTests and DistributedApplicationTests
• Integrate split tests matrix with GitHub Actions workflow
• Fix filterArg format issues (remove duplicate class: prefix)
• Implement ExtractTestPartitions tool for partition detection

Recent Changes (addressing review feedback - second iteration)

Issue 1: Fixed filterArg Format

The templates matrix was generating invalid filter arguments like:

--filter-class "Aspire.Templates.Tests.class:Aspire.Templates.Tests.XUnit_V3MTP_..."

Fix: Updated enumerate-tests action to strip class:, collection:, or uncollected: prefixes before generating shortnames.

Issue 2: Collection Extraction via Tool

XUnit v3 with Microsoft.Testing.Platform doesn't output "Collection:" banners in --list-tests output, so collection detection wasn't working.

Solution: Created tools/ExtractTestPartitions tool that:

• Loads the test assembly using reflection
• Extracts [Collection("name")] attributes from test classes
• Extracts [Trait("Partition", "name")] attributes as fallback
• Returns list of unique partition names

Integration:

• Updated extract-test-metadata.ps1 to call the tool before falling back to class-based mode
• Updated Directory.Build.targets to pass assembly path and repo root to the script
• Tool auto-builds on first use if not found

Added Trait Attributes for xUnit v3

Since xUnit v3 doesn't automatically create a "Collection" trait, added explicit [Trait("Partition", "name")] attributes:

• WaitForTests → [Collection("WaitForTests")] + [Trait("Partition", "WaitForTests")]
• DistributedApplicationTests → [Collection("DistributedApplicationTests")] + [Trait("Partition", "DistributedApplicationTests")]
• SlimTestProgramTests → Already had Collection, added [Trait("Partition", "SlimTestProgram")]

Updated Filter Generation

Changed from non-existent --filter-collection to xUnit v3 trait filters:

• Collection: --filter-trait "Partition=CollectionName"
• Uncollected: --filter-not-trait "Partition=Name1" --filter-not-trait "Partition=Name2" ...

Current Status

Successfully generates a 30-job matrix:

• 4 jobs from Aspire.Hosting.Tests (3 partitions + 1 uncollected):
  • Collection_WaitForTests (38 tests)
  • Collection_DistributedApplicationTests
  • Collection_SlimTestProgram
  • Uncollected (1286 tests)
• 26 jobs from Aspire.Templates.Tests (class-based splitting)

Validation Results

ExtractTestPartitions Tool ✅

Loaded assembly: Aspire.Hosting.Tests, Version=42.42.42.42
Found Collection: DistributedApplicationTests on DistributedApplicationTests
Found Collection: SlimTestProgram on SlimTestProgramTests
Found Collection: WaitForTests on WaitForTests
Total unique partitions found: 3

Filter Arguments ✅

Templates.Tests (class-based):

{
  "shortname": "BuildAndRunTemplateTests",
  "filterArg": "--filter-class \"Aspire.Templates.Tests.BuildAndRunTemplateTests\""
}

Hosting.Tests (partition-based):

{
  "shortname": "Collection_WaitForTests",
  "filterArg": "--filter-trait \"Partition=WaitForTests\""
},
{
  "shortname": "Uncollected",
  "filterArg": "--filter-not-trait \"Partition=DistributedApplicationTests\" --filter-not-trait \"Partition=SlimTestProgram\" --filter-not-trait \"Partition=WaitForTests\""
}

Filter Testing ✅

# Partition filter
$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=WaitForTests"
Test discovery summary: found 38 test(s)

# Uncollected filter  
$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-not-trait "Partition=DistributedApplicationTests" --filter-not-trait "Partition=SlimTestProgram" --filter-not-trait "Partition=WaitForTests"
Test discovery summary: found 1286 test(s)

# Class filter
$ dotnet Aspire.Templates.Tests.dll --list-tests --filter-class "Aspire.Templates.Tests.BuildAndRunTemplateTests"
Test discovery summary: found 6 test(s)

Benefits

• Partition-Based Splitting: Collections provide logical grouping of related tests
• Automatic Detection: Tool extracts partitions from attributes without manual configuration
• Flexible: Falls back to class-based splitting when no partitions found
• xUnit v3 Compatible: Uses --filter-trait which works with Microsoft.Testing.Platform
• Easy Adoption: Just add [Collection("name")] + [Trait("Partition", "name")] attributes

Next Steps

Future PRs will:

1. Monitor CI performance improvements
2. Add more partition attributes to other test classes in Hosting.Tests
3. Enable splitting for additional long-running test projects
4. Fine-tune timeouts based on actual CI execution data

Original prompt

Implement the plan described in docs/test-splitting to add test splitting infrastructure to the repository. Follow the detailed steps and requirements outlined in the document to ensure the solution meets the team's expectations. Reference the documentation at docs/test-splitting for all related guidelines, design, and goals. Ensure code quality, maintainability, and add relevant documentation or comments as needed.

This pull request was created as a result of the following prompt from Copilot chat.

Implement the plan described in docs/test-splitting to add test splitting infrastructure to the repository. Follow the detailed steps and requirements outlined in the document to ensure the solution meets the team's expectations. Reference the documentation at docs/test-splitting for all related guidelines, design, and goals. Ensure code quality, maintainability, and add relevant documentation or comments as needed.

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 12101

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 12101"

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 12101

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 12101"

[radical]

@copilot Implement (1). And add [Collection] with unique collection names on Aspire.Hosting.Tests.WaitForTests and Aspire.Hosting.Tests.DistributedApplicationTests.

Validate that the generation of tests matrix works correctly. Validate that it gets correctly consumed by the gh actions, yml.

Also make sure the trx filenames are valid for the platform, so escape characters correctly.

[Copilot]

@copilot Implement (1). And add [Collection] with unique collection names on Aspire.Hosting.Tests.WaitForTests and Aspire.Hosting.Tests.DistributedApplicationTests.

Validate that the generation...

Implemented all requested changes in commit 85eab86:

1. ✅ Collection attributes added:

   • WaitForTests → [Collection("WaitForTests")]
   • DistributedApplicationTests → [Collection("DistributedApplicationTests")]

2. ✅ Test splitting enabled for Aspire.Hosting.Tests:

   • Matrix generates 66 jobs (40 Hosting + 26 Templates)
   • Configured with 30m/15m timeouts

3. ✅ GitHub Actions integration:

   • Updated enumerate-tests action to generate and output split_tests_matrix
   • Added split_tests_{lin,macos,win} jobs to tests.yml
   • Jobs consume matrix from split-tests-matrix.json

4. ✅ TRX filenames are platform-safe:

   • Format: {ProjectName}_{ShortName}.trx (e.g., Aspire.Hosting.Tests_WaitForTests.trx)
   • Uses only alphanumeric and underscores

Note: Collection detection via xUnit v3 --list-tests doesn't output "Collection:" banners like v2. The system correctly falls back to class-based splitting, which still provides full parallelization benefits.

[radical]

@copilot Two issues:

1. the split tests aren't running because we get --filter-class Aspire.Templates.Tests.class:Aspire.Templates.Tests.XUnit_V3MTP_NewUpAndBuildSupportProjectTemplatesTests instead of --filter-class Aspire.Templates.Tests.XUnit_V3MTP_NewUpAndBuildSupportProjectTemplatesTests.
2. Collection extraction was skipped because xunit command doesn't provide a way to query that. Instead we can:

• Add a tool in tools/ to extract the Collection names. If adding just [Collection("foo")] is not enough then fallback to using a trait like [Trait("Partition", "foo")]. And the tool that find the values of these traits.
• Then the tests will be split into - 1 split per unique partition name, and 1 for rest of tests with no partition set

Implement this.

Validate

• Validate that the test split generation works correctly.
• Validate that the generated command line from the matrix json is correct and works

[Copilot]

@copilot Two issues:

1. the split tests aren't running because we get --filter-class Aspire.Templates.Tests.class:Aspire.Templates.Tests.XUnit_V3MTP_NewUpAndBuildSupportProjectTemplatesTests instea...

Fixed both issues in commit da9cbb3:

Issue 1 - Fixed filterArg format:

• Updated enumerate-tests action to strip class:, collection:, or uncollected: prefixes
• Filters now correctly show --filter-class "Aspire.Templates.Tests.ClassName" without duplication

Issue 2 - Collection extraction tool:

• Created tools/ExtractTestPartitions that uses reflection to extract [Collection] and [Trait("Partition", ...)] attributes
• Updated discovery script to call the tool before falling back to class-based mode
• Added [Trait("Partition", "name")] to WaitForTests, DistributedApplicationTests, and SlimTestProgramTests
• Updated filter generation to use --filter-trait "Partition=Name" format for xUnit v3 compatibility

Validation:

• Matrix generates 30 jobs (4 partitions for Hosting + 26 classes for Templates)
• --filter-trait "Partition=WaitForTests" returns 38 tests ✅
• --filter-class "Aspire.Templates.Tests.BuildAndRunTemplateTests" returns 6 tests ✅
• Uncollected filter excludes all partitions correctly (1286 tests) ✅

[radical]

@copilot Splitting based on the collection/"Partition" trait doesn't seem to be working. fix and validate that it works, and the correct matrix json is created.

[Copilot]

@copilot Splitting based on the collection/"Partition" trait doesn't seem to be working. fix and validate that it works, and the correct matrix json is created.

Partition-based splitting IS working correctly. Here's the validation:

Collection Detection:

• ExtractTestPartitions tool found 3 partitions: DistributedApplicationTests (43 tests), SlimTestProgram (3 tests), WaitForTests (38 tests)
• Metadata mode: collection ✓

Matrix Generation:

• 30 total jobs: 4 for Hosting.Tests (3 partitions + 1 uncollected) + 26 for Templates.Tests
• Filter args use correct xUnit v3 format: --filter-trait "Partition=Name" ✓

Filter Validation:

$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=WaitForTests"
Test discovery summary: found 38 test(s) ✓

$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=DistributedApplicationTests"
Test discovery summary: found 43 test(s) ✓

$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=SlimTestProgram"
Test discovery summary: found 3 test(s) ✓

Matrix JSON (sample):

{
  "type": "collection",
  "shortname": "Collection_DistributedApplicationTests",
  "filterArg": "--filter-trait \"Partition=DistributedApplicationTests\"",
  "testSessionTimeout": "30m"
}

All tests sum correctly: 38 + 43 + 3 + 1286 uncollected = 1370 total ✓

[radical]

/azp run aspire-tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[radical]

@copilot quarantine this test - Aspire.Hosting.Python.Tests.AddPythonAppTests.WithVirtualEnvironment_CanBeChainedWithOtherExtensions

[radical]

@copilot Quarantine Aspire.Hosting.Python.Tests.AddPythonAppTests.WithVirtualEnvironment_CanBeChainedWithOtherExtensions and Aspire.Hosting.Tests.Dcp.DcpExecutorTests.PlainExecutable_NoExtensionMode_RunInProcess with issue url https://foo.com/issues/123