Remove RunTestsAsTool Concept #53010

MichaelSimons · 2026-02-12T18:48:01Z

The RunTestAsTool concept was necessary when there was separate dotnet/sdk and dotnet/installer repos. The installer ran several of the sdk tests using this mechanism. Now that the repos have been consolidated, this infrastructure is no longer necessary. Removing it cleans up the infrastructure and removes an entire leg from CI (TestAsTools). This infrastructure was requiring workarounds for the xunit v3 upgrade

Copilot

Pull request overview

Removes the legacy “RunTestsAsTool/TestAsTool” infrastructure (originally used for running SDK tests from the old installer repo) and cleans up associated MSBuild and pipeline wiring, with some related test-context simplification.

Changes:

Removes CanRunTestAsTool usage from test/benchmark projects and deletes the TestAsTool MSBuild target + OverrideTest.targets.
Drops the AzDO “TestAsTools” job category and removes runTestsAsTool pipeline plumbing.
Simplifies SdkTestContext.Initialize path inference for test assets and execution directories.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/trustedroots.Tests/trustedroots.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/dotnet.Tests/dotnet.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/dotnet-MsiInstallation.Tests/dotnet-MsiInstallation.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/System.CommandLine.StaticCompletions.Tests/System.CommandLine.StaticCompletions.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/SDDLTests/SDDLTests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/Microsoft.Win32.Msi.Tests/Microsoft.Win32.Msi.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/Microsoft.Win32.Msi.Manual.Tests/Microsoft.Win32.Msi.Manual.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/Microsoft.WebTools.AspireService.Tests/Microsoft.WebTools.AspireService.Tests.csproj	Adds `IsPackable` (needs review vs removal of tool packing).
test/Microsoft.NET.TestFramework/SdkTestContext.cs	Removes “run as tool” branching and simplifies directory resolution logic.
test/Microsoft.DotNet.PackageInstall.Tests/Microsoft.DotNet.PackageInstall.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/EndToEnd.Tests/EndToEnd.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
test/Directory.Build.targets	Removes `TestAsTool` MSBuild target and tool-packing property group.
test/ArgumentForwarding.Tests/ArgumentForwarding.Tests.csproj	Removes obsolete `CanRunTestAsTool` property.
eng/pipelines/templates/jobs/sdk-job-matrix.yml	Removes “TestAsTools” job category from the matrix.
eng/pipelines/templates/jobs/sdk-build.yml	Removes `runTestsAsTool` parameter and MSBuild property flow.
benchmarks/MicroBenchmark/MicroBenchmark.csproj	Removes obsolete `CanRunTestAsTool` property.
OverrideTest.targets	Deletes the target override used to route `Test` → `TestAsTool`.
Directory.Build.targets	Removes conditional import of `OverrideTest.targets`.

Copilot · 2026-02-12T22:32:44Z

test/Microsoft.NET.TestFramework/SdkTestContext.cs

+            testContext.TestAssetsDirectory =
+                (Directory.Exists(basePath) ? basePath : null)
+                ?? Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_ASSETS_DIRECTORY")


The new coalesce chain uses Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_ASSETS_DIRECTORY") directly. If that variable is set but empty, TestAssetsDirectory becomes "" (or whitespace) and later path operations can fail in non-obvious ways. Consider using string.IsNullOrEmpty (or trimming) so empty values are treated the same as null and the fallback (FindFolderInTree) is used instead.

Suggested change

testContext.TestAssetsDirectory =

(Directory.Exists(basePath) ? basePath : null)

?? Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_ASSETS_DIRECTORY")

string? testAssetsDirectoryEnvironment = Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_ASSETS_DIRECTORY");

if (string.IsNullOrWhiteSpace(testAssetsDirectoryEnvironment))

{

testAssetsDirectoryEnvironment = null;

}

testContext.TestAssetsDirectory =

(Directory.Exists(basePath) ? basePath : null)

?? testAssetsDirectoryEnvironment

Copilot · 2026-02-12T22:32:45Z

test/Microsoft.NET.TestFramework/SdkTestContext.cs

+            testContext.TestExecutionDirectory =
+                commandLine.TestExecutionDirectory
+                ?? Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_EXECUTION_DIRECTORY")
+                ?? Path.Combine(FindFolderInTree("artifacts", AppContext.BaseDirectory)!, "tmp", repoConfiguration, "testing");



Similarly, TestExecutionDirectory now accepts DOTNET_SDK_TEST_EXECUTION_DIRECTORY as-is. If it is set to an empty string, Directory.CreateDirectory(TestExecutionDirectory) will throw (ArgumentException). Consider ignoring empty/whitespace values and falling back to the artifacts-based path.

Suggested change

testContext.TestExecutionDirectory =

commandLine.TestExecutionDirectory

?? Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_EXECUTION_DIRECTORY")

?? Path.Combine(FindFolderInTree("artifacts", AppContext.BaseDirectory)!, "tmp", repoConfiguration, "testing");

string? testExecutionDirectory = commandLine.TestExecutionDirectory;

if (string.IsNullOrWhiteSpace(testExecutionDirectory))

{

string? envTestExecutionDirectory = Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_EXECUTION_DIRECTORY");

if (!string.IsNullOrWhiteSpace(envTestExecutionDirectory))

{

testExecutionDirectory = envTestExecutionDirectory;

}

}

if (string.IsNullOrWhiteSpace(testExecutionDirectory))

{

testExecutionDirectory = Path.Combine(FindFolderInTree("artifacts", AppContext.BaseDirectory)!, "tmp", repoConfiguration, "testing");

}

testContext.TestExecutionDirectory = testExecutionDirectory;

Copilot · 2026-02-12T22:32:45Z

test/Microsoft.WebTools.AspireService.Tests/Microsoft.WebTools.AspireService.Tests.csproj

    <TargetFramework>$(SdkTargetFramework)</TargetFramework>
    <RootNamespace>Microsoft.WebTools.AspireServer.UnitTests</RootNamespace>
    <OutputType>Exe</OutputType>
+    <IsPackable>true</IsPackable>


This test project now explicitly sets true. With the PR removing the “tests as tools” packing flow, making a test project packable again seems unintended and can cause it to be included in repo-wide pack/signing steps. If there’s no longer a need to generate a NuGet package here, this should likely be removed or set to false to match other test projects in this repo.

Suggested change

<IsPackable>true</IsPackable>

<IsPackable>false</IsPackable>

Yeah, it's not clear why or if this is needed.

MiYanni

Thank you for doing this. One of those things on the list of "stuff after repo merge" I never got around to investigating. ❤️

dsplaisted · 2026-02-12T23:45:58Z

test/Microsoft.NET.TestFramework/SdkTestContext.cs

+            testContext.TestExecutionDirectory =
+                commandLine.TestExecutionDirectory
+                ?? Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_EXECUTION_DIRECTORY")
+                ?? Path.Combine(FindFolderInTree("artifacts", AppContext.BaseDirectory)!, "tmp", repoConfiguration, "testing");


It looks like this could throw a NullReferenceException if FindFolderInTree returns null.

FindFolderInTree throws by default if the folder is not found - https://github.com/dotnet/sdk/blob/main/test/Microsoft.NET.TestFramework/SdkTestContext.cs#L405

The FindFolderInTree api could maybe be designed a little better but that is outside the scope of this PR.

dsplaisted · 2026-02-12T23:47:30Z

test/Microsoft.WebTools.AspireService.Tests/Microsoft.WebTools.AspireService.Tests.csproj

    <TargetFramework>$(SdkTargetFramework)</TargetFramework>
    <RootNamespace>Microsoft.WebTools.AspireServer.UnitTests</RootNamespace>
    <OutputType>Exe</OutputType>
+    <IsPackable>true</IsPackable>


Yeah, it's not clear why or if this is needed.

Remove RunTestsAsTool Concept

4bca56f

github-actions bot added the Area-Infrastructure label Feb 12, 2026

MichaelSimons added 2 commits February 12, 2026 20:34

Add IsPackable to AspireService.Tests

bc8276f

Merge branch 'main' into remove-test-as-tool

6460ea6

MichaelSimons marked this pull request as ready for review February 12, 2026 22:22

MichaelSimons requested a review from MiYanni as a code owner February 12, 2026 22:22

Copilot AI review requested due to automatic review settings February 12, 2026 22:22

MichaelSimons requested a review from MiYanni as a code owner February 12, 2026 22:22

MichaelSimons requested a review from dsplaisted February 12, 2026 22:23

Copilot started reviewing on behalf of MichaelSimons February 12, 2026 22:23 View session

MichaelSimons requested a review from a team February 12, 2026 22:23

Copilot AI reviewed Feb 12, 2026

View reviewed changes

MiYanni approved these changes Feb 12, 2026

View reviewed changes

dsplaisted approved these changes Feb 12, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove RunTestsAsTool Concept #53010

Remove RunTestsAsTool Concept #53010

Uh oh!

MichaelSimons commented Feb 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

dsplaisted Feb 12, 2026

Uh oh!

MiYanni left a comment

Uh oh!

dsplaisted Feb 12, 2026

Uh oh!

MichaelSimons Feb 12, 2026

Uh oh!

dsplaisted Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-            testContext.TestExecutionDirectory =
-                commandLine.TestExecutionDirectory
-                ?? Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_EXECUTION_DIRECTORY")
-                ?? Path.Combine(FindFolderInTree("artifacts", AppContext.BaseDirectory)!, "tmp", repoConfiguration, "testing");
+            string? testExecutionDirectory = commandLine.TestExecutionDirectory;
+            if (string.IsNullOrWhiteSpace(testExecutionDirectory))
+            {
+                string? envTestExecutionDirectory = Environment.GetEnvironmentVariable("DOTNET_SDK_TEST_EXECUTION_DIRECTORY");
+                if (!string.IsNullOrWhiteSpace(envTestExecutionDirectory))
+                {
+                    testExecutionDirectory = envTestExecutionDirectory;
+                }
+            }
+            if (string.IsNullOrWhiteSpace(testExecutionDirectory))
+            {
+                testExecutionDirectory = Path.Combine(FindFolderInTree("artifacts", AppContext.BaseDirectory)!, "tmp", repoConfiguration, "testing");
+            }
+            testContext.TestExecutionDirectory = testExecutionDirectory;

Remove RunTestsAsTool Concept #53010

Are you sure you want to change the base?

Remove RunTestsAsTool Concept #53010

Uh oh!

Conversation

MichaelSimons commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

dsplaisted Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

MiYanni left a comment

Choose a reason for hiding this comment

Uh oh!

dsplaisted Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

MichaelSimons Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

dsplaisted Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MichaelSimons commented Feb 12, 2026 •

edited

Loading