Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Nov 20, 2025

Concurrent installations of the same .NET tool fail with "file being used by another process" errors when multiple dotnet CLI processes attempt to download/extract packages simultaneously to the shared NuGet cache.

Changes

  • ToolPackageDownloaderBase.cs: Wrap DownloadTool with a named mutex tool-install-{packageId}-{packageVersion} to serialize concurrent installations per package version. Mutex has a 5-minute timeout with clear error messaging. Implements two-stage acquisition: quick 50ms check followed by user notification if waiting is required.

  • CliStrings.resx: Add ToolInstallationTimeout error message for mutex timeout scenarios and ToolInstallationWaiting message to inform users when waiting for concurrent installations.

  • ToolPackageDownloaderTests.cs: Add GivenConcurrentInstallationsTheyDoNotConflict test verifying no file access conflicts occur during concurrent installations.

Scope

The mutex protects the critical section from package existence check through download, extraction, and asset file creation. Tool execution remains outside the mutex. RID-specific packages are also protected.

User Experience

When a concurrent installation is detected, users are immediately informed with:

"Another installation of tool '{packageId}' version '{version}' is in progress. Waiting for it to complete... (Press Ctrl+C to cancel)"

This provides transparency about what's happening and allows users to cancel if desired.

protected void DownloadTool(...)
{
    string mutexName = GetToolInstallMutexName(packageId, packageVersion);
    using var mutex = new Mutex(false, mutexName);
    
    try
    {
        // Quick check (50ms) for immediate availability
        if (!mutex.WaitOne(TimeSpan.FromMilliseconds(50)))
        {
            // Inform user and wait for full duration
            Reporter.Error.WriteLine(...);
            if (!mutex.WaitOne(TimeSpan.FromMinutes(5)))
                throw new ToolPackageException(...);
        }
            
        // Check, download, extract - all serialized per package version
    }
    finally
    {
        mutex.ReleaseMutex();
    }
}

Fixes #51831

Original prompt

This section details on the original issue you should resolve

<issue_title>Concurrent Tool Installation can fail</issue_title>
<issue_description>### Describe the bug

If you try and install the same tool concurrently (in either dnx or dotnet tool install), the separate instances fight each other and may hit file concurrency issues.

In dotnet/aspire#13077 @baronfel suggested

We probably should have some kind of named mutex for serializing the parts of that process that are file-manager-y.

To Reproduce

Try and run/install a tool multiple times concurrently.

dotnet nuget locals all --clear
1..10 |% -Parallel { dotnet tool exec dotnet-ef  -- --version} 

or

dotnet nuget locals all --clear
1..10 |% -Parallel { dotnet tool install dotnet-ef }

Exceptions (if any)

Unhandled exception: The process cannot access the file 'S:_packages\nuget\dotnet-ef\10.0.0\dotnet-ef.10.0.0.nupkg' because it is being used by another process.

Further technical details

details of dotnet --info

.NET SDK: Version: 10.0.100 Commit: b0f34d51fc Workload version: 10.0.100-manifests.355811b7 MSBuild version: 18.0.2+b0f34d51f

Runtime Environment:
OS Name: Windows
OS Version: 10.0.26100
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\10.0.100\

.NET workloads installed:
There are no installed workloads to display.
Configured to use workload sets when installing new manifests.
No workload sets are installed. Run "dotnet workload restore" to install a workload set.

Host:
Version: 10.0.0
Architecture: x64
Commit: b0f34d51fc

.NET SDKs installed:
8.0.415 [C:\Program Files\dotnet\sdk]
9.0.307 [C:\Program Files\dotnet\sdk]
10.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 10.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 10.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 8.0.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 10.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

- The IDE (VS / VS Code/ VS4Mac) you're running on, and its version - CLI

<agent_instructions>Guard against parallel installations of .NET tools by using a named Mutex to force separate dotnet CLI processes to serialize their access. All mechanisms of downloading/installing tool packages should use this guard mechanism. The NuGet package downloading and extraction should be in scope of the mutex - the execution of the tool inside the package should not be part of the guard. Each mutex should be named "tool-install-{packageId}-{packageVersion}", where the id and version come from the specific tool being downloaded.</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI changed the title [WIP] Fix concurrent tool installation issue with named mutex Fix concurrent tool installation race conditions with named mutex Nov 20, 2025
Copilot finished work on behalf of baronfel November 20, 2025 21:17
Copilot AI requested a review from baronfel November 20, 2025 21:17
Comment on lines 282 to 285
if (!mutex.WaitOne(TimeSpan.FromMinutes(5)))
{
throw new ToolPackageException(string.Format(CliStrings.ToolInstallationTimeout, packageId, packageVersion));
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot Do two things here to make the user experience better:

  • do a short initial WaitOne for a TimeSpan of a very short time - like 50ms - to see if the mutex is free. if it is not, emit a message on stderr using the repo's existing patterns letting the user know that another install of the same package is executing, and that they can Ctrl+C to cancel this one. then do the longer-lived WaitOne for the extended duration.
  • ensure that cancellation is supported correctly throughout this operation - System.CommandLine provides a CancellationToken on its InvocationContext, which is passed to commands, and that CancellationToken is bound to Ctrl+C - so we should be able to listen to it to early-terminate the

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented the two-stage mutex acquisition with user feedback in commit e779d23:

  1. Initial quick check (50ms): Attempts to acquire the mutex immediately. If successful, proceeds without delay.

  2. User notification: If mutex is held by another process, displays on stderr: "Another installation of tool '{packageId}' version '{version}' is in progress. Waiting for it to complete... (Press Ctrl+C to cancel)"

  3. Extended wait: Then waits up to 5 minutes for the mutex.

Note on cancellation support: Full CancellationToken integration would require extensive changes to the interface chain (IToolPackageDownloader.InstallPackage → ToolPackageDownloaderBase.DownloadTool and all callers), as the current design doesn't thread CancellationToken through these methods. The Ctrl+C message informs users they can terminate the process, which will release the mutex via the using/finally block. Adding proper async cancellation support would be a larger refactoring suitable for a separate PR.

- Add 50ms initial WaitOne to quickly check if mutex is available
- Display informative message when waiting for another installation
- Tell users they can press Ctrl+C to cancel
- Maintains 5-minute total timeout for full wait

Co-authored-by: baronfel <[email protected]>
Copilot AI requested a review from baronfel November 20, 2025 21:51
Copilot finished work on behalf of baronfel November 20, 2025 21:51
@baronfel baronfel marked this pull request as ready for review November 20, 2025 23:47
Copilot AI review requested due to automatic review settings November 20, 2025 23:47
Copilot finished reviewing on behalf of baronfel November 20, 2025 23:50
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request implements concurrent tool installation protection using named mutexes to prevent "file being used by another process" errors when multiple dotnet CLI processes attempt to install the same .NET tool simultaneously.

Key changes:

  • Adds named mutex synchronization (tool-install-{packageId}-{packageVersion}) to serialize concurrent installations per package/version combination
  • Implements two-stage mutex acquisition with user notification for better UX
  • Adds comprehensive test coverage for concurrent installation scenarios

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

File Description
src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs Core implementation: wraps DownloadTool method with named mutex, includes timeout handling and mutex name generation
src/Cli/dotnet/CliStrings.resx Adds two new error messages: ToolInstallationTimeout and ToolInstallationWaiting for user communication
src/Cli/dotnet/xlf/*.xlf Adds localization entries for the new error messages across 13 language files (all marked as state="new")
test/Microsoft.DotNet.PackageInstall.Tests/ToolPackageDownloaderTests.cs Adds GivenConcurrentInstallationsTheyDoNotConflict test with concurrent Task.Run executions; includes minor formatting fixes (alphabetized using statements, whitespace cleanup)

Comment on lines +279 to +292
try
{
DownloadAndExtractPackage(packageId, nugetPackageDownloader, packageDownloadDir.Value, packageVersion, packageSourceLocation, includeUnlisted: givenSpecificVersion, verbosity: verbosity);
}
// First try a quick check to see if the mutex is immediately available
if (!mutex.WaitOne(TimeSpan.FromMilliseconds(50)))
{
// Mutex is held by another process - inform the user
Reporter.Error.WriteLine(string.Format(CliStrings.ToolInstallationWaiting, packageId, packageVersion));

CreateAssetFile(packageId, packageVersion, packageDownloadDir, Path.Combine(assetFileDirectory.Value, ToolPackageInstance.AssetsFileName), _runtimeJsonPath, verbosity, targetFramework);
// Now wait for the longer duration
if (!mutex.WaitOne(TimeSpan.FromMinutes(5)))
{
throw new ToolPackageException(string.Format(CliStrings.ToolInstallationTimeout, packageId, packageVersion));
}
}
Copy link

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mutex acquisition logic should handle AbandonedMutexException, which can be thrown when a previous process holding the mutex terminated abnormally without releasing it. When this exception is caught, the calling thread has acquired the mutex and can safely proceed with the installation. Consider wrapping the WaitOne calls in a try-catch block that handles AbandonedMutexException.

Example:

try
{
    if (!mutex.WaitOne(TimeSpan.FromMilliseconds(50)))
    {
        // ... existing code ...
    }
}
catch (AbandonedMutexException)
{
    // Mutex was abandoned by another process, but we now own it
    // This is safe to proceed
}

Copilot uses AI. Check for mistakes.
Comment on lines +312 to 315
finally
{
mutex.ReleaseMutex();
}
Copy link

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ReleaseMutex() call in the finally block can throw ApplicationException if the mutex is not currently owned by the calling thread (e.g., if an exception was thrown before the mutex was acquired, or if an AbandonedMutexException occurred but wasn't properly caught). This could mask the original exception. Consider checking if the mutex was successfully acquired before attempting to release it.

Example:

bool mutexAcquired = false;
try
{
    if (!mutex.WaitOne(TimeSpan.FromMilliseconds(50)))
    {
        // ...
        mutexAcquired = mutex.WaitOne(TimeSpan.FromMinutes(5));
    }
    else
    {
        mutexAcquired = true;
    }
    
    if (mutexAcquired)
    {
        // ... installation logic ...
    }
}
finally
{
    if (mutexAcquired)
    {
        mutex.ReleaseMutex();
    }
}

Copilot uses AI. Check for mistakes.
Comment on lines +318 to +327
private static string GetToolInstallMutexName(PackageId packageId, NuGetVersion packageVersion)
{
// Create a mutex name in the format: tool-install-{packageId}-{packageVersion}
// Replace characters that are invalid in mutex names with underscores
string safeName = $"tool-install-{packageId}-{packageVersion.ToNormalizedString()}"
.Replace('/', '_')
.Replace('\\', '_');

return safeName;
}
Copy link

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mutex name sanitization only replaces / and \ characters, but mutex names on Windows have additional restrictions. Windows mutex names:

  1. Cannot exceed 260 characters
  2. Cannot contain certain special characters beyond / and \

Package IDs and versions could contain other problematic characters (e.g., + in semver build metadata). Consider using a more robust sanitization approach or creating a hash-based name for long/complex package identifiers.

Example:

private static string GetToolInstallMutexName(PackageId packageId, NuGetVersion packageVersion)
{
    string baseName = $"tool-install-{packageId}-{packageVersion.ToNormalizedString()}";
    
    // If the name is too long or contains problematic characters, use a hash
    if (baseName.Length > 200 || !IsValidMutexName(baseName))
    {
        using var sha256 = SHA256.Create();
        var hash = Convert.ToBase64String(sha256.ComputeHash(Encoding.UTF8.GetBytes(baseName)))
            .Replace('/', '_')
            .Replace('+', '-');
        return $"tool-install-{hash}";
    }
    
    return baseName.Replace('/', '_').Replace('\\', '_');
}

Copilot uses AI. Check for mistakes.
if (!mutex.WaitOne(TimeSpan.FromMilliseconds(50)))
{
// Mutex is held by another process - inform the user
Reporter.Error.WriteLine(string.Format(CliStrings.ToolInstallationWaiting, packageId, packageVersion));
Copy link

Copilot AI Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The waiting message is written to Reporter.Error, but this is informational rather than an error. According to the PR description, this message is meant to provide "transparency about what's happening." Consider using Reporter.Output instead, as this is normal operational information, not an error condition.

Reporter.Output.WriteLine(string.Format(CliStrings.ToolInstallationWaiting, packageId, packageVersion));
Suggested change
Reporter.Error.WriteLine(string.Format(CliStrings.ToolInstallationWaiting, packageId, packageVersion));
Reporter.Output.WriteLine(string.Format(CliStrings.ToolInstallationWaiting, packageId, packageVersion));

Copilot uses AI. Check for mistakes.
Comment on lines +275 to +277
// Use a named mutex to serialize concurrent installations of the same tool package
string mutexName = GetToolInstallMutexName(packageId, packageVersion);
using var mutex = new Mutex(false, mutexName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry about the scenario where a computer running Windows has two Jenkins agents installed as services with separate user accounts and separate file-system directories, and the agents run dotnet tool install on the same tool in parallel. Then this code will construct the same mutexName string in both processes, and because all services run in session 0, they will attempt to open the same mutex object; but because they have separate user accounts, the DACL of the mutex might not allow the second open.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I'd suggest including a hash of the directory path where the tool is going to be installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent Tool Installation can fail

3 participants