Update to msbuild 17.13 breaks builds

[kewdgr]

Describe the Bug

All our builds have been working for years with tags up to 3.5-20250114-windowsservercore-ltsc2019.
It seems that the switch to VS Build Tools 17.13 introduced a breaking change for our builds

message: Exception has been thrown by the target of an invocation. Could not load file or assembly 'Microsoft.VisualStudio.SolutionPersistence, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

Does anybody have a suggestion for a fix? A Tag for a container using older msbuild?

Steps to Reproduce

Just starting MSBuild included in this container 3.5-20250211-windowsservercore-ltsc2019 was enough.
It works perfectly with the same project in this container : 3.5-20250114-windowsservercore-ltsc2019

Other Information

message: Exception has been thrown by the target of an invocation. Could not load file or assembly 'Microsoft.VisualStudio.SolutionPersistence, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

Output of docker version

Server: Docker Desktop 4.38.0 (181591)
Engine:
Version: 27.5.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.11
Git commit: 4c9b3b0
Built: Wed Jan 22 13:40:41 2025
OS/Arch: windows/amd64
Experimental: false

Output of docker info

Client:
Version: 27.5.1
Context: desktop-windows
Debug Mode: false
Plugins:
ai: Ask Gordon - Docker Agent (Docker Inc.)
Version: v0.7.3
Path: C:\Users\user.docker\cli-plugins\docker-ai.exe
buildx: Docker Buildx (Docker Inc.)
Version: v0.20.1-desktop.2
Path: C:\Users\user.docker\cli-plugins\docker-buildx.exe
compose: Docker Compose (Docker Inc.)
Version: v2.32.4-desktop.1
Path: C:\Users\user.docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.38
Path: C:\Users\user.docker\cli-plugins\docker-debug.exe
desktop: Docker Desktop commands (Beta) (Docker Inc.)
Version: v0.1.4
Path: C:\Users\user.docker\cli-plugins\docker-desktop.exe
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: C:\Users\user.docker\cli-plugins\docker-dev.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.27
Path: C:\Users\user.docker\cli-plugins\docker-extension.exe
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: C:\Users\user.docker\cli-plugins\docker-feedback.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: C:\Users\user.docker\cli-plugins\docker-init.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Users\user.docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.16.1
Path: C:\Users\user.docker\cli-plugins\docker-scout.exe

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 27.5.1
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 22631 (22621.1.amd64fre.ni_release.220506-1250)
Operating System: Microsoft Windows Version 23H2 (OS Build 22631.4602)
OSType: windows
Architecture: x86_64
CPUs: 12
Total Memory: 31.67GiB
Name: PC
ID: 30fc25ba-fa04-4f36-8b51-f5153b58f2b8
Docker Root Dir: C:\ProgramData\Docker
Debug Mode: false
Labels:
com.docker.desktop.address=npipe://\.\pipe\docker_cli
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

[mthalman]

cc @rainersigwald, @baronfel

@kewdgr - More information is needed on a repro. The tests that get run prior to publishing these images verified that MSBuild was working, at least for that test scenario. See here as an example:

dotnet-framework-docker/tests/Microsoft.DotNet.Framework.Docker.Tests/projects/dotnetapp-3.5/Dockerfile

Line 8 in 624787e

RUN ["msbuild.exe", "dotnetapp-3.5.csproj", "/p:Configuration=Release"]

So there seems to be scenarios where MSBuild works and others where it doesn't. Can you please provide more info on how you're executing MSBuild and your target project?

Here is the output of running that test I mentioned:

Executing: docker pull dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/sdk:3.5-windowsservercore-ltsc2019
3.5-windowsservercore-ltsc2019: Pulling from build-staging/2640615/dotnet/framework/sdk
Digest: sha256:3e96e8f423f2bb1a4b9d85efe3323d415bf824d88962ad2a37a3eb03935fca22
Status: Image is up to date for dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/sdk:3.5-windowsservercore-ltsc2019
dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/sdk:3.5-windowsservercore-ltsc2019
Execution Elapsed Time: 00:00:00.6048557
Executing: docker pull dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/runtime:3.5-windowsservercore-ltsc2019
3.5-windowsservercore-ltsc2019: Pulling from build-staging/2640615/dotnet/framework/runtime
803f4a9590cb: Already exists
c3af2bd0a196: Already exists
e4055959666b: Already exists
2194a26937d1: Already exists
Digest: sha256:2b64d82befff0f026e04f62adcd07b8b6a230ee4f0ae04d8d53fda5d4fb89211
Status: Downloaded newer image for dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/runtime:3.5-windowsservercore-ltsc2019
dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/runtime:3.5-windowsservercore-ltsc2019
Execution Elapsed Time: 00:00:01.2399116
Executing: docker build -t dotnetapp-133837962683122340 --build-arg BASE_BUILD_IMAGE=dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/sdk:3.5-windowsservercore-ltsc2019 --build-arg BASE_RUNTIME_IMAGE=dotnetdockerstaging.azurecr.io/build-staging/2640615/dotnet/framework/runtime:3.5-windowsservercore-ltsc2019 -f D:\a\_work\1\a\bin\Microsoft.DotNet.Framework.Docker.Tests\release\projects\dotnetapp-3.5\Dockerfile D:\a\_work\1\a\bin\Microsoft.DotNet.Framework.Docker.Tests\release\projects\dotnetapp-3.5
Sending build context to Docker daemon   7.68kB

Step 1/10 : ARG BASE_BUILD_IMAGE
Step 2/10 : ARG BASE_RUNTIME_IMAGE
Step 3/10 : FROM $BASE_BUILD_IMAGE as builder
 ---> 3373570731b6
Step 4/10 : WORKDIR /app
 ---> Running in 9632b417e1e9
 ---> Removed intermediate container 9632b417e1e9
 ---> 639668d38668
Step 5/10 : COPY . ./
 ---> da2c69f1ed89
Step 6/10 : RUN ["msbuild.exe", "dotnetapp-3.5.csproj", "/p:Configuration=Release"]
 ---> Running in 17731f3ff78b
MSBuild version 17.13.9+e0f243f1e for .NET Framework
Build started 2/12/2025 1:11:13 AM.

Project "C:\app\dotnetapp-3.5.csproj" on node 1 (default targets).
PrepareForBuild:
  Creating directory "bin\Release\".
  Creating directory "obj\Release\".
CoreCompile:
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\Roslyn\csc.exe /noconfig /nowarn:1701,1702 /fullpaths /nostdlib+ /platform:AnyCPU /errorreport:prompt /warn:4 /define:TRACE /highentropyva- /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscorlib.dll /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll" /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Data.DataSetExtensions.dll" /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Data.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Xml.dll /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Xml.Linq.dll" /debug:pdbonly /filealign:512 /optimize+ /out:obj\Release\dotnetapp-3.5.exe /target:exe /utf8output /langversion:7.3 Program.cs
  CompilerServer: server - server processed compilation - dotnetapp-3.5
_CopyAppConfigFile:
  Copying file from "C:\app\App.config" to "C:\app\bin\Release\dotnetapp-3.5.exe.config".
CopyFilesToOutputDirectory:
  Copying file from "C:\app\obj\Release\dotnetapp-3.5.exe" to "C:\app\bin\Release\dotnetapp-3.5.exe".
  dotnetapp-3.5 -> C:\app\bin\Release\dotnetapp-3.5.exe
  Copying file from "C:\app\obj\Release\dotnetapp-3.5.pdb" to "C:\app\bin\Release\dotnetapp-3.5.pdb".
Done Building Project "C:\app\dotnetapp-3.5.csproj" (default targets).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.72
 ---> Removed intermediate container 17731f3ff78b
 ---> c013001f2f79
Step 7/10 : FROM $BASE_RUNTIME_IMAGE
 ---> 6be13f34da99
Step 8/10 : WORKDIR /app
 ---> Running in ddb01c472c27
 ---> Removed intermediate container ddb01c472c27
 ---> a4e0f8cbf906
Step 9/10 : COPY --from=builder /app/bin/Release .
 ---> a34230502e44
Step 10/10 : ENTRYPOINT ["dotnetapp-3.5.exe"]
 ---> Running in 80c09df7e764
 ---> Removed intermediate container 80c09df7e764
 ---> b7460e1f038f
Successfully built b7460e1f038f
Successfully tagged dotnetapp-133837962683122340:latest
Execution Elapsed Time: 00:00:14.0412208
Executing: docker run --name dotnetapp-133837962683122340 --rm  dotnetapp-133837962683122340 
Hello Docker World from .NET 3.5!
Execution Elapsed Time: 00:00:03.0726308
Executing: docker image rm -f dotnetapp-133837962683122340
Untagged: dotnetapp-133837962683122340:latest
Deleted: sha256:b7460e1f038f46e35202c0c26d642b1daa6593543112080a4fe23a88131de836
Deleted: sha256:7f54a6643fb7a7aeaca635a28f8764899bc205bbfb81d4b458a6eb9f617c51e0
Deleted: sha256:a34230502e44248c802afc5346e6804fe018d80dab2d9186621c81ce42a1a7d4
Deleted: sha256:403bf14345694486a28cbe80ec4e65371d8fa02d083e8ad90dee3d2c133e05d9
Deleted: sha256:a4e0f8cbf906476b337328b4d3fc5b9cf19df4397e39b5f73ff5f93c0e1622cc
Deleted: sha256:52950c2ad24ff5de6f6b56869bae51dd959b76ea56525924497e5f1aa8c0f5e2
Execution Elapsed Time: 00:00:00.3939812

[rainersigwald]

Yeah, I think the most interesting info would likely be "what is the command line and full output of the thing that is failing?"

We do have a new dependency on that library in 17.13, but I would expect it to be present and working as part of the install.

One thing that's maybe different between your scenario and the test is that the test directly builds a .csproj without a .sln. Trying to see if that is just always broken in #1214.

cc @surayya-MS (due to the solution persistence library)

[willemk]

I'm also having issues with the new version of the docker images. Not sure if they are related.

I'm trying to do a docker build of a .Net framework 4.8 project (using the sdk:4.8.1 / aspnet:4.8.1 tags) , and it fails at the nuget restore step. If I comment out the nugest restore step, it fails at the MSBuild step.

The error thrown on both Run and Build commands is:
docker: Error response from daemon: container 201fca03db29805588aac8369f8bd435a9a9628e86f4145d6cdbe86075b764e3 encountered an error during hcs::System::Start: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106).

Checking the event viewer, it gives me a bit more details:

failed to start container [container=201fca03db29805588aac8369f8bd435a9a9628e86f4145d6cdbe86075b764e3 namespace=moby module=libcontainerd error=container 201fca03db29805588aac8369f8bd435a9a9628e86f4145d6cdbe86075b764e3 encountered an error during hcs::System::Start: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106)]

and

failed to cleanup after a failed Start [module=libcontainerd container=201fca03db29805588aac8369f8bd435a9a9628e86f4145d6cdbe86075b764e3 error=container 201fca03db29805588aac8369f8bd435a9a9628e86f4145d6cdbe86075b764e3 encountered an error during hcs::System::waitBackground: failure in a Windows system call: The virtual machine or container exited unexpectedly. (0xc0370106) namespace=moby]

I'm building this on Server 2025, build 26100.2894 running in a VM.

If I switch to the -20250114-windowsservercore-ltsc2025 it works fine.

[MichaelSimons]

@willemk - I think your issue is different. Can you log a new issue for this to keep better separation of issues? I will respond with a suggestion once you open it.

[Mik4sa]

I got the same error as @kewdgr. Our image is based on 4.8.1-20250211-windowsservercore-ltsc2022
Which info do you need to look further into this?

This is the command we run:
C:\BuildAgent\tools\NuGet.CommandLine.5.4.0\tools\NuGet.exe restore C:\BuildAgent\work\19a02017e0f563c3\Master.sln

And the output:

09:30:18     MSBuild auto-detection: using msbuild version '17.13.9.7704' from 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\amd64'.
09:30:18     Error parsing solution file at C:\BuildAgent\work\19a02017e0f563c3\Master.sln: Exception has been thrown by the target of an invocation.  Could not load file or assembly 'Microsoft.VisualStudio.SolutionPersistence, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
09:30:18     Process exited with code 1

[kewdgr]

That was a very interesting point, you are also using an "old" version of Nuget together with the container, and that seems to be the problem. When we are using the Version installed into the container it is working. It seems that the container has a problem with older Nuget versions ore the other way around.

[Mik4sa]

Nice, thanks. This indeed worked for me too. The version was controlled by my CI platform. I always thought that I couldn't change this version but indeed I can :) Thanks again

[rainersigwald]

I think you need a nuget.exe that has (at least) NuGet/NuGet.Client#4975 to support newer MSBuild/VS versions.

[kewdgr]

@rainersigwald Thanks for the hint! It seems a little bit odd that a stand alone executable throws an error like this on nuget restore. Not so stand-alone it seems :-)

[lbussell]

@kewdgr it seems like you found a solution. Can we close the issue?