Build problems with BenchmarkDotNet 0.14.0 and Grpc.Tools 2.67.0

[brgirgis]

Upgrading to latest BenchmarkDotNet causes projects with protobuf files to fail. Last known good version is 0.13.12. The difference seems to be related to the newly added parameters for the build step:

• 0.13.12:

  start dotnet  build -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /path/to/PerformanceTests/bin/Release/net8.0/945898c9-e8ca-4ce1-afdf-c4f6c3f66d8b
  

• Problematic 0.14.0

  start dotnet  build -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:IntermediateOutputPath="/path/to/PerformanceTests/bin/Release/net8.0/2faf0c9d-78ff-4730-8350-c4a193d9211f/obj/Release/net8.0/" /p:OutDir="/path/to/PerformanceTests/bin/Release/net8.0/2faf0c9d-78ff-4730-8350-c4a193d9211f/bin/Release/net8.0/" /p:OutputPath="/path/to/PerformanceTests/bin/Release/net8.0/2faf0c9d-78ff-4730-8350-c4a193d9211f/bin/Release/net8.0/" --output "/path/to/PerformanceTests/bin/Release/net8.0/2faf0c9d-78ff-4730-8350-c4a193d9211f/bin/Release/net8.0/" in /path/to/PerformanceTests/bin/Release/net8.0/2faf0c9d-78ff-4730-8350-c4a193d9211f
  

It looks like the new parameters are confusing the Grpc.Tools package somehow:

• /p:IntermediateOutputPath
• /p:OutDir
• /p:OutputPath
• --output

It would be great to be able to restore the old behavior somehow.

[timcassell]

Can you please share a minimal repro project?

[muss0v]

Same problem with version 0.14.0 and Grpc.Tools. But I cannot seem to reproduce this in a minimal repo project. Error I'm getting:

Details

Setup power plan (GUID: 8c5e7ega-e8bf-4a96-9a85-a623a8c635c FriendlyName: High performance)
// Build Error: Standard output:

Standard error:
C:\Users\test.nuget\packages\grpc.tools\2.67.0\build_protobuf\Google.Protobuf.Tools.targets(341,5): warning : Some expected protoc outputs were not generated. [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]
C:\Users\test.nuget\packages\grpc.tools\2.67.0\build_protobuf\Google.Protobuf.Tools.targets(341,5): warning : [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]
C:\Users\test.nuget\packages\grpc.tools\2.67.0\build_protobuf\Google.Protobuf.Tools.targets(341,5): warning : C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\3faecf9f-d773-4edf-b06d-e0610594ee22\obj\Release\net8.0/Protos\Exchange.cs [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]
C:\Users\test.nuget\packages\grpc.tools\2.67.0\build_protobuf\Google.Protobuf.Tools.targets(341,5): warning : C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\3faecf9f-d773-4edf-b06d-e0610594ee22\obj\Release\net8.0/Protos\ExchangeGrpc.cs [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]

[timcassell]

Can you build your project with custom output paths without BenchmarkDotNet (passed to command line, not in csproj)? It's possibly an issue with Grpc.Tools itself.

[muss0v]

Can you build your project with custom output paths without BenchmarkDotNet (passed to command line, not in csproj)? It's possibly an issue with Grpc.Tools itself.

Well, just building the project is fine. But when you run the benchmark project it will give an error after step "Setup power plan ...". I also noticed that the previous version (0.13.12) has an extra build command. Check console outputs when running the Benchmark project:

0.13.12:

Running benchmark project!
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 4 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet  restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\2f55aa58-798a-41b5-9a80-6f274986e122
// command took 2.14 sec and exited with 0
// start dotnet  build -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\2f55aa58-798a-41b5-9a80-6f274986e122
// command took 3.07 sec and exited with 1
// start dotnet  build -c Release --no-restore --no-dependencies /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\2f55aa58-798a-41b5-9a80-6f274986e122
// command took 4.2 sec and exited with 0
// ***** Done, took 00:00:09 (9.61 sec)   *****
// Found 4 benchmarks:
//   JsonConvertDeserializeTest.Custom: DefaultJob
//   JsonConvertDeserializeTest.Custom2: DefaultJob
//   JsonConvertDeserializeTest.Custom3: DefaultJob
//   JsonConvertDeserializeTest.TestClassic: DefaultJob

Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: JsonConvertDeserializeTest.Custom: DefaultJob
// *** Execute ***
// Launch: 1 / 1
// Execute: dotnet 2f55aa58-798a-41b5-9a80-6f274986e122.dll --anonymousPipes 3200 1308 --benchmarkName TestProject.Benchmark.Tests.JsonConvertDeserializeTest.Custom --job Default --benchmarkId 0 in C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\2f55aa58-798a-41b5-9a80-6f274986e122\bin\Release\net8.0
// BeforeAnythingElse

// Benchmark Process Environment Information:
// BenchmarkDotNet v0.13.12
// Runtime=.NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
// GC=Concurrent Workstation
// HardwareIntrinsics=AVX2,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT VectorSize=256
// Job: DefaultJob

...

0.14.0:

Running benchmark project!
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 4 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet  restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:IntermediateOutputPath="C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\obj\Release\net8.0/" /p:OutDir="C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\bin\Release\net8.0/" /p:OutputPath="C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\bin\Release\net8.0/" in C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04
// command took 2.23 sec and exited with 0
// start dotnet  build -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:IntermediateOutputPath="C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\obj\Release\net8.0/" /p:OutDir="C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\bin\Release\net8.0/" /p:OutputPath="C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\bin\Release\net8.0/" --output "C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\bin\Release\net8.0/" in C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04
// command took 11.43 sec and exited with 1
// ***** Done, took 00:00:13 (13.85 sec)   *****
// Found 4 benchmarks:
//   JsonConvertDeserializeTest.Custom: DefaultJob
//   JsonConvertDeserializeTest.Custom2: DefaultJob
//   JsonConvertDeserializeTest.Custom3: DefaultJob
//   JsonConvertDeserializeTest.TestClassic: DefaultJob

Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// Build Error: Standard output:

 Standard error:
 C:\Users\test\.nuget\packages\grpc.tools\2.67.0\build\_protobuf\Google.Protobuf.Tools.targets(341,5): warning : Some expected protoc outputs were not generated. [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]
C:\Users\test\.nuget\packages\grpc.tools\2.67.0\build\_protobuf\Google.Protobuf.Tools.targets(341,5): warning :  [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]
C:\Users\test\.nuget\packages\grpc.tools\2.67.0\build\_protobuf\Google.Protobuf.Tools.targets(341,5): warning :     C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\obj\Release\net8.0/Protos\Test.cs [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]
C:\Users\test\.nuget\packages\grpc.tools\2.67.0\build\_protobuf\Google.Protobuf.Tools.targets(341,5): warning :     C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Benchmark\bin\Release\net8.0\9778aead-6650-4e9a-b880-365c8220dc04\obj\Release\net8.0/Protos\TestGrpc.cs [C:\Users\test\Source\Repos\TestProject\TestProject\TestProject.Shared\TestProject.Shared.csproj]

...

[timcassell]

I also noticed that the previous version (0.13.12) has an extra build command.

The first build actually failed, but the log didn't tell you the reason // command took 3.07 sec and exited with 1. v0.13.12 and older would retry the build with --no-dependencies because of pathing issues. Those extra output arguments were added in 0.14.0 to fix that, which is why that step was removed. Possibly it was trying to build on top of the running exe, but it could be for any other reason.

Well, just building the project is fine.

I meant building it manually, e.g.

dotnet build /p:IntermediateOutputPath={intermediatePath} /p:OutDir={binaryPath} /p:OutputPath={binaryPath} --output={binaryPath}

It would also help if you could share a repro project so I can debug it myself (it doesn't have to be minimal).

[muss0v]

I also noticed that the previous version (0.13.12) has an extra build command.

The first build actually failed, but the log didn't tell you the reason // command took 3.07 sec and exited with 1. v0.13.12 and older would retry the build with --no-dependencies because of pathing issues. Those extra output arguments were added in 0.14.0 to fix that, which is why that step was removed. Possibly it was trying to build on top of the running exe, but it could be for any other reason.

Well, just building the project is fine.

I meant building it manually, e.g.

dotnet build /p:IntermediateOutputPath={intermediatePath} /p:OutDir={binaryPath} /p:OutputPath={binaryPath} --output={binaryPath}
It would also help if you could share a repro project so I can debug it myself (it doesn't have to be minimal).

I discovered that the issue was caused by a COM reference to IWshRuntimeLibrary in one of the referenced projects. After removing this reference, the build works again. I'm not sure if this was related to the original OP/Grpc.Tools issue, but at least it's resolved for me now.

[brgirgis]

The real wonder for me is really why there is a build step at all. Obviously the exe has built, why does the benchmark code need to mess with it at all?

[brgirgis]

As a user, I should also have the option to opt out of the build step if I so choose.

[timcassell]

The real wonder for me is really why there is a build step at all. Obviously the exe has built, why does the benchmark code need to mess with it at all?

To achieve process-level isolation, BenchmarkDotNet generates, builds and executes a new console app per every benchmark.

As a user, I should also have the option to opt out of the build step if I so choose.

You can, using [InProcess] toolchain.

https://benchmarkdotnet.org/articles/configs/toolchains.html

---

@brgirgis Are you able to share a project to reproduce the build error?

[codito]

This issue might be due to additional slash (Path.AltDirectorySeparatorChar) appended to the IntermediateOutputPath and OutputPath in the build command:

BenchmarkDotNet/src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommand.cs

Lines 265 to 269 in 804482d

	.AppendArgument($"/p:IntermediateOutputPath=\"{artifactsPaths.IntermediateDirectoryPath}{Path.AltDirectorySeparatorChar}\"")
	.AppendArgument($"/p:OutDir=\"{artifactsPaths.BinariesDirectoryPath}{Path.AltDirectorySeparatorChar}\"")
	// OutputPath is legacy, per-project version of OutDir. We set both just in case. https://github.com/dotnet/msbuild/issues/87
	.AppendArgument($"/p:OutputPath=\"{artifactsPaths.BinariesDirectoryPath}{Path.AltDirectorySeparatorChar}\"")
	.AppendArgument(isRestore ? string.Empty : $"--output \"{artifactsPaths.BinariesDirectoryPath}{Path.AltDirectorySeparatorChar}\"");

Related comment from the grpc tools repo: grpc/grpc#19816 (comment)

[mfeemster]

Just wanted to confirm I am seeing this too. I couldn't figure out why my benchmarks would fail when referencing another project.
Now I see that project references Interop.IWshRuntimeLibrary and the only way to get the benchmarks running again is to use [InProcess] as mentioned above.