SimpleJob attribute: how so specify RuntimeMoniker equivalent to net6.0-windows

[DanStevens]

I want to benchmark classes in a WPF Class Library that multi-targets both .NET 6.0 and .NET Framework 4.8. Here's project file:

WpfClassLibrary.cs:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net6.0-windows;net4.8</TargetFrameworks>
    <UseWPF>true</UseWPF>
  </PropertyGroup>
</Project>

The class I want to test, simple instantiates a System.Windows.Documents.FlowDocument object:

namespace WpfClassLibrary
{
    using System.Windows.Documents;

    public class WpfClass
    {
        public FlowDocument Document { get; } = new FlowDocument();
    }
}

Here's the project file for the benchmarking project:

BenchmarkingWpfClassLibrary.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net6.0-windows;net4.8</TargetFrameworks>
    <UseWPF>true</UseWPF>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.7" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\WpfClassLibrary\WpfClassLibrary.csproj" />
  </ItemGroup>
</Project>

And the benchmark itself:

Program.cs:

namespace Benchmarks
{
    using BenchmarkDotNet.Attributes;
    using BenchmarkDotNet.Jobs;
    using BenchmarkDotNet.Running;
    using WpfClassLibrary;

    internal class Program
    {
        static void Main(string[] args)
        {
            var results = BenchmarkRunner.Run<WpfClassBenchmarks>();
        }
    }

    //[SimpleJob(RuntimeMoniker.Net48)]
    //[SimpleJob(RuntimeMoniker.Net60)]
    public class WpfClassBenchmarks
    {
        [Benchmark]
        public void WpfClass()
        {
            _ = new WpfClass();
        }
    }
}

If I run this as-is the benchmark builds and runs fine. However, if I comment out the two SimpleJob attributes, the .NET Framework 4.8 benchmark builds and runs fine, but the .NET 6.0 benchmark does not. I get the following error:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'PresentationFramework, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
File name: 'PresentationFramework, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
   at WpfClassLibrary.WpfClass..ctor()
   at Benchmarks.WpfClassBenchmarks.WpfClass() in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\Program.cs:line 23
   at BenchmarkDotNet.Autogenerated.Runnable_0.WorkloadActionNoUnroll(Int64 invokeCount) in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\bin\Release\net6.0-windows\7eaf257f-9929-493b-9995-b7827893b4db\7eaf257f-9929-493b-9995-b7827893b4db.notcs:line 311
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName) in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\bin\Release\net6.0-windows\7eaf257f-9929-493b-9995-b7827893b4db\7eaf257f-9929-493b-9995-b7827893b4db.notcs:line 176
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\Users\stevensd\source\BenchmarkingWpfClassLibrary\Benchmarks\bin\Release\net6.0-windows\7eaf257f-9929-493b-9995-b7827893b4db\7eaf257f-9929-493b-9995-b7827893b4db.notcs:line 57

I think what is happening is that when specifying no SimpleJob attribute, BenchmarkDotNet is building benchmark using the appropriate net6.0-windows moniker. However, when using the SimpleJob attribute with a moniker RuntimeMoniker.Net60 it's being built with the incorrect net6.0.

How can I specify a simple job that uses the runtime moniker of net60-windows and not net60?

Source code: https://github.com/DanStevens/BenchmarkingWpfClassLibrary

[adamsitnik]

Hi @DanStevens

We don't provide API for that, the best I can do right now is to provide a workaround (see the explanation in comments):

var net48 = Job.Default.WithRuntime(ClrRuntime.Net48).AsBaseline().WithId(".NET 4.8");
// It's impossible to specify "net6.0-windows" via the existing APIs.
// But if the process is run as net6.0, the default BDN Job will properly recognize and use "net6.0-windows" moniker.
var net60windows = Job.Default.WithId(".NET 6.0");

var config = DefaultConfig.Instance
    .AddJob(net48)
    .AddJob(net60windows);

BenchmarkRunner.Run<WpfClassBenchmarks>(config);

dotnet run -c Release -f net6.0

[timcassell]

You could manually create the toolchain like this:

Job.Default.WithToolchain(CsProjCoreToolchain.From(new NetCoreAppSettings("net6.0-windows", null, ".NET 6.0 Windows")))

That doesn't rely on the host process.