Setting global verify setting fails on startup in Verify.XunitV3

[benmccallum]

Hey @SimonCropp ,

We have the following, ad-hoc but recurring (sort of one in every 2 CI builds) issue:

The API 'AutoGuru.Shared.TestUtilities.TestContextNormalizingHelper.InitializeVerifierSettings' must be called prior to any Verify has run. Usually this is done in a [ModuleInitializer].

The thing is:

1. We are calling TestContextNormalizingHelper.InitializeVerifierSettings from a module initializer that looks like this:

internal static class ModuleInitializer
{
    [ModuleInitializer]
    internal static void Init()
    {
        TestContextNormalizingHelper.Init();
    }
}

2. This never fails in local development (Windows) and only fails in CI (Linux on GHAs).
3. We do the same thing for other test projects, but it's just this one.
4. I've also tried using XUnit V3's pipeline startup and an interlock to force this to happen before tests run, but it still happens on CI, it's weird! If I debug this with a breakpoint in Init() and in StartAsync, I see that Init() is called first implying that the module initializer is calls it before the startup pipeline, so my attempted fix is also useless.

[assembly: Xunit.v3.TestPipelineStartup(typeof(AutoGuru.TestPipelineStartup))]

namespace AutoGuru;

#pragma warning disable IDE0065 // Misplaced using directive
using System.Runtime.CompilerServices;
using System.Threading.Tasks;
using AutoGuru.Shared.TestUtilities;
using Xunit.Sdk;

internal static class ModuleInitializer
{
    private static int _hasRun = 0;

    [ModuleInitializer]
    internal static void Init()
    {
        if (Interlocked.Exchange(ref _hasRun, 1) == 0)
        {
            TestContextNormalizingHelper.Init();
        }
    }
}

public class TestPipelineStartup : Xunit.v3.ITestPipelineStartup
{

    public ValueTask StartAsync(IMessageSink diagnosticMessageSink)
    {
        ModuleInitializer.Init();
        return default;
    }

    public ValueTask StopAsync()
    {
        return default;
    }
}

Any suggestions!? I'm not sure if this is a Verify issue, Xunit V3 issue or even .net runtime issue, because we upgraded all three at the same time.

Current stack:

1. Verify.XunitV3 - 28.15.0
2. xunit.v3 - 3.0.0
3. .NET 9 (latest SDK does the CI build)

[benmccallum]

I think I might've realised that the test project in question depends on another test project which annoyingly has the same module initializer calling the same TestContextNormalizingHelper.Init();.

That project is effectively a test utils and tests containing project which is never a good thing.

I can solve it with an interlocked exchange in the TestContextNormalizingHelper.Init(); method so init only ever happens once. Will see how that goes and report back if it happens again.