Question on adding new test projects to the ci.yml

[clrudolphi]

I am attempting to add a test project for the xUnit3 Generator Plugin. I have created a new project within the solution, in a new sub-folder of the Tests/ folder.

When adding this test project to the ci.yml, I (admittedly in a cargo-cult fashion) tried to mimic the other invocations of test projects. When executed however, I am getting an error about the GitHubActions logger not being found.

Run dotnet test ./Tests/TestFrameworkPlugins/Reqnroll.xUnit3.Generator.ReqnrollPluginTests/Reqnroll.xUnit3.Generator.ReqnrollPluginTests.csproj --logger "trx;LogFileName=/home/runner/work/Reqnroll/Reqnroll/TestResults/xunit3-generator-plugintests-results.trx" --no-restore --verbosity minimal --logger "GitHubActions;summary.includePassedTests=true;summary.includeSkippedTests=true;annotations.titleFormat=[@traits.Category] @test;annotations.messageFormat=@error\n@trace" -- RunConfiguration.CollectSourceInformation=true RunConfiguration.TreatNoTestsAsError=true
  Reqnroll.Utils -> /home/runner/work/Reqnroll/Reqnroll/Reqnroll.Utils/bin/Debug/netstandard2.0/Reqnroll.Utils.dll
  Reqnroll -> /home/runner/work/Reqnroll/Reqnroll/Reqnroll/bin/Debug/netstandard2.0/Reqnroll.dll
  Reqnroll.xUnit3.ReqnrollPlugin -> /home/runner/work/Reqnroll/Reqnroll/Plugins/Reqnroll.xUnit3.ReqnrollPlugin/bin/Debug/netstandard2.0/Reqnroll.xUnit3.ReqnrollPlugin.dll
  Reqnroll.Parser -> /home/runner/work/Reqnroll/Reqnroll/Reqnroll.Parser/bin/Debug/netstandard2.0/Reqnroll.Parser.dll
  Reqnroll.Generator -> /home/runner/work/Reqnroll/Reqnroll/Reqnroll.Generator/bin/Debug/netstandard2.0/Reqnroll.Generator.dll
  Reqnroll.xUnit3.Generator.ReqnrollPlugin -> /home/runner/work/Reqnroll/Reqnroll/Plugins/Reqnroll.xUnit3.Generator.ReqnrollPlugin/bin/Debug/netstandard2.0/Reqnroll.xUnit3.Generator.ReqnrollPlugin.dll
  The package Reqnroll.xunit.v3.3.2.2-local is missing a readme. Go to https://aka.ms/nuget/authoring-best-practices/readme to learn why package readmes are important.
  Successfully created package '/home/runner/work/Reqnroll/Reqnroll/Plugins/Reqnroll.xUnit3.Generator.ReqnrollPlugin/bin/Debug/Reqnroll.xunit.v3.3.2.2-local.nupkg'.
  The package Reqnroll.xunit.v3.3.2.2-local is missing a readme. Go to https://aka.ms/nuget/authoring-best-practices/readme to learn why package readmes are important.
  Successfully created package '/home/runner/work/Reqnroll/Reqnroll/Plugins/Reqnroll.xUnit3.Generator.ReqnrollPlugin/bin/Debug/Reqnroll.xunit.v3.3.2.2-local.snupkg'.
  Reqnroll.xUnit3.Generator.ReqnrollPluginTests -> /home/runner/work/Reqnroll/Reqnroll/Tests/TestFrameworkPlugins/Reqnroll.xUnit3.Generator.ReqnrollPluginTests/bin/Debug/net8.0/Reqnroll.xUnit3.Generator.ReqnrollPluginTests.dll
Test run for /home/runner/work/Reqnroll/Reqnroll/Tests/TestFrameworkPlugins/Reqnroll.xUnit3.Generator.ReqnrollPluginTests/bin/Debug/net8.0/Reqnroll.xUnit3.Generator.ReqnrollPluginTests.dll (.NETCoreApp,Version=v8.0)
VSTest version 18.0.1 (x64)

Starting test execution, please wait...
Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'GitHubActions'.

Any suggestions on what mistake I've made and how to fix?

[gasparnagy]

To answer the technical part of the question, that dependency has to be added to the project file itself as a conditional dependency that is only active during CI, see for example here.

BUT: I would NOT add a separate test project for the plugins. The unit tests for the plugins (generator or runtime) should go to Reqnroll.PluginTests. The end-to-end system/integration tests should go to Reqnroll.SystemTests/Plugins in general (for xUnit3 to the XUnit3GenerationTest class).

[clrudolphi]

Can you help me past a blocker then?
I first attempted to test this in Reqnroll.GeneratorTests. I've also just now attempted to move this to Reqnroll.PluginTests (as suggested above). In both situations, there is a blocker in that by referencing both xUnit and xUnit3, the compiler gets confused about how to resolve references to xunit-related attributes. (The type 'FactAttribute' exists in both 'xunit.core, Version=2.9.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' and 'xunit.v3.core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c')

I had concluded that we cannot test xunit3 code with the xunitv2. That was one of the motivators behind adding a new project.

[gasparnagy]

Oh... I see. Let me think on this and come back.

[obligaron]

It's possible to avoid the compile time conflict with PackageReference aliases.
How good this will work at runtime is another thing (e.g. assembly name conflict).

Another ideas:

• We could add SystemTests that check the result (e.g. that a test is not parallel executed) instead of checking the generated code (see SystemTests: add a test for non parallelizable tag #936 for example).
• We could generated the code outproc (like in the SystemTests) and inspect the generated code that is stored on the drive instead of in-memory.

[Code-Grump]

Now I have a working (but not yet reviewed) branch with project-based packing, it may be somewhat easier to create and execute these kinds of tests due to some of the structural changes and dependency flowing that's now possible.