MSTest Discovery Broken with 1.1.x

[michael-hawker]

Describe the bug

After updating to 1.1.1/1.1.2 from 1.0.3 MSTest discovery is now broken. This is a blocking issue for us.

We couldn't test in our CI before because we were blocked by #2631 as well. I tried in the CI, but it seems there may have been something missed with the fix for #2631 as it's trying to install 1.1.1 still for MSTest, see comment here on that issue.

Locally I see the same behavior still I did with 1.1.1:

========== Starting test discovery ==========
[MSTest][Discovery][H:\...\tests\SizerBase.UnitTests.WinAppSdk\bin\x64\Debug\net6.0-windows10.0.19041.0\SizerBase.UnitTests.WinAppSdk.dll] Failed to discover tests from assembly H:\...\tests\SizerBase.UnitTests.WinAppSdk\bin\x64\Debug\net6.0-windows10.0.19041.0\SizerBase.UnitTests.WinAppSdk.dll. Reason:File does not exist: H:\...\tests\SizerBase.UnitTests.WinAppSdk\bin\x64\Debug\net6.0-windows10.0.19041.0\SizerBase.UnitTests.WinAppSdk.dll
No test is available in H:\...\tests\SizerBase.UnitTests.WinAppSdk\bin\x64\Debug\net6.0-windows10.0.19041.0\SizerBase.UnitTests.WinAppSdk.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
========== Test discovery finished: 0 Tests found in 5.1 sec ==========

The file exists on disk, everything built. Only change is updating NuGet reference from 1.0.3 to 1.1.x causes tests now to not be discovered properly/run.

Steps to reproduce the bug

1. Create a MSTest based unit test project for WinAppSDK with 1.0.3
2. Run tests.
3. Close VS
4. Clean environment
5. Update to 1.1.x
6. Re-run tests (fail)

Note, you have to close VS/clean environment as VS caches test discovery, didn't detect this issue at first on initial update as VS used prior cache to run tests. Issue seems to be with test discovery itself.

Expected behavior

Able to run MSTest projects as was able to with 1.0.3.

Screenshots

No response

NuGet package version

1.1.1

Packaging type

No response

Windows version

Windows 10 version 21H2 (19044, November 2021 Update)

IDE

Visual Studio 2022

Additional context

No response

[riverar]

Can reproduce. Here's log output from my test project w/ verbosity cranked up. Not very helpful.

[7/5/2022 2:45:33.994 PM]  Operation Dequeue : 'RunFromCodeLensOperation'
[7/5/2022 2:45:33.999 PM]  Building Test Projects
[7/5/2022 2:45:34.070 PM]  Start building 1 containers.
[7/5/2022 2:45:34.070 PM]  Triggering build for 1 IProjectBasedTestContainers.
[7/5/2022 2:45:35.599 PM]  No IBuildableTestContainers were found.
[7/5/2022 2:45:35.599 PM]  Completed building containers.
[7/5/2022 2:45:35.599 PM]  Start updating 1 containers.
[7/5/2022 2:45:35.599 PM]  Updating container C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll.
[7/5/2022 2:45:35.615 PM]  Successfully refreshed container: C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll
[7/5/2022 2:45:35.615 PM]  Completed updating containers.
[7/5/2022 2:45:35.615 PM]  DiscoveryOperation<RunFromCodeLensOperation> Changed container count is 1
[7/5/2022 2:45:35.654 PM]  Test run animation has started.
[7/5/2022 2:45:35.663 PM]  Test run progress has started.
[7/5/2022 2:45:35.663 PM]  Discovering the following containers :
[7/5/2022 2:45:35.663 PM]  	C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll
[7/5/2022 2:45:35.765 PM]  Starting test discovery for requested test run
[7/5/2022 2:45:35.921 PM]  Project testproject references test adapter: MSTest.TestAdapter, version 2.2.8
[7/5/2022 2:45:36.123 PM]  ========== Starting test discovery ==========
[7/5/2022 2:45:36.124 PM]  File timestamp changed from 7/5/2022 2:34:56 PM to 7/5/2022 2:45:35 PM for C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll
[7/5/2022 2:45:36.126 PM]  Grouped C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll : (AnyCPU, FrameworkCore10, net6.0, )
[7/5/2022 2:45:36.128 PM]  Test discovery settings for C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll:
 <RunSettings>
  <BoostTestInternalSettings xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <VSProcessId>28340</VSProcessId>
  </BoostTestInternalSettings>
  <RunConfiguration>
    <ResultsDirectory>C:\Sources\was-repros\was_issue_2706\TestResults</ResultsDirectory>
    <SolutionDirectory>C:\Sources\was-repros\was_issue_2706\</SolutionDirectory>
    <CollectSourceInformation>False</CollectSourceInformation>
  </RunConfiguration>
  <GoogleTestAdapterSettings xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <SolutionSettings>
      <Settings />
    </SolutionSettings>
    <ProjectSettings />
  </GoogleTestAdapterSettings>
</RunSettings>.
[7/5/2022 2:45:36.565 PM] [Warning] [MSTest][Discovery][C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll] Failed to discover tests from assembly C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll. Reason:File does not exist: C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll
[7/5/2022 2:45:36.566 PM] [Warning] No test is available in C:\Sources\was-repros\was_issue_2706\tests\testproject\bin\Debug\net6.0-windows10.0.19041.0\testproject.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[7/5/2022 2:45:36.590 PM]  ========== Test discovery finished: 0 Tests found in 466.1 ms ==========
[7/5/2022 2:45:36.902 PM]  ========== Starting test run ==========
[7/5/2022 2:45:36.904 PM]  Attachments processing not needed
[7/5/2022 2:45:36.906 PM]  ========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in < 1 ms ==========
[7/5/2022 2:45:36.989 PM]  Test run animation has stopped.
[7/5/2022 2:45:37.021 PM]  Test run progress has stopped.

[michael-hawker]

Thanks @riverar for confirmation on another machine. I was hoping to get more details from the CI on our GitHub action as well, but that's been stuck because of #2631 as well (which seems resolved for our main build with 1.1.2 but is failing for the test project still - it seems that 1.1.2 is still trying to install the 1.1.1 package still in that case?).

[DefaultRyan]

I tried to reproduce this issue using MSTest 2.2.10, and the test discovery seemed to work.

Were you using MSTest 2.2.10? If 2.2.10 still doesn't work, could you share a minimal repro project? (I'm new to using MSTest).

[riverar]

Concur @DefaultRyan. Looks like this is fixed in 2.2.10, suspect it was related to microsoft/testfx#1064 or microsoft/testfx@6901b31. @michael-hawker tag you're it.

[michael-hawker]

Thanks folks, we were on 2.2.8 indeed. I believe I'll still be blocked from fully validating in our CI by the remaining issue with the fix for #2631 though.

I tried locally, and I see test discovery may have found them now, but didn't appear to run them:

Log level is set to Informational (Default).
Connected to test environment '< Local Windows Environment >'
Test data store opened in 0.023 sec.
Loaded 0 test records from the solution cache in 0.416 sec.
========== Starting test discovery ==========
========== Test discovery finished: 5 Tests found in 5.6 sec ==========
Starting test discovery for requested test run
========== Starting test discovery ==========
========== Test discovery finished: 5 Tests found in 20.4 sec ==========
========== Starting test run ==========
========== Test run finished: 10 Tests (5 Passed, 5 Failed, 0 Skipped) run in 424 ms ==========

Though I don't see anything listed in the Test Explorer now either:

I'm on 17.2.4 of VS, so I'll try updating that as well.

Was able to grab the message from the bubble over the method:

Message: 
    Initialization method SizerBase.Tests.ExampleSizerBaseTestClass.TestSetup threw exception. System.TypeInitializationException: The type initializer for 'CommunityToolkit.Labs.UnitTests.App' threw an exception. ---> System.TypeInitializationException: The type initializer for 'WinRT.ActivationFactory`1' threw an exception. ---> System.Runtime.InteropServices.COMException: Class not registered (0x80040154 (REGDB_E_CLASSNOTREG)).
    TestCleanup method SizerBase.Tests.ExampleSizerBaseTestClass.TestCleanup threw exception. System.TypeInitializationException: System.TypeInitializationException: The type initializer for 'CommunityToolkit.Labs.UnitTests.App' threw an exception. ---> System.TypeInitializationException: The type initializer for 'WinRT.ActivationFactory`1' threw an exception. ---> System.Runtime.InteropServices.COMException: Class not registered (0x80040154 (REGDB_E_CLASSNOTREG)).

  Stack Trace: 
    BaseActivationFactory.ctor(String typeNamespace, String typeFullName)
    ActivationFactory`1.ctor()
    ActivationFactory`1.cctor()
    --- End of inner exception stack trace ---
    ActivationFactory`1.As(Guid iid)
    Window.Make___objRef_global__Microsoft_UI_Xaml_IWindowStatics()
    Window.get__objRef_global__Microsoft_UI_Xaml_IWindowStatics()
    Window.get_Current()
    App.cctor() line 41

Looks like something may be going on with accessing the Window.Current now that wasn't happening with 1.0.3:

public sealed partial class App : Application
{
#if WINAPPSDK
    private static Microsoft.UI.Xaml.Window currentWindow = Microsoft.UI.Xaml.Window.Current; // Line 41

@DefaultRyan I've invited you to our repo, so you can see the PR with the WinAppSDK and MSTest updates here: CommunityToolkit/Labs-Windows#183 (it's working on our main with WAS 1.0.3 and MSTest 2.2.8).

I'd be curious if you just add the line above with the reference to Window.Current in your setup for App.xaml.cs, if you'll hit the same issue though?

I tried putting breakpoints on this line and in the constructor, but I don't see them being hit when I ask it to debug the test from the code bubble.

[michael-hawker]

I updated to 17.2.6 and I can see the tests again now in Test Explorer, but still hit the WinRT activation issue now with all the WinUI tests.

[DefaultRyan]

@DrusTheAxe and @huichen123 may have seen something similar. Could one of you respond here regarding the class activation issue?

Is there a WindowsAppSDK issue, an MSTest issue, or is there some manual configuration that is expected for our customers to use 1.1 with MSTest (and if so, is it documented)?

[mikebattista]

Is the app project using MSIX? What about the test project?

[michael-hawker]

We're using single-package MSIX projects for both app and tests:

  <ItemGroup Condition="'$(DisableMsixProjectCapabilityAddedByProject)'!='true' and '$(EnablePreviewMsixTooling)'=='true'">
    <ProjectCapability Include="Msix" />
  </ItemGroup>