[BUG]: InvalidLoggerException: Could not find a test logger with FriendlyName 'microsoft-playwright-testing' when executed tests by VSTest@3 task in Azure Pipelines

[kmarek]

Describe the bug

C# nUnit/Playwright tests run by VSTest task results in following error:
##[error]DiscoveryMessage : Microsoft.VisualStudio.TestPlatform.Common.Exceptions.InvalidLoggerException: Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'microsoft-playwright-testing'.

To Reproduce
Steps to reproduce the behavior:

1. Step 1 - configure test solution

• latest package Azure.Developer.MicrosoftPlaywrightTesting.NUnit installed in nUnit project
• class PlaywrightServiceSetup added to project

using Azure.Developer.MicrosoftPlaywrightTesting.NUnit;

namespace CompanyName.TestAutomation.AppName;

[SetUpFixture]
public class PlaywrightServiceSetup : PlaywrightServiceNUnit { };

• azurepts.runsettings file added to repo

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <TestRunParameters>
        <Parameter name="Os" value="linux" />
        <Parameter name="ServiceAuthType" value="AccessToken" />
        <Parameter name="UseCloudHostedBrowsers" value="true" />
        <Parameter name="EnableGitHubSummary" value="false" />
    </TestRunParameters>
  <!-- NUnit adapter -->  
  <NUnit>
    <NumberOfTestWorkers>20</NumberOfTestWorkers>
    <TestOutputXml>TestResults</TestOutputXml>
    <SkipExecutionWhenNoTests>True</SkipExecutionWhenNoTests>
    <AllowParallelWithDebugger>True</AllowParallelWithDebugger>
  </NUnit>
  <RunConfiguration>
    <EnvironmentVariables>
        <DEBUG>pw:api*</DEBUG>
        <PLAYWRIGHT_SERVICE_URL>wss://<region>.microsoft.com/accounts/<workspace_ID>/browsers</PLAYWRIGHT_SERVICE_URL>
        <PLAYWRIGHT_SERVICE_OS>linux</PLAYWRIGHT_SERVICE_OS>
    </EnvironmentVariables>
  </RunConfiguration>
  <Playwright>
    <BrowserName>chromium</BrowserName>
    <ExpectTimeout>30000</ExpectTimeout>
    <LaunchOptions>
      <Headless>false</Headless>
    </LaunchOptions>
  </Playwright>
    <LoggerRunSettings>
        <Loggers>            
            <Logger friendlyName="microsoft-playwright-testing" enabled="True" />           
            <Logger friendlyName="trx" enabled="false" />
        </Loggers>
    </LoggerRunSettings>
</RunSettings>

3. Yaml Pipeline created in Azure Pipelines

vmImage: 'windows-latest'

related steps

- task: DotNetCoreCLI@2
      displayName: 'Dotnet Build'
      inputs:
        projects: |
          **/*.csproj

- task: VSTest@3
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **/CompanyName.TestAutomation.*.dll
      !**/CompanyName.TestAutomation.Framework.*.dll
    testRunTitle: ${{ parameters.testRunName }}
    searchFolder: '$(System.DefaultWorkingDirectory)'
    resultsFolder: '$(Agent.TempDirectory)/TestResults'
    runSettingsFile: src/azurepts.runsettings
    runInParallel: true
    platform: 'Any CPU'
    configuration: 'Release'
    rerunType: 'basedOnTestFailurePercentage' #'basedOnTestFailureCount'
    rerunFailedThreshold: ${{ parameters.rerunFailedThreshold }}
    #rerunFailedTestCasesMaxLimit: 20
    rerunFailedTests: true
    rerunMaxAttempts: ${{ parameters.rerunMaxAttempts }}
    testFiltercriteria: ${{ parameters.filter }}
    diagnosticsEnabled: true    
  env:
    E2E-ENVIRONMENT: ${{ parameters.environment }}
    PLAYWRIGHT_SERVICE_ACCESS_TOKEN: $(MPT_TOKEN)

where MPT_TOKEN is pipeline secret variable

5. Execute pipeline
6. See error

Count of test sources found: 1
=================================================================
Discovering tests from sources
Using new test platform for test execution
Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
##[error]DiscoveryMessage : Microsoft.VisualStudio.TestPlatform.Common.Exceptions.InvalidLoggerException: Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'microsoft-playwright-testing'.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.TestLoggerManager.Initialize(String runSettings) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/TestLoggerManager.cs:line 183
   at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateDiscoveryRequest(IRequestData requestData, DiscoveryCriteria discoveryCriteria, TestPlatformOptions options, Dictionary`2 sourceToSourceDetailMap, IWarningLogger warningLogger) in /_/src/Microsoft.TestPlatform.Client/TestPlatform.cs:line 89
   at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.DiscoverTests(DiscoveryRequestPayload discoveryPayload, ITestDiscoveryEventsRegistrar discoveryEventsRegistrar, ProtocolConfig protocolConfig) in /_/src/vstest.console/TestPlatformHelpers/TestRequestManager.cs:line 245
   at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.<>c__DisplayClass29_0.<StartDiscovery>b__0() in /_/src/Microsoft.TestPlatform.Client/DesignMode/DesignModeClient.cs:line 530
Number of testcases discovered : 0
Discovered tests 0 from sources
=================================================================
##[error]The slice of type 'Discovery' is 'Aborted' because of the error : System.Exception: Microsoft.VisualStudio.TestPlatform.Common.Exceptions.InvalidLoggerException: Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'microsoft-playwright-testing'.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.TestLoggerManager.Initialize(String runSettings) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/TestLoggerManager.cs:line 183
   at Microsoft.VisualStudio.TestPlatform.Client.TestPlatform.CreateDiscoveryRequest(IRequestData requestData, DiscoveryCriteria discoveryCriteria, TestPlatformOptions options, Dictionary`2 sourceToSourceDetailMap, IWarningLogger warningLogger) in /_/src/Microsoft.TestPlatform.Client/TestPlatform.cs:line 89
   at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.DiscoverTests(DiscoveryRequestPayload discoveryPayload, ITestDiscoveryEventsRegistrar discoveryEventsRegistrar, ProtocolConfig protocolConfig) in /_/src/vstest.console/TestPlatformHelpers/TestRequestManager.cs:line 245
   at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.<>c__DisplayClass29_0.<StartDiscovery>b__0() in /_/src/Microsoft.TestPlatform.Client/DesignMode/DesignModeClient.cs:line 530
Received the command : Stop
TestExecutionHost.ProcessCommand. Stop Command handled
SliceFetch Aborted. Moving to the TestHostEnd phase
Test run '1371174' is in 'Aborted' state.
##[error]Test run is aborted. Logging details of the run logs.
##[error]System.Exception: The test run was aborted, failing the task.

Expected behavior
Tests are executed using Playwright Testing service and report is added to dashboard

Setup information (please complete the following information):

• Azure Subscription Id: fb313fed-52f0-48fe-aa8d-74dc8458fab7
• Workspace Name: WS0DEV
• Service trace Id (if available):
• Log snippet (if available):
• Browser on which tests are executed (if applicable): chrome
• Time range (in PST) when the issue was experienced:
• Playwright Version:

 Azure.Developer.MicrosoftPlaywrightTesting.NUnit: 1.0.0-beta.4
 Microsoft.Playwright.NUnit: 1.49.0
 NUnit: 4.3.2

• Client machine OS: vmImage: 'windows-latest'
• OS requested on Service: linux

Additional context
NOTE 1:
When I delete following section from azurepts.runsettings file

<LoggerRunSettings>
        <Loggers>            
            <Logger friendlyName="microsoft-playwright-testing" enabled="True" />           
            <Logger friendlyName="trx" enabled="false" />
        </Loggers>
    </LoggerRunSettings>

tests are executed successfully on Azure Playwright Services but report is not created in workspace

NOTE 2:
When I use AzureCLI. tests are executed successfully and report is created in workspace:

- task: AzureCLI@2
  displayName: Run Playwright Test  
  env:
    PLAYWRIGHT_SERVICE_URL: 'wss://<region>.api.playwright.microsoft.com/accounts/<workspace_ID>/browsers'
    E2E-ENVIRONMENT: ${{ parameters.environment }}
  inputs:
    azureSubscription: ${{ parameters.subscriptionName }} 
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      dotnet test CompanyName.TestAutomation.dll --settings:src/azurepts.runsettings --filter ${{ parameters.filter }} --logger "microsoft-playwright-testing" -- NUnit.NumberOfTestWorkers=10
    addSpnToEnvironment: true
    workingDirectory: '$(System.DefaultWorkingDirectory)' 

also tests are executed successfully and report is generated in workspace when executed locally using command dotnet test --logger "microsoft-playwright-testing" -- NUnit.NumberOfTestWorkers=10

[puagarwa]

Hi @kmarek ,
Currently we have known issue in vsts console exe while using logger due to conflict in dotnet and netstandard resolution. So it wont work. I will suggest to use dotnet cli. I understand you may have usecase where you need to use vstest but unfortunately its limitation with our logger.

[kmarek]

Thanks for the info. The main advantage of vstest over dotnet test is its ability to automatically re-run failed tests.

Is there an estimated timeline for fixing the known issue in the vstest.console.exe?

[puagarwa]

@kmarek Unfortunately, No ETA yet on this.