Merge pull request #286300 from vvs11/main

Playwright Testing documentation updates for reporting and Entra ID support

Author: ttorble

articles/playwright-testing/how-to-configure-visual-comparisons.md

@@ -44,40 +44,29 @@ Example service config that runs visual comparisons and configures the path for
 
 ```typeScript
 import { defineConfig } from '@playwright/test';
+import { getServiceConfig, ServiceOS } from '@azure/microsoft-playwright-testing';
 import config from './playwright.config';
-import dotenv from 'dotenv';
-
-dotenv.config();
-
-// Name the test run if it's not named yet.
-process.env.PLAYWRIGHT_SERVICE_RUN_ID = process.env.PLAYWRIGHT_SERVICE_RUN_ID || new Date().toISOString();
-
-// Can be 'linux' or 'windows'.
-const os = process.env.PLAYWRIGHT_SERVICE_OS || 'linux';
-
-export default defineConfig(config, {
-  workers: 20,
-
-  // Enable screenshot testing and configure directory with expectations.
-  ignoreSnapshots: false,
-  snapshotPathTemplate: `{testDir}/__screenshots__/{testFilePath}/${os}/{arg}{ext}`,
-
-  use: {
-    // Specify the service endpoint.
-    connectOptions: {
-      wsEndpoint: `${process.env.PLAYWRIGHT_SERVICE_URL}?cap=${JSON.stringify({
-        os,
-        runId: process.env.PLAYWRIGHT_SERVICE_RUN_ID
-      })}`,
-      timeout: 30000,
-      headers: {
-        'x-mpt-access-key': process.env.PLAYWRIGHT_SERVICE_ACCESS_TOKEN!
-      },
-      // Allow service to access the localhost.
-      exposeNetwork: '<loopback>'
-    }
+
+/* Learn more about service configuration at https://aka.ms/mpt/config */
+export default defineConfig(
+  config,
+  getServiceConfig(config, {
+    exposeNetwork: '<loopback>',
+    timeout: 30000,
+    os: ServiceOS.LINUX
+  }),
+  {
+    /* 
+    Playwright Testing service reporter is added by default.
+    This will override any reporter options specified in the base playwright config.
+    If you are using more reporters, please update your configuration accordingly.
+    */
+    reporter: [["list"], ['@azure/microsoft-playwright-testing/reporter']],
+    ignoreSnapshots: false,
+    // Enable screenshot testing and configure directory with expectations. 
+    snapshotPathTemplate: `{testDir}/__screenshots__/{testFilePath}/${ServiceOS.LINUX}/{arg}{ext}`,
   }
-});
+);
 ```
 
 ## Related content

articles/playwright-testing/how-to-manage-access-tokens.md

@@ -105,3 +105,4 @@ You can only delete access tokens that you created in a workspace. To create an
 ## Related content
 
 - Learn more about [managing access to a workspace](./how-to-manage-workspace-access.md).
+- Learn more about [managing authentication to the workspace](./how-to-manage-authentication.md)

articles/playwright-testing/how-to-manage-authentication.md

@@ -0,0 +1,96 @@
+---
+title: Microsoft Playwright Testing authentication
+description: Learn how to manage authentication and authorization for Microsoft Playwright Testing preview
+ms.topic: how-to
+ms.date: 09/07/2024
+ms.custom: playwright-testing-preview
+---
+
+# Manage authentication and authorization for Microsoft Playwright Testing preview
+
+In this article, you learn how to manage authentication and authorization for Microsoft Playwright Testing preview. Authentication is required to run Playwright tests on cloud-hosted browsers and to publish test results and artifacts to the service.
+
+By default, [Microsoft Entra ID](/entra/identity/) is used for authentication. This method is more secure and is the recommended authentication method. You can't disable authentication using Microsoft Entra ID. However, you can also use access tokens to authenticate and authorize.
+
+
+> [!IMPORTANT]
+> Microsoft Playwright Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
+
+## Background  
+
+Microsoft Playwright Testing Preview is built on the Playwright open-source framework. It runs Playwright tests on cloud-hosted browsers and publishes reports and artifacts back to the service. 
+
+To use the service, the client must authenticate with the service to access the browsers. Similarly, publishing results and artifacts requires authenticated API interactions. The service offers two authentication methods: Microsoft Entra ID and access tokens.
+
+Microsoft Entra ID uses your Azure credentials, requiring a sign-in to your Azure account for secure access. Alternatively, you can generate an access token from your Playwright workspace and use it in your setup. However, we strongly recommend Microsoft Entra ID for authentication due to its enhanced security. Access tokens, while convenient, function like long-lived passwords and are more susceptible to being compromised.
+
+## Enable authentication using access-tokens
+
+Microsoft Playwright Testing service also supports authentication using access tokens. This authentication method is less secure. We recommend using Microsoft Entra ID to authenticate to the service. 
+
+> [!CAUTION]
+> Your workspace access tokens are similar to a password for your Microsoft Playwright Testing workspace. Always be careful to protect your access tokens. Avoid distributing access tokens to other users, hard-coding them, or saving them anywhere in plain text that is accessible to others.
+
+Revoke and recreate your tokens if you believe they are compromised.
+
+To enable authentication using access tokens:
+
+1. Sign in to the [Playwright portal](https://aka.ms/mpt/portal) with your Azure account and select your workspace.
+
+1. Select the settings icon on the home page to go to the workspace settings.
+
+1. Select the **Authentication** page and turn on **Enable authentication using Access tokens**
+
+
+    :::image type="content" source="./media/how-to-manage-authentication/playwright-testing-enable-access-token.png" alt-text="Screenshot that shows the access tokens settings page in the Playwright portal." lightbox="./media/how-to-manage-authentication/playwright-testing-enable-access-token.png":::
+
+> [!CAUTION]
+> Authentication using access tokens is less secure. [Learn how to manage access tokens](./how-to-manage-access-tokens.md)
+
+## Set up authentication using access-tokens
+
+1. While running the tests, enable access token auth in the `playwright.service.config.ts` file in your setup. 
+
+    ```typescript
+    /* Learn more about service configuration at https://aka.ms/mpt/config */
+    export default defineConfig(config, getServiceConfig( config {
+        serviceAuthType:'ACCESS_TOKEN'
+    }));
+    ```
+
+1. Create access token 
+
+    Follow the steps to [create an access token](./how-to-manage-access-tokens.md#generate-a-workspace-access-token)
+
+1. Set up your environment
+
+    To set up your environment, you have to configure the `PLAYWRIGHT_SERVICE_ACCESS_TOKEN` environment variable with the value you obtained in the previous steps.
+
+    We recommend that you use the `dotenv` module to manage your environment. With `dotenv`, you define your environment variables in the `.env` file.
+
+    1. Add the `dotenv` module to your project:
+
+        ```shell
+        npm i --save-dev dotenv
+        ```
+
+    1. Create a `.env` file alongside the `playwright.config.ts` file in your Playwright project:
+        
+        ```
+        PLAYWRIGHT_SERVICE_ACCESS_TOKEN={MY-ACCESS-TOKEN}
+        ```
+
+        Make sure to replace the `{MY-ACCESS-TOKEN}` text placeholder with the value you copied earlier.
+
+
+## Run tests on the service and publish results
+
+Run Playwright tests against cloud-hosted browsers and publish the results to the service using the configuration you created above.
+
+```typescript
+npx playwright test --config=playwright.service.config.ts --workers=20
+```
+
+## Related content
+
+- Learn more about [managing access tokens](./how-to-manage-access-tokens.md).

articles/playwright-testing/how-to-test-local-applications.md

@@ -24,25 +24,18 @@ You can specify one or multiple networks by using a list of rules. For example,
 You can configure the `exposeNetwork` option in `playwright.service.config.ts`. The following example shows how to expose the `localhost` network by using the [`<loopback>`](https://en.wikipedia.org/wiki/Loopback) rule:
 
 ```typescript
-export default defineConfig(config, {
-    workers: 20,
-    use: {
-        // Specify the service endpoint.
-        connectOptions: {
-            wsEndpoint: `${process.env.PLAYWRIGHT_SERVICE_URL}?cap=${JSON.stringify({
-                // Can be 'linux' or 'windows'.
-                os: process.env.PLAYWRIGHT_SERVICE_OS || 'linux',
-                runId: process.env.PLAYWRIGHT_SERVICE_RUN_ID
-            })}`,
-            timeout: 30000,
-            headers: {
-                'x-mpt-access-key': process.env.PLAYWRIGHT_SERVICE_ACCESS_TOKEN!
-            },
-            // Allow service to access the localhost.
-            exposeNetwork: '<loopback>'
-        }
-    }
-});
+import { getServiceConfig, ServiceOS } from "@azure/microsoft-playwright-testing";
+import { defineConfig } from "@playwright/test";
+import { AzureCliCredential } from "@azure/identity";
+import config from "./playwright.config";
+
+export default defineConfig(
+  config,
+  getServiceConfig(config, {
+    exposeNetwork: '<loopback>', // Allow service to access the localhost.
+  }),
+);
+
 ```
 
 You can now reference `localhost` in the Playwright test code, and run the tests on cloud-hosted browsers with Microsoft Playwright Testing:

articles/playwright-testing/how-to-try-playwright-testing-free.md

@@ -8,7 +8,7 @@ ms.custom: playwright-testing-preview
 
 # Try Microsoft Playwright Testing Preview for free
 
-Microsoft Playwright Testing Preview is a fully managed service for end-to-end testing built on top of Playwright. With the free trial, you can try Microsoft Playwright Testing for free for 30 days and 100 test minutes. In this article, you learn about the limits of the free trial, how to get started, and how to track your free trial usage.
+Microsoft Playwright Testing Preview is a fully managed service for end-to-end testing built on top of Playwright. With the free trial, you can try Microsoft Playwright Testing for free for 30 days, 100 test minutes, and 1,000 test results. In this article, you learn about the limits of the free trial, how to get started, and how to track your free trial usage.
 
 > [!IMPORTANT]
 > Microsoft Playwright Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
@@ -26,9 +26,10 @@ The following table lists the limits for the Microsoft Playwright Testing free t
 |-|-|
 | Duration of trial | 30 days |
 | Total test minutes¹ | 100 minutes |
+| Total test results¹ | 1,000 results |
 | Number of workspaces²³ | 1 |
 
-¹ If you run a test that exceeds the free trial test minute limit, only the overage test minutes account towards the pay-as-you-go billing model.
+¹ If your usage exceeds either the free test minute limit or the free test result limit, only the overage counts toward the pay-as-you-go billing model. The two features are billed separately. See [Microsoft Playwright Testing preview pricing](https://aka.ms/mpt/pricing)
 
 ² These limits only apply to the *first* workspace you create in your Azure subscription. Any subsequent workspaces you create in the subscription automatically uses the pay-as-you-go billing model.
 
@@ -82,7 +83,7 @@ In the list of all workspaces, you can view a banner message that indicates if a
 
 When you exceed any of the limits of the free trial, your workspace is automatically converted to the pay-as-you-go billing model. 
 
-All test runs, access tokens, and other artifacts linked to your workspace remain available.
+All test runs, test results, and other artifacts linked to your workspace remain available.
 
 ## Next step
 

articles/playwright-testing/how-to-use-service-config-file.md

@@ -0,0 +1,139 @@
+---
+title: Microsoft Playwright Testing service configuration file options
+description: Learn how to use options available in configuration file with Microsoft Playwright Testing preview
+ms.topic: how-to
+ms.date: 09/07/2024
+ms.custom: playwright-testing-preview
+---
+# Use options available in configuration file with Microsoft Playwright Testing preview
+
+This article shows you how to use the options available in the `playwright.service.config.ts` file that was generated for you. 
+If you don't have this file in your code, follow the QuickStart guide, see [Quickstart: Run end-to-end tests at scale with Microsoft Playwright Testing Preview](./quickstart-run-end-to-end-tests.md) 
+
+> [!IMPORTANT]
+> Microsoft Playwright Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
+
+## Prerequisites
+
+* Follow the Quickstart guide and set up a project to run with Microsoft Playwright Testing service. See, [Quickstart: Run end-to-end tests at scale with Microsoft Playwright Testing Preview](./quickstart-run-end-to-end-tests.md) 
+
+Here's version of the `playwright.service.config.ts` file with all the available options:
+
+```typescript
+import { getServiceConfig, ServiceOS } from "@azure/microsoft-playwright-testing";
+import { defineConfig } from "@playwright/test";
+import { AzureCliCredential } from "@azure/identity";
+import config from "./playwright.config";
+
+export default defineConfig(
+  config,
+  getServiceConfig(config, {
+    serviceAuthType:'ACCESS_TOKEN' // Use this option when you want to authenticate using access tokens. This mode of auth should be enabled for the workspace.
+    os: ServiceOS.WINDOWS, // Select the operating system where you want to run tests.
+    runId: new Date().toISOString(), // Set a unique ID for every test run to distinguish them in the service portal.
+    credential: new AzureCliCredential(), // Select the authentication method you want to use with Entra.
+    useCloudHostedBrowsers: true, // Select if you want to use cloud-hosted browsers to run your Playwright tests.
+    exposeNetwork: '<loopback>', // Use this option to connect to local resources from your Playwright test code without having to configure additional firewall settings.
+    timeout: 30000 // Set the timeout for your tests.
+  }),
+  {
+    reporter: [
+      ["list"],
+      [
+        "@azure/microsoft-playwright-testing/reporter",
+        {
+          enableGitHubSummary: true, // Enable/disable GitHub summary in GitHub Actions workflow.
+        },
+      ],
+    ],
+  },
+);
+
+```
+
+## Settings in `playwright.service.config.ts` file
+
+* **`serviceAuthType`**:
+    - **Description**: This setting allows you to choose the authentication method you want to use for your test run. 
+    - **Available Options**:
+        - `ACCESS_TOKEN` to use access tokens. You need to enable authentication using access tokens if you want to use this option, see [manage authentication](./how-to-manage-authentication.md).
+        - `ENTRA_ID` to use Microsoft Entra ID for authentication. It's the default mode. 
+    - **Default Value**: `ENTRA_ID`
+    - **Example**:
+      ```typescript
+      serviceAuthType:'ENTRA_ID'
+      ```
+
+
+* **`os`**:
+    - **Description**: This setting allows you to choose the operating system where the browsers running Playwright tests are hosted.
+    - **Available Options**:
+        - `ServiceOS.WINDOWS` for Windows OS.
+        - `ServiceOS.LINUX` for Linux OS.
+    - **Default Value**: `ServiceOS.LINUX`
+    - **Example**:
+      ```typescript
+      os: ServiceOS.WINDOWS
+      ```
+
+* **`runId`**:
+    - **Description**: This setting allows you to set a unique ID for every test run to distinguish them in the service portal.
+    - **Example**:
+      ```typescript
+      runId: new Date().toISOString()
+      ```
+
+* **`credential`**:
+    - **Description**: This setting allows you to select the authentication method you want to use with Microsoft Entra ID.
+    - **Example**:
+      ```typescript
+      credential: new AzureCliCredential()
+      ```
+
+* **`useCloudHostedBrowsers`**
+    - **Description**: This setting allows you to choose whether to use cloud-hosted browsers or the browsers on your client machine to run your Playwright tests. If you disable this option, your tests run on the browsers of your client machine instead of cloud-hosted browsers, and you don't incur any charges.
+    - **Default Value**: true
+    - **Example**:
+      ```typescript
+      useCloudHostedBrowsers: true
+      ```
+
+* **`exposeNetwork`**
+    - **Description**: This setting allows you to connect to local resources from your Playwright test code without having to configure another firewall settings. To learn more, see [how to test local applications](./how-to-test-local-applications.md)
+    - **Example**:
+      ```typescript
+      exposeNetwork: '<loopback>'
+      ```
+
+* **`timeout`**
+    - **Description**: This setting allows you to set timeout for your tests connecting to the cloud-hosted browsers. 
+    - **Example**:
+      ```typescript
+      timeout: 30000,
+      ```     
+
+* **`reporter`**
+    - **Description**: The `playwright.service.config.ts` file extends the playwright config file of your setup. This option overrides the existing reporters and sets Microsoft Playwright Testing reporter. You can add or modify this list to include the reporters that you want to use. You're billed for Microsoft Playwright Testing reporting if you add `@azure/microsoft-playwright-testing/reporter`. 
+    - **Default Value**: ["@azure/microsoft-playwright-testing/reporter"]
+    - **Example**:
+      ```typescript
+      reporter: [
+      ["list"],
+      ["@azure/microsoft-playwright-testing/reporter"],
+      ```
+* **`enableGitHubSummary`**:
+    - **Description**: This setting allows you to configure the Microsoft Playwright Testing service reporter. You can choose whether to include the test run summary in the GitHub summary when running in GitHub Actions.
+    - **Default Value**: true
+    - **Example**:
+    ```typescript
+      reporter: [
+        ["list"],
+        [
+          "@azure/microsoft-playwright-testing/reporter",
+          {
+            enableGitHubSummary: true,
+          },
+        ],
+      ]
+    ```
+