Skip to content

Commit 8d7c221

Browse files
vvs11vvs11
committed
added reporting docs
1 parent 917a8c0 commit 8d7c221

8 files changed

+227
-8
lines changed

‎articles/playwright-testing/includes/include-playwright-portal-view-test-results.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
title: "Include file"
3+
description: "Include file"
4+
ms.custom: "include file"
5+
ms.topic: "include"
6+
ms.date: 09/23/2024
7+
---
8+
9+
10+
1. Open the link generated at the end of test run in the console to see the results on Playwright portal. The portal contains the CI information, test run status, and commit id associated with the test run.
11+
12+
:::image type="content" source="./media/include-playwright-portal-view-test-results/playwright-testing-open-test-run.png" alt-text="Screenshot that shows list of tests in the test run." lightbox="./media/include-playwright-portal-view-test-results/playwright-testing-open-test-run.png":::
13+
14+
3. Open any failed test to see all the required information to troubleshoot. Switch between retries, see error logs, test steps and attachments such as screenshots and video. Navigate to the trace viewer from here.
15+
16+
:::image type="content" source="./media/include-playwright-portal-view-test-results/playwright-testing-open-test.png" alt-text="Screenshot that shows the preview of a test" lightbox="./media/include-playwright-portal-view-test-results/playwright-testing-open-test.png":::
17+
18+
4. Use the trace viewer to visually step through your test execution, or use the timeline to hover over steps and reveal the page state before and after each action. Detailed logs, DOM snapshot, network activity, errors, and console output are available at each test step.
19+
20+
:::image type="content" source="./media/include-playwright-portal-view-test-results/playwright-testing-trace-viewer.png" alt-text="Screenshot that shows the trace viewer" lightbox="./media/include-playwright-portal-view-test-results/playwright-testing-trace-viewer.png":::
21+

‎articles/playwright-testing/media/include-playwright-portal-view-test-results/playwright-testing-open-test-run.png

109 KB
Loading

‎articles/playwright-testing/media/include-playwright-portal-view-test-results/playwright-testing-open-test.png

137 KB
Loading

‎articles/playwright-testing/media/include-playwright-portal-view-test-results/playwright-testing-trace-viewer.png

597 KB
Loading

‎articles/playwright-testing/media/include-playwright-portal-view-test-results/playwright-testing-view-test-results.png

26.2 KB
Loading

‎articles/playwright-testing/quickstart-automate-end-to-end-testing.md

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,11 +75,23 @@ If you haven't configured your Playwright tests yet for running them on cloud-ho
7575

7676
Update the `package.json` file in your repository to add details about Microsoft Playwright Testing service package in `devDependencies` section.
7777

78-
```json
78+
```typescript
7979
"devDependencies": {
8080
"@azure/microsoft-playwright-testing": "^1.0.0-beta.3"
8181
}
8282
```
83+
84+
## Enable artifacts in Playwright configuration
85+
86+
In the `playwright.config.ts` file of your project, make sure you are collecting all the required artifacts.
87+
```typescript
88+
use: {
89+
trace: 'on-first-retry',
90+
video:'retain-on-failure',
91+
screenshot:'on'
92+
},
93+
```
94+
8395
## Update the workflow definition
8496

8597
Update the CI workflow definition to run your Playwright tests with the Playwright CLI. Pass the [service configuration file](#add-service-configuration-file) as an input parameter for the Playwright CLI. You configure your environment by specifying environment variables.
@@ -95,7 +107,7 @@ Update the CI workflow definition to run your Playwright tests with the Playwrig
95107
```yml
96108

97109
# This step is to sign-in to Azure to run tests from GitHub Action workflow.
98-
# You can choose how set up Authentication to Azure from GitHub Actions, this is one example.
110+
# You can choose how to set up Authentication to Azure from GitHub Actions, this is one example.
99111
- name: Login to Azure with AzPowershell (enableAzPSSession true)
100112
uses: azure/login@v2
101113
with:
@@ -176,10 +188,20 @@ Update the CI workflow definition to run your Playwright tests with the Playwrig
176188
>
177189
> ```npx playwright test {name-of-file.spec.ts} --config=playwright.service.config.ts```
178190

191+
## View test runs and results in the Playwright portal
192+
193+
To view results and troubleshoot the test run with Microsoft Playwright Testing,
194+
195+
[!INCLUDE [View test runs and results in the Playwright portal](./includes/include-playwright-portal-view-test-results.md)]
196+
179197

180198
> [!TIP]
181199
> You can use Microsoft Playwright Testing service features independently. You can publish test results to the portal without using the cloud-hosted browsers feature and you can also use only cloud-hosted browsers to expedite your test suite without publishing test results.
182200

201+
> [!NOTE]
202+
> The test results and artifacts that you publish are retained on the service for 90 days. After that, they are automatically deleted.
203+
204+
183205
## Related content
184206

185207
You've successfully set up a continuous end-to-end testing workflow to run your Playwright tests at scale on cloud-hosted browsers.

‎articles/playwright-testing/quickstart-generate-rich-reports-for-tests.md

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
---
2+
title: 'Quickstart: Generate rich reports for Playwright tests'
3+
description: 'This quickstart shows how to troubleshoot your test runs using Microsoft Playwright Testing Preview.
4+
ms.topic: quickstart
5+
ms.date: 09/23/2024
6+
ms.custom: playwright-testing-preview
7+
---
8+
9+
# Quickstart: Troubleshoot tests with Microsoft Playwright Testing Preview
10+
11+
In this quickstart, you learn how to troubleshoot your Playwright tests easily using reports and artifacts published on Microsoft Playwright Testing Preview.
12+
13+
After you complete this quickstart, you have a Microsoft Playwright Testing workspace to view test results and artifacts in the service portal.
14+
15+
> [!IMPORTANT]
16+
> Microsoft Playwright Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
17+
18+
## Prerequisites
19+
20+
* An Azure account with an active subscription. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.
21+
* Your Azure account needs the [Owner](/azure/role-based-access-control/built-in-roles#owner), [Contributor](/azure/role-based-access-control/built-in-roles#contributor), or one of the [classic administrator roles](/azure/role-based-access-control/rbac-and-directory-admin-roles#classic-subscription-administrator-roles).
22+
* A Playwright project. If you don't have project, create one by using the [Playwright getting started documentation](https://playwright.dev/docs/intro) or use our [Microsoft Playwright Testing sample project](https://github.com/microsoft/playwright-testing-service/tree/main/samples/get-started).
23+
* Azure CLI. If you don't have Azure CLI, see [Install Azure CLI](/cli/azure/install-azure-cli).
24+
25+
## Create a workspace
26+
27+
To get started with running your Playwright tests at scale on cloud browsers, you first create a Microsoft Playwright Testing workspace in the Playwright portal.
28+
29+
[!INCLUDE [Create workspace in Playwright portal](./includes/include-playwright-portal-create-workspace.md)]
30+
31+
When the workspace creation finishes, you're redirected to the setup guide.
32+
33+
## Install Microsoft Playwright Testing package
34+
35+
To use the service, install Microsoft Playwright Testing package.
36+
37+
```npm
38+
npm init @azure/microsoft-playwright-testing
39+
```
40+
41+
This generates `playwright.service.config.ts` file which serves to:
42+
43+
- Direct and authenticate Playwright to the Microsoft Playwright Testing service.
44+
- Adds a reporter to publish test results and artifacts.
45+
46+
If you already have this file, the package asks you to override it.
47+
48+
Once you have the `playwright.service.config.ts` file, disable cloud-hosted browsers.
49+
50+
```typescript
51+
useCloudHostedBrowsers: false
52+
```
53+
Setting this as `false` ensures that cloud-hosted browsers are not used to run the tests. The tests run on your local machine but the results and artifacts are published on the service.
54+
55+
## Configure the service region endpoint
56+
57+
In your setup, you have to provide the region-specific service endpoint. The endpoint depends on the Azure region you selected when creating the workspace.
58+
59+
To get the service endpoint URL, perform the following steps:
60+
61+
1. In **Add region endpoint in your setup**, copy the region endpoint for your workspace.
62+
63+
The endpoint URL matches the Azure region that you selected when creating the workspace.
64+
65+
:::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-region-endpoint.png" alt-text="Screenshot that shows how to copy the workspace region endpoint in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-region-endpoint.png":::
66+
67+
## Set up your environment
68+
69+
To set up your environment, you have to configure the `PLAYWRIGHT_SERVICE_URL` environment variable with the value you obtained in the previous steps.
70+
71+
We recommend that you use the `dotenv` module to manage your environment. With `dotenv`, you define your environment variables in the `.env` file.
72+
73+
1. Add the `dotenv` module to your project:
74+
75+
```shell
76+
npm i --save-dev dotenv
77+
```
78+
79+
1. Create a `.env` file alongside the `playwright.config.ts` file in your Playwright project:
80+
81+
```
82+
PLAYWRIGHT_SERVICE_URL={MY-REGION-ENDPOINT}
83+
```
84+
85+
Make sure to replace the `{MY-REGION-ENDPOINT}` text placeholder with the value you copied earlier.
86+
87+
88+
## Set up Authentication
89+
90+
To run your Playwright tests in your Microsoft Playwright Testing workspace, you need to authenticate the Playwright client where you're running the tests with the service. This could be your local dev machine or CI machine.
91+
92+
The service offers two authentication methods: Microsoft Entra ID and Access Tokens.
93+
94+
Microsoft Entra ID uses your Azure credentials, requiring a sign-in to your Azure account for secure access. Alternatively, you can generate an access token from your Playwright workspace and use it in your setup.
95+
96+
##### Set up authentication using Microsoft Entra ID
97+
98+
Microsoft Entra ID is the default and recommended authentication for the service. From your local dev machine, you can use [Azure CLI](/cli/azure/install-azure-cli) to sign-in
99+
100+
```CLI
101+
az login
102+
```
103+
> [!NOTE]
104+
> If you're a part of multiple Microsoft Entra tenants, make sure you sign in to the tenant where your workspace belongs. You can get the tenant ID from Azure portal. See [Find your Microsoft Entra Tenant](/azure/azure-portal/get-subscription-tenant-id#find-your-microsoft-entra-tenant). Once you get the ID, sign-in using the command `az login --tenant <TenantID>`
105+
106+
##### Set up authentication using access tokens
107+
108+
You can generate an access token from your Playwright Testing workspace and use it in your setup. However, we strongly recommend Microsoft Entra ID for authentication due to its enhanced security. Access tokens, while convenient, function like long-lived passwords and are more susceptible to being compromised.
109+
110+
1. Authentication using access tokens is disabled by default. To use, [Enable access-token based authentication](./how-to-manage-authentication.md#enable-authentication-using-access-tokens)
111+
112+
2. [Set up authentication using access tokens](./how-to-manage-authentication.md#set-up-authentication-using-access-tokens)
113+
114+
> [!CAUTION]
115+
> We strongly recommend using Microsoft Entra ID for authentication to the service. If you are using access tokens, see [How to Manage Access Tokens](./how-to-manage-access-tokens.md)
116+
117+
## Enable artifacts in Playwright configuration
118+
In the `playwright.config.ts` file of your project, make sure you are collecting all the required artifacts.
119+
```typescript
120+
use: {
121+
trace: 'on-first-retry',
122+
video:'retain-on-failure',
123+
screenshot:'on'
124+
},
125+
```
126+
127+
## Run your tests and publish results on Microsoft Playwright Testing
128+
129+
You've now prepared the configuration for publishing test results with Microsoft Playwright Testing. Run tests using the newly created `playwright.service.config.ts` file and publish test results to the service.
130+
131+
```bash
132+
npx playwright test --config=playwright.service.config.ts
133+
```
134+
> [!NOTE]
135+
> For the Reporting feature of Microsoft Playwright Testing, you get charged based on the number test results published. If you're a first-time user or [getting started with a free trial](./how-to-try-playwright-testing-free.md), you might start with publishing single test result instead of your full test suite to avoid exhausting your free trial limits.
136+
137+
After the test completes, you can view the test status in the terminal.
138+
139+
```output
140+
Running 6 test using 2 worker
141+
5 passed, 1 failed (20.2s)
142+
143+
Test report: https://playwright.microsoft.com/workspaces/<workspace-id>/runs/<run-id>
144+
```
145+
146+
> [!CAUTION]
147+
> Depending on the size of your test suite, you might incur additional charges for the test minutes beyond your allotted free test minutes and free test results.
148+
149+
## View test runs and results in the Playwright portal
150+
151+
To view results and troubleshoot the test run with Microsoft Playwright Testing,
152+
153+
[!INCLUDE [View test runs and results in the Playwright portal](./includes/include-playwright-portal-view-test-results.md)]
154+
155+
156+
> [!TIP]
157+
> You can also use Microsoft Playwright Testing service to run tests in parallel using cloud-hosted browsers. Both Reporting and cloud-hosted browsers are independent features and are billed separately. You can use either of these or both.
158+
159+
> [!NOTE]
160+
> The test results and artifacts that you publish are retained on the service for 90 days. After that, they are automatically deleted.
161+
162+
163+
## Next step
164+
165+
You've successfully created a Microsoft Playwright Testing workspace in the Playwright portal and run your Playwright tests on cloud browsers.
166+
167+
Advance to the next quickstart to set up continuous end-to-end testing by running your Playwright tests in your CI/CD workflow.
168+
169+
> [!div class="nextstepaction"]
170+
> [Set up continuous end-to-end testing in CI/CD](./quickstart-automate-end-to-end-testing.md)

‎articles/playwright-testing/quickstart-run-end-to-end-tests.md

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,15 @@ You can generate an access token from your Playwright Testing workspace and use
107107
> [!CAUTION]
108108
> We strongly recommend using Microsoft Entra ID for authentication to the service. If you are using access tokens, see [How to Manage Access Tokens](./how-to-manage-access-tokens.md)
109109

110+
## Enable artifacts in Playwright configuration
111+
In the `playwright.config.ts` file of your project, make sure you are collecting all the required artifacts.
112+
```typescript
113+
use: {
114+
trace: 'on-first-retry',
115+
video:'retain-on-failure',
116+
screenshot:'on'
117+
}
118+
```
110119
## Run your tests at scale with Microsoft Playwright Testing
111120

112121
You've now prepared the configuration for running your Playwright tests in the cloud with Microsoft Playwright Testing. You can either use the Playwright CLI to run your tests, or use the [Playwright Test Visual Studio Code extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).
@@ -213,9 +222,7 @@ When you run multiple Playwright tests or a full test suite with Microsoft Playw
213222
Running 6 tests using 6 workers
214223
6 passed (18.2s)
215224
216-
To open last HTML report run:
217-
218-
npx playwright show-report
225+
Test report: https://playwright.microsoft.com/workspaces/<workspace-id>/runs/<run-id>
219226
```
220227
221228
# [Visual Studio Code](#tab/vscode)
@@ -243,11 +250,10 @@ To run your Playwright test suite in Visual Studio Code with Microsoft Playwrigh
243250
244251
## View test runs and results in the Playwright portal
245252
246-
Go to the [Playwright portal](https://aka.ms/mpt/portal) to view the test runs and test results for your workspace.
253+
To view results and troubleshoot the test run with Microsoft Playwright Testing,
247254
248-
:::image type="content" source="./media/quickstart-run-end-to-end-tests/playwright-testing-test-run-page.png" alt-text="Screenshot that shows the test runs for a workspace in the Playwright Testing portal." lightbox="./media/quickstart-run-end-to-end-tests/playwright-testing-test-run-page.png":::
255+
[!INCLUDE [View test runs and results in the Playwright portal](./includes/include-playwright-portal-view-test-results.md)]
249256
250-
The test run contains the CI information, test run status, workers used, duration, and billable minutes. If you open a test run, you can see the results and artifacts for each test along with other information.
251257
252258
> [!TIP]
253259
> You can use Microsoft Playwright Testing service features independently. You can publish test results to the portal without using the cloud-hosted browsers feature and you can also use only cloud-hosted browsers to expedite your test suite without publishing test results.

0 commit comments

Comments
 (0)