Skip to content

Commit 62b0350

Browse files
ninallamninallam
committed
Add Playwright Workspaces articles to App Testing
1 parent 11dd891 commit 62b0350

File tree

42 files changed

+2437
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+2437
-0
lines changed

‎articles/app-testing/playwright-workspaces/concept-determine-optimal-configuration.md

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
---
2+
title: Optimal test suite configuration
3+
titleSuffix: Playwright Workspaces
4+
description: Learn about the factors that affect test completion time in Playwright Workspaces. Get practical steps to determine the optimal Playwright test project configuration.
5+
ms.service: azure-app-testing
6+
ms.subservice: playwright-workspaces
7+
author: ninallam
8+
ms.author: ninallam
9+
ms.topic: conceptual
10+
ms.date: 08/07/2025
11+
ms.custom: playwright-workspaces-preview
12+
---
13+
14+
# Determine the optimal test suite configuration for Playwright Workspaces
15+
16+
Playwright Workspaces Preview enables you to speed up your Playwright test execution by increasing parallelism at cloud scale. Several factors affect the completion time for your test suite. Determining the optimal configuration for reducing test suite completion time is application-specific and requires experimentation. This article explains the different levels to configure parallelism for your tests, the factors that influence test duration, and how to determine your optimal configuration to minimize test completion time.
17+
18+
In Playwright, you can run tests in parallel by using worker processes. By using Playwright Workspaces, you can further increase parallelism by using cloud-hosted browsers. In general, adding more parallelism reduces the time to complete your test suite. However, adding more worker processes doesn't always result in shorter test suite completion times. For example, the client machine computing resources, network latency, or test complexity might also affect test duration.
19+
20+
The following chart gives an example of running a test suite. By running the test suite with Playwright Workspaces instead of locally, you can significantly increase the parallelism and reduce the test completion time. Notice that, when running with the service, the completion time reaches a minimum limit, after which adding more workers only has a minimal effect. The chart also shows how using more computing resources on the client machine positively affects the test completion time for tests running with the service.
21+
22+
![Line chart that shows the relation between the number of parallel workers and the test suite completion time for different run environments.](./media/concept-determine-optimal-configuration/playwright-workspaces-parallelization-chart.png)
23+
24+
## Worker processes
25+
26+
In [Playwright](https://playwright.dev/docs/intro), all tests run in worker processes. These processes are OS processes, running independently, in parallel, orchestrated by the Playwright Test runner. All workers have identical environments and each process starts its own browser.
27+
28+
Generally, increasing the number of parallel workers can reduce the time it takes to complete the full test suite. You can learn more about [Playwright Test parallelism](https://playwright.dev/docs/test-parallel) in the Playwright documentation.
29+
30+
As previously shown in the chart, the test suite completion time doesn't continue to decrease as you add more worker processes. There are [other factors that influence the test suite duration](#factors-that-influence-completion-time).
31+
32+
### Run tests locally
33+
34+
By default, `@playwright/test` limits the number of workers to 1/2 of the number of CPU cores on your machine. You can override the number of workers for running your test.
35+
36+
When you run tests locally, the number of worker processes is limited to the number of CPU cores on your machine. Beyond a certain point, adding more workers leads to resource contention, which slows down each worker and introduces test flakiness.
37+
38+
To override the number of workers using the [`--workers` command line flag](https://playwright.dev/docs/test-cli#reference):
39+
40+
```bash
41+
npx playwright test --workers=10
42+
```
43+
44+
To specify the number of workers in `playwright.config.ts` using the `workers` setting:
45+
46+
```typescript
47+
export default defineConfig({
48+
...
49+
workers: 10,
50+
...
51+
});
52+
```
53+
54+
### Run tests with the service
55+
56+
When you use Playwright Workspaces, you can increase the number of workers at cloud-scale to larger numbers. When you use the service, the worker processes continue to run locally, but the resource-intensive browser instances are now running remotely in the cloud.
57+
58+
Because the worker processes still run on the client machine (developer workstation or CI agent machine), the client machine might still become a bottleneck for scalable execution as you add more workers. Learn how you can [determine the optimal configuration](#workflow-for-determining-your-optimal-configuration).
59+
60+
You can specify the number of workers on the command line with the `--workers` flag:
61+
62+
```bash
63+
npx playwright test --config=playwright.service.config.ts --workers=30
64+
```
65+
66+
Alternately you can specify the number of workers in `playwright.service.config.ts` using the `workers` setting:
67+
68+
```typescript
69+
export default defineConfig({
70+
...
71+
workers: 30,
72+
...
73+
});
74+
```
75+
76+
## Factors that influence completion time
77+
78+
In addition to the number of parallel worker processes, there are several factors that influence the test suite completion time.
79+
80+
| Factor | Effects on test duration |
81+
|-|-|
82+
| **Client machine compute resources** | The worker processes still run on the client machine (developer workstation or CI agent machine) and need to communicate with the remote browsers. Increasing the number of parallel workers might result in resource contention on the client machine, and slow down tests. |
83+
| **Complexity of the test code** | As the complexity of the test code increases, the time to complete the tests might also increase. |
84+
| **Latency between the client machine and the remote browsers** | The workers run on the client machine and communicate with the remote browsers. Depending on the Azure region where the browsers are hosted, the network latency might increase. Learn how you can [optimize regional latency in Playwright Workspaces](./how-to-optimize-regional-latency.md). |
85+
| **Playwright configuration settings** | Playwright settings such as service timeouts, retries, or tracing can adversely affect the test completion time. Experiment with the optimal configuration for these settings when running your tests in the cloud. |
86+
| **Target application's load-handling capacity** | Running tests with Playwright Workspaces enables you to run with higher parallelism, which results in a higher load on the target application. Verify that the application can handle the load that is generated by running your Playwright tests. |
87+
88+
Learn more about the [workflow for determining the optimal configuration](#workflow-for-determining-your-optimal-configuration) for minimizing the test suite duration.
89+
90+
## Workflow for determining your optimal configuration
91+
92+
The optimal configuration for minimizing the test suite completion time is specific to your application and environment. To determine your optimal configuration, experiment with different levels of parallelization, client machine hardware configuration, or test suite setup.
93+
94+
The following approach can help you find the optimal configuration for running your tests with Playwright Workspaces:
95+
96+
### 1. Determine your test completion time goal
97+
98+
Determine what is an acceptable test suite completion time and associated cost per test run.
99+
100+
Depending on the scenario, your requirements for test completion might be different. When you're running your end-to-end tests with every code change, as part of a continuous integration (CI) workflow, minimizing test completion time is essential. When you schedule your end-to-end tests in a (nightly) batch run, you might have requirements that are less demanding.
101+
102+
### 2. Verify that your tests run correctly on the client machine
103+
104+
Before you run your Playwright test suite with Playwright Workspaces, make sure that your tests run correctly on your client machine. If you run your tests as part of a CI workflow, validate that your tests run correctly on the CI agent machine. Ensure that you run your tests with a minimum of two parallel workers to verify that your tests are properly configured for parallel execution. Learn more about [parallelism in Playwright](https://playwright.dev/docs/test-parallel).
105+
106+
### 3. Run with cloud-hosted browsers on Playwright Workspaces
107+
108+
Once your tests run correctly, add the service configuration to run your tests on cloud-hosted browsers with the service. Validate that your tests continue to run correctly from your client machine (developer workstation or CI agent machine).
109+
110+
Get started with the [Quickstart: run Playwright tests at scale with Playwright Workspaces](./quickstart-run-end-to-end-tests.md).
111+
112+
### 4. Verify the Azure region remote browsers
113+
114+
Playwright Workspaces can use remote browsers in the Azure region that's nearest to your client machine, or use the fixed region on which your workspace was created.
115+
116+
Learn how you can [optimize regional latency for your workspace](./how-to-optimize-regional-latency.md).
117+
118+
### 5. Experiment with the number of parallel workers
119+
120+
Experiment with the number of parallel workers to run your tests. Measure the test completion time and compare against the target goal you set previously.
121+
122+
Notice at which point the test completion time no longer reduces as you add more workers. Move to the next step to further optimize your setup.
123+
124+
> [!NOTE]
125+
> While the service is in preview, the number of [parallel workers per workspace is limited](./resource-limits-quotas-capacity.md) to 50. You can [request an increase of this limit for your workspace](https://aka.ms/mpt/feedback).
126+
127+
### 6. Scale the client
128+
129+
As you increase parallelism, the client machine might experience compute resource contention. Increase the computing resources on the client machine, for example by selecting [larger GitHub-hosted runners](https://docs.github.com/actions/using-github-hosted-runners/about-larger-runners).
130+
131+
Alternatively, if you have hardware limitations, you can [shard](https://playwright.dev/docs/test-sharding) your client tests.
132+
133+
Rerun your tests and experiment with the number of parallel workers.
134+
135+
### 7. Update your Playwright test configuration settings
136+
137+
Configure your Playwright test configuration settings, such as test [timeouts](https://playwright.dev/docs/test-timeouts), [trace](https://playwright.dev/docs/api/class-testoptions#test-options-trace) settings, or [retries](https://playwright.dev/docs/test-retries).
138+
139+
## Related content
140+
141+
- [Run your Playwright tests at scale with Playwright Workspaces](./quickstart-run-end-to-end-tests.md)
142+
- [What is Playwright Workspaces](./overview-what-is-microsoft-playwright-workspaces.md)

‎articles/app-testing/playwright-workspaces/how-to-configure-visual-comparisons.md

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
---
2+
title: Configure visual comparisons
3+
titleSuffix: Playwright Workspaces
4+
description: Learn how to configure visual comparisons with Playwright Workspaces.
5+
ms.topic: how-to
6+
ms.service: azure-app-testing
7+
ms.subservice: playwright-workspaces
8+
author: ninallam
9+
ms.author: ninallam
10+
ms.date: 08/07/2025
11+
ms.custom: playwright-workspaces-preview
12+
---
13+
14+
# Configure visual comparisons with Playwright Workspaces
15+
16+
In this article, you learn how to properly configure Playwright's visual comparison tests when using Playwright Workspaces. Unexpected test failures may occur because Playwright's snapshots differ between local and remote browsers.
17+
18+
> [!IMPORTANT]
19+
> Playwright Workspaces is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
20+
21+
## Background
22+
23+
The Playwright Test runner uses the host OS as a part of the expected screenshot path. If you're running tests using remote browsers on a different OS than your host machine, the visual comparison tests fail. Our recommendation is to only run visual comparisons when using the service. If you're taking screenshots on the service, there's no need to compare them to your local setup since they don't match.
24+
25+
## Configure ignoreSnapshots
26+
27+
You can use the [`ignoreSnapshots` option](https://playwright.dev/docs/api/class-testconfig#test-config-ignore-snapshots) to only run visual comparisons when using Playwright Workspaces.
28+
29+
1. Set `ignoreSnapshots: true` in the original `playwright.config.ts` that doesn't use the service.
30+
1. Set `ignoreSnapshots: false` in `playwright.service.config.ts`.
31+
32+
When you're using the service, its configuration overrides `playwright.config.ts`, and runs visual comparisons.
33+
34+
## Configure the snapshot path
35+
36+
To configure snapshot paths for a particular project or the whole config, you can set [`snapshotPathTemplate` option](https://playwright.dev/docs/api/class-testproject#test-project-snapshot-path-template).
37+
38+
```js
39+
// This path is exactly like the default path, but replaces OS with hardcoded value that is used on the service (linux).
40+
config.snapshotPathTemplate = '{snapshotDir}/{testFileDir}/{testFileName}-snapshots/{arg}{-projectName}-linux{ext}'
41+
42+
// This is an alternative path where you keep screenshots in a separate directory, one per service OS (linux in this case).
43+
config.snapshotPathTemplate = '{testDir}/__screenshots__/{testFilePath}/linux/{arg}{ext}';
44+
```
45+
46+
## Example service config
47+
48+
Example service config that runs visual comparisons and configures the path for `snapshotPathTemplate`:
49+
50+
```typeScript
51+
import { defineConfig } from '@playwright/test';
52+
import { getServiceConfig, ServiceOS } from '@azure/playwright';
53+
import { DefaultAzureCredential } from '@azure/identity';
54+
import config from './playwright.config';
55+
56+
/* Learn more about service configuration at https://aka.ms/mpt/config */
57+
export default defineConfig(
58+
config,
59+
getServiceConfig(config, {
60+
exposeNetwork: '<loopback>',
61+
timeout: 30000,
62+
os: ServiceOS.LINUX,
63+
credential: new DefaultAzureCredential()
64+
}),
65+
{
66+
ignoreSnapshots: false,
67+
// Enable screenshot testing and configure directory with expectations. 
68+
snapshotPathTemplate: `{testDir}/__screenshots__/{testFilePath}/${ServiceOS.LINUX}/{arg}{ext}`,
69+
}
70+
);
71+
```
72+
73+
## Related content
74+
75+
- Learn more about [Playwright Visual Comparisons](https://playwright.dev/docs/test-snapshots).

0 commit comments

Comments
 (0)