Skip to content

Commit 1c42e1b

Browse files
vvs11vvs11
committed
updated ci article
1 parent 03f9ca3 commit 1c42e1b

File tree

2 files changed

+187
-91
lines changed

2 files changed

+187
-91
lines changed

‎articles/playwright-testing/playwright-testing-reporting-with-sharding.md

Lines changed: 63 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ In this article, you learn how to use the Microsoft Playwright Testing service's
1616

1717
Playwright's sharding enables you to split your test suite to run across multiple machines simultaneously. This feature helps running tests in parallel.
1818

19-
You can use Playwright Testing's reporting feature to get a consolidated report of a test run with sharding. You need to make sure you set the variable `PLAYWRIGHT_SERVICE_RUN_ID` so that it remains same across all shards.
19+
You can use Playwright Testing's reporting feature to get a consolidated report of a test run with sharding. You need to make sure the `runId` of the test run is same across all shards.
2020

2121
> [!IMPORTANT]
2222
> Microsoft Playwright Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
@@ -27,65 +27,82 @@ You can use Playwright Testing's reporting feature to get a consolidated report
2727

2828
* Set up continuous end-to-end testing. Complete the [Quickstart: Set up continuous end-to-end testing with Microsoft Playwright Testing Preview](./quickstart-automate-end-to-end-testing.md) to set up continuous integration (CI) pipeline.
2929

30+
## Update playwright service configuration file
31+
32+
Add or update `runId` to `playwright.service.config.ts` file in your setup.
33+
34+
```typescript
35+
.
36+
.
37+
.
38+
export default defineConfig(
39+
config,
40+
getServiceConfig(config, {
41+
runId: process.env.PLAYWRIGHT_SERVICE_RUN_ID, //Set runId for the test run
42+
}),
43+
);
44+
45+
```
46+
You can use `PLAYWRIGHT_SERVICE_RUN_ID` variable in your setup to ensure the `runId` remains same across all shards.
3047

3148
## Set up variables
3249

33-
The `PLAYWRIGHT_SERVICE_RUN_ID` variable is an identifier that is used by Playwright Testing service to distinguish between test runs. The results from multiple runs with same `RUN_ID` are reported to the same run on the Playwright portal.
50+
The `runId` setting is be used as an identifier by Playwright Testing service to distinguish between test runs. The results from multiple runs with same `runId` are reported to the same run on the Playwright portal.
3451

35-
By default, a test run that uses reporting feature automatically generates a unique `RUN_ID` unless you explicitly set the value yourself. If the value of the variable remains same across runs, the results are reported together in the same run on the Playwright portal.
52+
By default, a test run that uses reporting feature automatically generates a unique `runId` unless you explicitly set the value yourself. If the value of the variable remains same across runs, the results are reported together in the same run on the Playwright portal.
3653

3754
> [!Tip]
3855
> If you use the cloud-hosted browsers provided by Microsoft Playwright Testing service to run your tests, you might have already set this variable. To avoid overwrites, make sure you set it only once.
3956
4057

41-
While using sharding, make sure the same `RUN_ID` is set across all the shards for the results to be reported together.
58+
While using sharding, make sure the same `runId` is set across all the shards for the results to be reported together. This can be achieved by using a variable `PLAYWRIGHT_SERVICE_RUN_ID` and setting this value same across all shards.
4259

4360
Here's an example of how you can set it in your pipeline via GitHub Actions.
4461

4562
```yml
46-
name: Playwright Tests
47-
on:
48-
push:
49-
branches: [ main, master ]
50-
pull_request:
51-
branches: [ main, master ]
52-
workflow_dispatch:
53-
strategy:
54-
fail-fast: false
55-
matrix:
56-
shardIndex: [1, 2, 3, 4]
57-
shardTotal: [4]
58-
permissions:
59-
id-token: write
60-
contents: read
61-
jobs:
62-
test:
63-
timeout-minutes: 60
64-
runs-on: ubuntu-latest
65-
steps:
66-
- uses: actions/checkout@v3
67-
- uses: actions/setup-node@v3
68-
with:
69-
node-version: 18
70-
# This step is to sign-in to Azure to run tests from GitHub Action workflow.
71-
# You can choose how set up Authentication to Azure from GitHub Actions, this is one example.
72-
- name: Login to Azure with AzPowershell (enableAzPSSession true)
73-
uses: azure/login@v2
74-
with:
75-
client-id: ${{ secrets.AZURE_CLIENT_ID }}
76-
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
77-
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
78-
enable-AzPSSession: true
79-
80-
- name: Install dependencies
63+
name: Playwright Tests (Microsoft Playwright Testing)
64+
on:
65+
push:
66+
branches: [ main, master ]
67+
pull_request:
68+
branches: [ main, master ]
69+
workflow_dispatch:
70+
strategy:
71+
fail-fast: false
72+
matrix:
73+
shardIndex: [1, 2, 3, 4]
74+
shardTotal: [4]
75+
permissions: # Required when using Microsoft Entra ID to authenticate
76+
id-token: write
77+
contents: read
78+
jobs:
79+
test:
80+
timeout-minutes: 60
81+
runs-on: ubuntu-latest
82+
steps:
83+
- uses: actions/checkout@v3
84+
- uses: actions/setup-node@v3
85+
with:
86+
node-version: 18
87+
# This step is to sign-in to Azure to run tests from GitHub Action workflow.
88+
# You can choose how set up Authentication to Azure from GitHub Actions, this is one example.
89+
- name: Login to Azure with AzPowershell (enableAzPSSession true)
90+
uses: azure/login@v2
91+
with:
92+
client-id: ${{ secrets.AZURE_CLIENT_ID }} # Required when using Microsoft Entra ID to authenticate
93+
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
94+
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
95+
enable-AzPSSession: true
96+
97+
- name: Install dependencies
8198
working-directory: path/to/playwright/folder # update accordingly
82-
run: npm ci
99+
run: npm ci
83100

84-
- name: Run Playwright tests
101+
- name: Run Playwright tests
85102
working-directory: path/to/playwright/folder # update accordingly
86-
env:
87-
# Regional endpoint for Microsoft Playwright Testing
88-
PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
89-
PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha } #This Run_ID will be unique and will remain same across all shards
90-
run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
103+
env:
104+
# Regional endpoint for Microsoft Playwright Testing
105+
PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
106+
PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha } #This Run_ID will be unique and will remain same across all shards
107+
run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
91108
```

‎articles/playwright-testing/quickstart-automate-end-to-end-testing.md

Lines changed: 124 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,79 @@ Enable artifacts such as screenshot, videos and traces to be captured by Playwri
158158
Once you collect these artifacts, attach them to the `TestContext` to ensure they are available in your test reports. For more information, see our [sample project for NUnit](https://aka.ms/mpt/nunit-sample)
159159
::: zone-end
160160

161+
## Set up authentication
162+
163+
The CI machine running Playwright tests needs to authenticate with Playwright Testing service to get the browsers to run the tests and to publish the test results and arifacts back to the service.
164+
165+
The service offers two authentication methods: Microsoft Entra ID and Access Tokens. We strongly recommend using Microsoft Entra ID to authenticate your pipelines.
166+
167+
#### Set up authentication using Microsoft Entra ID
168+
169+
# [GitHub Actions](#tab/github)
170+
171+
If you are using GitHub Actions, you can connect to the service using GitHub OpenID Connect. Follow the steps below to set up the integration:
172+
173+
##### Prerequisites
174+
175+
**Option 1: Microsoft Entra application**
176+
177+
* Create a Microsoft Entra application with a service principal by [Azure portal](/entra/identity-platform/howto-create-service-principal-portal#register-an-application-with-microsoft-entra-id-and-create-a-service-principal), [Azure CLI](/cli/azure/azure-cli-sp-tutorial-1#create-a-service-principal), or [Azure PowerShell](/powershell/azure/create-azure-service-principal-azureps#create-a-service-principal).
178+
179+
* Copy the values for **Client ID**, **Subscription ID**, and **Directory (tenant) ID** to use later in your GitHub Actions workflow.
180+
181+
* Assign the `Owner` or `Contributor` role to the service principal created in the previous step. These roles must be assigned on the Playwright Testing workspace. For more details, refer to [How to manage access](./how-to-manage-access-tokens.md).
182+
183+
* [Configure a federated identity credential on a Microsoft Entra application](/entra/workload-id/workload-identity-federation-create-trust) to trust tokens issued by GitHub Actions to your GitHub repository.
184+
185+
**Option 2: User-assigned managed identity**
186+
187+
* [Create a user-assigned managed identity](/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities#create-a-user-assigned-managed-identity).
188+
189+
* Copy the values for **Client ID**, **Subscription ID**, and **Directory (tenant) ID** to use later in your GitHub Actions workflow.
190+
191+
* Assign the `Owner` or `Contributor` role to the user-assigned managed identity created in the previous step. These roles must be assigned on the Playwright Testing workspace. For more details, refer to [How to manage access](./how-to-manage-access-tokens.md).
192+
193+
* [Configure a federated identity credential on a user-assigned managed identity](/entra/workload-id/workload-identity-federation-create-trust-user-assigned-managed-identity) to trust tokens issued by GitHub Actions to your GitHub repository.
194+
195+
##### Create GitHub secrets
196+
197+
1. Add the values you got in the previous step as secrets to your GitHub repository. See [set up GitHub Action Secret](/azure/developer/github/connect-from-azure-openid-connect?branch=main#create-github-secrets). These variables will be used in the GitHub Action workflow in subsequest steps.
198+
199+
| GitHub Secret | Source (Microsoft Entra Application or Managed Identity) |
200+
|---------------------|----------------------------------------------------------|
201+
| `AZURE_CLIENT_ID` | Client ID |
202+
| `AZURE_SUBSCRIPTION_ID` | Subscription ID |
203+
| `AZURE_TENANT_ID` | Directory (Tenant) ID |
204+
205+
> **Note:**
206+
> For enhanced security, it is strongly recommended to use GitHub Secrets to store sensitive values rather than including them directly in your workflow file.
207+
208+
# [Azure Pipelines](#tab/pipelines)
209+
210+
If you are using Azure Pipelines, you can connect to the service using Service Connections. Follow the steps below to set up the integration:
211+
212+
1. [Create an app registration with workload identity federation](/azure/devops/pipelines/library/connect-to-azure?view=azure-devops#create-an-app-registration-with-workload-identity-federation-automatic). Select the subscription and resource group associated with your Playwright Testing workspace. Typically, the resource group has the same name as the Playwright Testing workspace.
213+
214+
2. Use this service connection in Azure Pipeline yaml file as shown in subsequent steps.
215+
216+
#### Set up authentication using access tokens
217+
218+
> [!CAUTION]
219+
> We strongly recommend using Microsoft Entra ID for authentication to the service. If you are using access tokens, see [How to Manage Access Tokens](./how-to-manage-access-tokens.md)
220+
221+
You can generate an access token from your Playwright Testing workspace and use it in your setup. However, we strongly recommend Microsoft Entra ID for authentication due to its enhanced security. Access tokens, while convenient, function like long-lived passwords and are more susceptible to being compromised.
222+
223+
1. Authentication using access tokens is disabled by default. To use, [Enable access-token based authentication](./how-to-manage-authentication.md#enable-authentication-using-access-tokens)
224+
225+
2. [Set up authentication using access tokens](./how-to-manage-authentication.md#set-up-authentication-using-access-tokens)
226+
227+
3. Store the access token in a CI workflow secret and use it in the GitHub Actions workflow or Azure Pipeline yaml file.
228+
229+
| Secret name | Value |
230+
| ----------- | ------------ |
231+
| *PLAYWRIGHT_SERVICE_ACCESS_TOKEN* | Paste the value of Access Token you created previously. |
232+
233+
161234
## Update the workflow definition
162235

163236
::: zone pivot="playwright-test-runner"
@@ -175,47 +248,51 @@ Update the CI workflow definition to run your Playwright tests with the Playwrig
175248

176249
# This step is to sign-in to Azure to run tests from GitHub Action workflow.
177250
# Choose how to set up authentication to Azure from GitHub Actions. This is one example.
178-
on:
179-
push:
180-
branches: [ main, master ]
181-
pull_request:
182-
branches: [ main, master ]
183-
permissions: # Required when using Microsoft Entra ID to authenticate
184-
id-token: write
185-
contents: read
186-
jobs:
187-
test:
188-
timeout-minutes: 60
189-
runs-on: ubuntu-latest
251+
name: Playwright Tests (Microsoft Playwright Testing)
252+
on:
253+
push:
254+
branches: [main, master]
255+
pull_request:
256+
branches: [main, master]
257+
258+
permissions: # Required when using Microsoft Entra ID to authenticate
259+
id-token: write
260+
contents: read
261+
262+
jobs:
263+
test:
264+
timeout-minutes: 60
265+
runs-on: ubuntu-latest
190266
steps:
191-
- uses: actions/checkout@v4
192-
- name: Login to Azure with AzPowershell (enableAzPSSession true)
193-
uses: azure/login@v2
194-
with:
195-
client-id: ${{ secrets.AZURE_CLIENT_ID }}
196-
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
197-
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
198-
enable-AzPSSession: true
199-
200-
- name: Install dependencies
201-
working-directory: path/to/playwright/folder # update accordingly
202-
run: npm ci
203-
204-
- name: Run Playwright tests
205-
working-directory: path/to/playwright/folder # update accordingly
206-
env:
207-
# Regional endpoint for Microsoft Playwright Testing
208-
PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
209-
PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
210-
run: npx playwright test -c playwright.service.config.ts --workers=20
211-
212-
- name: Upload Playwright report
213-
uses: actions/upload-artifact@v3
214-
if: always()
215-
with:
216-
name: playwright-report
217-
path: path/to/playwright/folder/playwright-report/ # update accordingly
218-
retention-days: 10
267+
- uses: actions/checkout@v4
268+
269+
- name: Login to Azure with AzPowershell (enableAzPSSession true)
270+
uses: azure/login@v2
271+
with:
272+
client-id: ${{ secrets.AZURE_CLIENT_ID }} # GitHub Open ID connect values copied in previous steps
273+
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
274+
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
275+
enable-AzPSSession: true
276+
277+
- name: Install dependencies
278+
working-directory: path/to/playwright/folder # update accordingly
279+
run: npm ci
280+
281+
- name: Run Playwright tests
282+
working-directory: path/to/playwright/folder # update accordingly
283+
env:
284+
# Regional endpoint for Microsoft Playwright Testing
285+
PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
286+
# PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }} # Not recommended, use Microsoft Entra ID authentication.
287+
run: npx playwright test -c playwright.service.config.ts --workers=20
288+
289+
- name: Upload Playwright report
290+
uses: actions/upload-artifact@v3
291+
if: always()
292+
with:
293+
name: playwright-report
294+
path: path/to/playwright/folder/playwright-report/ # update accordingly
295+
retention-days: 10
219296
```
220297
221298
# [Azure Pipelines](#tab/pipelines)
@@ -233,7 +310,8 @@ Update the CI workflow definition to run your Playwright tests with the Playwrig
233310
displayName: Run Playwright Test
234311
env:
235312
PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL)
236-
PLAYWRIGHT_SERVICE_RUN_ID: ${{ parameters.runIdPrefix }}$(Build.DefinitionName) - $(Build.BuildNumber) - $(System.JobAttempt)
313+
# PLAYWRIGHT_SERVICE_ACCESS_TOKEN: $(PLAYWRIGHT_SERVICE_ACCESS_TOKEN) # Not recommended, use Microsoft Entra ID authentication.
314+
237315
inputs:
238316
azureSubscription: My_Service_Connection # Service connection used to authenticate this pipeline with Azure to use the service
239317
scriptType: 'pscore'
@@ -287,7 +365,7 @@ Update the CI workflow definition to run your Playwright tests with the Playwrig
287365
- name: Login to Azure with AzPowershell (enableAzPSSession true)
288366
uses: azure/login@v2
289367
with:
290-
client-id: ${{ secrets.AZURE_CLIENT_ID }}
368+
client-id: ${{ secrets.AZURE_CLIENT_ID }} # GitHub Open ID connect values copied in previous steps
291369
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
292370
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
293371
enable-AzPSSession: true
@@ -310,7 +388,7 @@ Update the CI workflow definition to run your Playwright tests with the Playwrig
310388
env:
311389
# Regional endpoint for Microsoft Playwright Testing
312390
PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
313-
PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ github.sha }}
391+
# PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{ secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }} # Not recommended, use Microsoft Entra ID authentication.
314392
run: dotnet test --settings:.runsettings --logger "microsoft-playwright-testing" -- NUnit.NumberOfTestWorkers=20
315393
316394
- name: Upload Playwright report
@@ -345,9 +423,10 @@ Update the CI workflow definition to run your Playwright tests with the Playwrig
345423
displayName: Run Playwright Test
346424
env:
347425
PLAYWRIGHT_SERVICE_URL: $(PLAYWRIGHT_SERVICE_URL)
348-
PLAYWRIGHT_SERVICE_RUN_ID: ${{ parameters.runIdPrefix }}$(Build.DefinitionName) - $(Build.BuildNumber) - $(System.JobAttempt)
426+
# PLAYWRIGHT_SERVICE_ACCESS_TOKEN: $(PLAYWRIGHT_SERVICE_ACCESS_TOKEN) # Not recommended, use Microsoft Entra ID authentication.
427+
349428
inputs:
350-
azureSubscription: My_Service_Connection # Service connection used to authenticate this pipeline with Azure to use the service
429+
azureSubscription: My_Service_Connection # Service connection used to authenticate this pipeline with Azure to use the service using Microsoft Entra ID.
351430
scriptType: 'pscore'
352431
scriptLocation: 'inlineScript'
353432
inlineScript: |

0 commit comments

Comments
 (0)