Skip to content

Commit 597bec7

Browse files
test(docs): add info on how to run tests and update screenshots (#28229)
Issue number: N/A --------- <!-- Please do not submit updates to dependencies unless it fixes an issue. --> <!-- Please try to limit your pull request to one type (bugfix, feature, etc). Submit multiple pull requests if needed. --> ## What is the current behavior? <!-- Please describe the current behavior that you are modifying. --> During planning the team discussed that some of our internal docs are outdated. Additionally, some of this info would be useful for contributors. ## What is the new behavior? <!-- Please describe the behavior or changes that are being added by this PR. --> - Ported remaining E2E docs to GitHub - Added info on installing deps, running tests, and updating screenshots. API and best practices are already on the repo. ## Does this introduce a breaking change? - [ ] Yes - [x] No <!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. --> ## Other information <!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. --> --------- Co-authored-by: Brandy Carney <[email protected]>
1 parent eb41b55 commit 597bec7

File tree

2 files changed

+109
-0
lines changed

2 files changed

+109
-0
lines changed

core/src/utils/test/playwright/docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,6 @@ This directory contains information on how to get the most out of Ionic's E2E te
66

77
| Directory | Description |
88
| - | - |
9+
| [Usage Instructions](./usage-instructions.md) | How to run tests and update screenshots |
910
| [Best Practices](./best-practices.md) | Contains information on conventions to follow as well as pitfalls to avoid when writing tests |
1011
| [API](./api.md) | Documents the custom functionality that has been built on top of Playwright |
Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Usage Instructions
2+
3+
E2E tests verify Ionic components in a real browser. This is useful for testing user interaction and catching visual regressions. We use Playwright as it allows us to test in multiple browsers. Tests can be written and run using Playwright's public API.
4+
5+
## Table of Contents
6+
7+
- [Installing Dependencies](#installing-dependencies)
8+
- [Running Tests](#running-tests)
9+
- [Managing Screenshots](#managing-screenshots)
10+
- [Further Reading](#further-reading)
11+
12+
## Installing Dependencies
13+
14+
Follow these steps to install Playwright dependencies. These steps must also be run whenever the installed version of Playwright changes to ensure that you are testing with the correct browser binaries.
15+
16+
1. Install the Playwright dependency in the `core` directory: `npm ci`
17+
2. Download the correct browsers: `npx playwright install`
18+
19+
## Running Tests
20+
21+
### Running All Test Files
22+
23+
All E2E tests can be run using the following command:
24+
25+
```shell
26+
npm run test.e2e
27+
```
28+
29+
> [!NOTE]
30+
> This command is a wrapper for `npx playwright test`. All data passed to `npm run test.e2e` can also be passed to `npx playwright test`.
31+
32+
### Running Specific Test Files
33+
34+
Specific test files can be run by passing the file paths or a directory that contains multiple test files. See [Managing Screenshots](#managing-screenshots) for generating ground truths before running screenshot tests.
35+
36+
**Specific Test Files**
37+
38+
```shell
39+
npm run test.e2e src/components/button/test/basic/button.e2e.ts src/components/button/test/a11y/button.e2e.ts
40+
```
41+
42+
**Test Directory with Multiple Files**
43+
44+
```shell
45+
# Will run all the test files in the `test` directory
46+
npm run test.e2e src/components/button/test
47+
```
48+
49+
## Managing Screenshots
50+
51+
### Generating or Updating Ground Truths (Local Development)
52+
53+
If you are running a test that takes a screenshot, you must first generate the reference screenshot from your reference branch. This is known as generating a "ground truth screenshot". All other screenshots will be compared to this ground truth. Alternatively, if the reference branch has changed since the last time you generated ground truths you may need to update your local ground truths.
54+
55+
For most types of work the reference branch is typically `main`. Features are merged into a different branch, so developers should use that as the reference branch. For example, if branch `foo` will be merged into `bar`, then the reference branch is `bar`.
56+
57+
The examples provided in the [Running Tests](#running-tests) section also apply here, allowing you to update screenshots for a specific test file.
58+
59+
Note that since you are generating the reference branch ground truth screenshots, you must be on the reference branch locally. Don't forget to pull the latest reference branch changes and then re-build using `npm run build`.
60+
61+
```shell
62+
npm run test.e2e -- --update-snapshots
63+
```
64+
65+
From here, you can switch back to your branch and run the tests.
66+
67+
> [!NOTE]
68+
> Locally generated ground truths should not be committed to the repo. The `.gitignore` file prevents this from accidentally happening.
69+
70+
### Updating Ground Truths (CI)
71+
72+
> [!IMPORTANT]
73+
> Only Ionic Team members can update ground truths on the main repo. Ground truths cannot be updated on forked versions of the repo.
74+
75+
When making an intentional visual change, you will need to update the ground truth screenshots or add new ones. It is important that the ground truth and comparison screenshots are taken in the same environment, so do not update the ground truth screenshots locally and commit them to the repo.
76+
77+
Instead, use the [Update Reference Screenshots GitHub Action](https://github.com/ionic-team/ionic-framework/actions/workflows/update-screenshots.yml).
78+
79+
1. Click the **Run workflow** dropdown.
80+
2. Select your branch.
81+
3. Click **Run workflow**.
82+
83+
This workflow will re-run the screenshot tests. Instead of failing any tests with mismatched screenshots, it will take new ground truth screenshots. These ground truth screenshots will be pushed as a single commit to your branch once the workflow is completed.
84+
85+
### Verifying Screenshot Differences
86+
87+
When any of the screenshot tests fail, it means a potential regression was caught. Developers must manually verify the difference in the Playwright test report.
88+
89+
If the screenshots fail on CI then developers must download the build artifact. On the **Summary** page for a particular workflow, find the **Artifacts** section. Screenshot tests are currently parallelized across several test runners, and the results from each of those runners is included in an artifact with the following naming scheme:
90+
91+
```
92+
test-results-[current shard]-[total shards]
93+
94+
Example:
95+
96+
test-results-2-5 --> Test results from job runner 2 out of 5.
97+
```
98+
99+
Download the appropriate artifact and unzip the file.
100+
101+
In the newly created directory, open the `playwright-report/index.html` in your browser. From here, you will be able to see the tests that failed as well as the expected screenshot, the actual screenshot, and the pixel differences.
102+
103+
> [!WARNING]
104+
> It is recommended to verify the screenshot difference within the Playwright test report first. If you choose to try and reproduce the difference in a browser manually, make sure you are using the **exact** same browser version that Playwright is using.
105+
106+
## Further Reading
107+
108+
For more info on how to use Playwright, please see the [Playwright documentation](https://playwright.dev/docs/intro).

0 commit comments

Comments
 (0)