|
| 1 | +# Usage Instructions |
| 2 | + |
| 3 | +E2E tests verify Ionic components in a real browser. This is useful for testing user interaction and catching visual regressions. We use Playwright as it allows us to test in multiple browsers. Tests can be written and run using Playwright's public API. |
| 4 | + |
| 5 | +## Table of Contents |
| 6 | + |
| 7 | +- [Installing Dependencies](#installing-dependencies) |
| 8 | +- [Running Tests](#running-tests) |
| 9 | +- [Managing Screenshots](#managing-screenshots) |
| 10 | +- [Further Reading](#further-reading) |
| 11 | + |
| 12 | +## Installing Dependencies |
| 13 | + |
| 14 | +Follow these steps to install Playwright dependencies. These steps must also be run whenever the installed version of Playwright changes to ensure that you are testing with the correct browser binaries. |
| 15 | + |
| 16 | +1. Install the Playwright dependency in the `core` directory: `npm ci` |
| 17 | +2. Download the correct browsers: `npx playwright install` |
| 18 | + |
| 19 | +## Running Tests |
| 20 | + |
| 21 | +### Running All Test Files |
| 22 | + |
| 23 | +All E2E tests can be run using the following command: |
| 24 | + |
| 25 | +```shell |
| 26 | +npm run test.e2e |
| 27 | +``` |
| 28 | + |
| 29 | +> [!NOTE] |
| 30 | +> This command is a wrapper for `npx playwright test`. All data passed to `npm run test.e2e` can also be passed to `npx playwright test`. |
| 31 | +
|
| 32 | +### Running Specific Test Files |
| 33 | + |
| 34 | +Specific test files can be run by passing the file paths or a directory that contains multiple test files. See [Managing Screenshots](#managing-screenshots) for generating ground truths before running screenshot tests. |
| 35 | + |
| 36 | +**Specific Test Files** |
| 37 | + |
| 38 | +```shell |
| 39 | +npm run test.e2e src/components/button/test/basic/button.e2e.ts src/components/button/test/a11y/button.e2e.ts |
| 40 | +``` |
| 41 | + |
| 42 | +**Test Directory with Multiple Files** |
| 43 | + |
| 44 | +```shell |
| 45 | +# Will run all the test files in the `test` directory |
| 46 | +npm run test.e2e src/components/button/test |
| 47 | +``` |
| 48 | + |
| 49 | +## Managing Screenshots |
| 50 | + |
| 51 | +### Generating or Updating Ground Truths (Local Development) |
| 52 | + |
| 53 | +If you are running a test that takes a screenshot, you must first generate the reference screenshot from your reference branch. This is known as generating a "ground truth screenshot". All other screenshots will be compared to this ground truth. Alternatively, if the reference branch has changed since the last time you generated ground truths you may need to update your local ground truths. |
| 54 | + |
| 55 | +For most types of work the reference branch is typically `main`. Features are merged into a different branch, so developers should use that as the reference branch. For example, if branch `foo` will be merged into `bar`, then the reference branch is `bar`. |
| 56 | + |
| 57 | +The examples provided in the [Running Tests](#running-tests) section also apply here, allowing you to update screenshots for a specific test file. |
| 58 | + |
| 59 | +Note that since you are generating the reference branch ground truth screenshots, you must be on the reference branch locally. Don't forget to pull the latest reference branch changes and then re-build using `npm run build`. |
| 60 | + |
| 61 | +```shell |
| 62 | +npm run test.e2e -- --update-snapshots |
| 63 | +``` |
| 64 | + |
| 65 | +From here, you can switch back to your branch and run the tests. |
| 66 | + |
| 67 | +> [!NOTE] |
| 68 | +> Locally generated ground truths should not be committed to the repo. The `.gitignore` file prevents this from accidentally happening. |
| 69 | +
|
| 70 | +### Updating Ground Truths (CI) |
| 71 | + |
| 72 | +> [!IMPORTANT] |
| 73 | +> Only Ionic Team members can update ground truths on the main repo. Ground truths cannot be updated on forked versions of the repo. |
| 74 | +
|
| 75 | +When making an intentional visual change, you will need to update the ground truth screenshots or add new ones. It is important that the ground truth and comparison screenshots are taken in the same environment, so do not update the ground truth screenshots locally and commit them to the repo. |
| 76 | + |
| 77 | +Instead, use the [Update Reference Screenshots GitHub Action](https://github.com/ionic-team/ionic-framework/actions/workflows/update-screenshots.yml). |
| 78 | + |
| 79 | +1. Click the **Run workflow** dropdown. |
| 80 | +2. Select your branch. |
| 81 | +3. Click **Run workflow**. |
| 82 | + |
| 83 | +This workflow will re-run the screenshot tests. Instead of failing any tests with mismatched screenshots, it will take new ground truth screenshots. These ground truth screenshots will be pushed as a single commit to your branch once the workflow is completed. |
| 84 | + |
| 85 | +### Verifying Screenshot Differences |
| 86 | + |
| 87 | +When any of the screenshot tests fail, it means a potential regression was caught. Developers must manually verify the difference in the Playwright test report. |
| 88 | + |
| 89 | +If the screenshots fail on CI then developers must download the build artifact. On the **Summary** page for a particular workflow, find the **Artifacts** section. Screenshot tests are currently parallelized across several test runners, and the results from each of those runners is included in an artifact with the following naming scheme: |
| 90 | + |
| 91 | +``` |
| 92 | +test-results-[current shard]-[total shards] |
| 93 | +
|
| 94 | +Example: |
| 95 | +
|
| 96 | +test-results-2-5 --> Test results from job runner 2 out of 5. |
| 97 | +``` |
| 98 | + |
| 99 | +Download the appropriate artifact and unzip the file. |
| 100 | + |
| 101 | +In the newly created directory, open the `playwright-report/index.html` in your browser. From here, you will be able to see the tests that failed as well as the expected screenshot, the actual screenshot, and the pixel differences. |
| 102 | + |
| 103 | +> [!WARNING] |
| 104 | +> It is recommended to verify the screenshot difference within the Playwright test report first. If you choose to try and reproduce the difference in a browser manually, make sure you are using the **exact** same browser version that Playwright is using. |
| 105 | +
|
| 106 | +## Further Reading |
| 107 | + |
| 108 | +For more info on how to use Playwright, please see the [Playwright documentation](https://playwright.dev/docs/intro). |
0 commit comments