|
1 | 1 | # Automated Testing |
2 | 2 |
|
3 | | -## Why is Puppeteer the tool of choice for end-to-end tests? |
| 3 | +## Why is Playwright the tool of choice for end-to-end tests? |
4 | 4 |
|
5 | | -There exists a rich ecosystem of tooling available for web-based end-to-end automated testing. Thus, it's a common question: "Why does Gutenberg use [Puppeteer](https://developers.google.com/web/tools/puppeteer/) instead of ([Cypress](https://cypress.io/), [Selenium](https://www.selenium.dev/), [Playwright](https://github.com/microsoft/playwright), etc)?". Given some historical unreliability of the build results associated with end-to-end tests, it's especially natural to weigh this question in considering whether our tools are providing more value than the effort required in maintaining them. While we should always be comfortable in reevaluating earlier decisions, there were and continue to be many reasons that Puppeteer is the best compromise of the options available for end-to-end testing. |
| 5 | +There exists a rich ecosystem of tooling available for web-based end-to-end automated testing. Gutenberg uses [Playwright](https://playwright.dev/) as its end-to-end testing framework. The project previously used [Puppeteer](https://pptr.dev/) but has since fully migrated to Playwright. Here are the reasons Playwright was chosen: |
6 | 6 |
|
7 | | -These include: |
| 7 | +- **Multi-browser support**. Playwright supports Chromium, Firefox, and WebKit out of the box, providing broader browser coverage compared to Puppeteer's Chrome-only approach. |
| 8 | +- **Built-in test runner**. `@playwright/test` provides a powerful test runner with parallel execution, [fixtures](https://playwright.dev/docs/test-fixtures), and auto-waiting, reducing flakiness and improving developer experience. |
| 9 | +- **Auto-waiting and web-first assertions**. Playwright automatically waits for elements to be actionable before performing actions, reducing the need for manual `waitFor*` calls while still surfacing legitimate performance issues that affect users. |
| 10 | +- **Better debugging tools**. Playwright offers a [trace viewer](https://playwright.dev/docs/trace-viewer), [UI mode](https://playwright.dev/docs/test-ui-mode), and a built-in [inspector](https://playwright.dev/docs/debug#playwright-inspector) for step-by-step debugging. |
| 11 | +- **Fixtures over global variables**. Playwright's fixture model injects `page`, `browser`, and other parameters into tests, making it easier to work with multiple pages or tabs and to run tests in parallel. |
| 12 | +- **Page Object Model**. Playwright encourages the [Page Object Model](https://playwright.dev/docs/pom) pattern for reusable utility functions, improving test readability and maintainability. |
8 | 13 |
|
9 | | -- **Interoperability with existing testing framework**. Puppeteer is "just" a tool for controlling a Chrome browser, and makes no assumptions about how it's integrated into a testing environment. While this requires some additional effort in ensuring the test environment is available, it also allows for cohesion in how it integrates with an existing setup. Gutenberg is able to consistently use Jest for both unit testing and end-to-end testing. This is contrasted with other solutions like Cypress, which provide their own testing framework and assertion library as part of an all-in-one solution. |
10 | | -- **An expressive but predictable API**. Puppeteer strikes a nice balance between low-level access to browser behavior, while retaining an expressive API for issuing and awaiting responses to those commands using modern JavaScript [`async` and `await` syntax](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await). This is contrasted with other solutions, which either don't support or leverage native language async functionality, don't expose direct access to the browser, or leverage custom domain-specific language syntaxes for expressing browser commands and assertions. The fact that Puppeteer largely targets the Chrome browser is non-ideal in how it does not provide full browser coverage. On the other hand, the limited set of browser targets offers more consistent results and stronger guarantees about how code is evaluated in the browser environment. |
11 | | -- **Surfacing bugs, not obscuring them**. Many alternative solutions offer options to automatically await settled network requests or asynchronous appearance of elements on the page. While this can serve as a convenience in accounting for unpredictable delays, it can also unknowingly cause oversight of legitimate user-facing issues. For example, if an element will only appear on the page after some network request or computation has completed, it may be easy to overlook that these delays can cause unpredictable and frustrating behavior for users ([example](https://github.com/WordPress/gutenberg/pull/11287)). Given that developers often test on high-end hardware and stable network connections, consideration of resiliency on low-end hardware or spotty network availability is not always on the forefront of one's considerations. Puppeteer forces us to acknowledge these delays with explicit `waitFor*` expressions, putting us in much greater alignment with the real-world experience of an end-user. |
12 | | -- **Debugging**. It's important that in that case that a test fails, there should be straight-forward means to diagnose and resolve the issue. While its offerings are rather simplistic relative to the competition, Puppeteer does expose options to run tests as "headful" (with the browser visible) and with delayed actions. Combined with the fact that it interoperates well with native language / runtime features (e.g. debugger statements or breakpoints), this provides developers with sufficient debugging access. |
13 | | - |
14 | | -For more context, refer to the following resources: |
15 | | - |
16 | | -- [Testing Overview: End-to-End Testing](/docs/contributors/code/testing-overview.md#end-to-end-testing) |
17 | | -- [Testing: Experiment with Puppeteer for E2E testing](https://github.com/WordPress/gutenberg/pull/5618) |
18 | | - - In early iterations, the contributing team opted to use Cypress for end-to-end testing. This pull request outlines problems with the approach, and proposed the initial transition to Puppeteer. |
19 | | -- [JavaScript Chat Summary: January 28, 2020](https://make.wordpress.org/core/2020/02/04/javascript-chat-summary-january-28-2020/) |
20 | | - - Playwright is a new offering created by many of the original contributors to Puppeteer. It offers increased browser coverage and improved reliability of tests. While still early in development at the time of this writing, there has been some interest in evaluating it for future use as an end-to-end testing solution. |
| 14 | +For more details on writing end-to-end tests, see the [End-to-End Testing guide](/docs/contributors/code/e2e/README.md). |
0 commit comments