Integrate Test262 into WPT

rfcs/test262_integration.md

@@ -0,0 +1,83 @@
+# RFC 229: Integrating Test262
+
+## Summary
+
+This RFC integrates the Test262 (ECMAScript) test suite into WPT by vendoring it directly under `third_party/test262`. Execution will be on by default.
+
+## Motivation
+
+*   **Browser-based Testing:** Run Test262 in real browsers (stable and experimental), not just JS shells. This provides a more complete conformance signal for how ECMAScript behaves in the full web platform environment.
+*   **Unified Infrastructure:** Use the existing WPT infrastructure (`wpt`, CI) and `wpt.fyi` for test execution and results analysis.
+*   **Operational Efficiency:** Remove the need for vendors to maintain separate infrastructure for running Test262 and correlating its results with WPT.
+*   **Interop:** Allow ECMAScript features to be proposed and selected for inclusion in cross-browser Interop efforts.
+
+## Detailed Design
+
+### 1. Source Management
+
+The Test262 test suite will be vendored directly into the WPT repository under `third_party/test262/`. This establishes a clear convention for third-party test suites. Other suites, such as WebAssembly (`wasm/core/`), could be moved under `third_party/` in the future for consistency.
+
+*   A `vendored.toml` file will be located at `third_party/test262/vendored.toml`. This file will contain the source repository URL and the specific commit SHA of the vendored revision, acting as a source of truth. The format will be:
+    ```toml
+    [test262]
+    source = "https://github.com/tc39/test262"
+    rev = "<sha1>"
+    ```
+*   A dedicated CI job will keep this vendored copy up-to-date. By having a separate configuration file for each vendored suite, automated update PRs for different suites will not conflict with each other. The CI job will periodically:
+    1.  Fetch the latest commit SHA from the official `tc39/test262` repository.
+    2.  Copy the `test/` and `harness/` directories into `third_party/test262`.
+    3.  Update the `rev` in `vendored.toml`.
+    4.  Open a PR with all the above changes.
+*   The update PR will include a smoketest to ensure Test262 integration is not broken. This small, dedicated test suite in `infrastructure/test262/` will verify key integration points (e.g., YAML frontmatter parsing for `negative` tests/`flags`, `assert.js` harness injection). Expected results will be in corresponding `.ini` files within `infrastructure/metadata/test262/` (e.g., `infrastructure/testharness/lone-surrogates.html` metadata in `infrastructure/metadata/infrastructure/testharness/lone-surrogates.html.ini`). This smoketest is the *only* blocking Test262-related check in the PR; full test runs occur in scheduled nightly runs, populating wpt.fyi.
+*   This approach ensures WPT consumers have no novel requirements beyond selecting the test type.
+
+### 2. Test Integration
+
+WPT's manifest generation will be extended to support Test262.
+
+*   **Manifest Generation:** `wpt manifest` will, by default, recognize `third_party/test262`, traverse its `test` directory, and add discovered tests to `MANIFEST.json`.
+*   **Metadata Parsing:** A new Test262-specific parser will read YAML frontmatter from `.js` files to extract metadata (e.g., ES features, negative expectations, execution flags). Unlike `wasm/core` tests, which are pre-processed, Test262 metadata will be used by `wptserve` handlers at runtime.
+*   **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use parsed metadata to construct the execution environment. Test modes will use distinct URL conventions based on file extensions (e.g., `path/to/test.js` served as `path/to/test.test262.html` or `path/to/test.test262-module.html`).
+
+### 3. Execution Control
+
+Running Test262 is **on by default**.
+
+*   **Default Behavior:** `wpt run` will include Test262 tests by default, consistent with all other test types.
+*   **Excluding Tests:** Users who wish to exclude Test262 (or any other type) can do so by specifying the types they *do* want to run (e.g., `wpt run --test-types=testharness,reftest`). A dedicated `--exclude-test-types` flag may be added in the future.
+*   **CI Integration:** CI systems will run Test262 tests by default. No special flags are needed to enable them.
+
+## Alternatives Considered
+
+### Storing Test262 Metadata in the Manifest
+
+Storing parsed YAML metadata from Test262 files directly within `MANIFEST.json` was considered. This is the standard WPT architecture for other dynamic test types (e.g., Workers, Window tests) to optimize runtime performance.
+
+The current dynamic approach requires two disk I/O operations per test: one for the `wptserve` handler to parse YAML frontmatter, and a second for the browser to fetch the `.js` content. Storing metadata in `MANIFEST.json` would eliminate the first disk I/O and YAML parsing, effectively halving disk I/O operations per test during runtime.
+
+However, this optimization is deferred for initial implementation simplicity. Metadata will be parsed dynamically by `wptserve` handlers at runtime.
+
+### Pre-generating HTML Wrappers
+
+Pre-generating static `.js.html` files for each Test262 test (as done for some other vendored suites like WebAssembly) was considered but rejected due to:
+
+*   **Complexity of Multiple Contexts:** Test262 tests execute in various contexts (e.g., module, strict). Pre-generating wrappers for all combinations would lead to a combinatorial explosion of static files, adding potentially far more than 50,000 new files and creating a significant management burden.
+
+Dynamic HTML wrapper generation at serve-time, driven by Test262's metadata, offers a more flexible and efficient solution without duplicating repository files.
+
+### Using a dedicated `--test262` flag
+
+This was considered to provide a clear separation from WPT's internal test types. However, using the existing `--test-types` flag is more consistent with how other test types are selected and is the preferred approach.
+
+## Implementation Considerations & Risks
+
+*   **Manifest Generation Performance:** Including the 50,000+ Test262 tests in the manifest by default will increase the time required to run `wpt manifest`. This performance cost is accepted to ensure a simpler user experience, where a single, default manifest contains all tests.
+*   **Repository Size:** Vendoring Test262 will significantly increase WPT repository size (~50,000 files), a manageable increase.
+*   **CI Performance:** Full Test262 runs are lengthy. They will be scheduled during the existing nighly builds, not on every pull request.
+*   **Upstream Changes:** Updating vendored Test262 tests requires care to ensure changes to the test harness or metadata format are handled correctly. The CI job's update PRs will include a smoketest for basic integration issues.
+*   **Unit Tests for New Code:** New Python code for Test262 integration (e.g., manifest parsing, serving logic) will have dedicated unit tests, separate from integration smoketests.
+
+## Proof of Concept
+
+A prototype Test262 integration exists in:
+[https://github.com/web-platform-tests/wpt/pull/55997](https://github.com/web-platform-tests/wpt/pull/55997)