Skip to content

Commit 64af5da

Browse files
Further updates to e2e test running helpers and instructions (#9969)
* further updates to e2e test running helpers and instructions * Update packages/wrangler/e2e/README.md Co-authored-by: Somhairle MacLeòid <[email protected]> --------- Co-authored-by: Somhairle MacLeòid <[email protected]>
1 parent d6a1b9b commit 64af5da

File tree

5 files changed

+86
-76
lines changed

5 files changed

+86
-76
lines changed

CONTRIBUTING.md

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -334,21 +334,23 @@ A summary of this repositories actions can be found [in the `.github/workflows`
334334

335335
## Running E2E tests locally
336336

337-
A large number of Wrangler & Vite's E2E tests don't require any authentication, and can be run with no Cloudflare account credentials. These can be run as follows:
337+
A large number of Wrangler, C3 & Vite's E2E tests don't require any authentication, and can be run with no Cloudflare account credentials. These can be run as follows, optionally providing [`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` environment variables.](#creating-an-api-token):
338338

339-
- **Vite:** `pnpm test:e2e -F @cloudflare/vite-plugin`, optionally providing [`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` environment variables.](#creating-an-api-token)
339+
- **Vite:** `pnpm test:e2e -F @cloudflare/vite-plugin`
340340

341-
- **Wrangler:** `pnpm test:e2e:wrangler`, optionally providing [`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` environment variables.](#creating-an-api-token)
341+
You may optionally want to append a filename pattern to limit which e2e tests are run. Also you may want to set `--bail=n` to limit the number of fails tests to show the error before the rest of the tests finish running and to limit the noise in that output:
342342

343-
You may optionally want to append a filename pattern to limit which e2e tests are run. Also you may want to set `--bail=n` to limit the number of fails tests to show the error before the rest of the tests finish running and to limit the noise in that output:
343+
```sh
344+
pnpm test:e2e -F @cloudflare/vite-plugin [file-pattern] --bail=1
345+
```
344346

345-
```sh
346-
# Vite
347-
pnpm test:e2e -F @cloudflare/vite-plugin [file-pattern] --bail=1
347+
- **C3:** `pnpm test:e2e -F create-cloudflare`
348348

349-
# Wrangler
350-
pnpm test:e2e:wrangler [file-pattern] --bail=1
351-
```
349+
See [C3 E2E tests README](packages/create-cloudflare/e2e/README.md) for more information.
350+
351+
- **Wrangler:** `pnpm test:e2e -F wrangler` or `pnpm test:e2e:wrangler`
352+
353+
See [Wrangler E2E tests README](packages/wrangler/e2e/README.md) for more information.
352354

353355
### Creating an API Token
354356

@@ -367,10 +369,13 @@ Once you've created the token, you can use it when running E2E tests to test aga
367369

368370
```sh
369371
# Vite
370-
CLOUDFLARE_ACCOUNT_ID="<Account ID for the token you just created>" CLOUDFLARE_API_TOKEN="<Token you just created>" pnpm test:e2e -F @cloudflare/vite-plugin [file-pattern] --bail=1
372+
CLOUDFLARE_ACCOUNT_ID="<Account ID for the token you just created>" CLOUDFLARE_API_TOKEN="<Token you just created>" pnpm test:e2e -F @cloudflare/vite-plugin
373+
374+
# C3
375+
CLOUDFLARE_ACCOUNT_ID="<Account ID for the token you just created>" CLOUDFLARE_API_TOKEN="<Token you just created>" pnpm test:e2e -F @create-cloudflare
371376

372377
# Wrangler
373-
CLOUDFLARE_ACCOUNT_ID="<Account ID for the token you just created>" CLOUDFLARE_API_TOKEN="<Token you just created>" pnpm test:e2e:wrangler [file-pattern] --bail=1
378+
CLOUDFLARE_ACCOUNT_ID="<Account ID for the token you just created>" CLOUDFLARE_API_TOKEN="<Token you just created>" pnpm test:e2e:wrangler
374379
```
375380

376381
> [!NOTE]

packages/wrangler/e2e/README.md

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,37 @@ This folder contains e2e tests for Wrangler.
44

55
## Run the tests
66

7-
Run each of the test files as a separately cached turbo task.
7+
Run each of the e2e test files as a separately cached turbo task. This is what we run in the CI.
8+
9+
This is the most effective way to run the Wrangler e2e tests. If any of the tests flake, the ones that passed will be cached and will be skipped on re-runs.
810

911
```zsh
1012
pnpm test:e2e:wrangler
1113
```
1214

15+
You can also run the turbo task directly if you want more fine grained control although this is not as resilient to flakes and is less effective at caching:
16+
17+
```zsh
18+
pnpm test:e2e -F wrangler -- <extra-vitest-params>
19+
```
20+
1321
## Configuration
1422

15-
You can configure how these e2e tests are run, in terms of the backend Cloudflare account, Wrangler and Miniflare distributable binaries and libraries, and which test file to run.
23+
You can configure how these e2e tests are run:
24+
25+
- Vitest configuration
26+
- Cloudflare credentials
27+
- The e2e test file to run
28+
29+
### Vitest configuration
30+
31+
Any params after a `--` will be passed to the Vitest runner, so you can use this to configure the test run.
32+
33+
For example to update the snapshots for all Wrangler e2e tests and bail after only 1 error, you can run:
34+
35+
```zsh
36+
pnpm test:e2e:wrangler -- -u --bail=1
37+
```
1638

1739
### Cloudflare Credentials
1840

packages/wrangler/e2e/helpers/wrangler.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@ import type { CommandOptions } from "./command";
88
// in scripts doesn't break.
99
export const WRANGLER =
1010
process.env.WRANGLER?.replaceAll("\\", "/") ??
11-
`node ${resolve("packages/wrangler/bin/wrangler.js")}`;
11+
`node --no-warnings ${resolve(__dirname, "../../bin/wrangler.js")}`;
1212
export const WRANGLER_IMPORT = pathToFileURL(
1313
process.env.WRANGLER_IMPORT?.replaceAll("\\", "/") ??
14-
resolve("packages/wrangler/wrangler-dist/cli.js")
14+
resolve(__dirname, "../../wrangler-dist/cli.js")
1515
);
1616
export const MINIFLARE_IMPORT = pathToFileURL(
1717
process.env.MINIFLARE_IMPORT?.replaceAll("\\", "/") ??
18-
resolve("packages/miniflare/dist/src/index.js")
18+
resolve(__dirname, "../../../miniflare/dist/src/index.js")
1919
);
2020

2121
export type WranglerCommandOptions = CommandOptions & { debug?: boolean };

packages/wrangler/e2e/validate-environment.ts

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,14 @@ if (!process.env.WRANGLER_IMPORT) {
1111

1212
if (!process.env.CLOUDFLARE_ACCOUNT_ID) {
1313
console.warn(
14-
"No CLOUDFLARE_ACCOUNT_ID variable provided, skipping API tests"
14+
"No `CLOUDFLARE_ACCOUNT_ID` variable provided, skipping API tests"
1515
);
1616
}
1717

1818
if (!process.env.CLOUDFLARE_API_TOKEN) {
19-
console.warn("No CLOUDFLARE_API_TOKEN variable provided, skipping API tests");
19+
console.warn(
20+
"No `CLOUDFLARE_API_TOKEN` variable provided, skipping API tests"
21+
);
2022
}
2123

2224
// Exporting noop vitest setup function allows it to be loaded as a setup file.

tools/e2e/runIndividualE2EFiles.ts

Lines changed: 38 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,70 +1,51 @@
1-
// Turbo's env var linting isn't very sophisticated.
2-
// This file adds environment variables that are declared in Wrangler's turbo.json
3-
/* eslint-disable turbo/no-undeclared-env-vars */
4-
5-
/**
6-
* Turbo only supports caching on the individual task level, but for Wrangler's
7-
* e2e tests we want to support caching on a more granular basis—at the file level.
8-
*
9-
* As such, we run the `test:e2e` turbo task multiple times—once per e2e test file
10-
* with different arguments, ensuring that each file's tests can be cached individually.
11-
*
12-
* The intended flow here is that CI will run this file, which will trigger turbo to run
13-
* an individual task for each Wrangler e2e test file, using `execSync`.
14-
*/
151
import { execSync } from "child_process";
162
import { readdirSync } from "fs";
17-
18-
// Get a list of e2e test files, each of which should have an associated script
19-
const e2eTests = readdirSync("packages/wrangler/e2e");
20-
21-
const tasks = new Set<string>();
22-
23-
for (const file of e2eTests) {
24-
// Ignore other files in the e2e directory (the README, for instance)
25-
if (file.endsWith(".test.ts")) {
26-
tasks.add(`e2e/${file}`);
27-
}
28-
}
3+
import type { ExecSyncOptionsWithBufferEncoding } from "child_process";
4+
5+
// Turbo only supports caching on the individual task level, but for Wrangler's
6+
// e2e tests we want to support caching on a more granular basis - at the file level.
7+
//
8+
// As such, we run the `test:e2e` turbo task multiple times — once per e2e test file so that each file's tests can be cached individually.
9+
// We use the `WRANGLER_E2E_TEST_FILE` environment variable to pass the specific test file to the e2e test runner so that it reuses the cached build tasks.
10+
// If you use a command line argument to do this turbo will create a different cache key for the build tasks.
11+
//
12+
// The intended flow here is that CI will run this file, which will trigger turbo to run
13+
// an individual task for each Wrangler e2e test file, using `execSync`.
14+
//
15+
// Any params after a `--` will be passed to the Vitest runner, so you can use this to configure the test run.
16+
// For example to update the snapshots for all Wrangler e2e tests, you can run:
17+
//
18+
// ```bash
19+
// pnpm test:e2e:wrangler -- -u
20+
// ```
21+
22+
const extraParamsIndex = process.argv.indexOf("--");
23+
const extraParams =
24+
extraParamsIndex === -1 ? [] : process.argv.slice(extraParamsIndex);
25+
const command =
26+
`pnpm test:e2e --log-order=stream --output-logs=new-only --summarize --filter wrangler ` +
27+
extraParams.join(" ");
2928

3029
const failed: string[] = [];
30+
for (const file of readdirSync("packages/wrangler/e2e")) {
31+
if (!file.endsWith(".test.ts")) {
32+
// Ignore other files in the e2e directory (the README, for instance)
33+
continue;
34+
}
3135

32-
// If the user passes arguments to this script, use those to configure Vitest rather than running each test file individually
33-
const hasCustomVitestArguments = process.argv[2];
34-
35-
const commandSuffix = hasCustomVitestArguments
36-
? ` -- --no-file-parallelism ${process.argv.slice(2).join(" ")}`
37-
: "";
38-
const command = `pnpm test:e2e --log-order=stream --output-logs=new-only --summarize --filter wrangler${commandSuffix}`;
39-
40-
// Add the default environment configuration for E2E tests.
41-
// Most of these rely on Turbo being set up correctly to build Wrangler & C3:
42-
// https://github.com/cloudflare/workers-sdk/tree/main/packages/wrangler/turbo.json#L61-L62
43-
process.env.WRANGLER ??= `node --no-warnings ${process.cwd()}/packages/wrangler/bin/wrangler.js`;
44-
process.env.WRANGLER_IMPORT ??= `${process.cwd()}/packages/wrangler/wrangler-dist/cli.js`;
45-
process.env.MINIFLARE_IMPORT ??= `${process.cwd()}/packages/miniflare/dist/src/index.js`;
46-
47-
if (hasCustomVitestArguments) {
48-
execSync(command, {
36+
const WRANGLER_E2E_TEST_FILE = `e2e/${file}`;
37+
const options: ExecSyncOptionsWithBufferEncoding = {
4938
stdio: "inherit",
50-
env: { ...process.env },
51-
});
52-
process.exit(0);
53-
}
54-
for (const file of tasks) {
55-
console.log("::group::Testing: " + file);
39+
env: { ...process.env, WRANGLER_E2E_TEST_FILE },
40+
};
41+
42+
console.log("::group::Testing: " + WRANGLER_E2E_TEST_FILE);
5643
try {
57-
execSync(command, {
58-
stdio: "inherit",
59-
env: { ...process.env, WRANGLER_E2E_TEST_FILE: file },
60-
});
44+
execSync(command, options);
6145
} catch {
6246
console.error("Task failed - retrying");
6347
try {
64-
execSync(command, {
65-
stdio: "inherit",
66-
env: { ...process.env, WRANGLER_E2E_TEST_FILE: file },
67-
});
48+
execSync(command, options);
6849
} catch {
6950
console.error("Still failed, moving on");
7051
failed.push(file);

0 commit comments

Comments
 (0)