From 0a7466454dc821bd5d2157978509dcd487bfd740 Mon Sep 17 00:00:00 2001 From: Raul Macarie Date: Mon, 13 Oct 2025 14:12:58 +0200 Subject: [PATCH 01/72] feat(browser): support custom screenshot comparison algorithms (#8687) --- guide/browser/config.md | 113 +++++++++++++++++++++ guide/browser/visual-regression-testing.md | 18 ++++ 2 files changed, 131 insertions(+) diff --git a/guide/browser/config.md b/guide/browser/config.md index c1f8c724..7dd724a7 100644 --- a/guide/browser/config.md +++ b/guide/browser/config.md @@ -492,3 +492,116 @@ For example, to store diffs in a subdirectory of attachments: resolveDiffPath: ({ arg, attachmentsDir, browserName, ext, root, testFileName }) => `${root}/${attachmentsDir}/screenshot-diffs/${testFileName}/${arg}-${browserName}${ext}` ``` + +#### browser.expect.toMatchScreenshot.comparators + +- **Type:** `Record` + +Register custom screenshot comparison algorithms, like [SSIM](https://en.wikipedia.org/wiki/Structural_similarity_index_measure) or other perceptual similarity metrics. + +To create a custom comparator, you need to register it in your config. If using TypeScript, declare its options in the `ScreenshotComparatorRegistry` interface. + +```ts +import { defineConfig } from 'vitest/config' + +// 1. Declare the comparator's options type +declare module 'vitest/browser' { + interface ScreenshotComparatorRegistry { + myCustomComparator: { + sensitivity?: number + ignoreColors?: boolean + } + } +} + +// 2. Implement the comparator +export default defineConfig({ + test: { + browser: { + expect: { + toMatchScreenshot: { + comparators: { + myCustomComparator: async ( + reference, + actual, + { + createDiff, // always provided by Vitest + sensitivity = 0.01, + ignoreColors = false, + } + ) => { + // ...algorithm implementation + return { pass, diff, message } + }, + }, + }, + }, + }, + }, +}) +``` + +Then use it in your tests: + +```ts +await expect(locator).toMatchScreenshot({ + comparatorName: 'myCustomComparator', + comparatorOptions: { + sensitivity: 0.08, + ignoreColors: true, + }, +}) +``` + +**Comparator Function Signature:** + +```ts +type Comparator = ( + reference: { + metadata: { height: number; width: number } + data: TypedArray + }, + actual: { + metadata: { height: number; width: number } + data: TypedArray + }, + options: { + createDiff: boolean + } & Options +) => Promise<{ + pass: boolean + diff: TypedArray | null + message: string | null +}> | { + pass: boolean + diff: TypedArray | null + message: string | null +} +``` + +The `reference` and `actual` images are decoded using the appropriate codec (currently only PNG). The `data` property is a flat `TypedArray` (`Buffer`, `Uint8Array`, or `Uint8ClampedArray`) containing pixel data in RGBA format: + +- **4 bytes per pixel**: red, green, blue, alpha (from `0` to `255` each) +- **Row-major order**: pixels are stored left-to-right, top-to-bottom +- **Total length**: `width × height × 4` bytes +- **Alpha channel**: always present. Images without transparency have alpha values set to `255` (fully opaque) + +::: tip Performance Considerations +The `createDiff` option indicates whether a diff image is needed. During [stable screenshot detection](/guide/browser/visual-regression-testing#how-visual-tests-work), Vitest calls comparators with `createDiff: false` to avoid unnecessary work. + +**Respect this flag to keep your tests fast**. +::: + +::: warning Handle Missing Options +The `options` parameter in `toMatchScreenshot()` is optional, so users might not provide all your comparator options. Always make them optional with default values: + +```ts +myCustomComparator: ( + reference, + actual, + { createDiff, threshold = 0.1, maxDiff = 100 }, +) => { + // ...comparison logic +} +``` +::: diff --git a/guide/browser/visual-regression-testing.md b/guide/browser/visual-regression-testing.md index 5049c272..101253f8 100644 --- a/guide/browser/visual-regression-testing.md +++ b/guide/browser/visual-regression-testing.md @@ -121,6 +121,24 @@ $ vitest --update Review updated screenshots before committing to make sure changes are intentional. +## How Visual Tests Work + +Visual regression tests need stable screenshots to compare against. But pages aren't instantly stable as images load, animations finish, fonts render, and layouts settle. + +Vitest handles this automatically through "Stable Screenshot Detection": + +1. Vitest takes a first screenshot (or uses the reference screenshot if available) as baseline +1. It takes another screenshot and compares it with the baseline + - If the screenshots match, the page is stable and testing continues + - If they differ, Vitest uses the newest screenshot as the baseline and repeats +1. This continues until stability is achieved or the timeout is reached + +This ensures that transient visual changes (like loading spinners or animations) don't cause false failures. If something never stops animating though, you'll hit the timeout, so consider [disabling animations during testing](#disable-animations). + +If a stable screenshot is captured after retries (one or more) and a reference screenshot exists, Vitest performs a final comparison with the reference using `createDiff: true`. This will generate a diff image if they don't match. + +During stability detection, Vitest calls comparators with `createDiff: false` since it only needs to know if screenshots match. This keeps the detection process fast. + ## Configuring Visual Tests ### Global Configuration From 1b92f369ddbff9d35be758ee3c6259404713bee2 Mon Sep 17 00:00:00 2001 From: Vladimir Date: Mon, 13 Oct 2025 16:48:31 +0200 Subject: [PATCH 02/72] feat: add `displayAnnotations` option to `github-options` (#8706) --- guide/reporters.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/guide/reporters.md b/guide/reporters.md index b411babb..5d705046 100644 --- a/guide/reporters.md +++ b/guide/reporters.md @@ -564,6 +564,18 @@ export default defineConfig({ }) ``` +If you are using [Annotations API](/guide/test-annotations), the reporter will automatically inline them in the GitHub UI. You can disable this by setting `displayAnnotations` option to `false`: + +```ts +export default defineConfig({ + test: { + reporters: [ + ['github-actions', { displayAnnotations: false }], + ], + }, +}) +``` + ### Blob Reporter Stores test results on the machine so they can be later merged using [`--merge-reports`](/guide/cli#merge-reports) command. From a216aa687812e7b0b12168ff92dee69c6f396d92 Mon Sep 17 00:00:00 2001 From: Romain Hamel Date: Mon, 13 Oct 2025 19:03:23 +0200 Subject: [PATCH 03/72] docs: update `TestSuite` meta example (#8708) --- advanced/api/test-suite.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/advanced/api/test-suite.md b/advanced/api/test-suite.md index bc3e2ab4..6894178a 100644 --- a/advanced/api/test-suite.md +++ b/advanced/api/test-suite.md @@ -197,14 +197,16 @@ Note that errors are serialized into simple objects: `instanceof Error` will alw function meta(): TaskMeta ``` -Custom [metadata](/advanced/metadata) that was attached to the suite during its execution or collection. The meta can be attached by assigning a property to the `task.meta` object during a test run: +Custom [metadata](/advanced/metadata) that was attached to the suite during its execution or collection. The meta can be attached by assigning a property to the `suite.meta` object during a test run: -```ts {5,10} +```ts {7,12} import { test } from 'vitest' +import { getCurrentSuite } from 'vitest/suite' -describe('the validation works correctly', (task) => { +describe('the validation works correctly', () => { // assign "decorated" during collection - task.meta.decorated = false + const { suite } = getCurrentSuite() + suite!.meta.decorated = true test('some test', ({ task }) => { // assign "decorated" during test run, it will be available From 514deb0fb0f314cc99c538a48cfbf1952744e8bd Mon Sep 17 00:00:00 2001 From: Chris Date: Tue, 14 Oct 2025 10:24:00 +0200 Subject: [PATCH 04/72] feat: add schema validation matchers (#8527) --- api/expect.md | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/api/expect.md b/api/expect.md index e577949c..06901025 100644 --- a/api/expect.md +++ b/api/expect.md @@ -1688,6 +1688,42 @@ test('variety ends with "re"', () => { You can use `expect.not` with this matcher to negate the expected value. ::: +## expect.schemaMatching + +- **Type:** `(expected: StandardSchemaV1) => any` + +When used with an equality check, this asymmetric matcher will return `true` if the value matches the provided schema. The schema must implement the [Standard Schema v1](https://standardschema.dev/) specification. + +```ts +import { expect, test } from 'vitest' +import { z } from 'zod' +import * as v from 'valibot' +import { type } from 'arktype' + +test('email validation', () => { + const user = { email: 'john@example.com' } + + // using Zod + expect(user).toEqual({ + email: expect.schemaMatching(z.string().email()), + }) + + // using Valibot + expect(user).toEqual({ + email: expect.schemaMatching(v.pipe(v.string(), v.email())) + }) + + // using ArkType + expect(user).toEqual({ + email: expect.schemaMatching(type('string.email')), + }) +}) +``` + +:::tip +You can use `expect.not` with this matcher to negate the expected value. +::: + ## expect.addSnapshotSerializer - **Type:** `(plugin: PrettyFormatPlugin) => void` From dcb91075ef7d2f822191d727b12faceb561369fa Mon Sep 17 00:00:00 2001 From: Vladimir Date: Tue, 14 Oct 2025 10:24:20 +0200 Subject: [PATCH 05/72] docs(browser): remove the "experimental" tag (#8707) --- advanced/api/plugin.md | 2 +- advanced/api/reporters.md | 2 +- advanced/guide/tests.md | 2 -- advanced/metadata.md | 4 ---- config/index.md | 6 +----- guide/browser/config.md | 1 + guide/browser/index.md | 4 ++-- guide/browser/why.md | 2 +- 8 files changed, 7 insertions(+), 16 deletions(-) diff --git a/advanced/api/plugin.md b/advanced/api/plugin.md index 039f9a9e..dd64a5fa 100644 --- a/advanced/api/plugin.md +++ b/advanced/api/plugin.md @@ -11,7 +11,7 @@ This is an advanced API. If you just want to [run tests](/guide/), you probably This guide assumes you know how to work with [Vite plugins](https://vite.dev/guide/api-plugin.html). ::: -Vitest supports an experimental `configureVitest` [plugin](https://vite.dev/guide/api-plugin.html) hook hook since version 3.1. Any feedback regarding this API is welcome in [GitHub](https://github.com/vitest-dev/vitest/discussions/7104). +Vitest supports a `configureVitest` [plugin](https://vite.dev/guide/api-plugin.html) hook hook since version 3.1. ::: code-group ```ts [only vitest] diff --git a/advanced/api/reporters.md b/advanced/api/reporters.md index 552f2fd8..ba87fc34 100644 --- a/advanced/api/reporters.md +++ b/advanced/api/reporters.md @@ -83,7 +83,7 @@ export default new MyReporter() ``` ::: -## onBrowserInit experimental {#onbrowserinit} +## onBrowserInit {#onbrowserinit} ```ts function onBrowserInit(project: TestProject): Awaitable diff --git a/advanced/guide/tests.md b/advanced/guide/tests.md index a37facf6..8bfc5060 100644 --- a/advanced/guide/tests.md +++ b/advanced/guide/tests.md @@ -2,8 +2,6 @@ ::: warning This guide explains how to use the advanced API to run tests via a Node.js script. If you just want to [run tests](/guide/), you probably don't need this. It is primarily used by library authors. - -Breaking changes might not follow SemVer, please pin Vitest's version when using the experimental API. ::: Vitest exposes two methods to initiate Vitest: diff --git a/advanced/metadata.md b/advanced/metadata.md index 883eb70c..b86faa64 100644 --- a/advanced/metadata.md +++ b/advanced/metadata.md @@ -1,9 +1,5 @@ # Task Metadata -::: warning -Vitest exposes experimental private API. Breaking changes might not follow SemVer, please pin Vitest's version when using it. -::: - If you are developing a custom reporter or using Vitest Node.js API, you might find it useful to pass data from tests that are being executed in various contexts to your reporter or custom Vitest handler. To accomplish this, relying on the [test context](/guide/test-context) is not feasible since it cannot be serialized. However, with Vitest, you can utilize the `meta` property available on every task (suite or test) to share data between your tests and the Node.js process. It's important to note that this communication is one-way only, as the `meta` property can only be modified from within the test context. Any changes made within the Node.js context will not be visible in your tests. diff --git a/config/index.md b/config/index.md index f49f1e20..52948261 100644 --- a/config/index.md +++ b/config/index.md @@ -1610,17 +1610,13 @@ Open Vitest UI (WIP) Listen to port and serve API. When set to true, the default port is 51204 -### browser experimental {#browser} +### browser {#browser} - **Default:** `{ enabled: false }` - **CLI:** `--browser=`, `--browser.name=chrome --browser.headless` Configuration for running browser tests. Please, refer to the ["Browser Config Reference"](/guide/browser/config) article. -::: warning -This is an experimental feature. Breaking changes might not follow SemVer, please pin Vitest's version when using it. -::: - ### clearMocks - **Type:** `boolean` diff --git a/guide/browser/config.md b/guide/browser/config.md index 7dd724a7..2cae76fe 100644 --- a/guide/browser/config.md +++ b/guide/browser/config.md @@ -191,6 +191,7 @@ The custom provider API is highly experimental and can change between patches. I export interface BrowserProvider { name: string mocker?: BrowserModuleMocker + readonly initScripts?: string[] /** * @experimental opt-in into file parallelisation */ diff --git a/guide/browser/index.md b/guide/browser/index.md index 29e19a2c..bcda934d 100644 --- a/guide/browser/index.md +++ b/guide/browser/index.md @@ -3,9 +3,9 @@ title: Browser Mode | Guide outline: deep --- -# Browser Mode Experimental {#browser-mode} +# Browser Mode {#browser-mode} -This page provides information about the experimental browser mode feature in the Vitest API, which allows you to run your tests in the browser natively, providing access to browser globals like window and document. This feature is currently under development, and APIs may change in the future. +This page provides information about the browser mode feature in the Vitest API, which allows you to run your tests in the browser natively, providing access to browser globals like window and document. ::: tip If you are looking for documentation for `expect`, `vi` or any general API like test projects or type testing, refer to the ["Getting Started" guide](/guide/). diff --git a/guide/browser/why.md b/guide/browser/why.md index 73201e7c..d84c8ffb 100644 --- a/guide/browser/why.md +++ b/guide/browser/why.md @@ -7,7 +7,7 @@ outline: deep ## Motivation -We developed the Vitest browser mode feature to help improve testing workflows and achieve more accurate and reliable test results. This experimental addition to our testing API allows developers to run tests in a native browser environment. In this section, we'll explore the motivations behind this feature and its benefits for testing. +We developed the Vitest browser mode feature to help improve testing workflows and achieve more accurate and reliable test results. This addition to our testing API allows developers to run tests in a native browser environment. In this section, we'll explore the motivations behind this feature and its benefits for testing. ### Different Ways of Testing From 0a6fe8523d9a12af82f7a73c7d6fe4c4d263d243 Mon Sep 17 00:00:00 2001 From: Nozomu Ikuta <16436160+nozomuikuta@users.noreply.github.com> Date: Thu, 16 Oct 2025 22:07:12 +0900 Subject: [PATCH 06/72] docs: add warning about local expect when relying on test state (#8716) --- config/index.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/config/index.md b/config/index.md index 52948261..27bf7416 100644 --- a/config/index.md +++ b/config/index.md @@ -1925,6 +1925,10 @@ Whether to randomize tests. If you want tests to run in parallel, you can enable it with this option, or CLI argument [`--sequence.concurrent`](/guide/cli). +::: warning +When you run tests with `sequence.concurrent` and `expect.requireAssertions` set to `true`, you should use [local expect](/guide/test-context.html#expect) instead of the global one. Otherwise, this may cause false negatives in [some situations (#8469)](https://github.com/vitest-dev/vitest/issues/8469). +::: + #### sequence.seed - **Type**: `number` @@ -2410,6 +2414,10 @@ This only works with Vitest's `expect`. If you use `assert` or `.should` asserti You can change the value of this by calling `vi.setConfig({ expect: { requireAssertions: false } })`. The config will be applied to every subsequent `expect` call until the `vi.resetConfig` is called manually. ::: +::: warning +When you run tests with `sequence.concurrent` and `expect.requireAssertions` set to `true`, you should use [local expect](/guide/test-context.html#expect) instead of the global one. Otherwise, this may cause false negatives in [some situations (#8469)](https://github.com/vitest-dev/vitest/issues/8469). +::: + #### expect.poll Global configuration options for [`expect.poll`](/api/expect#poll). These are the same options you can pass down to `expect.poll(condition, options)`. From 1aadf05fdcd054edc44ca0e105c2b856e6a0d6b2 Mon Sep 17 00:00:00 2001 From: Vladimir Date: Thu, 16 Oct 2025 21:38:53 +0200 Subject: [PATCH 07/72] docs: add mintlify to sponsors (#8725) --- .vitepress/sponsors.ts | 18 +++++++++----- public/mintlify.svg | 53 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 6 deletions(-) create mode 100644 public/mintlify.svg diff --git a/.vitepress/sponsors.ts b/.vitepress/sponsors.ts index 7bf17112..5b74c09e 100644 --- a/.vitepress/sponsors.ts +++ b/.vitepress/sponsors.ts @@ -29,7 +29,13 @@ const vitestSponsors = { img: '/zammad.svg', }, ], - // platinum: [], + platinum: [ + { + name: 'mintlify', + url: 'https://www.mintlify.com/', + img: '/mintlify.svg', + }, + ], gold: [ { name: 'vital', @@ -65,11 +71,11 @@ export const sponsors = [ size: 'big', items: vitestSponsors.special, }, - // { - // tier: 'Platinum Sponsors', - // size: 'big', - // items: vitestSponsors.platinum, - // }, + { + tier: 'Platinum Sponsors', + size: 'big', + items: vitestSponsors.platinum, + }, { tier: 'Gold Sponsors', size: 'medium', diff --git a/public/mintlify.svg b/public/mintlify.svg new file mode 100644 index 00000000..8875780f --- /dev/null +++ b/public/mintlify.svg @@ -0,0 +1,53 @@ + + + + + + + + + + + + + + \ No newline at end of file From 94c9e32e43afdc8e7a73015a1aa6370d95e6bcb5 Mon Sep 17 00:00:00 2001 From: Ahiwe Onyebuchi Valentine Date: Mon, 20 Oct 2025 06:34:27 +0100 Subject: [PATCH 08/72] docs: typo in reporters guide (#8738) --- guide/reporters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/reporters.md b/guide/reporters.md index 5d705046..66b16356 100644 --- a/guide/reporters.md +++ b/guide/reporters.md @@ -141,7 +141,7 @@ Final output after tests have finished: Duration 1.26s (transform 35ms, setup 1ms, collect 90ms, tests 1.47s, environment 0ms, prepare 267ms) ``` -If there is only one test file running, Vitest will output the full test tree of that file, simillar to the [`tree`](#tree-reporter) reporter. The default reporter will also print the test tree if there is at least one failed test in the file. +If there is only one test file running, Vitest will output the full test tree of that file, similar to the [`tree`](#tree-reporter) reporter. The default reporter will also print the test tree if there is at least one failed test in the file. ```bash ✓ __tests__/file1.test.ts (2) 725ms From c54f3dba166cc7792248ce2fe192ae5c1fe719fb Mon Sep 17 00:00:00 2001 From: Ahiwe Onyebuchi Valentine Date: Tue, 21 Oct 2025 17:55:18 +0100 Subject: [PATCH 09/72] docs: correct typos in migration guide (#8750) --- guide/migration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/guide/migration.md b/guide/migration.md index cf8f86aa..bc491e82 100644 --- a/guide/migration.md +++ b/guide/migration.md @@ -73,7 +73,7 @@ See also new guides: ### `spyOn` and `fn` Support Constructors -Previously, if you tried to spy on a constructor with `vi.spyOn`, you would get an error like `Constructor requires 'new'`. Since Vitest 4, all mocks called with a `new` keyword construct the instance instead of callying `mock.apply`. This means that the mock implementation has to use either the `function` or the `class` keyword in these cases: +Previously, if you tried to spy on a constructor with `vi.spyOn`, you would get an error like `Constructor requires 'new'`. Since Vitest 4, all mocks called with a `new` keyword construct the instance instead of calling `mock.apply`. This means that the mock implementation has to use either the `function` or the `class` keyword in these cases: ```ts {12-14,16-20} const cart = { @@ -104,7 +104,7 @@ Note that now if you provide an arrow function, you will get [` is no ### Changes to Mocking -Alongside new features like supporting constructors, Vitest 4 creates mocks differently to address several module mocking issues that we received over the years. This release attemts to make module spies less confusing, especially when working with classes. +Alongside new features like supporting constructors, Vitest 4 creates mocks differently to address several module mocking issues that we received over the years. This release attempts to make module spies less confusing, especially when working with classes. - `vi.fn().getMockName()` now returns `vi.fn()` by default instead of `spy`. This can affect snapshots with mocks - the name will be changed from `[MockFunction spy]` to `[MockFunction]`. Spies created with `vi.spyOn` will keep using the original name by default for better debugging experience - `vi.restoreAllMocks` no longer resets the state of spies and only restores spies created manually with `vi.spyOn`, automocks are no longer affected by this function (this also affects the config option [`restoreMocks`](/config/#restoremocks)). Note that `.mockRestore` will still reset the mock implementation and clear the state From 17a699a2e9efcbc103aebc351dcc5e154b984a4e Mon Sep 17 00:00:00 2001 From: Alexander Karan <47707063+AlexanderKaran@users.noreply.github.com> Date: Wed, 22 Oct 2025 00:56:55 +0800 Subject: [PATCH 10/72] Update comparisons.md with Mocha Comparisons (#8740) --- guide/comparisons.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/guide/comparisons.md b/guide/comparisons.md index 773d8005..53d80d3b 100644 --- a/guide/comparisons.md +++ b/guide/comparisons.md @@ -53,3 +53,24 @@ For transforming your code, uvu relies on require and loader hooks. Vitest uses uvu does not provide an intelligent watch mode to rerun the changed tests, while Vitest gives you amazing DX thanks to the default watch mode using Vite instant Hot Module Reload (HMR). uvu is a fast option for running simple tests, but Vitest can be faster and more reliable for more complex tests and projects. + +## Mocha + +[Mocha](https://mochajs.org) is a test framework running on Node.js and in the browser. Mocha is a popular choice for server-side testing. Mocha is highly configurable and does not include certain features by default. For example, it does not come with an assertion library, with the idea being that Node's built-in assertion runner is good enough for most use cases. Another popular choice for assertions with Mocha is [Chai](https://www.chaijs.com). + +Vitest also provides out-of-the-box setup for a few other features, which take additional configuration or the addition of other libraries in Mocha, for example: + +- Snapshot testing +- TypeScript +- JSX support +- Code Coverage +- Mocking +- Smart watch mode (only re-runs affected tests) + +While Mocha supports Native ESM, it has limitations and configuration constraints. Watch mode does not work with ES Module files, for example. + +Performance-wise, Mocha runs tests serially by default but supports parallel execution with the `--parallel` flag (though some reporters and features don't work in parallel mode). + +If you're already using Vite in your build pipeline, Vitest allows you to reuse the same configuration and plugins for testing, whereas Mocha would require a separate test setup. Vitest provides a Jest-compatible API while also supporting Mocha's familiar `describe`, `it`, and hook syntax, making migration straightforward for most test suites. + +Mocha remains a solid choice for projects that need a minimal, flexible test runner with complete control over their testing stack. However, if you want a modern testing experience with everything included out of the box - especially for Vite-powered applications - Vitest has you covered. From 4d429eb20723bae13c78bc666ccc4f9dfc4ddd53 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ari=20Perkki=C3=B6?= Date: Tue, 21 Oct 2025 20:12:54 +0300 Subject: [PATCH 11/72] feat!: rewrite pools without `tinypool` (#8705) Co-authored-by: Vladimir Sheremet --- .vitepress/components/FeaturesList.vue | 1 - .vitepress/config.ts | 6 +- advanced/pool.md | 129 ++++++++++---- config/index.md | 237 ++----------------------- guide/cli-generated.md | 114 +----------- guide/debugging.md | 14 +- guide/features.md | 5 +- guide/improving-performance.md | 39 ++-- guide/migration.md | 102 ++++++++++- guide/parallelism.md | 2 +- guide/profiling-test-performance.md | 49 +---- guide/recipes.md | 55 ++++++ guide/test-context.md | 2 +- 13 files changed, 320 insertions(+), 435 deletions(-) create mode 100644 guide/recipes.md diff --git a/.vitepress/components/FeaturesList.vue b/.vitepress/components/FeaturesList.vue index a82fdd4f..3c633519 100644 --- a/.vitepress/components/FeaturesList.vue +++ b/.vitepress/components/FeaturesList.vue @@ -10,7 +10,6 @@ Component testing for Vue, React, Svelte, Lit, Marko and more Out-of-the-box TypeScript / JSX support ESM first, top level await - Workers multi-threading via Tinypool Benchmarking support with Tinybench Filtering, timeouts, concurrent for suite and tests Projects support diff --git a/.vitepress/config.ts b/.vitepress/config.ts index 7cc9ba86..f429b7b3 100644 --- a/.vitepress/config.ts +++ b/.vitepress/config.ts @@ -52,7 +52,7 @@ export default ({ mode }: { mode: string }) => { ['link', { rel: 'icon', href: '/favicon.ico', sizes: '48x48' }], ['link', { rel: 'icon', href: '/logo.svg', sizes: 'any', type: 'image/svg+xml' }], ['meta', { name: 'author', content: `${teamMembers.map(c => c.name).join(', ')} and ${vitestName} contributors` }], - ['meta', { name: 'keywords', content: 'vitest, vite, test, coverage, snapshot, react, vue, preact, svelte, solid, lit, marko, ruby, cypress, puppeteer, jsdom, happy-dom, test-runner, jest, typescript, esm, tinypool, tinyspy, node' }], + ['meta', { name: 'keywords', content: 'vitest, vite, test, coverage, snapshot, react, vue, preact, svelte, solid, lit, marko, ruby, cypress, puppeteer, jsdom, happy-dom, test-runner, jest, typescript, esm, tinyspy, node' }], ['meta', { property: 'og:title', content: vitestName }], ['meta', { property: 'og:description', content: vitestDescription }], ['meta', { property: 'og:url', content: ogUrl }], @@ -628,6 +628,10 @@ function guide(): DefaultTheme.SidebarItem[] { }, ], }, + { + text: 'Recipes', + link: '/guide/recipes', + }, ] } diff --git a/advanced/pool.md b/advanced/pool.md index bf63d9af..89fcdad7 100644 --- a/advanced/pool.md +++ b/advanced/pool.md @@ -1,10 +1,10 @@ # Custom Pool ::: warning -This is an advanced and very low-level API. If you just want to [run tests](/guide/), you probably don't need this. It is primarily used by library authors. +This is an advanced, experimental and very low-level API. If you just want to [run tests](/guide/), you probably don't need this. It is primarily used by library authors. ::: -Vitest runs tests in pools. By default, there are several pools: +Vitest runs tests in a pool. By default, there are several pool runners: - `threads` to run tests using `node:worker_threads` (isolation is provided with a new worker context) - `forks` to run tests using `node:child_process` (isolation is provided with a new `child_process.fork` process) @@ -12,21 +12,24 @@ Vitest runs tests in pools. By default, there are several pools: - `browser` to run tests using browser providers - `typescript` to run typechecking on tests -You can provide your own pool by specifying a file path: +::: tip +See [`vitest-pool-example`](https://www.npmjs.com/package/vitest-pool-example) for example of a custom pool runner implementation. +::: + +## Usage + +You can provide your own pool runner by a function that returns `PoolRunnerInitializer`. ```ts [vitest.config.ts] import { defineConfig } from 'vitest/config' +import customPool from './my-custom-pool.ts' export default defineConfig({ test: { // will run every file with a custom pool by default - pool: './my-custom-pool.ts', - // you can provide options using `poolOptions` object - poolOptions: { - myCustomPool: { - customProperty: true, - }, - }, + pool: customPool({ + customProperty: true, + }) }, }) ``` @@ -34,6 +37,8 @@ export default defineConfig({ If you need to run tests in different pools, use the [`projects`](/guide/projects) feature: ```ts [vitest.config.ts] +import customPool from './my-custom-pool.ts' + export default defineConfig({ test: { projects: [ @@ -43,6 +48,14 @@ export default defineConfig({ pool: 'threads', }, }, + { + extends: true, + test: { + pool: customPool({ + customProperty: true, + }) + } + } ], }, }) @@ -50,47 +63,85 @@ export default defineConfig({ ## API -The file specified in `pool` option should export a function (can be async) that accepts `Vitest` interface as its first option. This function needs to return an object matching `ProcessPool` interface: +The `pool` option accepts a `PoolRunnerInitializer` that can be used for custom pool runners. The `name` property should indicate name of the custom pool runner. It should be identical with your worker's `name` property. -```ts -import type { ProcessPool, TestSpecification } from 'vitest/node' +```ts [my-custom-pool.ts] +import type { PoolRunnerInitializer } from 'vitest/node' -export interface ProcessPool { - name: string - runTests: (files: TestSpecification[], invalidates?: string[]) => Promise - collectTests: (files: TestSpecification[], invalidates?: string[]) => Promise - close?: () => Promise +export function customPool(customOptions: CustomOptions): PoolRunnerInitializer { + return { + name: 'custom-pool', + createPoolWorker: options => new CustomPoolWorker(options, customOptions), + } } ``` -The function is called only once (unless the server config was updated), and it's generally a good idea to initialize everything you need for tests inside that function and reuse it when `runTests` is called. +In your `CustomPoolWorker` you need to define all required methods: + +```ts [my-custom-pool.ts] +import type { PoolOptions, PoolWorker, WorkerRequest } from 'vitest/node' -Vitest calls `runTest` when new tests are scheduled to run. It will not call it if `files` is empty. The first argument is an array of [TestSpecifications](/advanced/api/test-specification). Files are sorted using [`sequencer`](/config/#sequence-sequencer) before `runTests` is called. It's possible (but unlikely) to have the same file twice, but it will always have a different project - this is implemented via [`projects`](/guide/projects) configuration. +class CustomPoolWorker implements PoolWorker { + name = 'custom-pool' + private customOptions: CustomOptions -Vitest will wait until `runTests` is executed before finishing a run (i.e., it will emit [`onTestRunEnd`](/advanced/reporters) only after `runTests` is resolved). + constructor(options: PoolOptions, customOptions: CustomOptions) { + this.customOptions = customOptions + } -If you are using a custom pool, you will have to provide test files and their results yourself - you can reference [`vitest.state`](https://github.com/vitest-dev/vitest/blob/main/packages/vitest/src/node/state.ts) for that (most important are `collectFiles` and `updateTasks`). Vitest uses `startTests` function from `@vitest/runner` package to do that. + send(message: WorkerRequest): void { + // Provide way to send your worker a message + } -Vitest will call `collectTests` if `vitest.collect` is called or `vitest list` is invoked via a CLI command. It works the same way as `runTests`, but you don't have to run test callbacks, only report their tasks by calling `vitest.state.collectFiles(files)`. + on(event: string, callback: (arg: any) => void): void { + // Provide way to listen to your workers events, e.g. message, error, exit + } -To communicate between different processes, you can create methods object using `createMethodsRPC` from `vitest/node`, and use any form of communication that you prefer. For example, to use WebSockets with `birpc` you can write something like this: + off(event: string, callback: (arg: any) => void): void { + // Provide way to unsubscribe `on` listeners + } -```ts -import { createBirpc } from 'birpc' -import { parse, stringify } from 'flatted' -import { createMethodsRPC, TestProject } from 'vitest/node' + async start() { + // do something when the worker is started + } -function createRpc(project: TestProject, wss: WebSocketServer) { - return createBirpc( - createMethodsRPC(project), - { - post: msg => wss.send(msg), - on: fn => wss.on('message', fn), - serialize: stringify, - deserialize: parse, - }, - ) + async stop() { + // cleanup the state + } + + deserialize(data) { + return data + } } ``` -You can see a simple example of a pool made from scratch that doesn't run tests but marks them as collected in [pool/custom-pool.ts](https://github.com/vitest-dev/vitest/blob/main/test/cli/fixtures/custom-pool/pool/custom-pool.ts). +Your `CustomPoolRunner` will be controlling how your custom test runner worker life cycles and communication channel works. For example, your `CustomPoolRunner` could launch a `node:worker_threads` `Worker`, and provide communication via `Worker.postMessage` and `parentPort`. + +In your worker file, you can import helper utilities from `vitest/worker`: + +```ts [my-worker.ts] +import { init, runBaseTests } from 'vitest/worker' + +init({ + post: (response) => { + // Provide way to send this message to CustomPoolRunner's onWorker as message event + }, + on: (callback) => { + // Provide a way to listen CustomPoolRunner's "postMessage" calls + }, + off: (callback) => { + // Optional, provide a way to remove listeners added by "on" calls + }, + teardown: () => { + // Optional, provide a way to teardown worker, e.g. unsubscribe all the `on` listeners + }, + serialize: (value) => { + // Optional, provide custom serializer for `post` calls + }, + deserialize: (value) => { + // Optional, provide custom deserializer for `on` callbacks + }, + runTests: state => runBaseTests('run', state), + collectTests: state => runBaseTests('collect', state), +}) +``` diff --git a/config/index.md b/config/index.md index 27bf7416..93475595 100644 --- a/config/index.md +++ b/config/index.md @@ -674,7 +674,7 @@ Custom [reporters](/guide/reporters) for output. Reporters can be [a Reporter in Write test results to a file when the `--reporter=json`, `--reporter=html` or `--reporter=junit` option is also specified. By providing an object instead of a string you can define individual outputs when using multiple reporters. -### pool {#pool} +### pool - **Type:** `'threads' | 'forks' | 'vmThreads' | 'vmForks'` - **Default:** `'forks'` @@ -682,19 +682,19 @@ By providing an object instead of a string you can define individual outputs whe Pool used to run tests in. -#### threads +#### threads -Enable multi-threading using [tinypool](https://github.com/tinylibs/tinypool) (a lightweight fork of [Piscina](https://github.com/piscinajs/piscina)). When using threads you are unable to use process related APIs such as `process.chdir()`. Some libraries written in native languages, such as Prisma, `bcrypt` and `canvas`, have problems when running in multiple threads and run into segfaults. In these cases it is advised to use `forks` pool instead. +Enable multi-threading. When using threads you are unable to use process related APIs such as `process.chdir()`. Some libraries written in native languages, such as Prisma, `bcrypt` and `canvas`, have problems when running in multiple threads and run into segfaults. In these cases it is advised to use `forks` pool instead. -#### forks +#### forks -Similar as `threads` pool but uses `child_process` instead of `worker_threads` via [tinypool](https://github.com/tinylibs/tinypool). Communication between tests and main process is not as fast as with `threads` pool. Process related APIs such as `process.chdir()` are available in `forks` pool. +Similar as `threads` pool but uses `child_process` instead of `worker_threads`. Communication between tests and main process is not as fast as with `threads` pool. Process related APIs such as `process.chdir()` are available in `forks` pool. -#### vmThreads +#### vmThreads Run tests using [VM context](https://nodejs.org/api/vm.html) (inside a sandboxed environment) in a `threads` pool. -This makes tests run faster, but the VM module is unstable when running [ESM code](https://github.com/nodejs/node/issues/37648). Your tests will [leak memory](https://github.com/nodejs/node/issues/33439) - to battle that, consider manually editing [`poolOptions.vmThreads.memoryLimit`](#pooloptions-vmthreads-memorylimit) value. +This makes tests run faster, but the VM module is unstable when running [ESM code](https://github.com/nodejs/node/issues/37648). Your tests will [leak memory](https://github.com/nodejs/node/issues/33439) - to battle that, consider manually editing [`vmMemoryLimit`](#vmMemorylimit) value. ::: warning Running code in a sandbox has some advantages (faster tests), but also comes with a number of disadvantages. @@ -716,166 +716,28 @@ catch (err) { Please, be aware of these issues when using this option. Vitest team cannot fix any of the issues on our side. ::: -#### vmForks +#### vmForks -Similar as `vmThreads` pool but uses `child_process` instead of `worker_threads` via [tinypool](https://github.com/tinylibs/tinypool). Communication between tests and the main process is not as fast as with `vmThreads` pool. Process related APIs such as `process.chdir()` are available in `vmForks` pool. Please be aware that this pool has the same pitfalls listed in `vmThreads`. +Similar as `vmThreads` pool but uses `child_process` instead of `worker_threads`. Communication between tests and the main process is not as fast as with `vmThreads` pool. Process related APIs such as `process.chdir()` are available in `vmForks` pool. Please be aware that this pool has the same pitfalls listed in `vmThreads`. -### poolOptions {#pooloptions} - -- **Type:** `Record<'threads' | 'forks' | 'vmThreads' | 'vmForks', {}>` -- **Default:** `{}` - -#### poolOptions.threads - -Options for `threads` pool. - -```ts -import { defineConfig } from 'vitest/config' - -export default defineConfig({ - test: { - poolOptions: { - threads: { - // Threads related options here - } - } - } -}) -``` - -##### poolOptions.threads.maxThreads - -- **Type:** `number | string` -- **Default:** _available CPUs_ - -Maximum number or percentage of threads. You can also use `VITEST_MAX_THREADS` environment variable. - -##### poolOptions.threads.singleThread - -- **Type:** `boolean` -- **Default:** `false` - -Run all tests with the same environment inside a single worker thread. This will disable built-in module isolation (your source code or [inlined](#server-deps-inline) code will still be reevaluated for each test), but can improve test performance. - -:::warning -Even though this option will force tests to run one after another, this option is different from Jest's `--runInBand`. Vitest uses workers not only for running tests in parallel, but also to provide isolation. By disabling this option, your tests will run sequentially, but in the same global context, so you must provide isolation yourself. - -This might cause all sorts of issues, if you are relying on global state (frontend frameworks usually do) or your code relies on environment to be defined separately for each test. But can be a speed boost for your tests (up to 3 times faster), that don't necessarily rely on global state or can easily bypass that. -::: - -##### poolOptions.threads.useAtomics - -- **Type:** `boolean` -- **Default:** `false` - -Use Atomics to synchronize threads. - -This can improve performance in some cases, but might cause segfault in older Node versions. - -##### poolOptions.threads.isolate - -- **Type:** `boolean` -- **Default:** `true` - -Isolate environment for each test file. - -##### poolOptions.threads.execArgv - -- **Type:** `string[]` -- **Default:** `[]` - -Pass additional arguments to `node` in the threads. See [Command-line API | Node.js](https://nodejs.org/docs/latest/api/cli.html) for more information. - -:::warning -Be careful when using, it as some options may crash worker, e.g. --prof, --title. See https://github.com/nodejs/node/issues/41103. -::: - -#### poolOptions.forks - -Options for `forks` pool. - -```ts -import { defineConfig } from 'vitest/config' - -export default defineConfig({ - test: { - poolOptions: { - forks: { - // Forks related options here - } - } - } -}) -``` - -##### poolOptions.forks.maxForks - -- **Type:** `number | string` -- **Default:** _available CPUs_ - -Maximum number or percentage of forks. You can also use `VITEST_MAX_FORKS` environment variable. - -##### poolOptions.forks.isolate - -- **Type:** `boolean` -- **Default:** `true` - -Isolate environment for each test file. - -##### poolOptions.forks.singleFork - -- **Type:** `boolean` -- **Default:** `false` - -Run all tests with the same environment inside a single child process. This will disable built-in module isolation (your source code or [inlined](#server-deps-inline) code will still be reevaluated for each test), but can improve test performance. - -:::warning -Even though this option will force tests to run one after another, this option is different from Jest's `--runInBand`. Vitest uses child processes not only for running tests in parallel, but also to provide isolation. By disabling this option, your tests will run sequentially, but in the same global context, so you must provide isolation yourself. - -This might cause all sorts of issues, if you are relying on global state (frontend frameworks usually do) or your code relies on environment to be defined separately for each test. But can be a speed boost for your tests (up to 3 times faster), that don't necessarily rely on global state or can easily bypass that. -::: - -##### poolOptions.forks.execArgv +### execArgv - **Type:** `string[]` - **Default:** `[]` -Pass additional arguments to `node` process in the child processes. See [Command-line API | Node.js](https://nodejs.org/docs/latest/api/cli.html) for more information. +Pass additional arguments to `node` in the runner worker. See [Command-line API | Node.js](https://nodejs.org/docs/latest/api/cli.html) for more information. :::warning Be careful when using, it as some options may crash worker, e.g. --prof, --title. See https://github.com/nodejs/node/issues/41103. ::: -#### poolOptions.vmThreads - -Options for `vmThreads` pool. - -```ts -import { defineConfig } from 'vitest/config' - -export default defineConfig({ - test: { - poolOptions: { - vmThreads: { - // VM threads related options here - } - } - } -}) -``` - -##### poolOptions.vmThreads.maxThreads - -- **Type:** `number | string` -- **Default:** _available CPUs_ - -Maximum number or percentage of threads. You can also use `VITEST_MAX_THREADS` environment variable. - -##### poolOptions.vmThreads.memoryLimit +### vmMemoryLimit - **Type:** `string | number` - **Default:** `1 / CPU Cores` +This option affects only `vmForks` and `vmThreads` pools. + Specifies the memory limit for workers before they are recycled. This value heavily depends on your environment, so it's better to specify it manually instead of relying on the default. ::: tip @@ -900,70 +762,7 @@ The limit can be specified in a number of different ways and whatever the result Percentage based memory limit [does not work on Linux CircleCI](https://github.com/jestjs/jest/issues/11956#issuecomment-1212925677) workers due to incorrect system memory being reported. ::: -##### poolOptions.vmThreads.useAtomics - -- **Type:** `boolean` -- **Default:** `false` - -Use Atomics to synchronize threads. - -This can improve performance in some cases, but might cause segfault in older Node versions. - -##### poolOptions.vmThreads.execArgv - -- **Type:** `string[]` -- **Default:** `[]` - -Pass additional arguments to `node` process in the VM context. See [Command-line API | Node.js](https://nodejs.org/docs/latest/api/cli.html) for more information. - -:::warning -Be careful when using, it as some options may crash worker, e.g. --prof, --title. See https://github.com/nodejs/node/issues/41103. -::: - -#### poolOptions.vmForks - -Options for `vmForks` pool. - -```ts -import { defineConfig } from 'vitest/config' - -export default defineConfig({ - test: { - poolOptions: { - vmForks: { - // VM forks related options here - } - } - } -}) -``` - -##### poolOptions.vmForks.maxForks - -- **Type:** `number | string` -- **Default:** _available CPUs_ - -Maximum number or percentage of forks. You can also use `VITEST_MAX_FORKS` environment variable. - -##### poolOptions.vmForks.memoryLimit - -- **Type:** `string | number` -- **Default:** `1 / CPU Cores` - -Specifies the memory limit for workers before they are recycled. This value heavily depends on your environment, so it's better to specify it manually instead of relying on the default. How the value is calculated is described in [`poolOptions.vmThreads.memoryLimit`](#pooloptions-vmthreads-memorylimit) - -##### poolOptions.vmForks.execArgv - -- **Type:** `string[]` -- **Default:** `[]` - -Pass additional arguments to `node` process in the VM context. See [Command-line API | Node.js](https://nodejs.org/docs/latest/api/cli.html) for more information. - -:::warning -Be careful when using, it as some options may crash worker, e.g. --prof, --title. See https://github.com/nodejs/node/issues/41103. -::: - -### fileParallelism {#fileparallelism} +### fileParallelism - **Type:** `boolean` - **Default:** `true` @@ -975,11 +774,11 @@ Should all test files run in parallel. Setting this to `false` will override `ma This option doesn't affect tests running in the same file. If you want to run those in parallel, use `concurrent` option on [describe](/api/#describe-concurrent) or via [a config](#sequence-concurrent). ::: -### maxWorkers {#maxworkers} +### maxWorkers - **Type:** `number | string` -Maximum number or percentage of workers to run tests in. `poolOptions.{threads,vmThreads}.maxThreads`/`poolOptions.forks.maxForks` has higher priority. +Maximum number or percentage of workers to run tests in. ### testTimeout @@ -2342,7 +2141,7 @@ Run tests in an isolated environment. This option has no effect on `vmThreads` a Disabling this option might [improve performance](/guide/improving-performance) if your code doesn't rely on side effects (which is usually true for projects with `node` environment). ::: tip -You can disable isolation for specific pools by using [`poolOptions`](#pooloptions) property. +You can disable isolation for specific test files by using Vitest workspaces and disabling isolation per project. ::: ### includeTaskLocation {#includeTaskLocation} diff --git a/guide/cli-generated.md b/guide/cli-generated.md index 12430a3b..72e916ab 100644 --- a/guide/cli-generated.md +++ b/guide/cli-generated.md @@ -390,117 +390,19 @@ Enable trace view mode. Supported: "on", "off", "on-first-retry", "on-all-retrie Specify pool, if not running in the browser (default: `forks`) -### poolOptions.threads.isolate +### execArgv -- **CLI:** `--poolOptions.threads.isolate` -- **Config:** [poolOptions.threads.isolate](/config/#pooloptions-threads-isolate) +- **CLI:** `--execArgv