|
1 | 1 | # Concurrency |
| 2 | + |
| 3 | +Vitest runs each test case in a test file _sequentially_. We can speed up our test time by running tests _concurrently_ instead. |
| 4 | + |
| 5 | +```diff filename=your.test.ts remove=1 add=2 |
| 6 | +test(`${i}`, () => {}) |
| 7 | +test.concurrent(`${i}`, () => {}) |
| 8 | +// ππππππ |
| 9 | +``` |
| 10 | + |
| 11 | +By making our test cases concurrent, we can switch from a test waterfall to a flat test execution: |
| 12 | + |
| 13 | + |
| 14 | + |
| 15 | +Now that our test run at the same time, it is absolutely crucial we provision proper _test isolation_. We don't want tests to be stepping on each other's toes and becoming flaky as a result. This often comes down to eliminating or replacing any shared (or global) state in test cases. |
| 16 | + |
| 17 | +For example, ... |
| 18 | + |
| 19 | +```ts diff remove=1 add=3 |
| 20 | +test.concurrent(`${i}`, async () => { |
| 21 | +// ππππππ |
| 22 | +test.concurrent(`${i}`, async ({ expect }) => { |
| 23 | + await setTimeout(25) |
| 24 | + expect(i).toBe(i) |
| 25 | +}) |
| 26 | +``` |
| 27 | +
|
| 28 | +With these changes, our test time goes from 2.8 seconds to 271 _milliseconds_! π€― |
| 29 | +
|
| 30 | +```bash remove=7 add=8 |
| 31 | + β src/two.test.ts (100 tests) 56ms |
| 32 | + β src/one.test.ts (100 tests) 56ms |
| 33 | + |
| 34 | + Test Files 2 passed (2) |
| 35 | + Tests 200 passed (200) |
| 36 | + Start at 10:46:01 |
| 37 | + Duration 2.84s |
| 38 | + Duration 271ms |
| 39 | +``` |
| 40 | +
|
| 41 | +## Configuring concurrency |
| 42 | +
|
| 43 | +By default, Vitest runs **<u>5</u>** concurrent test cases at the same time. You can further increase that number based on your machine's capabilities using the [`test.maxConcurrency`](https://vitest.dev/config/#maxconcurrency) property in your Vitest configuration: |
| 44 | +
|
| 45 | +```ts filename=vitest.config.ts add=4 |
| 46 | +export default defineConfig({ |
| 47 | + test: { |
| 48 | + globals: true, |
| 49 | + maxConcurrency: 50, |
| 50 | + }, |
| 51 | +}) |
| 52 | +``` |
| 53 | +
|
| 54 | +> π¦ Bigger doesn't necessarily mean better with concurrency. There is a physical limit to any concurrency dictated by your hardware. If you set a `maxConcurrency` value higher than that limit, concurrent tests will be _queued_ until they have a slot to run. |
| 55 | +
|
| 56 | +By fine-tunning `maxConcurrency`, we are able to improve the test performance even further to the whooping 123ms! |
| 57 | +
|
| 58 | +```bash remove=1 add=2 |
| 59 | + Duration 271ms |
| 60 | + Duration 123ms |
| 61 | +``` |
| 62 | +
|
| 63 | +## Dangers & reliability |
| 64 | +
|
| 65 | +While concurrency may improve performance, it can also make your tests _flaky_. Keep in mind that the main price you pay for concurrency is _writing isolated tests_. |
| 66 | +
|
| 67 | +Here's a few guidelines on how to keep your tests concurrency-friendly: |
| 68 | +
|
| 69 | +- **Do not rely on _shared state_ of any kind**. Never have multiple test modify the same data (even the `expect` function can become a shared state!). In practice, you achieve this through actions like: |
| 70 | + - Striving toward self-contained tests (never have one test rely on the result of another); |
| 71 | + - Keeping the test setup next to the test itself; |
| 72 | + - Binding mocks (e.g. databases or network) to the test. |
| 73 | +- **Isolate side effects**. If your test absolutely must perform a side effect, like writing to a file, guarantee that those side effects are isolated and bound to the test (e.g. create a temporary file accessed only by this particular test). |
| 74 | +- **Abolish hard-coded values**. If two tests try to establish a mock server at the same port, one is destined to fail. Once again, procude test-specific fixtures (e.g. spawn a mock server at port `0` to get a random available port number). |
| 75 | +
|
| 76 | +It is worth mentioning that due to these considerations, not all test cases can be flipped to concurrent and call it a day. Concurrency can, however, be a useful factor to stress your tests and highlight the shortcomings in their design. You can then address those shortcomings in planned, incremental refactors to benefit from concurrency (among other things) once your tests are properly isolated. |
| 77 | +
|
| 78 | +<callout-warning> |
| 79 | + Concurrency is not a solution to slow test but a change in how your tests run. |
| 80 | +</callout-warning> |
| 81 | +
|
| 82 | +## When to use concurrent tests? |
| 83 | +
|
| 84 | +Given a hefty list of considerations surrounding the dangers of concurrency, it begs a question: _When should I use it, then?_ It is hard to give a definitive answer to this question. Ultimately, you remain the best judge of your test suite. That being said, I will try to list a few general recommendations on using concurrency below. |
| 85 | +
|
| 86 | +1. You should be fairly safe to use `test.concurrent` in your _unit tests_ (with variable performance impact, mind you). Those are isolated by design, and you should have little trouble enabling concurrency for them. Even if you discover problematic tests (congratulations!), refactoring them should prove a less time-consuming effort due to the nature of unit tests. |
| 87 | +1. Introduce concurrency _incrementally_, starting from the slowest test cases. The `test.concurrent` API literally allows you to toggle concurrency on a test basis. Use that! Take two, three, five slowest tests in your project and make them concurrent. Incremental approach here also helps with any potential refactors. |
| 88 | +1. Consider other techniques as well. Concurrent test runs isn't the only way to speed up your tests. You can also use things like disabling test isolation or sharding, both of which we will explore next in this block. |
| 89 | +
|
| 90 | +## Related materials |
| 91 | +
|
| 92 | +Take a look at these resources to learn more about concurrency in Vitest: |
| 93 | +
|
| 94 | +- [Running tests concurrently](https://vitest.dev/guide/features.html#running-tests-concurrently) |
| 95 | +- [`test.concurrent`](https://vitest.dev/api/#test-concurrent) |
0 commit comments