Skip to content

Commit 935d6a7

Browse files
authored
Add discussion about parallelization/concurrency and tools integration to vision document draft (#425)
In preparation for submitting the swift-testing vision document for wider review (as mentioned in this recent [Forum post](https://forums.swift.org/t/an-update-on-swift-testing-progress-and-stable-release-plans/71455)), this expands the document to include discussion about two important topics: - Parallelization and concurrency - Tools integration These topics are mentioned briefly in the document already, but these additions give a lot more detail about their design considerations. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.
1 parent 33ffe33 commit 935d6a7

File tree

1 file changed

+71
-0
lines changed

1 file changed

+71
-0
lines changed

Documentation/Vision.md

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,77 @@ re-running them requires some way to uniquely represent those arguments, which
409409
overlaps with some of the considerations discussed in
410410
[Test identity](#test-identity).
411411

412+
### Parallelization and concurrency
413+
414+
A modern testing system should make efficient use of the machine it runs on.
415+
Many tests can safely run in parallel, and the testing system should encourage
416+
this by enabling per-test parallelization by default. In addition to faster
417+
results and shorter iteration time, running tests in parallel can help identify
418+
bugs due to hidden dependencies between tests and encourage better state
419+
isolation.
420+
421+
However, some tests may need to disable parallelization and run one at a time.
422+
It should be possible to opt-out, and this may be especially useful while
423+
migrating from older testing systems which don't support parallelization.
424+
Although opting-out of this behavior should be possible, it should be narrowly
425+
scoped to not sacrifice other tests' ability to run in parallel.
426+
427+
In addition to running tests in parallel relative to each other, tests
428+
themselves should seamlessly support Swift's concurrency features. In particular,
429+
this means:
430+
431+
* Tests should be able to use async/await whenever necessary.
432+
* Tests should support isolation to a global actor such as `@MainActor`, but be
433+
nonisolated by default. (Isolation by default would undermine the goal of
434+
running tests in parallel by default.)
435+
* Values passed as arguments to parameterized tests should be `Sendable`, since
436+
they may cross between isolation domains within the testing system's execution
437+
machinery.
438+
* Types containing tests functions and their stored properties need not be
439+
`Sendable`, since they are only used from a single isolation domain while each
440+
test function is run.
441+
442+
### Tools integration
443+
444+
A well-rounded testing library should be integrated with popular tools used by
445+
the community. This integration should include some essential functionality such
446+
as:
447+
448+
* Building tests into products which can be be executed.
449+
* Running all built tests.
450+
* Showing per-test results, including details of each individual failure during
451+
a test.
452+
* Showing an aggregate summary of a test run, including failure statistics.
453+
454+
Beyond the essentials, tools may offer other useful features, such as:
455+
456+
* Filtering tests by name, specific traits (e.g. custom tags), or other criteria.
457+
* Outputting results to a standard format such as JUnit XML for importing into
458+
other tools.
459+
* Controlling runtime options such as whether parallel execution is enabled,
460+
whether failed tests should be reattempted, etc.
461+
* Relaunching a test executable after an unexpected crash.
462+
* Reporting "live" progress as each test finishes.
463+
464+
In order to deliver the functionality above, a testing system needs to track
465+
significant events which occur during execution, such as when each test starts
466+
and finishes or encounters a failure. There needs to be a structured,
467+
machine-readable representation of each event, and to provide granular progress
468+
updates, there should be an option for tools to observe a stream of such events.
469+
Any serialization formats involved should be versioned and provide some level of
470+
stability over time, to ensure compatibility with tools which may evolve on
471+
differing release schedules.
472+
473+
Some of the features outlined above are much easier to achieve if there are at
474+
least two processes involved: One which executes the tests themselves (which may
475+
be unstable and terminate unexpectedly), and another which launches and monitors
476+
the first process. In this document, we'll refer to this second process as the
477+
**harness**. A two-process architecture involving a supervisory harness helps
478+
enable functionality such as relaunching after a crash, live progress reporting,
479+
or outputting results to a standard format. A modern testing library should
480+
provide any necessary APIs and runtime support to facilitate integration with a
481+
harness.
482+
412483
## Today’s solution: XCTest
413484

414485
[XCTest](https://developer.apple.com/documentation/xctest) has historically

0 commit comments

Comments
 (0)