Skip to content

Commit 2afaf8e

Browse files
jerryjrchenjerryjrchen
committed
Interoperability between Swift Testing and XCTest
1 parent 79b68f9 commit 2afaf8e

File tree

1 file changed

+274
-0
lines changed

1 file changed

+274
-0
lines changed

‎proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md‎

Lines changed: 274 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,274 @@
1+
# Interoperability between Swift Testing and XCTest
2+
3+
- Proposal: [ST-NNNN](NNNN-xctest-interoperability.md)
4+
- Authors: [Jerry Chen](https://github.com/jerryjrchen)
5+
- Review Manager: TBD
6+
- Status: **Awaiting implementation**
7+
- Implementation: [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/pull/NNNNN)
8+
- Review: ([pitch](https://forums.swift.org/...))
9+
10+
## Introduction
11+
12+
Many projects want to migrate from XCTest to Swift Testing, and may be in an
13+
intermediate state where test helpers written using XCTest API are called from
14+
Swift Testing. Today, the Swift Testing and XCTest libraries stand mostly
15+
independently, which means an `XCTAssert` failure in a Swift Testing test is
16+
silently ignored. To address this, we formally declare a set of interoperability
17+
principles and propose updates to specific APIs that will enable users to
18+
migrate with confidence.
19+
20+
## Motivation
21+
22+
Unfortunately, mixing an API call from one framework with a test from the other
23+
framework may not work as expected. As a more concrete example, if you take an
24+
existing test helper function written for XCTest and call it in a Swift Testing
25+
test, it won't report the assertion failure:
26+
27+
```swift
28+
func assertUnique(_ elements: [Int]) {
29+
if Set(elements).count != elements.count {
30+
XCTFail("\(elements) has non unique elements")
31+
}
32+
}
33+
34+
// XCTest
35+
36+
class FooTests: XCTestCase {
37+
func testDups() {
38+
assertUnique([1, 2, 1]) // Fails as expected
39+
}
40+
}
41+
42+
// Swift Testing
43+
44+
@Test func `Duplicate elements`() {
45+
assertUnique([1, 2, 1]) // Passes? Oh no!
46+
}
47+
```
48+
49+
Generally, we get into trouble today when ALL the following conditions are met:
50+
51+
- You call XCTest API in a Swift Testing test, or call Swift Testing API in a
52+
XCTest test
53+
- The API doesn't function as expected in some or all cases, and
54+
- You get no notice at build or runtime about the malfunction
55+
56+
Example: calling `XCTAssertEqual(x, y)` in a Swift Testing test: if x does not
57+
equal y, it should report a failure but does nothing instead.
58+
59+
For the remainder of this proposal, we’ll describe tests which exhibit this
60+
problem as **lossy without interop**.
61+
62+
If you've switched completely to Swift Testing and don't expect to use XCTest in
63+
the future, this proposal includes a mechanism to **prevent you from
64+
inadvertently introducing XCTest APIs to your project, including via a testing
65+
library.**
66+
67+
## Proposed solution
68+
69+
At a high level, we propose the following **changes for APIs which are lossy
70+
without interop**:
71+
72+
- **XCTest API called in Swift Testing will work as expected.** In addition,
73+
runtime diagnostics will be included to highlight opportunities to adopt newer
74+
Swift Testing equivalent APIs.
75+
76+
- Conversely, **Swift Testing API called in XCTest will also work as expected.**
77+
You should always feel empowered to choose Swift Testing when writing new
78+
tests or test helpers, as it will work properly in both types of tests.
79+
80+
We don't propose supporting interoperability for APIs without risk of data loss,
81+
because they naturally have high visibility. For example, using `throw XCTSkip`
82+
in a Swift Testing test results in a test failure rather than a test skip,
83+
providing a clear indication that migration is needed.
84+
85+
## Detailed design
86+
87+
### Highlight usage of XCTest APIs in Swift Testing tests
88+
89+
We propose supporting the following XCTest APIs in Swift Testing:
90+
91+
- Assertions: `XCTAssert*` and [unconditional failure][] `XCTFail`
92+
- [Expected failures][], such as `XCTExpectFailure`: marking a Swift Testing
93+
issue in this way will generate a runtime issue.
94+
- `XCTAttachment`
95+
- [Issue handling traits][]: we will make our best effort to translate issues
96+
from XCTest to Swift Testing. Note that there are certain issue kinds that are
97+
new to Swift Testing and not expressible from XCTest.
98+
99+
Note that no changes are proposed for the `XCTSkip` API, because they already
100+
feature prominently as a test failure to be corrected when thrown in Swift
101+
Testing.
102+
103+
We also propose highlighting usage of above XCTest APIs in Swift Testing:
104+
105+
- **Report [runtime warning issues][]** for XCTest API usage in Swift Testing.
106+
This **applies to assertion successes AND failures**! We want to make sure you
107+
can identify opportunities to modernise even if your tests currently all pass.
108+
109+
- Opt-in **strict interop mode**, which will trigger a crash instead.
110+
111+
Here are some concrete examples:
112+
113+
| When running a Swift Testing test... | Current | Proposed (default) | Proposed (strict) |
114+
| ------------------------------------ | --------------- | -------------------------------------------- | ----------------- |
115+
| `XCTAssert` failure is a ... | ‼️ No-op | ❌ Test Failure and ⚠️ Runtime Warning Issue | 💥 Crash |
116+
| `XCTAssert` success is a ... | No-op | ⚠️ Runtime Warning Issue | 💥 Crash |
117+
| `throw XCTSkip` is a ... | ❌ Test Failure | ❌ Test Failure | ❌ Test Failure |
118+
119+
### Limited support for Swift Testing APIs in XCTest
120+
121+
We propose supporting the following Swift Testing APIs in XCTest:
122+
123+
- `#expect` and `#require`
124+
- Includes [exit testing][]
125+
- `withKnownIssue`: when this suppresses `XCTAssert` failures, it will still
126+
show a runtime warning issue.
127+
- Attachments
128+
- [Test cancellation][] (links to pitch)
129+
130+
We think developers will find utility in using Swift Testing APIs in XCTest. For
131+
example, you can replace `XCTAssert` with `#expect` in your XCTest tests and
132+
immediately get the benefits of the newer, more ergonomic API. In the meantime,
133+
you can incrementally migrate the rest of your test infrastructure to use Swift
134+
Testing at your own pace.
135+
136+
Present and future Swift Testing APIs will be supported in XCTest if the
137+
XCTest API _already_ provides similar functionality.
138+
139+
- For example, we plan on supporting the proposed Swift Testing [test
140+
cancellation][] feature in XCTest since it is analogous to `XCTSkip`
141+
142+
- On the other hand, [Traits][] are a powerful Swift Testing feature, and
143+
include the ability to [add tags][tags] to organise tests. Even though XCTest
144+
does not interact with tags, this is beyond the scope of interoperability
145+
because XCTest doesn't have existing “tag-like” behaviour to map onto.
146+
147+
Here are some concrete examples:
148+
149+
| When running a XCTest test... | Current | Proposed (default) | Proposed (strict) |
150+
| -------------------------------------------- | --------------- | ------------------------ | ----------------- |
151+
| `#expect` failure is a ... | ‼️ No-op | ❌ Test Failure | ❌ Test Failure |
152+
| `#expect` success is a ... | No-op | No-op | No-op |
153+
| `withKnownIssue` wrapping `XCTFail` is a ... | ❌ Test Failure | ⚠️ Runtime Warning Issue | 💥 Crash |
154+
155+
### Interoperability Modes
156+
157+
The default interoperability surfaces test failures that were previously
158+
ignored. We include two more permissible interoperability modes to avoid
159+
breaking projects that are dependent on this pre-interop behaviour.
160+
161+
- **Warning-only**: This is for projects which do not want to see new test
162+
failures surfaced due to interoperability.
163+
164+
- **None**: Some projects may additionally have issue handling trait that
165+
promote warnings to errors, which means that warning-only mode could still
166+
cause test failures.
167+
168+
For projects that want to bolster their Swift Testing adoption, there is also an
169+
opt-in strict interop mode.
170+
171+
- **Strict**: Warning issues included in the default mode can be easily
172+
overlooked, especially in CI. The strict mode guarantees that no XCTest API
173+
usage occurs when running Swift Testing tests by turning those warnings into a
174+
runtime crash.
175+
176+
Configure the interoperability mode when running tests using the
177+
`SWT_XCTEST_INTEROP_MODE` environment variable:
178+
179+
| Interop Mode | Issue behaviour across framework boundary | `SWT_XCTEST_INTEROP_MODE` |
180+
| ------------ | -------------------------------------------- | ------------------------------------------- |
181+
| Off | ‼️ No-op (status quo) | `off` |
182+
| Warning-only | ⚠️ Runtime Warning Issue | `warning` |
183+
| Default | ❌ Test Failure and ⚠️ Runtime Warning Issue | `default`, or empty value, or invalid value |
184+
| Strict | 💥 Crash | `strict` |
185+
186+
### Phased Rollout
187+
188+
When interoperability is first available, "default" will be the interop mode
189+
enabled for new projects. In a future release, "strict" will become the default
190+
interop mode.
191+
192+
## Source compatibility
193+
194+
As the main goal of interoperability is to change behaviour, this proposal will
195+
lead to situations where previously "passing" test code now starts showing
196+
failures. We believe this should be a net positive if it can highlight actual
197+
bugs you would have missed previously.
198+
199+
You can use `SWT_XCTEST_INTEROP_MODE=off` in the short-term to revert back to
200+
the current behaviour. Refer to the "Interoperability Modes" section for a full list
201+
of options.
202+
203+
## Integration with supporting tools
204+
205+
Interoperability will be first available in future toolchain version,
206+
hypothetically named `6.X`, where default interop mode will be enabled for
207+
projects. After that, a `6.Y` release would make strict interop mode the
208+
default.
209+
210+
- Swift Package Manager projects: `swift-tools-version` declared in
211+
Package.swift will be used to determine interop mode, regardless of the
212+
toolchain used to run tests.
213+
214+
- Xcode projects: Installed toolchain version will be used to determine interop
215+
mode.
216+
217+
- Any project can use `SWT_XCTEST_INTEROP_MODE` to override interop mode at
218+
runtime, provided they are on toolchain version `6.X` or newer
219+
220+
## Future directions
221+
222+
There's still more we can do to make it easier to migrate from XCTest to Swift
223+
Testing:
224+
225+
- Provide fixups at compile-time to replace usage of XCTest API with the
226+
corresponding Swift Testing API, e.g. replace `XCTAssert` with `#expect`.
227+
However, this would require introspection of the test body to look for XCTest
228+
API usage, which would be challenging to do completely and find usages of this
229+
API within helper methods.
230+
231+
- After new API added to SWT in future, will need to evaluate for
232+
interoperability with XCTest until strict mode is the default. "strict" is
233+
kind of saying "from this point forward, no new interop will be added" for new
234+
SWT features.
235+
236+
## Alternatives considered
237+
238+
### Arguments against interoperability
239+
240+
Frameworks that operate independently generally have no expectation of
241+
interoperability, and XCTest and Swift Testing are currently set up this way.
242+
Indeed, the end goal is not necessarily to support using XCTest APIs
243+
indefinitely within Swift Testing. Adding interoperability can be interpreted as
244+
going against that goal, and can enable users to delay migrating completely to
245+
Swift Testing.
246+
247+
However, we believe the benefits of fixing lossy without interop APIs and
248+
helping users catch more bugs are too important to pass up. We've also included
249+
a plan to increase the strictness of the interoperability mode over time, which
250+
should make it clear that this is not intended to be a permanent measure.
251+
252+
### Strict interop mode as the default
253+
254+
We believe that for projects using only Swift Testing, strict interop mode is
255+
the best choice. Making this the default would also send the clearest signal
256+
that we want users to migrate to Swift Testing.
257+
258+
However, we are especially sensitive to use cases that depend upon the currently
259+
lossy without interop APIs, and decided to prioritise the current default as a
260+
good balance between notifying users yet not breaking existing test suites.
261+
262+
## Acknowledgments
263+
264+
Thanks to Stuart Montgomery, Jonathan Grynspan, and Brian Croom for feedback on
265+
the proposal.
266+
267+
[unconditional failure]: https://developer.apple.com/documentation/xctest/unconditional-test-failures
268+
[runtime warning issues]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md
269+
[expected failures]: https://developer.apple.com/documentation/xctest/expected-failures
270+
[issue handling traits]: https://developer.apple.com/documentation/testing/issuehandlingtrait
271+
[test cancellation]: https://forums.swift.org/t/pitch-test-cancellation/81847
272+
[traits]: https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/traits
273+
[tags]: https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/addingtags
274+
[exit testing]: https://developer.apple.com/documentation/testing/exit-testing

0 commit comments

Comments
 (0)