From 2afaf8e49be940a150317f405960b55f5845bb69 Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Mon, 29 Sep 2025 16:55:00 -0700 Subject: [PATCH 01/12] Interoperability between Swift Testing and XCTest --- ...ift-testing-and-xctest-interoperability.md | 274 ++++++++++++++++++ 1 file changed, 274 insertions(+) create mode 100644 proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md diff --git a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md new file mode 100644 index 0000000000..cd8addb863 --- /dev/null +++ b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md @@ -0,0 +1,274 @@ +# Interoperability between Swift Testing and XCTest + +- Proposal: [ST-NNNN](NNNN-xctest-interoperability.md) +- Authors: [Jerry Chen](https://github.com/jerryjrchen) +- Review Manager: TBD +- Status: **Awaiting implementation** +- Implementation: [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/pull/NNNNN) +- Review: ([pitch](https://forums.swift.org/...)) + +## Introduction + +Many projects want to migrate from XCTest to Swift Testing, and may be in an +intermediate state where test helpers written using XCTest API are called from +Swift Testing. Today, the Swift Testing and XCTest libraries stand mostly +independently, which means an `XCTAssert` failure in a Swift Testing test is +silently ignored. To address this, we formally declare a set of interoperability +principles and propose updates to specific APIs that will enable users to +migrate with confidence. + +## Motivation + +Unfortunately, mixing an API call from one framework with a test from the other +framework may not work as expected. As a more concrete example, if you take an +existing test helper function written for XCTest and call it in a Swift Testing +test, it won't report the assertion failure: + +```swift +func assertUnique(_ elements: [Int]) { + if Set(elements).count != elements.count { + XCTFail("\(elements) has non unique elements") + } +} + +// XCTest + +class FooTests: XCTestCase { + func testDups() { + assertUnique([1, 2, 1]) // Fails as expected + } +} + +// Swift Testing + +@Test func `Duplicate elements`() { + assertUnique([1, 2, 1]) // Passes? Oh no! +} +``` + +Generally, we get into trouble today when ALL the following conditions are met: + +- You call XCTest API in a Swift Testing test, or call Swift Testing API in a + XCTest test +- The API doesn't function as expected in some or all cases, and +- You get no notice at build or runtime about the malfunction + +Example: calling `XCTAssertEqual(x, y)` in a Swift Testing test: if x does not +equal y, it should report a failure but does nothing instead. + +For the remainder of this proposal, we’ll describe tests which exhibit this +problem as **lossy without interop**. + +If you've switched completely to Swift Testing and don't expect to use XCTest in +the future, this proposal includes a mechanism to **prevent you from +inadvertently introducing XCTest APIs to your project, including via a testing +library.** + +## Proposed solution + +At a high level, we propose the following **changes for APIs which are lossy +without interop**: + +- **XCTest API called in Swift Testing will work as expected.** In addition, + runtime diagnostics will be included to highlight opportunities to adopt newer + Swift Testing equivalent APIs. + +- Conversely, **Swift Testing API called in XCTest will also work as expected.** + You should always feel empowered to choose Swift Testing when writing new + tests or test helpers, as it will work properly in both types of tests. + +We don't propose supporting interoperability for APIs without risk of data loss, +because they naturally have high visibility. For example, using `throw XCTSkip` +in a Swift Testing test results in a test failure rather than a test skip, +providing a clear indication that migration is needed. + +## Detailed design + +### Highlight usage of XCTest APIs in Swift Testing tests + +We propose supporting the following XCTest APIs in Swift Testing: + +- Assertions: `XCTAssert*` and [unconditional failure][] `XCTFail` +- [Expected failures][], such as `XCTExpectFailure`: marking a Swift Testing + issue in this way will generate a runtime issue. +- `XCTAttachment` +- [Issue handling traits][]: we will make our best effort to translate issues + from XCTest to Swift Testing. Note that there are certain issue kinds that are + new to Swift Testing and not expressible from XCTest. + +Note that no changes are proposed for the `XCTSkip` API, because they already +feature prominently as a test failure to be corrected when thrown in Swift +Testing. + +We also propose highlighting usage of above XCTest APIs in Swift Testing: + +- **Report [runtime warning issues][]** for XCTest API usage in Swift Testing. + This **applies to assertion successes AND failures**! We want to make sure you + can identify opportunities to modernise even if your tests currently all pass. + +- Opt-in **strict interop mode**, which will trigger a crash instead. + +Here are some concrete examples: + +| When running a Swift Testing test... | Current | Proposed (default) | Proposed (strict) | +| ------------------------------------ | --------------- | -------------------------------------------- | ----------------- | +| `XCTAssert` failure is a ... | ‼️ No-op | ❌ Test Failure and ⚠️ Runtime Warning Issue | 💥 Crash | +| `XCTAssert` success is a ... | No-op | ⚠️ Runtime Warning Issue | 💥 Crash | +| `throw XCTSkip` is a ... | ❌ Test Failure | ❌ Test Failure | ❌ Test Failure | + +### Limited support for Swift Testing APIs in XCTest + +We propose supporting the following Swift Testing APIs in XCTest: + +- `#expect` and `#require` + - Includes [exit testing][] +- `withKnownIssue`: when this suppresses `XCTAssert` failures, it will still + show a runtime warning issue. +- Attachments +- [Test cancellation][] (links to pitch) + +We think developers will find utility in using Swift Testing APIs in XCTest. For +example, you can replace `XCTAssert` with `#expect` in your XCTest tests and +immediately get the benefits of the newer, more ergonomic API. In the meantime, +you can incrementally migrate the rest of your test infrastructure to use Swift +Testing at your own pace. + +Present and future Swift Testing APIs will be supported in XCTest if the +XCTest API _already_ provides similar functionality. + +- For example, we plan on supporting the proposed Swift Testing [test + cancellation][] feature in XCTest since it is analogous to `XCTSkip` + +- On the other hand, [Traits][] are a powerful Swift Testing feature, and + include the ability to [add tags][tags] to organise tests. Even though XCTest + does not interact with tags, this is beyond the scope of interoperability + because XCTest doesn't have existing “tag-like” behaviour to map onto. + +Here are some concrete examples: + +| When running a XCTest test... | Current | Proposed (default) | Proposed (strict) | +| -------------------------------------------- | --------------- | ------------------------ | ----------------- | +| `#expect` failure is a ... | ‼️ No-op | ❌ Test Failure | ❌ Test Failure | +| `#expect` success is a ... | No-op | No-op | No-op | +| `withKnownIssue` wrapping `XCTFail` is a ... | ❌ Test Failure | ⚠️ Runtime Warning Issue | 💥 Crash | + +### Interoperability Modes + +The default interoperability surfaces test failures that were previously +ignored. We include two more permissible interoperability modes to avoid +breaking projects that are dependent on this pre-interop behaviour. + +- **Warning-only**: This is for projects which do not want to see new test + failures surfaced due to interoperability. + +- **None**: Some projects may additionally have issue handling trait that + promote warnings to errors, which means that warning-only mode could still + cause test failures. + +For projects that want to bolster their Swift Testing adoption, there is also an +opt-in strict interop mode. + +- **Strict**: Warning issues included in the default mode can be easily + overlooked, especially in CI. The strict mode guarantees that no XCTest API + usage occurs when running Swift Testing tests by turning those warnings into a + runtime crash. + +Configure the interoperability mode when running tests using the +`SWT_XCTEST_INTEROP_MODE` environment variable: + +| Interop Mode | Issue behaviour across framework boundary | `SWT_XCTEST_INTEROP_MODE` | +| ------------ | -------------------------------------------- | ------------------------------------------- | +| Off | ‼️ No-op (status quo) | `off` | +| Warning-only | ⚠️ Runtime Warning Issue | `warning` | +| Default | ❌ Test Failure and ⚠️ Runtime Warning Issue | `default`, or empty value, or invalid value | +| Strict | 💥 Crash | `strict` | + +### Phased Rollout + +When interoperability is first available, "default" will be the interop mode +enabled for new projects. In a future release, "strict" will become the default +interop mode. + +## Source compatibility + +As the main goal of interoperability is to change behaviour, this proposal will +lead to situations where previously "passing" test code now starts showing +failures. We believe this should be a net positive if it can highlight actual +bugs you would have missed previously. + +You can use `SWT_XCTEST_INTEROP_MODE=off` in the short-term to revert back to +the current behaviour. Refer to the "Interoperability Modes" section for a full list +of options. + +## Integration with supporting tools + +Interoperability will be first available in future toolchain version, +hypothetically named `6.X`, where default interop mode will be enabled for +projects. After that, a `6.Y` release would make strict interop mode the +default. + +- Swift Package Manager projects: `swift-tools-version` declared in + Package.swift will be used to determine interop mode, regardless of the + toolchain used to run tests. + +- Xcode projects: Installed toolchain version will be used to determine interop + mode. + +- Any project can use `SWT_XCTEST_INTEROP_MODE` to override interop mode at + runtime, provided they are on toolchain version `6.X` or newer + +## Future directions + +There's still more we can do to make it easier to migrate from XCTest to Swift +Testing: + +- Provide fixups at compile-time to replace usage of XCTest API with the + corresponding Swift Testing API, e.g. replace `XCTAssert` with `#expect`. + However, this would require introspection of the test body to look for XCTest + API usage, which would be challenging to do completely and find usages of this + API within helper methods. + +- After new API added to SWT in future, will need to evaluate for + interoperability with XCTest until strict mode is the default. "strict" is + kind of saying "from this point forward, no new interop will be added" for new + SWT features. + +## Alternatives considered + +### Arguments against interoperability + +Frameworks that operate independently generally have no expectation of +interoperability, and XCTest and Swift Testing are currently set up this way. +Indeed, the end goal is not necessarily to support using XCTest APIs +indefinitely within Swift Testing. Adding interoperability can be interpreted as +going against that goal, and can enable users to delay migrating completely to +Swift Testing. + +However, we believe the benefits of fixing lossy without interop APIs and +helping users catch more bugs are too important to pass up. We've also included +a plan to increase the strictness of the interoperability mode over time, which +should make it clear that this is not intended to be a permanent measure. + +### Strict interop mode as the default + +We believe that for projects using only Swift Testing, strict interop mode is +the best choice. Making this the default would also send the clearest signal +that we want users to migrate to Swift Testing. + +However, we are especially sensitive to use cases that depend upon the currently +lossy without interop APIs, and decided to prioritise the current default as a +good balance between notifying users yet not breaking existing test suites. + +## Acknowledgments + +Thanks to Stuart Montgomery, Jonathan Grynspan, and Brian Croom for feedback on +the proposal. + +[unconditional failure]: https://developer.apple.com/documentation/xctest/unconditional-test-failures +[runtime warning issues]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md +[expected failures]: https://developer.apple.com/documentation/xctest/expected-failures +[issue handling traits]: https://developer.apple.com/documentation/testing/issuehandlingtrait +[test cancellation]: https://forums.swift.org/t/pitch-test-cancellation/81847 +[traits]: https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/traits +[tags]: https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/addingtags +[exit testing]: https://developer.apple.com/documentation/testing/exit-testing From dc24e27b2853a5c49d9e1b280f9c53ce1c0a16e4 Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Thu, 2 Oct 2025 11:32:48 -0700 Subject: [PATCH 02/12] Minor changes --- ...ift-testing-and-xctest-interoperability.md | 111 +++++++++++------- 1 file changed, 67 insertions(+), 44 deletions(-) diff --git a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md index cd8addb863..0ca222f4c1 100644 --- a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md +++ b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md @@ -1,4 +1,4 @@ -# Interoperability between Swift Testing and XCTest +# Targeted Interoperability between Swift Testing and XCTest - Proposal: [ST-NNNN](NNNN-xctest-interoperability.md) - Authors: [Jerry Chen](https://github.com/jerryjrchen) @@ -51,18 +51,15 @@ Generally, we get into trouble today when ALL the following conditions are met: - You call XCTest API in a Swift Testing test, or call Swift Testing API in a XCTest test - The API doesn't function as expected in some or all cases, and -- You get no notice at build or runtime about the malfunction - -Example: calling `XCTAssertEqual(x, y)` in a Swift Testing test: if x does not -equal y, it should report a failure but does nothing instead. +- You get no notice at build time or runtime about the malfunction For the remainder of this proposal, we’ll describe tests which exhibit this problem as **lossy without interop**. If you've switched completely to Swift Testing and don't expect to use XCTest in the future, this proposal includes a mechanism to **prevent you from -inadvertently introducing XCTest APIs to your project, including via a testing -library.** +inadvertently introducing XCTest APIs to your project**, including via a testing +library. ## Proposed solution @@ -90,7 +87,7 @@ We propose supporting the following XCTest APIs in Swift Testing: - Assertions: `XCTAssert*` and [unconditional failure][] `XCTFail` - [Expected failures][], such as `XCTExpectFailure`: marking a Swift Testing - issue in this way will generate a runtime issue. + issue in this way will generate a runtime warning issue. - `XCTAttachment` - [Issue handling traits][]: we will make our best effort to translate issues from XCTest to Swift Testing. Note that there are certain issue kinds that are @@ -104,16 +101,18 @@ We also propose highlighting usage of above XCTest APIs in Swift Testing: - **Report [runtime warning issues][]** for XCTest API usage in Swift Testing. This **applies to assertion successes AND failures**! We want to make sure you - can identify opportunities to modernise even if your tests currently all pass. + can identify opportunities to modernise even if your tests currently pass. -- Opt-in **strict interop mode**, which will trigger a crash instead. +- Opt-in **strict interop mode**, where XCTest API usage will result in + `fatalError("Usage of XCTest API in a Swift Testing context is not +supported")`. Here are some concrete examples: | When running a Swift Testing test... | Current | Proposed (default) | Proposed (strict) | | ------------------------------------ | --------------- | -------------------------------------------- | ----------------- | -| `XCTAssert` failure is a ... | ‼️ No-op | ❌ Test Failure and ⚠️ Runtime Warning Issue | 💥 Crash | -| `XCTAssert` success is a ... | No-op | ⚠️ Runtime Warning Issue | 💥 Crash | +| `XCTAssert` failure is a ... | ‼️ No-op | ❌ Test Failure and ⚠️ Runtime Warning Issue | 💥 `fatalError` | +| `XCTAssert` success is a ... | No-op | ⚠️ Runtime Warning Issue | 💥 `fatalError` | | `throw XCTSkip` is a ... | ❌ Test Failure | ❌ Test Failure | ❌ Test Failure | ### Limited support for Swift Testing APIs in XCTest @@ -122,8 +121,8 @@ We propose supporting the following Swift Testing APIs in XCTest: - `#expect` and `#require` - Includes [exit testing][] -- `withKnownIssue`: when this suppresses `XCTAssert` failures, it will still - show a runtime warning issue. +- `withKnownIssue`: marking an XCTest issue in this way will generate a runtime + warning issue. In strict interop mode, this becomes a `fatalError`. - Attachments - [Test cancellation][] (links to pitch) @@ -136,12 +135,12 @@ Testing at your own pace. Present and future Swift Testing APIs will be supported in XCTest if the XCTest API _already_ provides similar functionality. -- For example, we plan on supporting the proposed Swift Testing [test - cancellation][] feature in XCTest since it is analogous to `XCTSkip` +- For example, we support the proposed Swift Testing [test cancellation][] + feature in XCTest since it is analogous to `XCTSkip`. - On the other hand, [Traits][] are a powerful Swift Testing feature, and include the ability to [add tags][tags] to organise tests. Even though XCTest - does not interact with tags, this is beyond the scope of interoperability + does not interact with tags, **this is beyond the scope of interoperability** because XCTest doesn't have existing “tag-like” behaviour to map onto. Here are some concrete examples: @@ -150,38 +149,33 @@ Here are some concrete examples: | -------------------------------------------- | --------------- | ------------------------ | ----------------- | | `#expect` failure is a ... | ‼️ No-op | ❌ Test Failure | ❌ Test Failure | | `#expect` success is a ... | No-op | No-op | No-op | -| `withKnownIssue` wrapping `XCTFail` is a ... | ❌ Test Failure | ⚠️ Runtime Warning Issue | 💥 Crash | +| `withKnownIssue` wrapping `XCTFail` is a ... | ❌ Test Failure | ⚠️ Runtime Warning Issue | 💥 `fatalError` | ### Interoperability Modes -The default interoperability surfaces test failures that were previously -ignored. We include two more permissible interoperability modes to avoid -breaking projects that are dependent on this pre-interop behaviour. +The default interoperability mode surfaces test failures that were previously +ignored. We also include two alternative interoperability modes: - **Warning-only**: This is for projects which do not want to see new test failures surfaced due to interoperability. -- **None**: Some projects may additionally have issue handling trait that - promote warnings to errors, which means that warning-only mode could still - cause test failures. - -For projects that want to bolster their Swift Testing adoption, there is also an -opt-in strict interop mode. - - **Strict**: Warning issues included in the default mode can be easily overlooked, especially in CI. The strict mode guarantees that no XCTest API usage occurs when running Swift Testing tests by turning those warnings into a - runtime crash. + `fatalError`. Configure the interoperability mode when running tests using the -`SWT_XCTEST_INTEROP_MODE` environment variable: +`SWIFT_TESTING_XCTEST_INTEROP_MODE` environment variable: + +| Interop Mode | Issue behaviour across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | +| ------------ | ----------------------------------------------------------------- | ------------------------------------------- | +| Warning-only | ⚠️ Runtime Warning Issue for XCTest API usage | `warning` | +| Default | ❌ Test Failure and ⚠️ Runtime Warning Issue for XCTest API usage | `default`, or empty value, or invalid value | +| Strict | 💥 `fatalError` for XCTest API usage | `strict` | -| Interop Mode | Issue behaviour across framework boundary | `SWT_XCTEST_INTEROP_MODE` | -| ------------ | -------------------------------------------- | ------------------------------------------- | -| Off | ‼️ No-op (status quo) | `off` | -| Warning-only | ⚠️ Runtime Warning Issue | `warning` | -| Default | ❌ Test Failure and ⚠️ Runtime Warning Issue | `default`, or empty value, or invalid value | -| Strict | 💥 Crash | `strict` | + ### Phased Rollout @@ -196,7 +190,7 @@ lead to situations where previously "passing" test code now starts showing failures. We believe this should be a net positive if it can highlight actual bugs you would have missed previously. -You can use `SWT_XCTEST_INTEROP_MODE=off` in the short-term to revert back to +You can use `SWIFT_TESTING_XCTEST_INTEROP_MODE=off` in the short-term to revert back to the current behaviour. Refer to the "Interoperability Modes" section for a full list of options. @@ -204,7 +198,7 @@ of options. Interoperability will be first available in future toolchain version, hypothetically named `6.X`, where default interop mode will be enabled for -projects. After that, a `6.Y` release would make strict interop mode the +projects. After that, a `7.Y` release would make strict interop mode the default. - Swift Package Manager projects: `swift-tools-version` declared in @@ -214,8 +208,8 @@ default. - Xcode projects: Installed toolchain version will be used to determine interop mode. -- Any project can use `SWT_XCTEST_INTEROP_MODE` to override interop mode at - runtime, provided they are on toolchain version `6.X` or newer +- Any project can use `SWIFT_TESTING_XCTEST_INTEROP_MODE` to override interop + mode at runtime, provided they are on toolchain version `6.X` or newer ## Future directions @@ -228,10 +222,9 @@ Testing: API usage, which would be challenging to do completely and find usages of this API within helper methods. -- After new API added to SWT in future, will need to evaluate for - interoperability with XCTest until strict mode is the default. "strict" is - kind of saying "from this point forward, no new interop will be added" for new - SWT features. +- After new API is added to Swift Testing in future, will need to evaluate for + interoperability with XCTest. Once strict mode is the default, we will no + longer include interoperability for new Swift Testing features. ## Alternatives considered @@ -249,6 +242,18 @@ helping users catch more bugs are too important to pass up. We've also included a plan to increase the strictness of the interoperability mode over time, which should make it clear that this is not intended to be a permanent measure. +### Opt-out of interoperability + +In a similar vein, we considered `SWIFT_TESTING_XCTEST_INTEROP_MODE=off` as a +way to completely turn off interoperability. Some projects may additionally have +issue handling trait that promote warnings to errors, which means that +warning-only mode could still cause test failures. + +However, in the scenario above, we think users who set up tests to elevate +warnings as errors would be interested in increased visibility of testing issues +surfaced by interop. We're open to feedback about other scenarios where a +"interop off" mode would be preferred. + ### Strict interop mode as the default We believe that for projects using only Swift Testing, strict interop mode is @@ -259,6 +264,24 @@ However, we are especially sensitive to use cases that depend upon the currently lossy without interop APIs, and decided to prioritise the current default as a good balance between notifying users yet not breaking existing test suites. +### Alternative methods to control interop mode + +- **Build setting:** e.g. a new `SwiftSetting` that can be included in + Package.swift or an Xcode project. A project could then configure their test + targets to have a non-default interop mode. + + However, interop is a runtime concept, and would be difficult or at least + non-idiomatic to modify with a build setting. + +- **CLI option through SwiftPM:** + + ``` + swift test --interop-mode=warning-only + ``` + + This could be offered in addition to the proposed environment variable option, + although it would be unclear which one should take precedence. + ## Acknowledgments Thanks to Stuart Montgomery, Jonathan Grynspan, and Brian Croom for feedback on From 1e721f848d0e2b95c46673575dbafe3ce45c5276 Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Thu, 2 Oct 2025 11:32:48 -0700 Subject: [PATCH 03/12] Update naming of default interop mode -> permissive --- ...ift-testing-and-xctest-interoperability.md | 33 +++++++++---------- 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md index 0ca222f4c1..2833de2f6d 100644 --- a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md +++ b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md @@ -109,7 +109,7 @@ supported")`. Here are some concrete examples: -| When running a Swift Testing test... | Current | Proposed (default) | Proposed (strict) | +| When running a Swift Testing test... | Current | Proposed | Proposed (strict) | | ------------------------------------ | --------------- | -------------------------------------------- | ----------------- | | `XCTAssert` failure is a ... | ‼️ No-op | ❌ Test Failure and ⚠️ Runtime Warning Issue | 💥 `fatalError` | | `XCTAssert` success is a ... | No-op | ⚠️ Runtime Warning Issue | 💥 `fatalError` | @@ -145,7 +145,7 @@ XCTest API _already_ provides similar functionality. Here are some concrete examples: -| When running a XCTest test... | Current | Proposed (default) | Proposed (strict) | +| When running a XCTest test... | Current | Proposed | Proposed (strict) | | -------------------------------------------- | --------------- | ------------------------ | ----------------- | | `#expect` failure is a ... | ‼️ No-op | ❌ Test Failure | ❌ Test Failure | | `#expect` success is a ... | No-op | No-op | No-op | @@ -153,13 +153,14 @@ Here are some concrete examples: ### Interoperability Modes -The default interoperability mode surfaces test failures that were previously -ignored. We also include two alternative interoperability modes: - - **Warning-only**: This is for projects which do not want to see new test failures surfaced due to interoperability. -- **Strict**: Warning issues included in the default mode can be easily +- **Permissive**: This is the default interoperability mode, which surfaces test + failures that were previously ignored. It also includes runtime warning issues + for XCTest API usage in a Swift Testing context. + +- **Strict**: Warning issues included in the permissive mode can be easily overlooked, especially in CI. The strict mode guarantees that no XCTest API usage occurs when running Swift Testing tests by turning those warnings into a `fatalError`. @@ -167,19 +168,15 @@ ignored. We also include two alternative interoperability modes: Configure the interoperability mode when running tests using the `SWIFT_TESTING_XCTEST_INTEROP_MODE` environment variable: -| Interop Mode | Issue behaviour across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | -| ------------ | ----------------------------------------------------------------- | ------------------------------------------- | -| Warning-only | ⚠️ Runtime Warning Issue for XCTest API usage | `warning` | -| Default | ❌ Test Failure and ⚠️ Runtime Warning Issue for XCTest API usage | `default`, or empty value, or invalid value | -| Strict | 💥 `fatalError` for XCTest API usage | `strict` | - - +| Interop Mode | Issue behaviour across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | +| ------------ | ----------------------------------------------------------------- | ---------------------------------------------- | +| Warning-only | XCTest API: ⚠️ Runtime Warning Issue | `warning-only` | +| Permissive | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ❌ Test Failure | `permissive`, or empty value, or invalid value | +| Strict | XCTest API: 💥 `fatalError`. Swift Testing API: ❌ Test Failure | `strict` | ### Phased Rollout -When interoperability is first available, "default" will be the interop mode +When interoperability is first available, "permissive" will be the default interop mode enabled for new projects. In a future release, "strict" will become the default interop mode. @@ -197,7 +194,7 @@ of options. ## Integration with supporting tools Interoperability will be first available in future toolchain version, -hypothetically named `6.X`, where default interop mode will be enabled for +hypothetically named `6.X`, where permissive interop mode will be enabled for projects. After that, a `7.Y` release would make strict interop mode the default. @@ -276,7 +273,7 @@ good balance between notifying users yet not breaking existing test suites. - **CLI option through SwiftPM:** ``` - swift test --interop-mode=warning-only + swift test --interop-mode=warning ``` This could be offered in addition to the proposed environment variable option, From 4325ef5f6d158ea5a72c38047862a397f3c8974e Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Thu, 2 Oct 2025 11:32:48 -0700 Subject: [PATCH 04/12] Update interop support philosophy --- ...ift-testing-and-xctest-interoperability.md | 32 +++++++++---------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md index 2833de2f6d..2de93dac4a 100644 --- a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md +++ b/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md @@ -63,16 +63,14 @@ library. ## Proposed solution -At a high level, we propose the following **changes for APIs which are lossy -without interop**: +- **XCTest APIs which are lossy without interop will work as expected when + called in Swift Testing.** In addition, runtime diagnostics will be included + to highlight opportunities to adopt newer Swift Testing equivalent APIs. -- **XCTest API called in Swift Testing will work as expected.** In addition, - runtime diagnostics will be included to highlight opportunities to adopt newer - Swift Testing equivalent APIs. - -- Conversely, **Swift Testing API called in XCTest will also work as expected.** - You should always feel empowered to choose Swift Testing when writing new - tests or test helpers, as it will work properly in both types of tests. +- Conversely, **Swift Testing API called in XCTest will work as expected if + XCTest already provides similar functionality.** You should always feel + empowered to choose Swift Testing when writing new tests or test helpers, as + it will work properly in both types of tests. We don't propose supporting interoperability for APIs without risk of data loss, because they naturally have high visibility. For example, using `throw XCTSkip` @@ -81,7 +79,7 @@ providing a clear indication that migration is needed. ## Detailed design -### Highlight usage of XCTest APIs in Swift Testing tests +### Highlight and support XCTest APIs which are lossy without interop We propose supporting the following XCTest APIs in Swift Testing: @@ -115,7 +113,7 @@ Here are some concrete examples: | `XCTAssert` success is a ... | No-op | ⚠️ Runtime Warning Issue | 💥 `fatalError` | | `throw XCTSkip` is a ... | ❌ Test Failure | ❌ Test Failure | ❌ Test Failure | -### Limited support for Swift Testing APIs in XCTest +### Targeted support for Swift Testing APIs with XCTest API equivalents We propose supporting the following Swift Testing APIs in XCTest: @@ -176,9 +174,9 @@ Configure the interoperability mode when running tests using the ### Phased Rollout -When interoperability is first available, "permissive" will be the default interop mode -enabled for new projects. In a future release, "strict" will become the default -interop mode. +When interoperability is first available, "permissive" will be the default +interop mode enabled for new projects. In a future release, "strict" will become +the default interop mode. ## Source compatibility @@ -187,9 +185,9 @@ lead to situations where previously "passing" test code now starts showing failures. We believe this should be a net positive if it can highlight actual bugs you would have missed previously. -You can use `SWIFT_TESTING_XCTEST_INTEROP_MODE=off` in the short-term to revert back to -the current behaviour. Refer to the "Interoperability Modes" section for a full list -of options. +You can use `SWIFT_TESTING_XCTEST_INTEROP_MODE=off` in the short-term to revert +back to the current behaviour. Refer to the "Interoperability Modes" section for +a full list of options. ## Integration with supporting tools From 3ecd884991b9200ad6b354e7f1532260dd040443 Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Thu, 2 Oct 2025 13:48:06 -0700 Subject: [PATCH 05/12] Rename doc --- ...=> NNNN-targeted-interoperability-swift-testing-and-xctest.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename proposals/testing/{NNNN-swift-testing-and-xctest-interoperability.md => NNNN-targeted-interoperability-swift-testing-and-xctest.md} (100%) diff --git a/proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md similarity index 100% rename from proposals/testing/NNNN-swift-testing-and-xctest-interoperability.md rename to proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md From b275e55d2ed00f0732411c872af40b29af1aba7b Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Thu, 2 Oct 2025 17:18:43 -0700 Subject: [PATCH 06/12] Respond to PR feedback --- ...teroperability-swift-testing-and-xctest.md | 100 +++++++++--------- 1 file changed, 52 insertions(+), 48 deletions(-) diff --git a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md index 2de93dac4a..950be4b9a7 100644 --- a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md +++ b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md @@ -14,13 +14,13 @@ intermediate state where test helpers written using XCTest API are called from Swift Testing. Today, the Swift Testing and XCTest libraries stand mostly independently, which means an `XCTAssert` failure in a Swift Testing test is silently ignored. To address this, we formally declare a set of interoperability -principles and propose updates to specific APIs that will enable users to -migrate with confidence. +principles and propose changes to the handling of specific APIs that will enable +users to migrate with confidence. ## Motivation -Unfortunately, mixing an API call from one framework with a test from the other -framework may not work as expected. As a more concrete example, if you take an +Calling XCTest or Swift Testing API within a test from the opposite framework +may not always work as expected. As a more concrete example, if you take an existing test helper function written for XCTest and call it in a Swift Testing test, it won't report the assertion failure: @@ -49,17 +49,16 @@ class FooTests: XCTestCase { Generally, we get into trouble today when ALL the following conditions are met: - You call XCTest API in a Swift Testing test, or call Swift Testing API in a - XCTest test + XCTest test, - The API doesn't function as expected in some or all cases, and - You get no notice at build time or runtime about the malfunction -For the remainder of this proposal, we’ll describe tests which exhibit this +For the remainder of this proposal, we’ll describe test APIs which exhibit this problem as **lossy without interop**. -If you've switched completely to Swift Testing and don't expect to use XCTest in -the future, this proposal includes a mechanism to **prevent you from -inadvertently introducing XCTest APIs to your project**, including via a testing -library. +This problem risks regressing test coverage for projects which migrate to Swift +Testing. Furthermore, projects that have switched completely to Swift Testing +may want to go and ensure they don't inadvertently add XCTest API. ## Proposed solution @@ -72,10 +71,10 @@ library. empowered to choose Swift Testing when writing new tests or test helpers, as it will work properly in both types of tests. -We don't propose supporting interoperability for APIs without risk of data loss, -because they naturally have high visibility. For example, using `throw XCTSkip` -in a Swift Testing test results in a test failure rather than a test skip, -providing a clear indication that migration is needed. +We don't propose supporting interoperability for APIs which are not lossy +without interop, because they naturally have high visibility. For example, using +`throw XCTSkip` in a Swift Testing test results in a test failure rather than a +test skip, providing a clear indication that migration is needed. ## Detailed design @@ -83,27 +82,26 @@ providing a clear indication that migration is needed. We propose supporting the following XCTest APIs in Swift Testing: -- Assertions: `XCTAssert*` and [unconditional failure][] `XCTFail` +- [Assertions][XCTest Assertions]: `XCTAssert*` and [unconditional failure][] + `XCTFail` - [Expected failures][], such as `XCTExpectFailure`: marking a Swift Testing issue in this way will generate a runtime warning issue. -- `XCTAttachment` +- [`XCTAttachment`][XCTest attachments] - [Issue handling traits][]: we will make our best effort to translate issues - from XCTest to Swift Testing. Note that there are certain issue kinds that are - new to Swift Testing and not expressible from XCTest. + from XCTest to Swift Testing. For issue details unique to XCTest, we will + include them as a comment when constructing the Swift Testing issue. Note that no changes are proposed for the `XCTSkip` API, because they already -feature prominently as a test failure to be corrected when thrown in Swift -Testing. +feature prominently as a test failure when thrown in Swift Testing. We also propose highlighting usage of above XCTest APIs in Swift Testing: - **Report [runtime warning issues][]** for XCTest API usage in Swift Testing. - This **applies to assertion successes AND failures**! We want to make sure you - can identify opportunities to modernise even if your tests currently pass. + This **applies to both assertion failures _and successes_**! This notifies you + about opportunities to modernise even if your tests currently pass. - Opt-in **strict interop mode**, where XCTest API usage will result in - `fatalError("Usage of XCTest API in a Swift Testing context is not -supported")`. + `fatalError("Usage of XCTest API in a Swift Testing context is unsupported")`. Here are some concrete examples: @@ -122,7 +120,7 @@ We propose supporting the following Swift Testing APIs in XCTest: - `withKnownIssue`: marking an XCTest issue in this way will generate a runtime warning issue. In strict interop mode, this becomes a `fatalError`. - Attachments -- [Test cancellation][] (links to pitch) +- [Test cancellation][] (currently pitched) We think developers will find utility in using Swift Testing APIs in XCTest. For example, you can replace `XCTAssert` with `#expect` in your XCTest tests and @@ -133,13 +131,13 @@ Testing at your own pace. Present and future Swift Testing APIs will be supported in XCTest if the XCTest API _already_ provides similar functionality. -- For example, we support the proposed Swift Testing [test cancellation][] - feature in XCTest since it is analogous to `XCTSkip`. +For example, the recently-pitched [test cancellation][] feature in Swift Testing +is analogous to `XCTSkip`. If that pitch were accepted, this proposal would +support interop of the new API with XCTest. -- On the other hand, [Traits][] are a powerful Swift Testing feature, and - include the ability to [add tags][tags] to organise tests. Even though XCTest - does not interact with tags, **this is beyond the scope of interoperability** - because XCTest doesn't have existing “tag-like” behaviour to map onto. +On the other hand, [traits][] are a powerful Swift Testing feature which is not +related to any functionality in XCTest. Therefore, there would be +interoperability for traits under this proposal. Here are some concrete examples: @@ -151,8 +149,10 @@ Here are some concrete examples: ### Interoperability Modes -- **Warning-only**: This is for projects which do not want to see new test - failures surfaced due to interoperability. +- **Warning-only**: Test failures that were previously ignored are reported as + runtime warning issues. It also includes runtime warning issues for XCTest API + usage in a Swift Testing context. This is for projects which do not want to + see new test failures surfaced due to interoperability. - **Permissive**: This is the default interoperability mode, which surfaces test failures that were previously ignored. It also includes runtime warning issues @@ -166,11 +166,11 @@ Here are some concrete examples: Configure the interoperability mode when running tests using the `SWIFT_TESTING_XCTEST_INTEROP_MODE` environment variable: -| Interop Mode | Issue behaviour across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | -| ------------ | ----------------------------------------------------------------- | ---------------------------------------------- | -| Warning-only | XCTest API: ⚠️ Runtime Warning Issue | `warning-only` | -| Permissive | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ❌ Test Failure | `permissive`, or empty value, or invalid value | -| Strict | XCTest API: 💥 `fatalError`. Swift Testing API: ❌ Test Failure | `strict` | +| Interop Mode | Issue behaviour across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | +| ------------ | -------------------------------------------------------------------------- | ---------------------------------------------- | +| Warning-only | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ⚠️ Runtime Warning Issue | `warning-only` | +| Permissive | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ❌ Test Failure | `permissive`, or empty value, or invalid value | +| Strict | XCTest API: 💥 `fatalError`. Swift Testing API: ❌ Test Failure | `strict` | ### Phased Rollout @@ -185,9 +185,9 @@ lead to situations where previously "passing" test code now starts showing failures. We believe this should be a net positive if it can highlight actual bugs you would have missed previously. -You can use `SWIFT_TESTING_XCTEST_INTEROP_MODE=off` in the short-term to revert -back to the current behaviour. Refer to the "Interoperability Modes" section for -a full list of options. +You can use `SWIFT_TESTING_XCTEST_INTEROP_MODE=warning-only` in the short-term +to revert any changes to test pass/fail outcomes as a result of +interoperability. ## Integration with supporting tools @@ -218,8 +218,7 @@ Testing: API within helper methods. - After new API is added to Swift Testing in future, will need to evaluate for - interoperability with XCTest. Once strict mode is the default, we will no - longer include interoperability for new Swift Testing features. + interoperability with XCTest. ## Alternatives considered @@ -241,7 +240,7 @@ should make it clear that this is not intended to be a permanent measure. In a similar vein, we considered `SWIFT_TESTING_XCTEST_INTEROP_MODE=off` as a way to completely turn off interoperability. Some projects may additionally have -issue handling trait that promote warnings to errors, which means that +an issue handling trait that promotes warnings to errors, which means that warning-only mode could still cause test failures. However, in the scenario above, we think users who set up tests to elevate @@ -256,8 +255,12 @@ the best choice. Making this the default would also send the clearest signal that we want users to migrate to Swift Testing. However, we are especially sensitive to use cases that depend upon the currently -lossy without interop APIs, and decided to prioritise the current default as a -good balance between notifying users yet not breaking existing test suites. +lossy without interop APIs. With strict interop mode, the test process will +crash on the first instance of XCTest API usage in Swift Testing, completely +halting testing. In this same scenario, the default permissive interop mode +would record a runtime warning issue and continue the remaining test, which we +believe strikes a better balance between notifying users yet not being totally +disruptive to the testing flow. ### Alternative methods to control interop mode @@ -271,7 +274,7 @@ good balance between notifying users yet not breaking existing test suites. - **CLI option through SwiftPM:** ``` - swift test --interop-mode=warning + swift test --interop-mode=warning-only ``` This could be offered in addition to the proposed environment variable option, @@ -282,11 +285,12 @@ good balance between notifying users yet not breaking existing test suites. Thanks to Stuart Montgomery, Jonathan Grynspan, and Brian Croom for feedback on the proposal. +[XCTest assertions]: https://developer.apple.com/documentation/xctest/equality-and-inequality-assertions +[XCTest attachments]: https://developer.apple.com/documentation/xctest/adding-attachments-to-tests-activities-and-issues [unconditional failure]: https://developer.apple.com/documentation/xctest/unconditional-test-failures [runtime warning issues]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md [expected failures]: https://developer.apple.com/documentation/xctest/expected-failures [issue handling traits]: https://developer.apple.com/documentation/testing/issuehandlingtrait [test cancellation]: https://forums.swift.org/t/pitch-test-cancellation/81847 [traits]: https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/traits -[tags]: https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/addingtags [exit testing]: https://developer.apple.com/documentation/testing/exit-testing From bb5f67e206f4dde95cb42c3bab2486bcd58e4269 Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Fri, 3 Oct 2025 09:26:48 -0700 Subject: [PATCH 07/12] Review feedback --- ...nteroperability-swift-testing-and-xctest.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md index 950be4b9a7..acb7a41e8b 100644 --- a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md +++ b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md @@ -78,6 +78,16 @@ test skip, providing a clear indication that migration is needed. ## Detailed design +> [!NOTE] +> This proposal refers to XCTest in the abstract. There are two different +> implementations of XCTest: the open source [Corelibs XCTest][] and a +> [proprietary XCTest][Xcode XCTest] that is shipped as part of Xcode. The Swift +> evolution process governs changes to the former only. Therefore, +> this proposal is targeted for Corelibs XCTest. + +[Corelibs XCTest]: https://github.com/swiftlang/swift-corelibs-xctest +[Xcode XCTest]: https://developer.apple.com/documentation/xctest + ### Highlight and support XCTest APIs which are lossy without interop We propose supporting the following XCTest APIs in Swift Testing: @@ -196,12 +206,10 @@ hypothetically named `6.X`, where permissive interop mode will be enabled for projects. After that, a `7.Y` release would make strict interop mode the default. -- Swift Package Manager projects: `swift-tools-version` declared in - Package.swift will be used to determine interop mode, regardless of the - toolchain used to run tests. +- Swift packages: `swift-tools-version` declared in Package.swift will be used + to determine interop mode, regardless of the toolchain used to run tests. -- Xcode projects: Installed toolchain version will be used to determine interop - mode. +- Otherwise, installed toolchain version will be used to determine interop mode. - Any project can use `SWIFT_TESTING_XCTEST_INTEROP_MODE` to override interop mode at runtime, provided they are on toolchain version `6.X` or newer From ed7e09a392f5e75ad8ff8488c5fb3bf49f39880a Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Fri, 3 Oct 2025 09:34:38 -0700 Subject: [PATCH 08/12] Warning-only => advisory --- ...rgeted-interoperability-swift-testing-and-xctest.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md index acb7a41e8b..92e1f6d35d 100644 --- a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md +++ b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md @@ -159,7 +159,7 @@ Here are some concrete examples: ### Interoperability Modes -- **Warning-only**: Test failures that were previously ignored are reported as +- **Advisory**: Test failures that were previously ignored are reported as runtime warning issues. It also includes runtime warning issues for XCTest API usage in a Swift Testing context. This is for projects which do not want to see new test failures surfaced due to interoperability. @@ -178,7 +178,7 @@ Configure the interoperability mode when running tests using the | Interop Mode | Issue behaviour across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | | ------------ | -------------------------------------------------------------------------- | ---------------------------------------------- | -| Warning-only | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ⚠️ Runtime Warning Issue | `warning-only` | +| Advisory | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ⚠️ Runtime Warning Issue | `warning-only` | | Permissive | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ❌ Test Failure | `permissive`, or empty value, or invalid value | | Strict | XCTest API: 💥 `fatalError`. Swift Testing API: ❌ Test Failure | `strict` | @@ -195,7 +195,7 @@ lead to situations where previously "passing" test code now starts showing failures. We believe this should be a net positive if it can highlight actual bugs you would have missed previously. -You can use `SWIFT_TESTING_XCTEST_INTEROP_MODE=warning-only` in the short-term +You can use `SWIFT_TESTING_XCTEST_INTEROP_MODE=advisory` in the short-term to revert any changes to test pass/fail outcomes as a result of interoperability. @@ -249,7 +249,7 @@ should make it clear that this is not intended to be a permanent measure. In a similar vein, we considered `SWIFT_TESTING_XCTEST_INTEROP_MODE=off` as a way to completely turn off interoperability. Some projects may additionally have an issue handling trait that promotes warnings to errors, which means that -warning-only mode could still cause test failures. +advisory mode could still cause test failures. However, in the scenario above, we think users who set up tests to elevate warnings as errors would be interested in increased visibility of testing issues @@ -282,7 +282,7 @@ disruptive to the testing flow. - **CLI option through SwiftPM:** ``` - swift test --interop-mode=warning-only + swift test --interop-mode=advisory ``` This could be offered in addition to the proposed environment variable option, From 07f45fadff5bae31c9a9216587d57e53da4d2de4 Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Fri, 3 Oct 2025 10:09:29 -0700 Subject: [PATCH 09/12] Update code example --- ...NNNN-targeted-interoperability-swift-testing-and-xctest.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md index 92e1f6d35d..ee91e79674 100644 --- a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md +++ b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md @@ -26,9 +26,7 @@ test, it won't report the assertion failure: ```swift func assertUnique(_ elements: [Int]) { - if Set(elements).count != elements.count { - XCTFail("\(elements) has non unique elements") - } + XCTAssertEqual(Set(elements).count, elements.count, "\(elements) has non unique elements") } // XCTest From f119329c83349ece828a9621af65762e13b02f9a Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Fri, 3 Oct 2025 10:14:51 -0700 Subject: [PATCH 10/12] Review feedback --- ...teroperability-swift-testing-and-xctest.md | 83 ++++++++++--------- 1 file changed, 46 insertions(+), 37 deletions(-) diff --git a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md index ee91e79674..08f0cfd8dc 100644 --- a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md +++ b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md @@ -12,10 +12,11 @@ Many projects want to migrate from XCTest to Swift Testing, and may be in an intermediate state where test helpers written using XCTest API are called from Swift Testing. Today, the Swift Testing and XCTest libraries stand mostly -independently, which means an `XCTAssert` failure in a Swift Testing test is -silently ignored. To address this, we formally declare a set of interoperability -principles and propose changes to the handling of specific APIs that will enable -users to migrate with confidence. +independently, which means an [`XCTAssert`][XCTest assertions] failure in a +Swift Testing test or an [`#expect`][Swift Testing expectations] failure in an +XCTest test is silently ignored. To address this, we formally declare a set of +interoperability principles and propose changes to the handling of specific APIs +that will enable users to migrate with confidence. ## Motivation @@ -44,7 +45,8 @@ class FooTests: XCTestCase { } ``` -Generally, we get into trouble today when ALL the following conditions are met: +Generally, you encounter the above limitation with testing APIs when _all_ the +following conditions are met: - You call XCTest API in a Swift Testing test, or call Swift Testing API in a XCTest test, @@ -52,11 +54,12 @@ Generally, we get into trouble today when ALL the following conditions are met: - You get no notice at build time or runtime about the malfunction For the remainder of this proposal, we’ll describe test APIs which exhibit this -problem as **lossy without interop**. +limitation as **lossy without interop**. -This problem risks regressing test coverage for projects which migrate to Swift -Testing. Furthermore, projects that have switched completely to Swift Testing -may want to go and ensure they don't inadvertently add XCTest API. +You could regress test coverage if you migrate to Swift Testing without +replacing usage of lossy without interop test APIs. Furthermore, you may want to +ensure you don't inadvertently introduce new XCTest API after completing your +Swift Testing migration. ## Proposed solution @@ -106,7 +109,7 @@ We also propose highlighting usage of above XCTest APIs in Swift Testing: - **Report [runtime warning issues][]** for XCTest API usage in Swift Testing. This **applies to both assertion failures _and successes_**! This notifies you - about opportunities to modernise even if your tests currently pass. + about opportunities to modernize even if your tests currently pass. - Opt-in **strict interop mode**, where XCTest API usage will result in `fatalError("Usage of XCTest API in a Swift Testing context is unsupported")`. @@ -123,7 +126,8 @@ Here are some concrete examples: We propose supporting the following Swift Testing APIs in XCTest: -- `#expect` and `#require` +- [`#expect` and `#require`][Swift Testing expectations] + - Includes [`#expect(throws:)`][testing for errors] - Includes [exit testing][] - `withKnownIssue`: marking an XCTest issue in this way will generate a runtime warning issue. In strict interop mode, this becomes a `fatalError`. @@ -144,7 +148,7 @@ is analogous to `XCTSkip`. If that pitch were accepted, this proposal would support interop of the new API with XCTest. On the other hand, [traits][] are a powerful Swift Testing feature which is not -related to any functionality in XCTest. Therefore, there would be +related to any functionality in XCTest. Therefore, there would not be interoperability for traits under this proposal. Here are some concrete examples: @@ -174,21 +178,18 @@ Here are some concrete examples: Configure the interoperability mode when running tests using the `SWIFT_TESTING_XCTEST_INTEROP_MODE` environment variable: -| Interop Mode | Issue behaviour across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | +| Interop Mode | Issue behavior across framework boundary | `SWIFT_TESTING_XCTEST_INTEROP_MODE` | | ------------ | -------------------------------------------------------------------------- | ---------------------------------------------- | -| Advisory | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ⚠️ Runtime Warning Issue | `warning-only` | +| Advisory | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ⚠️ Runtime Warning Issue | `advisory` | | Permissive | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ❌ Test Failure | `permissive`, or empty value, or invalid value | | Strict | XCTest API: 💥 `fatalError`. Swift Testing API: ❌ Test Failure | `strict` | -### Phased Rollout +## Source compatibility When interoperability is first available, "permissive" will be the default -interop mode enabled for new projects. In a future release, "strict" will become -the default interop mode. - -## Source compatibility +interop mode enabled for new projects. -As the main goal of interoperability is to change behaviour, this proposal will +As the main goal of interoperability is to change behavior, this proposal will lead to situations where previously "passing" test code now starts showing failures. We believe this should be a net positive if it can highlight actual bugs you would have missed previously. @@ -199,31 +200,34 @@ interoperability. ## Integration with supporting tools -Interoperability will be first available in future toolchain version, -hypothetically named `6.X`, where permissive interop mode will be enabled for -projects. After that, a `7.Y` release would make strict interop mode the -default. - - Swift packages: `swift-tools-version` declared in Package.swift will be used to determine interop mode, regardless of the toolchain used to run tests. + Specifically, it will use the default interop mode associated with that + toolchain version ("permissive" for the initial release version). + + We will work with the Swift Package Manager maintainers and the Ecosystem + Steering Group to make appropriate changes in other parts of the toolchain. -- Otherwise, installed toolchain version will be used to determine interop mode. +- Otherwise, the default interop mode associated with the installed toolchain + version will be used to determine interop mode. - Any project can use `SWIFT_TESTING_XCTEST_INTEROP_MODE` to override interop - mode at runtime, provided they are on toolchain version `6.X` or newer + mode at runtime. ## Future directions There's still more we can do to make it easier to migrate from XCTest to Swift Testing: +- In a future release, we would consider making strict interop mode the default. + - Provide fixups at compile-time to replace usage of XCTest API with the corresponding Swift Testing API, e.g. replace `XCTAssert` with `#expect`. However, this would require introspection of the test body to look for XCTest API usage, which would be challenging to do completely and find usages of this API within helper methods. -- After new API is added to Swift Testing in future, will need to evaluate for +- When new API is added to Swift Testing, we will need to evaluate it for interoperability with XCTest. ## Alternatives considered @@ -263,16 +267,16 @@ that we want users to migrate to Swift Testing. However, we are especially sensitive to use cases that depend upon the currently lossy without interop APIs. With strict interop mode, the test process will crash on the first instance of XCTest API usage in Swift Testing, completely -halting testing. In this same scenario, the default permissive interop mode -would record a runtime warning issue and continue the remaining test, which we -believe strikes a better balance between notifying users yet not being totally -disruptive to the testing flow. +halting testing. In this same scenario, the proposed default permissive interop +mode would record a runtime warning issue and continue the remaining test, which +we believe strikes a better balance between notifying users yet not being +totally disruptive to the testing flow. ### Alternative methods to control interop mode - **Build setting:** e.g. a new `SwiftSetting` that can be included in - Package.swift or an Xcode project. A project could then configure their test - targets to have a non-default interop mode. + Package.swift. A project could then configure their test targets to have a + non-default interop mode. However, interop is a runtime concept, and would be difficult or at least non-idiomatic to modify with a build setting. @@ -291,12 +295,17 @@ disruptive to the testing flow. Thanks to Stuart Montgomery, Jonathan Grynspan, and Brian Croom for feedback on the proposal. + [XCTest assertions]: https://developer.apple.com/documentation/xctest/equality-and-inequality-assertions [XCTest attachments]: https://developer.apple.com/documentation/xctest/adding-attachments-to-tests-activities-and-issues [unconditional failure]: https://developer.apple.com/documentation/xctest/unconditional-test-failures -[runtime warning issues]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md [expected failures]: https://developer.apple.com/documentation/xctest/expected-failures + +[Swift Testing expectations]: https://developer.apple.com/documentation/testing/expectations +[Testing for errors]: https://developer.apple.com/documentation/testing/testing-for-errors-in-swift-code +[exit testing]: https://developer.apple.com/documentation/testing/exit-testing [issue handling traits]: https://developer.apple.com/documentation/testing/issuehandlingtrait +[traits]: https://developer.apple.com/documentation/testing/traits + +[runtime warning issues]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md [test cancellation]: https://forums.swift.org/t/pitch-test-cancellation/81847 -[traits]: https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/traits -[exit testing]: https://developer.apple.com/documentation/testing/exit-testing From 4819c122e30a49a2ccd1c60e324abac1294a2fbc Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Fri, 3 Oct 2025 13:49:00 -0700 Subject: [PATCH 11/12] Review feedback --- ...N-targeted-interoperability-swift-testing-and-xctest.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md index 08f0cfd8dc..94b88a9f38 100644 --- a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md +++ b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md @@ -81,12 +81,12 @@ test skip, providing a clear indication that migration is needed. > [!NOTE] > This proposal refers to XCTest in the abstract. There are two different -> implementations of XCTest: the open source [Corelibs XCTest][] and a +> implementations of XCTest: the open source [swift-corelibs-xctest][] and a > [proprietary XCTest][Xcode XCTest] that is shipped as part of Xcode. The Swift > evolution process governs changes to the former only. Therefore, > this proposal is targeted for Corelibs XCTest. -[Corelibs XCTest]: https://github.com/swiftlang/swift-corelibs-xctest +[swift-corelibs-xctest]: https://github.com/swiftlang/swift-corelibs-xctest [Xcode XCTest]: https://developer.apple.com/documentation/xctest ### Highlight and support XCTest APIs which are lossy without interop @@ -97,7 +97,6 @@ We propose supporting the following XCTest APIs in Swift Testing: `XCTFail` - [Expected failures][], such as `XCTExpectFailure`: marking a Swift Testing issue in this way will generate a runtime warning issue. -- [`XCTAttachment`][XCTest attachments] - [Issue handling traits][]: we will make our best effort to translate issues from XCTest to Swift Testing. For issue details unique to XCTest, we will include them as a comment when constructing the Swift Testing issue. @@ -131,7 +130,6 @@ We propose supporting the following Swift Testing APIs in XCTest: - Includes [exit testing][] - `withKnownIssue`: marking an XCTest issue in this way will generate a runtime warning issue. In strict interop mode, this becomes a `fatalError`. -- Attachments - [Test cancellation][] (currently pitched) We think developers will find utility in using Swift Testing APIs in XCTest. For @@ -297,7 +295,6 @@ the proposal. [XCTest assertions]: https://developer.apple.com/documentation/xctest/equality-and-inequality-assertions -[XCTest attachments]: https://developer.apple.com/documentation/xctest/adding-attachments-to-tests-activities-and-issues [unconditional failure]: https://developer.apple.com/documentation/xctest/unconditional-test-failures [expected failures]: https://developer.apple.com/documentation/xctest/expected-failures From 0829ac134e0a1418964c521c5e57fbf2bd41cb66 Mon Sep 17 00:00:00 2001 From: Jerry Chen Date: Tue, 7 Oct 2025 09:42:17 -0700 Subject: [PATCH 12/12] Add notes on XCTestExpectation and XCTWaiter API --- ...teroperability-swift-testing-and-xctest.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md index 94b88a9f38..adf51cfbc0 100644 --- a/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md +++ b/proposals/testing/NNNN-targeted-interoperability-swift-testing-and-xctest.md @@ -104,6 +104,16 @@ We propose supporting the following XCTest APIs in Swift Testing: Note that no changes are proposed for the `XCTSkip` API, because they already feature prominently as a test failure when thrown in Swift Testing. +We also don't propose changes for [`XCTestExpectation`][XCTestExpectation] and +[`XCTWaiter`][XCTWaiter]. They cannot be used safely in a Swift concurrency +context when running Swift Testing tests. Instead, we recommend that users +migrate to Swift concurrency where possible. For code that does not easily map +to `async`/`await` semantics, use [continuations][] and [confirmations][] +instead. + +Refer to [Migrating a test from XCTest][XCTest migration validate async] for a +detailed discussion. + We also propose highlighting usage of above XCTest APIs in Swift Testing: - **Report [runtime warning issues][]** for XCTest API usage in Swift Testing. @@ -294,15 +304,25 @@ Thanks to Stuart Montgomery, Jonathan Grynspan, and Brian Croom for feedback on the proposal. + [XCTest assertions]: https://developer.apple.com/documentation/xctest/equality-and-inequality-assertions [unconditional failure]: https://developer.apple.com/documentation/xctest/unconditional-test-failures [expected failures]: https://developer.apple.com/documentation/xctest/expected-failures +[XCTWaiter]: https://developer.apple.com/documentation/xctest/xctwaiter +[XCTestExpectation]: https://developer.apple.com/documentation/xctest/xctestexpectation + + [Swift Testing expectations]: https://developer.apple.com/documentation/testing/expectations [Testing for errors]: https://developer.apple.com/documentation/testing/testing-for-errors-in-swift-code [exit testing]: https://developer.apple.com/documentation/testing/exit-testing [issue handling traits]: https://developer.apple.com/documentation/testing/issuehandlingtrait [traits]: https://developer.apple.com/documentation/testing/traits +[confirmations]: https://developer.apple.com/documentation/testing/confirmation +[XCTest migration validate async]: https://developer.apple.com/documentation/testing/migratingfromxctest#Validate-asynchronous-behaviors + + [runtime warning issues]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md +[continuations]: https://developer.apple.com/documentation/swift/checkedcontinuation [test cancellation]: https://forums.swift.org/t/pitch-test-cancellation/81847