Skip to content

Add support for .expectationFailed in interop and exit tests.#1608

Merged
grynspan merged 6 commits intomainfrom
jgrynspan/plumb-through-expectation-failed-to-interop-and-exit-tests
Mar 5, 2026
Merged

Add support for .expectationFailed in interop and exit tests.#1608
grynspan merged 6 commits intomainfrom
jgrynspan/plumb-through-expectation-failed-to-interop-and-exit-tests

Conversation

@grynspan
Copy link
Contributor

@grynspan grynspan commented Mar 5, 2026

This PR enhances the JSON event stream to allow us to recognize and recover issues of kind .expectationFailed. This then improves the fidelity of these issues when passed through XCTest interop and exit test process boundaries.

For example, given the following exit test:

@Test func foo() async {
  await #expect(processExitsWith: .success) {
    struct S: Equatable {
      var x: Int
      var y: String
    }
    let lhs = S(x: 1, y: "abc")
    let rhs = S(x: 2, y: "def")
    #expect(lhs == rhs)
  }
}

We'll now get a breakdown of the failed expectation (note I recorded this example with --verbose so type info would be visible too):

<X>  Test foo() recorded an issue at [...]: Expectation failed: lhs == rhs
`->  lhs == rhs: Swift.Bool → false
`->    lhs: ModuleName.S → S(x: 1, y: "abc")
`->      x: Swift.Int → 1
`->      y: Swift.String → "abc"
`->    rhs: ModuleName.S → S(x: 2, y: "def")
`->      x: Swift.Int → 2
`->      y: Swift.String → "def"

In the process of implementing this change, I was able to reconcile the issue-decoding logic used in interop and exit tests so they share an implementation.

This PR also resolves an issue running interop tests inside Xcode 26.4 where the tests could spuriously fail.

Checklist:

  • Code and documentation should follow the style of the Style Guide.
  • If public symbols are renamed or modified, DocC references should be updated.

This PR enhances the JSON event stream to allow us to recognize and recover
issues of kind `.expectationFailed`. This then improves the fidelity of these
issues when passed through XCTest interop and exit test process boundaries.

In the process of implementing this change, I was able to reconcile the
issue-decoding logic used in interop and exit tests so they share an
implementation.

This PR also resolves an issue running interop tests inside Xcode 26.4 where the
tests could spuriously fail.
@grynspan grynspan added this to the Swift 6.4.0 (main) milestone Mar 5, 2026
@grynspan grynspan self-assigned this Mar 5, 2026
@grynspan grynspan added the enhancement New feature or request label Mar 5, 2026
@grynspan grynspan added tools integration 🛠️ Integration of swift-testing into tools/IDEs exit-tests ☠️ Work related to exit tests issue-handling Related to Issue handling within the testing library command-line experience ⌨️ enhancements to the command line interface interoperability ↔️ Work related to Swift Testing and XCTest interoperability labels Mar 5, 2026
///
/// - Parameters:
/// - expression: The encoded expression to initialize this instance from.
init?<V>(decoding expression: ABI.EncodedExpression<V>) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ready to make this public/SPI yet.

///
/// If `event` does not represent a recorded issue, the initializer returns
/// `nil`.
init?<V>(decoding event: ABI.EncodedEvent<V>) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not ready to make these public/SPI yet.

Comment on lines +166 to +168
// The known issue comment, if there was one, is already included in
// the `comments` array above.
self.knownIssueContext = Issue.KnownIssueContext()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we separated the comment on withKnownIssue into a discrete field intentionally, so are we potentially conflating them by having them combined into a single field? (That might be a preexisting problem)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of right now, the known-issue comment isn't in the encoded form. It's something we can certainly look into as a separate PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The source comment itself is not new, but is out of date.)

@grynspan grynspan merged commit 22ffa1f into main Mar 5, 2026
53 of 54 checks passed
@grynspan grynspan deleted the jgrynspan/plumb-through-expectation-failed-to-interop-and-exit-tests branch March 5, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

command-line experience ⌨️ enhancements to the command line interface enhancement New feature or request exit-tests ☠️ Work related to exit tests interoperability ↔️ Work related to Swift Testing and XCTest interoperability issue-handling Related to Issue handling within the testing library tools integration 🛠️ Integration of swift-testing into tools/IDEs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants