diff --git a/Documentation/Proposals/0005-ranged-confirmations.md b/Documentation/Proposals/0005-ranged-confirmations.md new file mode 100644 index 000000000..7ba133c9f --- /dev/null +++ b/Documentation/Proposals/0005-ranged-confirmations.md @@ -0,0 +1,186 @@ +# Range-based confirmations + +* Proposal: [SWT-0005](0005-ranged-confirmations.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Accepted** +* Bug: rdar://138499457 +* Implementation: [swiftlang/swift-testing#598](https://github.com/swiftlang/swift-testing/pull/598), [swiftlang/swift-testing#689](https://github.com/swiftlang/swift-testing/pull689) +* Review: ([pitch](https://forums.swift.org/t/pitch-range-based-confirmations/74589)), + ([acceptance](https://forums.swift.org/t/pitch-range-based-confirmations/74589/7)) + +## Introduction + +Swift Testing includes [an interface](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)) +for checking that some asynchronous event occurs a given number of times +(typically exactly once or never at all.) This proposal enhances that interface +to allow arbitrary ranges of event counts so that a test can be written against +code that may not always fire said event the exact same number of times. + +## Motivation + +Some tests rely on fixtures or external state that is not perfectly +deterministic. For example, consider a test that checks that clicking the mouse +button will generate a `.mouseClicked` event. Such a test might use the +`confirmation()` interface: + +```swift +await confirmation(expectedCount: 1) { mouseClicked in + var eventLoop = EventLoop() + eventLoop.eventHandler = { event in + if event == .mouseClicked { + mouseClicked() + } + } + await eventLoop.simulate(.mouseClicked) +} +``` + +But what happens if the user _actually_ clicks a mouse button while this test is +running? That might trigger a _second_ `.mouseClicked` event, and then the test +will fail spuriously. + +## Proposed solution + +If the test author could instead indicate to Swift Testing that their test will +generate _one or more_ events, they could avoid spurious failures: + +```swift +await confirmation(expectedCount: 1...) { mouseClicked in + ... +} +``` + +With this proposal, we add an overload of `confirmation()` that takes any range +expression instead of a single integer value (which is still accepted via the +existing overload.) + +## Detailed design + +A new overload of `confirmation()` is added: + +```swift +/// Confirm that some event occurs during the invocation of a function. +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - expectedCount: A range of integers indicating the number of times the +/// expected event should occur when `body` is invoked. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to which any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Returns: Whatever is returned by `body`. +/// +/// - Throws: Whatever is thrown by `body`. +/// +/// Use confirmations to check that an event occurs while a test is running in +/// complex scenarios where `#expect()` and `#require()` are insufficient. For +/// example, a confirmation may be useful when an expected event occurs: +/// +/// - In a context that cannot be awaited by the calling function such as an +/// event handler or delegate callback; +/// - More than once, or never; or +/// - As a callback that is invoked as part of a larger operation. +/// +/// To use a confirmation, pass a closure containing the work to be performed. +/// The testing library will then pass an instance of ``Confirmation`` to the +/// closure. Every time the event in question occurs, the closure should call +/// the confirmation: +/// +/// ```swift +/// let minBuns = 5 +/// let maxBuns = 10 +/// await confirmation( +/// "Baked between \(minBuns) and \(maxBuns) buns", +/// expectedCount: minBuns ... maxBuns +/// ) { bunBaked in +/// foodTruck.eventHandler = { event in +/// if event == .baked(.cinnamonBun) { +/// bunBaked() +/// } +/// } +/// await foodTruck.bakeTray(of: .cinnamonBun) +/// } +/// ``` +/// +/// When the closure returns, the testing library checks if the confirmation's +/// preconditions have been met, and records an issue if they have not. +/// +/// If an exact count is expected, use +/// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` instead. +public func confirmation( + _ comment: Comment? = nil, + expectedCount: some RangeExpression & Sequence Sendable, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> sending R +) async rethrows -> R +``` + +### Ranges without lower bounds + +Certain types of range, specifically [`PartialRangeUpTo`](https://developer.apple.com/documentation/swift/partialrangeupto) +and [`PartialRangeThrough`](https://developer.apple.com/documentation/swift/partialrangethrough), +may have surprising behavior when used with this new interface because they +implicitly include `0`. If a test author writes `...10`, do they mean "zero to +ten" or "one to ten"? The programmatic meaning is the former, but some test +authors might mean the latter. If an event does not occur, a test using +`confirmation()` and this `expectedCount` value would pass when the test author +meant for it to fail. + +The unbounded range (`...`) type `UnboundedRange` is effectively useless when +used with this interface and any use of it here is almost certainly a programmer +error. + +`PartialRangeUpTo` and `PartialRangeThrough` conform to `RangeExpression`, but +not to `Sequence`, so they will be rejected at compile time. `UnboundedRange` is +a non-nominal type and will not match either. We will provide unavailable +overloads of `confirmation()` for these types with messages that explain why +they are unavailable, e.g.: + +```swift +@available(*, unavailable, message: "Unbounded range '...' has no effect when used with a confirmation.") +public func confirmation( + _ comment: Comment? = nil, + expectedCount: UnboundedRange, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> R +) async rethrows -> R +``` + +## Source compatibility + +This change is additive. Existing tests are unaffected. + +Code that refers to `confirmation(_:expectedCount:isolation:sourceLocation:_:)` +by symbol name may need to add a contextual type to disambiguate the two +overloads at compile time. + +## Integration with supporting tools + +The type of the associated value `expected` for the `Issue.Kind` case +`confirmationMiscounted(actual:expected:)` will change from `Int` to +`any RangeExpression & Sendable`[^1]. Tools that implement event handlers and +distinguish between `Issue.Kind` cases are advised not to assume the type of +this value is `Int`. + +## Alternatives considered + +- Doing nothing. We have identified real-world use cases for this interface + including in Swift Testing’s own test target. +- Allowing the use of any value as the `expectedCount` argument so long as it + conforms to a protocol `ExpectedCount` (we'd have range types and `Int` + conform by default.) It was unclear what this sort of flexibility would let + us do, and posed challenges for encoding and decoding events and issues when + using the JSON event stream interface. + +## Acknowledgments + +Thanks to the testing team for their help preparing this pitch! + +[^1]: In the future, this type will change to + `any RangeExpression & Sendable`. Compiler support is required + ([96960993](rdar://96960993)). diff --git a/Sources/Testing/Issues/Confirmation.swift b/Sources/Testing/Issues/Confirmation.swift index 48124c56e..95b7284b0 100644 --- a/Sources/Testing/Issues/Confirmation.swift +++ b/Sources/Testing/Issues/Confirmation.swift @@ -161,10 +161,9 @@ public func confirmation( /// /// If an exact count is expected, use /// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` instead. -@_spi(Experimental) public func confirmation( _ comment: Comment? = nil, - expectedCount: some RangeExpression & Sendable, + expectedCount: some RangeExpression & Sequence & Sendable, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: (Confirmation) async throws -> sending R @@ -174,7 +173,7 @@ public func confirmation( let actualCount = confirmation.count.rawValue if !expectedCount.contains(actualCount) { let issue = Issue( - kind: expectedCount.issueKind(forActualCount: actualCount), + kind: .confirmationMiscounted(actual: actualCount, expected: expectedCount), comments: Array(comment), sourceContext: .init(backtrace: .current(), sourceLocation: sourceLocation) ) @@ -184,7 +183,7 @@ public func confirmation( return try await body(confirmation) } -/// An overload of ``confirmation(_:expectedCount:sourceLocation:_:)-9bfdc`` +/// An overload of ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-6bkl6`` /// that handles the unbounded range operator (`...`). /// /// This overload is necessary because `UnboundedRange` does not conform to @@ -194,27 +193,41 @@ public func confirmation( public func confirmation( _ comment: Comment? = nil, expectedCount: UnboundedRange, + isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: (Confirmation) async throws -> R ) async rethrows -> R { fatalError("Unsupported") } -extension RangeExpression where Bound == Int, Self: Sendable { - /// Get an instance of ``Issue/Kind-swift.enum`` corresponding to this value. - /// - /// - Parameters: - /// - actualCount: The actual count for the failed confirmation. - /// - /// - Returns: An instance of ``Issue/Kind-swift.enum`` that describes `self`. - fileprivate func issueKind(forActualCount actualCount: Int) -> Issue.Kind { - switch self { - case let expectedCount as ClosedRange where expectedCount.lowerBound == expectedCount.upperBound: - return .confirmationMiscounted(actual: actualCount, expected: expectedCount.lowerBound) - case let expectedCount as Range where expectedCount.lowerBound == expectedCount.upperBound - 1: - return .confirmationMiscounted(actual: actualCount, expected: expectedCount.lowerBound) - default: - return .confirmationOutOfRange(actual: actualCount, expected: self) - } - } +/// An overload of ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-6bkl6`` +/// that handles the partial-range-through operator (`...n`). +/// +/// This overload is necessary because the lower bound of `PartialRangeThrough` +/// is ambiguous: does it start at `0` or `1`? Test authors should specify a +@available(*, unavailable, message: "Range expression '...n' is ambiguous without an explicit lower bound") +public func confirmation( + _ comment: Comment? = nil, + expectedCount: PartialRangeThrough, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> R +) async rethrows -> R { + fatalError("Unsupported") +} + +/// An overload of ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-6bkl6`` +/// that handles the partial-range-up-to operator (`..( + _ comment: Comment? = nil, + expectedCount: PartialRangeUpTo, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> R +) async rethrows -> R { + fatalError("Unsupported") } diff --git a/Sources/Testing/Issues/Issue.swift b/Sources/Testing/Issues/Issue.swift index fe3130567..886243cbe 100644 --- a/Sources/Testing/Issues/Issue.swift +++ b/Sources/Testing/Issues/Issue.swift @@ -32,27 +32,11 @@ public struct Issue: Sendable { /// - expected: The expected number of times /// ``Confirmation/confirm(count:)`` should have been called. /// - /// This issue can occur when calling - /// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` when the - /// confirmation passed to these functions' `body` closures is confirmed too - /// few or too many times. - indirect case confirmationMiscounted(actual: Int, expected: Int) - - /// An issue due to a confirmation being confirmed the wrong number of - /// times. - /// - /// - Parameters: - /// - actual: The number of times ``Confirmation/confirm(count:)`` was - /// actually called. - /// - expected: The expected number of times - /// ``Confirmation/confirm(count:)`` should have been called. - /// - /// This issue can occur when calling - /// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-9rt6m`` when - /// the confirmation passed to these functions' `body` closures is confirmed - /// too few or too many times. - @_spi(Experimental) - indirect case confirmationOutOfRange(actual: Int, expected: any RangeExpression & Sendable) + /// This issue can occur when calling ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-5mqz2`` + /// or ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-6bkl6`` + /// when the confirmation passed to these functions' `body` closures is + /// confirmed too few or too many times. + indirect case confirmationMiscounted(actual: Int, expected: any RangeExpression & Sendable) /// An issue due to an `Error` being thrown by a test function and caught by /// the testing library. @@ -186,6 +170,18 @@ extension Issue: CustomStringConvertible, CustomDebugStringConvertible { } } +/// An empty protocol defining a type that conforms to `RangeExpression`. +/// +/// In the future, when our minimum deployment target supports casting a value +/// to a constrained existential type ([SE-0353](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0353-constrained-existential-types.md#effect-on-abi-stability)), +/// we can remove this protocol and cast to `RangeExpression` instead. +private protocol _RangeExpressionOverIntValues: RangeExpression where Bound == Int {} +extension ClosedRange: _RangeExpressionOverIntValues {} +extension PartialRangeFrom: _RangeExpressionOverIntValues {} +extension PartialRangeThrough: _RangeExpressionOverIntValues {} +extension PartialRangeUpTo: _RangeExpressionOverIntValues {} +extension Range: _RangeExpressionOverIntValues {} + extension Issue.Kind: CustomStringConvertible { public var description: String { switch self { @@ -193,9 +189,9 @@ extension Issue.Kind: CustomStringConvertible { // Although the failure is unconditional at the point it is recorded, the // code that recorded the issue may not be unconditionally executing, so // we shouldn't describe it as unconditional (we just don't know!) - "Issue recorded" + return "Issue recorded" case let .expectationFailed(expectation): - if let mismatchedErrorDescription = expectation.mismatchedErrorDescription { + return if let mismatchedErrorDescription = expectation.mismatchedErrorDescription { "Expectation failed: \(mismatchedErrorDescription)" } else if let mismatchedExitConditionDescription = expectation.mismatchedExitConditionDescription { "Expectation failed: \(mismatchedExitConditionDescription)" @@ -203,19 +199,23 @@ extension Issue.Kind: CustomStringConvertible { "Expectation failed: \(expectation.evaluatedExpression.expandedDescription())" } case let .confirmationMiscounted(actual: actual, expected: expected): - "Confirmation was confirmed \(actual.counting("time")), but expected to be confirmed \(expected.counting("time"))" - case let .confirmationOutOfRange(actual: actual, expected: expected): - "Confirmation was confirmed \(actual.counting("time")), but expected to be confirmed \(String(describingForTest: expected)) time(s)" + if let expected = expected as? any _RangeExpressionOverIntValues { + let expected = expected.relative(to: []) + if expected.upperBound > expected.lowerBound && expected.lowerBound == expected.upperBound - 1 { + return "Confirmation was confirmed \(actual.counting("time")), but expected to be confirmed \(expected.lowerBound.counting("time"))" + } + } + return "Confirmation was confirmed \(actual.counting("time")), but expected to be confirmed \(String(describingForTest: expected)) time(s)" case let .errorCaught(error): - "Caught error: \(error)" + return "Caught error: \(error)" case let .timeLimitExceeded(timeLimitComponents: timeLimitComponents): - "Time limit was exceeded: \(TimeValue(timeLimitComponents))" + return "Time limit was exceeded: \(TimeValue(timeLimitComponents))" case .knownIssueNotRecorded: - "Known issue was not recorded" + return "Known issue was not recorded" case .apiMisused: - "An API was misused" + return "An API was misused" case .system: - "A system failure occurred" + return "A system failure occurred" } } } @@ -246,7 +246,7 @@ extension Issue { /// /// - Parameter issue: The original issue that gets snapshotted. public init(snapshotting issue: borrowing Issue) { - if case .confirmationOutOfRange = issue.kind { + if case .confirmationMiscounted = issue.kind { // Work around poor stringification of this issue kind in Xcode 16. self.kind = .unconditional self.comments = CollectionOfOne("\(issue.kind)") + issue.comments @@ -306,7 +306,7 @@ extension Issue.Kind { /// ``Confirmation/confirm(count:)`` should have been called. /// /// This issue can occur when calling - /// ``confirmation(_:expectedCount:sourceLocation:_:)`` when the + /// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` when the /// confirmation passed to these functions' `body` closures is confirmed too /// few or too many times. indirect case confirmationMiscounted(actual: Int, expected: Int) @@ -349,10 +349,8 @@ extension Issue.Kind { .unconditional case let .expectationFailed(expectation): .expectationFailed(Expectation.Snapshot(snapshotting: expectation)) - case let .confirmationMiscounted(actual: actual, expected: expected): - .confirmationMiscounted(actual: actual, expected: expected) - case let .confirmationOutOfRange(actual: actual, expected: _): - .confirmationMiscounted(actual: actual, expected: 0) + case .confirmationMiscounted: + .unconditional case let .errorCaught(error): .errorCaught(ErrorSnapshot(snapshotting: error)) case let .timeLimitExceeded(timeLimitComponents: timeLimitComponents): diff --git a/Sources/Testing/Testing.docc/Expectations.md b/Sources/Testing/Testing.docc/Expectations.md index 083e09c64..331150730 100644 --- a/Sources/Testing/Testing.docc/Expectations.md +++ b/Sources/Testing/Testing.docc/Expectations.md @@ -75,7 +75,8 @@ the test when the code doesn't satisfy a requirement, use ### Confirming that asynchronous events occur - -- ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` +- ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-5mqz2`` +- ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-6bkl6`` - ``Confirmation`` ### Retrieving information about checked expectations diff --git a/Sources/Testing/Testing.docc/MigratingFromXCTest.md b/Sources/Testing/Testing.docc/MigratingFromXCTest.md index 529c56595..6b63ddf10 100644 --- a/Sources/Testing/Testing.docc/MigratingFromXCTest.md +++ b/Sources/Testing/Testing.docc/MigratingFromXCTest.md @@ -428,7 +428,8 @@ Some tests, especially those that test asynchronously-delivered events, cannot be readily converted to use Swift concurrency. The testing library offers functionality called _confirmations_ which can be used to implement these tests. Instances of ``Confirmation`` are created and used within the scope of the -function ``confirmation(_:expectedCount:isolation:sourceLocation:_:)``. +functions ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-5mqz2`` +and ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-6bkl6``. Confirmations function similarly to the expectations API of XCTest, however, they don't block or suspend the caller while waiting for a condition to be fulfilled. diff --git a/Sources/Testing/Testing.docc/testing-asynchronous-code.md b/Sources/Testing/Testing.docc/testing-asynchronous-code.md index 548cf07b0..54e4b6a2f 100644 --- a/Sources/Testing/Testing.docc/testing-asynchronous-code.md +++ b/Sources/Testing/Testing.docc/testing-asynchronous-code.md @@ -31,7 +31,7 @@ expected event happens. ### Confirm that an event happens -Call ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` in your +Call ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-5mqz2`` in your asynchronous test function to create a `Confirmation` for the expected event. In the trailing closure parameter, call the code under test. Swift Testing passes a `Confirmation` as the parameter to the closure, which you call as a function in @@ -54,6 +54,35 @@ If you expect the event to happen more than once, set the test passes if the number of occurrences during the test matches the expected count, and fails otherwise. +You can also pass a range to ``confirmation(_:expectedCount:isolation:sourceLocation:_:)-6bkl6`` +if the exact number of times the event occurs may change over time or is random: + +```swift +@Test("Customers bought sandwiches") +func boughtSandwiches() async { + await confirmation(expectedCount: 0 ..< 1000) { boughtSandwich in + var foodTruck = FoodTruck() + foodTruck.orderHandler = { order in + if order.contains(.sandwich) { + boughtSandwich() + } + } + await FoodTruck.operate() + } +} +``` + +In this example, there may be zero customers or up to (but not including) 1,000 +customers who order sandwiches. Any [range expression](https://developer.apple.com/documentation/swift/rangeexpression) +which includes an explicit lower bound can be used: + +| Range Expression | Usage | +|-|-| +| `1...` | If an event must occur _at least_ once | +| `5...` | If an event must occur _at least_ five times | +| `1 ... 5` | If an event must occur at least once, but not more than five times | +| `0 ..< 100` | If an event may or may not occur, but _must not_ occur more than 99 times | + ### Confirm that an event doesn't happen To validate that a particular event doesn't occur during a test, diff --git a/Tests/TestingTests/ConfirmationTests.swift b/Tests/TestingTests/ConfirmationTests.swift index 7c2dca474..7fe824d71 100644 --- a/Tests/TestingTests/ConfirmationTests.swift +++ b/Tests/TestingTests/ConfirmationTests.swift @@ -29,25 +29,21 @@ struct ConfirmationTests { @Test("Unsuccessful confirmations") func unsuccessfulConfirmations() async { - await confirmation("Miscount recorded", expectedCount: 4) { miscountRecorded in - await confirmation("Out of range recorded", expectedCount: 5) { outOfRangeRecorded in - var configuration = Configuration() - configuration.eventHandler = { event, _ in - if case let .issueRecorded(issue) = event.kind { - switch issue.kind { - case .confirmationMiscounted: - miscountRecorded() - case .confirmationOutOfRange: - outOfRangeRecorded() - default: - break - } + await confirmation("Miscount recorded", expectedCount: 7) { miscountRecorded in + var configuration = Configuration() + configuration.eventHandler = { event, _ in + if case let .issueRecorded(issue) = event.kind { + switch issue.kind { + case .confirmationMiscounted: + miscountRecorded() + default: + break } } - let testPlan = await Runner.Plan(selecting: UnsuccessfulConfirmationTests.self) - let runner = Runner(plan: testPlan, configuration: configuration) - await runner.run() } + let testPlan = await Runner.Plan(selecting: UnsuccessfulConfirmationTests.self) + let runner = Runner(plan: testPlan, configuration: configuration) + await runner.run() } } @@ -119,8 +115,6 @@ struct UnsuccessfulConfirmationTests { 1 ... 2 as any ExpectedCount, 1 ..< 2, 1 ..< 3, - ..<2, - ...2, 999..., ]) func confirmedOutOfRange(_ range: any ExpectedCount) async { @@ -135,9 +129,7 @@ struct UnsuccessfulConfirmationTests { /// Needed since we don't have generic test functions, so we need a concrete /// argument type for `confirmedOutOfRange(_:)`, but we can't write /// `any RangeExpression & Sendable`. ([96960993](rdar://96960993)) -protocol ExpectedCount: RangeExpression, Sendable where Bound == Int {} +protocol ExpectedCount: RangeExpression, Sequence, Sendable where Bound == Int, Element == Int {} extension ClosedRange: ExpectedCount {} extension PartialRangeFrom: ExpectedCount {} -extension PartialRangeThrough: ExpectedCount {} -extension PartialRangeUpTo: ExpectedCount {} extension Range: ExpectedCount {} diff --git a/Tests/TestingTests/IssueTests.swift b/Tests/TestingTests/IssueTests.swift index 531b50d15..f2854cbfd 100644 --- a/Tests/TestingTests/IssueTests.swift +++ b/Tests/TestingTests/IssueTests.swift @@ -1455,7 +1455,6 @@ struct IssueCodingTests { private static let issueKinds: [Issue.Kind] = [ Issue.Kind.apiMisused, - Issue.Kind.confirmationMiscounted(actual: 13, expected: 42), Issue.Kind.errorCaught(NSError(domain: "Domain", code: 13, userInfo: ["UserInfoKey": "UserInfoValue"])), Issue.Kind.expectationFailed(Expectation(evaluatedExpression: .__fromSyntaxNode("abc"), isPassing: true, isRequired: true, sourceLocation: #_sourceLocation)), Issue.Kind.knownIssueNotRecorded,