Swift Testing aren't failing

[ekamalov]

Description

Running tests with TestStore using Swift Testing aren't failing.

Checklist

• I have determined whether this bug is also reproducible in a vanilla SwiftUI project.
• If possible, I've reproduced the issue using the main branch of this package.
• This issue hasn't been addressed in an existing GitHub issue or discussion.

Expected behavior

Demo reducer

@Reducer
struct CounterFeature {
  @ObservableState
  struct State: Equatable {
    var count = 0
  }

  enum Action {
    case decrementButtonTapped
    case incrementButtonTapped
  }

  var body: some Reducer<State, Action> {
    Reduce { state, action in
      switch action {
      case .decrementButtonTapped:
        state.count -= 1
        return .none
      case .incrementButtonTapped:
        state.count += 1
        return .none
      }
    }
  }
}

struct DemoTests {
    @Test
    func example() async {
        let store = await TestStore(initialState: CounterFeature.State()) {
             CounterFeature()
           }
        await store.send(.incrementButtonTapped) {
           $0.count = 0
         }
         await store.send(.decrementButtonTapped) {
           $0.count = 0
         }
    }
}

import XCTest

@MainActor
final class CounterFeatureTests: XCTestCase {
  func testCounter() async {
        let store = TestStore(initialState: CounterFeature.State()) {
            CounterFeature()
        }
        await store.send(.incrementButtonTapped) {
            $0.count = 0
        }
        await store.send(.decrementButtonTapped) {
            $0.count = 0
        }
    }
    
}

added testing with XCTest and swift testing

Actual behavior

It was expected that both tests would fail.
However, the results of Swift testing were successful in all cases.
XCTest is working correctly.

Steps to reproduce

No response

The Composable Architecture version information

1.14.0

Destination operating system

iOS 17

Xcode version information

Version 16.0 beta 6 (16A5230g)

Swift Compiler version information

swift-driver version: 1.115 Apple Swift version 6.0 (swiftlang-6.0.0.9.10 clang-1600.0.26.2)
Target: arm64-apple-macosx15.0

[mbrandonw]

Hi @ekamalov, I am not able to reproduce this. It seems to work just fine in Xcode 16 beta 6:

Can you provide a project that demonstrates the problem you are seeing? Also, by any chance are you explicitly depending on swift-testing rather than using the version that ships with Xcode? We have seen that cause problems before.

Since I don't believe this is an issue with the library I am going to convert this to a discussion. Please feel free to continue the conversation over there.

[ekamalov]

maybe the problem was in xcode, cleared all xcode caches, reinstalled and started working properly.

Thanks for the quick response.

[mmllr]

@ekamalov I had the same problem and found the issue going back to https://github.com/pointfreeco/swift-issue-reporting/blob/3fcc3f21695ad5bb889a024b1b046d61bebb1ef3/Sources/IssueReporting/Internal/SwiftTesting.swift#L61 (it could not resolve the method and then no issue was recorded). My first attempt to fix was to add the IssueReportingTestSupport target to the IssueReporting target but it turned out that the root of the problem was in the Xcode project (it was created with Xcode 14). After performing the suggested "Update to recommended settings" in Xcode 16 (select the target and then the suggestion warning bubble appears) the problem was fixed and the TCA tests where reporting errors with Swift Testing.

[ekamalov]

Hi @mbrandonw, I encountered another problem with Swift testing. The problem is that when I run the group test AuthVerifyOTPTests, one of the tests always fails. However, when I run each test individually, everything works fine. XCTest works.

@Reducer
public struct AuthVerifyOTP: Sendable {
    public init() { }
    
    @ObservableState
    public struct State: Equatable {
        var isTimerActive = false
        var currentCountDown: Int
        
        let countDown: Int
        
        public init(countDown: Int = 0) {
            self.countDown = countDown
            self.currentCountDown = countDown
        }
    }
    
    public enum Action: ViewAction {
        // MARK: - View Actions
        case view(View)
        
        @CasePathable
        public enum View {
            case onDisappear
            case onAppear
            case resendOTP
        }
        
        case timerTicked
        case timerStart
        case timerReset
        case timerStop
        
    }
    
    @Dependency(\.continuousClock) var clock
    private enum CancelID { case timer }
    
    public var body: some ReducerOf<Self> {
        self.core
    }
    
    var core: some ReducerOf<Self> {
        Reduce { state, action in
            switch action {
                    // MARK: - ViewActions
                case .view(.onAppear):
                    return .send(.timerStart)
                case .view(.onDisappear):
                    return .none
                case .view(.resendOTP):
                    return .run { send in
                        
                        await send(.timerReset)
                    }
                case .view:
                    return .none
                    // MARK: - Local
                case .timerTicked:
                    state.currentCountDown -= 1
                    if state.currentCountDown <= 0 {
                        return .send(.timerStop)
                    }
                    return .none
                case .timerStart:
                    state.currentCountDown = state.countDown
                    state.isTimerActive = true
                    return .run { [isTimerActive = state.isTimerActive] send in
                        guard isTimerActive else { return }
                        for await _ in self.clock.timer(interval: .seconds(1)) {
                            await send(.timerTicked, animation: .interpolatingSpring(stiffness: 3000, damping: 40))
                        }
                    }.cancellable(id: CancelID.timer, cancelInFlight: true)
                case .timerReset:
                    return .send(.timerStart)
                case .timerStop:
                    state.isTimerActive = false
                    return .cancel(id: CancelID.timer)
            }
        }
    }
}

@Suite
struct AuthVerifyOTPTests {
    
    @Test
    func testTimerStopsWhenCounterReachesZero() async {
        let clock = TestClock()
        
        let store = await TestStore(initialState: AuthVerifyOTP.State(countDown: 2)) {
            AuthVerifyOTP()
        } withDependencies: {
            $0.continuousClock = clock
        }
        
        await store.send(\.view.onAppear)
        
        await store.receive(\.timerStart) {
            $0.isTimerActive = true
        }
        
        await clock.advance(by: .seconds(2))
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 1
        }
        
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 0
        }
        
        await store.receive(\.timerStop) {
            $0.isTimerActive = false
        }
    }

    @Test
    func testResendOTP() async {
        let clock = TestClock()
        
        let store = await TestStore(initialState: AuthVerifyOTP.State(countDown: 1)) {
            AuthVerifyOTP()
        } withDependencies: {
            $0.continuousClock = clock
        }
        
        await store.send(\.view.onAppear)
        
        await store.receive(\.timerStart) {
            $0.isTimerActive = true
            $0.currentCountDown = 1
        }
        
        await clock.advance(by: .seconds(1))
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 0
        }
        
        await store.receive(\.timerStop) {
            $0.isTimerActive = false
        }
        
        await store.send(\.view.resendOTP)
        
        await store.receive(\.timerReset)
        await store.receive(\.timerStart) {
            $0.isTimerActive = true
            $0.currentCountDown = $0.countDown
        }
        
        await clock.advance(by: .seconds(1))
        
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 0
        }
        
        await store.receive(\.timerStop) {
            $0.isTimerActive = false
        }
    }

}

import XCTest

final class aaa: XCTestCase {
//    @Test
    func testTimerStopsWhenCounterReachesZero() async  {
        let clock = TestClock()
        
        let store = await TestStore(initialState: AuthVerifyOTP.State(countDown: 2)) {
            AuthVerifyOTP()
        } withDependencies: {
            $0.continuousClock = clock
        }
        
        await store.send(\.view.onAppear)
        
        await store.receive(\.timerStart) {
            $0.isTimerActive = true
        }
        
        await clock.advance(by: .seconds(2))
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 1
        }
        
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 0
        }
        
        await store.receive(\.timerStop) {
            $0.isTimerActive = false
        }
    }

//    @Test
    func testResendOTP() async  {
        let clock = TestClock()
        
        let store = await TestStore(initialState: AuthVerifyOTP.State(countDown: 1)) {
            AuthVerifyOTP()
        } withDependencies: {
            $0.continuousClock = clock
        }
        
        await store.send(\.view.onAppear)
        
        await store.receive(\.timerStart) {
            $0.isTimerActive = true
            $0.currentCountDown = 1
        }
        
        await clock.advance(by: .seconds(1))
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 0
        }
        
        await store.receive(\.timerStop) {
            $0.isTimerActive = false
        }
        
        await store.send(\.view.resendOTP)
        
        await store.receive(\.timerReset)
        await store.receive(\.timerStart) {
            $0.isTimerActive = true
            $0.currentCountDown = $0.countDown
        }
        
        await clock.advance(by: .seconds(1))
        
        await store.receive(\.timerTicked) {
            $0.currentCountDown = 0
        }
        
        await store.receive(\.timerStop) {
            $0.isTimerActive = false
        }
    }
}

[mbrandonw]

Hi @ekamalov, this is most likely happening because dependencies are bleeding out from one test to another while they run in parallel. The best tool to solve this problem is Testing's CustomExecutionTrait, but Apple still has not officially released it (see here). XCTest doesn't have this problem because parallel tests run in completely different processes.

Luckily we do have a fix for this, even without CustomExecutionTrait, and we plan on merging and releasing it soon. In the meantime, you can either run your tests in .serialized or wrap your test in withDependencies as described in these docs.

[ekamalov]

for some reason it doesn't help, or I'm doing something wrong. You can check it out, please.

[mbrandonw]

Hi @ekamalov, this is a good catch. We need to bucket cancel IDs for effects by test ID like we are doing in swift-dependencies. We will have a fix soon. Thanks!