TestStore.send not asserting properly

[mattdornfeld]

Description

I'm running TCA 0.42.0. I'm trying to write tests for my reducer. To make sure things are functioning properly I'm trying to intentionally write a failing test. I followed the examples in the documentation and here is my minimum reproducing example

import ComposableArchitecture
import Foundation
import XCTest

@testable import AppSource

@MainActor
class Test: XCTestCase {
    func test() async {
        let store = ComposableArchitecture.TestStore(
            initialState: PostViewReducer.State(),
            reducer: PostViewReducer()
        )
        
        print(store.state)
        await store.send(.loadCommentsPage) {
            $0.shouldLoadCommentsPage = false
        }
        print(store.state)
    }
}

And here are the logs from running the test

Test Suite 'Selected tests' started at 2022-10-17 02:20:46.431
Test Suite 'test_unit.xctest' started at 2022-10-17 02:20:46.432
Test Suite 'Test' started at 2022-10-17 02:20:46.432
Test Case '-[AppTests.Test test]' started.
State(shouldLoadCommentsPage: false)
State(shouldLoadCommentsPage: true)
Test Case '-[AppTests.Test test]' passed (0.016 seconds).
Test Suite 'Test' passed at 2022-10-17 02:20:46.449.
	 Executed 1 test, with 0 failures (0 unexpected) in 0.016 (0.017) seconds
Test Suite 'test_unit.xctest' passed at 2022-10-17 02:20:46.450.
	 Executed 1 test, with 0 failures (0 unexpected) in 0.016 (0.017) seconds
Test Suite 'Selected tests' passed at 2022-10-17 02:20:46.450.
	 Executed 1 test, with 0 failures (0 unexpected) in 0.016 (0.019) seconds
Program ended with exit code: 0

Checklist

• If possible, I've reproduced the issue using the main branch of this package.
• This issue hasn't been addressed in an existing GitHub issue or discussion.

Expected behavior

I inserted the two log statements to confirm the state is updated correctly. Initially shouldLoadCommentsPage is false. I sent the loadCommentsPage action and afterwards it's true. However I'm passing $0.shouldLoadCommentsPage = false into with the closure that's passed to send. I expect this to make the test fail since shouldLoadCommentsPage is updated to true. However the test seems to finish without any problems.

Actual behavior

The test finishes without any errors.

Steps to reproduce

No response

The Composable Architecture version information

0.42.0

Destination operating system

ios 15

Xcode version information

XCode 14

Swift Compiler version information

swift-driver version: 1.62.8 Apple Swift version 5.7 (swiftlang-5.7.0.127.4 clang-1400.0.29.50)
Target: x86_64-apple-macosx12.0

[mbrandonw]

It's hard to know what is going on without seeing some code. Can you share something that we can run locally to reproduce?

The only guess I have without seeing more code is that perhaps you are using a reference type for your state? Reference types have instance/identity equality and are not equal based on just the data they hold.

[mattdornfeld]

Sure, here's a simplified version of the ReducerProtocol I'm using. It has the same issue as the full one

import ComposableArchitecture

struct PostViewReducer: ReducerProtocol {
    struct State: Equatable {
        var shouldLoadCommentsPage: Bool = false
    }
    
    enum Action: Equatable {
        case loadCommentsPage
        case unloadCommentsPage
    }
    
    func reduce(into state: inout State, action: Action) -> Effect<Action, Never> {
        switch action {
        case .loadCommentsPage:
            state.shouldLoadCommentsPage = true
            return .none
        case .unloadCommentsPage:
            state.shouldLoadCommentsPage = false
            return .none
        }
    }
}

[tgrapperon]

Hey @mattdornfeld!
I'm not seeing anything wrong. If I run your example, it prints:

State(shouldLoadCommentsPage: false)

error: A state change does not match expectation: …

    − PostViewReducer.State(shouldLoadCommentsPage: false)
    + PostViewReducer.State(shouldLoadCommentsPage: true)

(Expected: −, Actual: +)

State(shouldLoadCommentsPage: true)

So it fails because shouldLoadCommentsPage is true, which is coherent with the logic and what's printed by your statements. If I update the expectation to shouldLoadCommentsPage = true, the test passes, again as expected.

If you can't reproduce, that's indeed very odd. I would probably try to run Clean Build Folder and try again in any case your app would be running incoherent data. Please let us know.

Here is the test I ran:

import XCTest
import ComposableArchitecture

struct PostViewReducer: ReducerProtocol {
  struct State: Equatable {
    var shouldLoadCommentsPage: Bool = false
  }

  enum Action: Equatable {
    case loadCommentsPage
    case unloadCommentsPage
  }

  func reduce(into state: inout State, action: Action) -> Effect<Action, Never> {
    switch action {
    case .loadCommentsPage:
      state.shouldLoadCommentsPage = true
      return .none
    case .unloadCommentsPage:
      state.shouldLoadCommentsPage = false
      return .none
    }
  }
}

@MainActor
class TestTestStoreIssue: XCTestCase {
  func test() async {
    let store = TestStore(
      initialState: PostViewReducer.State(),
      reducer: PostViewReducer()
    )

    print(store.state)
    await store.send(.loadCommentsPage) {
      $0.shouldLoadCommentsPage = false
    }
    print(store.state)
  }
}

[mbrandonw]

Hi @mattdornfeld, I'm going to move this to a discussion because I do not believe it is a bug in the library considering we have a ton of test coverage on this area of the library and have never witnessed this before. If you are still having trouble please feel free to share some code so that we can help diagnose the problem.

[Czajnikowski]

Okay, I have the same issue I think, something really weird is going on.

I can take this:

@MainActor
class ToastsReducerTests: XCTestCase {
    func testUndo() async {
        let store = TestStore(
            initialState: 1,
            reducer: Reducer<Int, String, Void> { state, action, _ in
                state += 1
                return .none
            },
            environment: ()
        )
        
        _ = await store.send("yo") { _ in }
    }
}

And now - when I stick it into my project it's ✅ (incorrect), when I stick it into TCA sample project its 🚫 (correct).

What the hell does that mean 😝? I know... It means that there is something specific to my (or our @mattdornfeld ) projects that really do not play well. But what's that 🤔🤔🤔?

To make things even funnier - my current logic is 100% TDD - I swear it was working before 😅

[Czajnikowski]

Oh, I also tried adding an ObjC file with a #define DEBUG - nope 🤐

[Czajnikowski]

Okay, so if the problem is that we can't/should not have XCT* in the framework then maybe we should really keep it outside 🙂?

I mean, I tried this super rough sketch, just to demonstrate an idea, and it works:

@MainActor
class ToastsReducerTests: XCTestCase {
    func testUndo() async {
        let store = TestStore(
            initialState: 1,
            reducer: Reducer<Int, String, Void> { state, action, _ in
                state += 1
                return .none
            },
            environment: ()
        )
        
        ComposableArchitecture.fail = { XCTFail("yo") }
        _ = await store.send("yo") { _ in }
    }
}

There are many options to solve the injection... What do you think guys?

[mbrandonw]

I'm not sure I understand how this is supposed to work. It is expected of people to manually override fail on all of their tests? And XCTFail isn't the only testing tool used inside TestStore. Soon it will also use XCTExpectFailure, which means that too would have to be injected?

It's certainly interesting, but I think you need to push it a bit further before we can consider it a viable alternative.

[Czajnikowski]

I'll push sth later today 🤙 The idea is to inject sth in the shape of Environment where we have closures that correspond to the XCTest APIs. And use them instead of XC*s

[Czajnikowski]

Okay, I pushed a bit more. Gosh... I didn't know that the codebase is so big 🥵
Main things, concerns:

• removed XCTestDynamicOverlay in favor of injection
• Working solution can be seen in AnimationTests (I pushed some code that makes tests red)
• Some tests still fail (related to runtimeWarn and maybe sth more?)
• Such pattern would likely have to be propagated to other dependencies, we'd rather have to have a func setXCTestFail and similar to handle that (for now I just brute-force commented out all the XCTFails in my local clones)
• I didn't have too much time to think about the big picture, more relying on some of you guys, who know the code a bit better than I. Looking forward to your judgment - if it even makes any sense or not, maybe there are some roadblocks ahead that I don't see yet?
• It's not a perfect solution for sure

Sorry for a wip/dirty code/indentation, but I didn't want to optimize prematurely 🙏