feat: matcher isolation trait

[Archdoog]

• Adds MatcherTrait for Swift Testing concurrency isolation.

Inspired by: https://github.com/hmlongco/Factory/blob/bcca76f9243ace59477c8d077d25a97795eab5c4/Sources/FactoryTesting/ContainerTrait.swift

[Archdoog]

@Kolos65 Same with this. If the linux stuff now passes, feel free to fix any remaining commit message issues and tweak the code if you see any issues.

[Kolos65]

Thanks for this as well @Archdoog!

I like the idea of the referenced ContainerTrait, but I'm not entirely sure about translating that to Matcher. I think the important question is this: Are matchers more often unique per test case / test suite or are they more global across a project? I think most of the time you would want your custom matchers to be available everywhere across the test suite and require a single setup point (just like a conformance to Equatable). This design was inspired from SwiftyMocky where it proved to be robust.

[Archdoog]

Thanks for this as well @Archdoog!

I like the idea of the referenced ContainerTrait, but I'm not entirely sure about translating that to Matcher. I think the important question is this: Are matchers more often unique per test case / test suite or are they more global across a project? I think most of the time you would want your custom matchers to be available everywhere across the test suite and require a single setup point (just like a conformance to Equatable). This design was inspired from SwiftyMocky where it proved to be robust.

That's a fair assessment and historically would've been totally correct. I think this is less of a feature question and more of a stability requirement for modern swift (modern concurrency and swift testing). Ironically, I actually have a single custom matcher setup function that's used for my entire testing environment. This was the end result of a few days debugging random CI crashes caused by the Matcher fatalError for missing type.

In summary, with Swift Testing's concurrency, this is basically a requirement to ensure reliably isolated tests. Without it, anyone using Mockable with swift testing and custom matcher calls will be stuck with singleton races causing flaky and inconsistent test crashes.

Deeper dive

There are multiple reasons for and notes about this:

1. Swift testing uses standard type lifecycles. There are no speciality setUp and tearDown methods that have bespoke lifecycles. In a testing target this means your minimum setup allowance is calling your custom matcher setup once per struct/@suite instead of once per test or once per full test target run.
2. Because of item 1, you cannot control when another test struct calls your setup (or the reset function). This means you will have race conditions. These occur when different test structs init and set up the Matcher at the exact moment another struct is expects to access a fully configured Matcher. In this case, the in process struct will crash with the fatalError. This can even happen for the default types included in the Matcher's default setup functions. Because another test can re-call the function at any moment during another test's execution.
3. The only time item 2 isn't a constant problem is if you don't customize the Matcher. This is because the first singleton init is truly the only time when setup can be called once per test target run.
4. @Suite(.serialized) for a single struct is not enough. This is because singletons scope the whole target. The only potential workaround here would be if could create a single @Suite(.serialized) for your entire test target (think one struct for all of your tests or if a single @Suite(.serialized) could be used across all tests/structs).

Before this, I was able to run my ~1500 unit test suite on repeat on my MacBook Pro thousands of times. Failure would occur in 1 or 2 tests occasionally. However, in a resource constrained CI environment, I was getting 20-50 tests every run that would crash and kill the entire run. Each time they were different tests and the missing matched type was either one of my own or one of the defaults.

Lastly, there's not really a drawback to this solution. @TaskLocal basically lets you assign a replacement to the singleton if you desire using .withValue. Without @Suite(.matcher) or without calling .withValue in another situation, you'll get the single memory address just like you do now. Meaning you can keep running older XCTest suites without any issue.

As an aside, it looks like SwiftMocky might encounter similar issues 😅. Looks like they have some old issues about parallelization (e.g. MakeAWishFoundation/SwiftyMocky#229). These seem likely to be a much larger problem as more teams realize the other benefits of swift testing (one of which is massive CI speed improvements).

Happy to answer any more questions here. I've actually got a keynote file about modern concurrency that I need to reformat and publish somewhere. There are so many subtle difference between modern concurrency and the old swift 5 stuff that make it clear why some of these new solutions exist and the problem(s) they solve.

[Archdoog]

One final note, the unit test that verifies this is making use of an anti-pattern just to verify that Matcher.current was actually scoped per test isolation.