Acceptance Test Suite for verifying supported configurations #190
Description
Currently, breaking the setup flow for Nimble is easy because of the many supported installation methods in various configurations:
Platforms - Supporting latest and latest - 1 when possible; Assumes the same version of Xcode can be used. Simulator-only is acceptable.
- iOS (7, 8, 9) - iOS 7 is incidentally supported in simulator only by setting min version target
- macOS (10.9, 10.10, 10.11) - Currently we can only cover one due to CI limitations
- tvOS (1)
- Linux
- macOS without Objective-C Runtime (SPM on mac)
- watchOS (Unverified)
Subject Target - The binary target that's being tested. Static libraries and CLI is currently unsupported. Build-processes differ between both targets that may break specific configurations.
- None (only tests code in the target target)
- Application Bundle
- Dynamic Framework - Assume restricted extensions API environment
Test Target - The target that Nimble assertions reside in. This target links directly to Nimble. CLI may be a future target.
- App Bundle - Since Nimble supports linking without XCTest
- Unit Test Bundle
- UI Test Bundle - only applies to app bundles?
Installation Method - How end-users install Nimble into their projects
- Embed as subproject (submodule)
- Cocoapods
- Carthage
- Swift Package Manager
Configurations - How end-users / consumers configure Nimble
- Linking with/without XCTest
- SwiftLint is/isn't installed
This is a combinatorial explosion that's difficult to ensure we don't break anything. Not all combinations are possible (UI Test Bundle w/o XCTest is impossible), but there's still a lot of possible configurations. And it will only get worse as time progresses.
A good first-step is to have an acceptance test suite that pulls in the latest commit of Nimble and runs sample projects in each of those configurations. I definitely don't expect we'll get this coverage with one large PR, but it doesn't hurt to build this over time.
To be clear, these sample projects should simply check that Nimble can be linked + run with those sample projects. There shouldn't be any significant assertions other than running some Nimble code to assert that Nimble successfully links and runs in the configuration.
Question for @Quick/contributors: should this be inside the Nimble project itself or as a separate project under the Quick org? A separate project would require triggering another build as needed (which is a bit more infrastructure work to setup), although I'm leaning towards that.
Some common configurations we should support (in order of importance):
- iOS 9; App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
- iOS 9; App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
- iOS 9; Framework; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
- iOS 9; Framework; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
- iOS 9; App bundle; UI Test Bundle; Cocoapods; with XCTest; with SwiftLint
- iOS 9; App bundle; UI Test Bundle; Carthage; with XCTest; with SwiftLint
- macOS (10.11); App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
- macOS (10.11); App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
- macOS (10.11); Framework; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
- macOS (10.11); Framework; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
- macOS (10.11); App bundle; UI Test Bundle; Cocoapods; with XCTest; with SwiftLint
- macOS (10.11); App bundle; UI Test Bundle; Carthage; with XCTest; with SwiftLint
- macOS (10.11); SwiftPM; with XCTest
- watchOS 2; App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
- watchOS 2; App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
- tvOS; App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
- tvOS; App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
- tvOS; App bundle; UI Test Bundle; Cocoapods; with XCTest; with SwiftLint
- tvOS; App bundle; UI Test Bundle; Carthage; with XCTest; with SwiftLint
- Linux; SwiftPM; with XCTest
- iOS 9; App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
- iOS 9; App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
- iOS 9; Framework; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
- iOS 9; Framework; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
- iOS 9; App bundle; UI Test Bundle; Cocoapods; with XCTest; without SwiftLint
- iOS 9; App bundle; UI Test Bundle; Carthage; with XCTest; without SwiftLint
- macOS (10.11); App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
- macOS (10.11); App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
- macOS (10.11); Framework; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
- macOS (10.11); Framework; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
- macOS (10.11); App bundle; UI Test Bundle; Cocoapods; with XCTest; without SwiftLint
- macOS (10.11); App bundle; UI Test Bundle; Carthage; with XCTest; without SwiftLint
- watchOS 2; App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
- watchOS 2; App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
- tvOS; App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
- tvOS; App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
- tvOS; App bundle; UI Test Bundle; Cocoapods; with XCTest; without SwiftLint
- tvOS; App bundle; UI Test Bundle; Carthage; with XCTest; without SwiftLint