Using xcodebuild's `-skipMacroValidation` gives unclear results

[czottmann]

xcsift 1.0.23.

When macros were changed and have approvals pending, xcsift will correctly report that, and mark the output as "status: failed", which is good:

❮ xcodebuild test -scheme "Listening Post" -destination "platform=macOS" 
status: failed
summary:
  errors: 2
  warnings: 0
  failed_tests: 0
  linker_errors: 0
errors[2]{file,line,message}:
  "/Users/czottmann/Library/Developer/Xcode/DerivedData/Listening_Post-egmhdlujnlujagfxqhuwzgrpivyu/SourcePackages/checkouts/swift-case-paths/Package@swift-6.0.swift:PACKAGE-TARGET:CasePathsMacros",null,Macro “CasePathsMacros” from package “swift-case-paths” was changed since a previous approval and must be enabled before it can be used
  "/Users/czottmann/Library/Developer/Xcode/DerivedData/Listening_Post-egmhdlujnlujagfxqhuwzgrpivyu/SourcePackages/checkouts/swift-dependencies/Package.swift:PACKAGE-TARGET:DependenciesMacrosPlugin",null,Macro “DependenciesMacrosPlugin” from package “swift-dependencies” was changed since a previous approval and must be enabled before it can be used

However, when I used xcodebuild's -skipMacroValidation flag while testing, the macro errors disappear (as they should) but the run is still marked as "status: failed", which is confusing my LLM to no end:

❮ xcodebuild test -scheme "Listening Post" -destination "platform=macOS" -skipMacroValidation
status: failed
summary:
  errors: 0
  warnings: 0
  failed_tests: 0
  linker_errors: 0

Thanks for xcsift! 💛

[alexey1312]

Thanks for reporting this!
To investigate this issue properly, could you provide the full raw xcodebuild output (without piping through xcsift)?

[czottmann]

Thanks, @alexey1312! I'm not comfortable w/ posting it publicly. Can I email it over?

[alexey1312]

Sure! You can email the full xcodebuild output to akakoulin.dev@gmail.com

Please make sure to remove any sensitive information from the logs before sending (project names, file paths with usernames, internal dependencies, etc.).

You can replace them with placeholders like
MyProject, /path/to/project, etc.

Thanks for helping to debug this!

[czottmann]

Done, thanks!

[czottmann]

Addendum: I dug a bit deeper, and it appears some tests are actually failing in the "status: failed" runs, yet xcsift listed 0 issues. So, I'm not sure anymore it's about the skipped macro validation.

Here's a log excerpt from the xcodebuild test output:

Testing started
Test suite 'OutboxPersistenceTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'OutboxPersistenceTests.testAllEntriesTerminalWithAllCompleted()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.541 seconds)
Test case 'OutboxPersistenceTests.testAllEntriesTerminalWithMixedStates()' passed on 'My Mac - 
Listening Post (Dev) (88125)' (0.041 seconds)
Test case 'OutboxPersistenceTests.testAllEntriesTerminalWithNoEntries()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.047 seconds)
Test case 'OutboxPersistenceTests.testAllEntriesTerminalWithPendingEntry()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.012 seconds)
Test case 'OutboxPersistenceTests.testDeleteForTrackRemovesAllEntries()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.010 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingExcludesCompletedEntries()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.035 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingExcludesEntriesBeforeDelayWindow()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.009 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingExcludesFailedEntries()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.013 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingExcludesInFlightEntries()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.014 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingIncludesEntriesAfterDelayWindow()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.109 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingIncludesEntriesAtDelayBoundary()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.014 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingReturnsOldestFirst()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.010 seconds)
Test case 'OutboxPersistenceTests.testFetchPendingReturnsPendingEntries()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.009 seconds)
Test case 'OutboxPersistenceTests.testMarkCompletedSetsTerminalState()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.012 seconds)
Test case 'OutboxPersistenceTests.testMarkFailedIncrementsAttempts()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.015 seconds)
Test case 'OutboxPersistenceTests.testMarkFailedWithMaxAttemptsSetsFailed()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.015 seconds)
Test case 'OutboxPersistenceTests.testMarkInFlightUpdatesStatus()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.011 seconds)
Test case 'OutboxPersistenceTests.testResetOrphanedInFlightOnlyResetsInFlight()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.014 seconds)
Test case 'OutboxPersistenceTests.testResetOrphanedInFlightResetsToPending()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.016 seconds)
Test suite 'MenuBarFeatureTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'MenuBarFeatureTests.testDeleteTrackRemovesFromState()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.014 seconds)
Test case 'MenuBarFeatureTests.testForgetTrackTriggersRefresh()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.012 seconds)
Test case 'MenuBarFeatureTests.testLikeTrackTriggersRefresh()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.029 seconds)
Test case 'MenuBarFeatureTests.testLoadRecentTracksFailure()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.010 seconds)
Test suite 'LastFMAuthFeatureTests' started on 'My Mac - Listening Post (Dev) (88131)'
Test case 'MenuBarFeatureTests.testLoadRecentTracksSuccess()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.109 seconds)
Test case 'LastFMAuthFeatureTests.testCancelAuthClickedResetsToIdle()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.111 seconds)
Test suite 'PublishingServiceTests' started on 'My Mac - Listening Post (Dev) (88132)'
Test case 'MenuBarFeatureTests.testRefreshSendsLoadRecentTracks()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.112 seconds)
Test case 'PublishingServiceTests.testProcessEntryCallsChannelWrite()' failed on 'My Mac - Listening Post (Dev) (88132)' (0.070 seconds)
Test case 'PublishingServiceTests.testProcessEntryMarksFailedAfterMaxRetries()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.038 seconds)
Test case 'PublishingServiceTests.testProcessEntryMarksFailedOnChannelError()' failed on 'My Mac - Listening Post (Dev) (88132)' (0.012 seconds)
Test case 'PublishingServiceTests.testProcessEntryMarksInFlightBeforeWrite()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.012 seconds)
Test case 'PublishingServiceTests.testProcessEntryMarksInFlightThenCompleted()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.016 seconds)
Test case 'MenuBarFeatureTests.testUnlikeTrackTriggersRefresh()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.114 seconds)
Test suite 'ErrorMappingTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'ErrorMappingTests.testLastFMMapBadGatewayToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapGatewayTimeoutToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapInternalServerErrorToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapNotConnectedToInternetToNetworkError()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapNotFoundToNetworkError()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapServiceUnavailableToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapTimeoutToNetworkError()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapTooManyRequestsToRateLimitExceeded()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testLastFMMapUnauthorizedToInvalidSessionKey()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorMappingTests.testLastFMNonRetryableErrorsAreCorrectlyMapped()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorMappingTests.testLastFMRetryableErrorsAreCorrectlyMapped()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorMappingTests.testListenBrainzApiErrorIncludesDescription()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.003 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapBadGatewayToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'PublishingServiceTests.testTrackMarkedPublishedWhenAllEntriesComplete()' failed on 'My Mac - Listening Post (Dev) (88132)' (0.050 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapGatewayTimeoutToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapInternalServerErrorToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapNotFoundToApiError()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapServiceUnavailableToServiceOffline()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test suite 'DataModelTests' started on 'My Mac - Listening Post (Dev) (88132)'
Test case 'DataModelTests.testChannelIDEncodingDecoding()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testChannelIDRawValues()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testChannelOutboxColumnsMapToSnakeCase()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'DataModelTests.testChannelOutboxDatabaseTableName()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapTimeoutToApiError()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapTooManyRequestsToRateLimitExceeded()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.014 seconds)
Test case 'ErrorMappingTests.testListenBrainzMapUnauthorizedToInvalidToken()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorMappingTests.testListenBrainzNonRetryableErrorsAreCorrectlyMapped()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorMappingTests.testListenBrainzRetryableErrorsAreCorrectlyMapped()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test suite 'ErrorTypeTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'ErrorTypeTests.testFilesChannelErrorCaseCount()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'DataModelTests.testChannelOutboxJSONRoundTrip()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.002 seconds)
Test case 'DataModelTests.testChannelOutboxStatusEncodingDecoding()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testOutboxStatusRawValues()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'DataModelTests.testRecognizedTrackColumnsMapToSnakeCase()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'DataModelTests.testRecognizedTrackDatabaseTableName()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testRecognizedTrackEquatable()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'DataModelTests.testRecognizedTrackJSONDecodingHandlesNilOptionals()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'ErrorTypeTests.testFilesChannelErrorDescriptions()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testFilesChannelErrorLocalizedDescription()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testFilesChannelErrorUserFriendlyMessages()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'DataModelTests.testRecognizedTrackJSONRoundTrip()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testSentimentValueEncoding()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testTrackEnrichmentColumnsMapToSnakeCase()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'DataModelTests.testTrackEnrichmentDatabaseTableName()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'ErrorTypeTests.testLastFMErrorDescriptionContent()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorTypeTests.testLastFMErrorDescriptions()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testLastFMErrorIsSendable()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorTypeTests.testLastFMErrorLocalizedDescription()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testLastFMErrorNotRetryableForPermanentErrors()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'DataModelTests.testTrackEnrichmentEquatable()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'DataModelTests.testTrackEnrichmentJSONRoundTrip()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.027 seconds)
Test case 'DataModelTests.testTrackEnrichmentWithAllNilOptionals()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testTrackSentimentColumnsMapToSnakeCase()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'ErrorTypeTests.testLastFMErrorRetryableForTransientErrors()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ErrorTypeTests.testListenBrainzErrorDescriptionContent()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testListenBrainzErrorDescriptions()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testListenBrainzErrorIsSendable()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testListenBrainzErrorLocalizedDescription()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testListenBrainzErrorNotRetryableForPermanentErrors()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ErrorTypeTests.testListenBrainzErrorRetryableForTransientErrors()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test suite 'FilesChannelTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'FilesChannelTests.testCSVHeadersAndRowsHaveSameCount()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'DataModelTests.testTrackSentimentDatabaseTableName()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testTrackSentimentJSONRoundTrip()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testTrackStatusEncodingDecoding()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testTrackStatusRawValues()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.002 seconds)
Test case 'DataModelTests.testTrackWithContextCoreAccessors()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.000 seconds)
Test case 'FilesChannelTests.testCSVRowHandlesNilOptionals()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'FilesChannelTests.testCSVRowOrderMatchesHeaders()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'FilesChannelTests.testExportableTrackMapsAllCoreFields()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'FilesChannelTests.testExportableTrackMapsEnrichmentFields()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'FilesChannelTests.testExportableTrackMapsNilEnrichmentFields()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'FilesChannelTests.testExportableTrackMapsSentimentAsIsLiked()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'DataModelTests.testTrackWithContextEnrichmentAccessors()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.034 seconds)
Test case 'DataModelTests.testTrackWithContextNilEnrichmentReturnsNil()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testTrackWithContextOutboxEntryAccessor()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'DataModelTests.testTrackWithContextSentimentAccessor()' passed on 'My Mac - Listening Post (Dev) (88132)' (0.001 seconds)
Test case 'FilesChannelTests.testJSONLEncodingRoundTrip()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'FilesChannelTests.testJSONLEncodingWithNilFields()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'FilesChannelTests.testMonthStringFirstDayOfMonth()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'FilesChannelTests.testMonthStringFormatsCorrectly()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'FilesChannelTests.testMonthStringLastDayOfMonth()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'FilesChannelTests.testMonthStringSingleDigitMonthIsPadded()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.048 seconds)
Test case 'FilesChannelTests.testMonthStringYearBoundary()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test suite 'ChannelAvailabilityTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'ChannelAvailabilityTests.testScrobbleAvailableForDifferentChannel()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleAvailableWhenAllConditionsMet()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleBlockedByCorrectChannelOutbox()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleDisabledOneSecondBeforeDelay()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleDisabledWhenAlreadyScrobbled()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleDisabledWhenInFlightOutboxEntry()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleDisabledWhenNotLicensed()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleDisabledWhenPendingOutboxEntry()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.013 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleDisabledWhenPublishDelayNotPassed()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleEnabledAtExactDelayBoundary()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleEnabledWithCompletedOutboxEntry()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ChannelAvailabilityTests.testScrobbleEnabledWithFailedOutboxEntry()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test suite 'TransformationTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'TransformationTests.testListenAdditionalInfoContainsISRC()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testListenAdditionalInfoSubmissionClient()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testListenEncodesToCorrectJSON()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testListenEncodesWithoutTimestampForNowPlaying()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testListenUsesUnknownForMissingArtist()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'TransformationTests.testListenUsesUnknownForMissingTitle()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'TransformationTests.testListenWithTimestampIncluded()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'TransformationTests.testListenWithTimestampOmitted()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'TransformationTests.testNowPlayingItemHasNoTimestamp()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testNowPlayingItemMapsAllFields()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testNowPlayingItemReturnsNilForMissingArtist()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'TransformationTests.testNowPlayingItemReturnsNilForMissingTitle()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testScrobbleItemMapsAllFields()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testScrobbleItemReturnsNilForMissingArtist()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testScrobbleItemReturnsNilForMissingBoth()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testScrobbleItemReturnsNilForMissingTitle()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'TransformationTests.testScrobbleItemTimestampMatchesRecognizedAt()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test suite 'LastFMSignatureTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'LastFMSignatureTests.testDifferentParamsProduceDifferentSignatures()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMAuthFeatureTests.testCompleteAuthClickedExchangesToken()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.091 seconds)
Test case 'LastFMSignatureTests.testDifferentSecretsProduceDifferentSignatures()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testEmptyParamsSignature()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.002 seconds)
Test case 'LastFMSignatureTests.testEmptyStringValues()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testKnownVector1()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testKnownVector2()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testLongValues()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testMultipleParamsSortedAlphabetically()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.023 seconds)
Test case 'LastFMSignatureTests.testParamsWithNumbers()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testParamsWithSpaces()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testParamsWithSpecialCharacters()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testSignatureIsDeterministic()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testSignatureIsLowercaseHex()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMSignatureTests.testSingleParamSignature()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.017 seconds)
Test suite 'ConfirmationStateMachineTests' started on 'My Mac - Listening Post (Dev) (88125)'
Test case 'ConfirmationStateMachineTests.testConfirmingWithDifferentTrackAtMaxAttemptsResets()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ConfirmationStateMachineTests.testConfirmingWithDifferentTrackSwitchesAndIncrementsAttempts()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'LastFMAuthFeatureTests.testCompleteAuthClickedWithExchangeFailure()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.351 seconds)
Test case 'ConfirmationStateMachineTests.testConfirmingWithSameTrackIncrementsMatchCount()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ConfirmationStateMachineTests.testConfirmingWithSameTrackKeepsConfirmingWhenNotEnoughMatches()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ConfirmationStateMachineTests.testExactlyAtConfidenceThresholdIsAccepted()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ConfirmationStateMachineTests.testIdleWithFirstMatchStartsConfirming()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ConfirmationStateMachineTests.testJustBelowConfidenceThresholdIsRejected()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.000 seconds)
Test case 'ConfirmationStateMachineTests.testLowConfidenceDoesNotChangeConfirmingState()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ConfirmationStateMachineTests.testLowConfidenceMatchIsRejected()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ConfirmationStateMachineTests.testMatchCountReachingRequiredMatchesConfirms()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ConfirmationStateMachineTests.testThreeConsecutiveMatchesWithRequiredThree()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.001 seconds)
Test case 'ConfirmationStateMachineTests.testTimeoutAfterMaxAttemptsWithDifferentTracks()' passed on 'My Mac - Listening Post (Dev) (88125)' (0.024 seconds)
Test case 'LastFMAuthFeatureTests.testCompleteAuthIgnoredWhenNotWaitingForBrowser()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.172 seconds)
Test case 'LastFMAuthFeatureTests.testConnectClickedTriggersTokenRequest()' failed on 'My Mac - Listening Post (Dev) (88131)' (0.368 seconds)
Test case 'LastFMAuthFeatureTests.testConnectClickedWithTokenFailure()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.030 seconds)
Test case 'LastFMAuthFeatureTests.testDisconnectClickedClearsState()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.042 seconds)
Test case 'LastFMAuthFeatureTests.testInitialStateIsIdle()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.001 seconds)
Test case 'LastFMAuthFeatureTests.testIsAuthenticatedProperty()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.000 seconds)
Test case 'LastFMAuthFeatureTests.testNeedsAuthenticationProperty()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.001 seconds)
Test case 'LastFMAuthFeatureTests.testOnAppearWithNoStoredCredentials()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.009 seconds)
Test case 'LastFMAuthFeatureTests.testPollingIgnoredWhenNotWaitingForBrowser()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.055 seconds)
Test case 'LastFMAuthFeatureTests.testPollTickCancelledWhenNotWaitingForBrowser()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.008 seconds)
Test case 'LastFMAuthFeatureTests.testPollTickFailureContinuesPolling()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.087 seconds)
Test case 'LastFMAuthFeatureTests.testPollTickSuccessTriggersSessionReceived()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.049 seconds)
Test case 'LastFMAuthFeatureTests.testTestConnectionResultWithInvalidSession()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.140 seconds)
Test case 'LastFMAuthFeatureTests.testTestConnectionResultWithValidSession()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.108 seconds)
Test case 'LastFMAuthFeatureTests.testTestConnectionWithValidSession()' passed on 'My Mac - Listening Post (Dev) (88131)' (0.011 seconds)
2026-01-21 16:16:56.497 xcodebuild[87403:17414379] [MT] IDETestOperationsObserverDebug: 15.694 elapsed -- Testing started completed.
2026-01-21 16:16:56.497 xcodebuild[87403:17414379] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2026-01-21 16:16:56.497 xcodebuild[87403:17414379] [MT] IDETestOperationsObserverDebug: 15.694 sec, +15.694 sec -- end

xcsift interprets it as such:

status: failed
summary:
  errors: 0
  warnings: 0
  failed_tests: 0
  linker_errors: 0

[alexey1312]

Both issues are fixed in PR #53:

1. Test case parsing - Now handles lowercase "Test case" format used by parallel/multi-destination testing (previously only matched "Test Case")

2. Coverage target extraction - Fixed extractTestedTarget() to handle lowercase "Test suite" format used in parallel testing output (previously only matched "Test Suite")