symbols not found for OrderedCollections.OrderedSet

[ferologics]

Describe the bug
While bumping the version of the TCA package from 0.20.0 to 0.25.1 we encountered an error with following console error:

"Undefined symbols for architecture arm64:
  "nominal type descriptor for OrderedCollections.OrderedSet" ... 

(find full log here and screenshot here)

For some reason the SPM linked dependency inside the TCA's @_exported import IdentifiedCollections fails to find "Swift Collections" package's OrderedCollections.OrderedSet symbols.

To Reproduce

1. Download the repro project from here
2. Open the project with latest Xcode 12
3. Build the “TCA_Collections (iOS)” target for “Any iOS Device (arm64)"
4. Observe failed build with described error

Expected behavior
A successful build.

Environment

• swift-composable-architecture version 0.25.1
• Xcode Version 12.5.1 (12E507)
• Apple Swift version 5.4.2 (swiftlang-1205.0.28.2 clang-1205.0.19.57) Target: x86_64-apple-darwin20.6.0
• macOS Big Sur 11.5.1 (20G80)

FYI: the issue reproduces on latest macOS Monterey Xcode 13 beta.

Additional context
While attempting to reproduce in separate project we noticed that the issue with symbols comes up specifically in framework targets. Using the "IdentifiedArray" in the main app target builds successfully. We annotate this in the code with a comment above the "foo" variable.

[stephencelis]

@ferologics Bummer! Though this does sounds like an Apple bug, especially if changing your project structure fixes, doesn't it? Or do you think we could work around it somehow?

If I understand correctly, it also sounds like you have a workaround in avoiding the framework and organizing things into SPM packages that can be directly embedded in your application.

And I wonder if it could also be worked around by explicitly depending on swift-identified-collections and/or explicitly importing IdentifiedCollections where you need it?

Have you filed feedback with Apple?

[ferologics]

Though this does sounds like an Apple bug, especially if changing your project structure fixes, doesn't it? Or do you think we could work around it somehow?

I also believe it's an Apple bug, unless ofc there's some special SPM flag that can be added to the package to make it magically work, which again, seems unlikely.

If I understand correctly, it also sounds like you have a workaround in avoiding the framework and organizing things into SPM packages that can be directly embedded in your application.

Not a workaround (although possibly?), more of an observation that the issue comes up once IdentifiedArray is used in a framework target, while in the main app target there's no issues with finding OrderedCollections.OrderedSet symbols.

And I wonder if it could also be worked around by explicitly depending on swift-identified-collections and/or explicitly importing IdentifiedCollections where you need it?

I already tired this and that the other day, but nether helped. Will try to do this again though, maybe something gives.

Have you filed feedback with Apple?

I submitted an Apple TSI earlier last week, only to get a (paraphrasing): "this isn't an issue with Apple / Xcode / SPM but with TCA, please resolve it with them, not our problem.". I asked them to reconsider this assessment, since to us it looks more on the Xcode / SPM side of things.

What's strange is that you didn't encounter this issue with Isowords. If I'm not mistaken behind the scenes SPM resolves multiply-linked static frameworks into dynamic ones, which in theory should trigger this bug, unless it's an SPM dependency graph issue with "Xcode generated dynamic frameworks".

[stephencelis]

I submitted an Apple TSI earlier last week, only to get a (paraphrasing): "this isn't an issue with Apple / Xcode / SPM but with TCA, please resolve it with them, not our problem.". I asked them to reconsider this assessment, since to us it looks more on the Xcode / SPM side of things.

Can you follow up and ask them how this is an issue with TCA and how we may resolve it? If it's a matter of removing @_exported we'll consider it, as it's a "private" feature, though plenty of other libraries out there (like Vapor) use it.

What's strange is that you didn't encounter this issue with Isowords. If I'm not mistaken behind the scenes SPM resolves multiply-linked static frameworks into dynamic ones, which in theory should trigger this bug, unless it's an SPM dependency graph issue with "Xcode generated dynamic frameworks".

We haven't done a release since introducing IdentifiedCollections, but we also use a single static library and do not use dynamic frameworks at all.

[stephencelis]

@ferologics As a follow up, we were able to archive/submit isowords with the latest TCA with no errors, so it appears that if you leverage SPM totally and avoid frameworks you may be able to work around the problem

[stephencelis]

Going to add the "Apple Bug" label on this for now till we know it's something we can fix.

[ferologics]

Thanks @stephencelis. I will update if/when they respond on the TSI email thread. In the meantime the 0.20.0 version works well for us.

if you leverage SPM totally and avoid frameworks you may be able to work around the problem

Unfortunately we can't rely on SPM solely because some of our dependencies don't support it (mixed Swift/Obj-C 🤷).

Also, I also tried naively adding swift-collections to the dependencies in TCA's package manifest, but that doesn't seem to give SPM the much need hint. 😕

[ferologics]

I took another look at the console output of the failed clang command and broke it down a bit. After a bit of digging I found that inserting the path to OrderedCollections_1222128174265B56_PackageProduct linked package arguments makes the command succeed (no errors are thrown). I annotated this line with the "👉" emoji bellow. (NB: here's a link to the command with aforementioned correction)

Ld /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct.framework/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct normal (in target 'ComposableArchitecture' from project 'swift-composable-architecture')
    cd /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/SourcePackages/checkouts/swift-composable-architecture

    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -target arm64-apple-ios13.0 -dynamiclib -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk -L/Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks -L/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/lib -F/Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks -F/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Frameworks -iframework /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Frameworks -iframework /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.5.sdk/Developer/Library/Frameworks -filelist /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Intermediates.noindex/swift-composable-architecture.build/Debug-iphoneos/ComposableArchitecture.build/Objects-normal/arm64/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct.LinkFileList -install_name @rpath/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct.framework/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct -dead_strip -Xlinker -object_path_lto -Xlinker /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Intermediates.noindex/swift-composable-architecture.build/Debug-iphoneos/ComposableArchitecture.build/Objects-normal/arm64/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct_lto.o -Xlinker -export_dynamic -Xlinker -no_deduplicate -fembed-bitcode-marker -fapplication-extension -fobjc-link-runtime -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphoneos -L/usr/lib/swift 

// list of linked package products that TCA depends on (including OrderedCollections)
/Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/CasePaths_-196C353DF25C847B_PackageProduct.framework/CasePaths_-196C353DF25C847B_PackageProduct 
/Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/CombineSchedulers_-54F188414C11E20_PackageProduct.framework/CombineSchedulers_-54F188414C11E20_PackageProduct 
/Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/CustomDump_169F2B173C94B61B_PackageProduct.framework/CustomDump_169F2B173C94B61B_PackageProduct 
/Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/IdentifiedCollections_-5C6E944B2019E177_PackageProduct.framework/IdentifiedCollections_-5C6E944B2019E177_PackageProduct 
/Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/XCTestDynamicOverlay_6187313334988FF6_PackageProduct.framework/XCTestDynamicOverlay_6187313334988FF6_PackageProduct 
👉 /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/OrderedCollections_1222128174265B56_PackageProduct.framework/OrderedCollections_1222128174265B56_PackageProduct

-Xlinker -dependency_info -Xlinker /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Intermediates.noindex/swift-composable-architecture.build/Debug-iphoneos/ComposableArchitecture.build/Objects-normal/arm64/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct_dependency_info.dat -o /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Products/Debug-iphoneos/PackageFrameworks/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct.framework/ComposableArchitecture_-2C5AE37F5F55102_PackageProduct -Xlinker -add_ast_path -Xlinker /Users/ferologics/Library/Developer/Xcode/DerivedData/TCA_Collections-bdsldmhkxbaixdemvamjmfkbqdjt/Build/Intermediates.noindex/swift-composable-architecture.build/Debug-iphoneos/ComposableArchitecture.build/Objects-normal/arm64/ComposableArchitecture.swiftmodule

Basically, SPM does some magic inside the Build/Products/Debug-iphoneos/PackageFrameworks directory which contains the package products that it needs to link to. Most apparently this magic is not fully battle tested just yet.

I believe this confirms the issue lies with Apple and how SPM / Xcode resolves the linked package products. I also think it is the SPM's responsibility to provide the full list of linked package products, and that we don't have any control over this particular TCA build step.

I'll file a feedback for the issue as well as another TSI report as they don't seem to be responding to the previous one any longer. I'd appreciate any suggestions on how to file a better report just so it doesn't end up neglected like the previous TSI due to "the nature of the issue being with a 3rd party resource". 👐

[stephencelis]

Thanks for the update! You may also want to file a bug at bugs.swift.org if you want more public visibility into the problem/fix.

[ferologics]

Small update on the old TSI:

... blah we can't support 3rd party tools blah... If the maintainers of the swift-composable-architecture library would like to discuss their use of Swift Package Manager with us, they can open their own TSI**, and we’ll be happy to answer any questions they have.

In other news, to avoid being shunned in my next TSI due to "using a 3rd party tool", I reproduced the problem without TCA, using local packages. I boiled the issue down to the following dependency setup:

App -> Framework Target -> PackageB -> PackageA -> OrderedCollections

With that in place the OrderedCollections symbols will always be missing. For reference here's the Xcode 13 RC project attached to the feedback FB9627252 I filed earlier today.

I tried your suggestions in the new project but without any success. Maybe there's a simple workaround that I'm missing here, but for now it looks like we'll have to wait for Apple's response on the issue.

[stephencelis]

Cool, I'm going to convert this to a discussion since it seems to be an Apple bug till they can confirm otherwise. Please keep up posted there!

[ferologics]

After the long winded wait, my feedback came through with a workaround:

The reason packages are involved is that it's a diamond-dependency situation [...]
A workaround seems to be to mark the PackageB [[read TCA]] product as dynamic:

.library(
  name: "PackageB",
  type: .dynamic,
  targets: ["PackageB"]
)

(by adding the type parameter). [...]
... In the meantime we will continue looking for a more permanent solution.

I have ported this workaround over to our fork of TCA here and 🎉 it works!

With that, I think that this case can be closed for the time being. I don't expect you to support this workaround, so until it is resolved with a more permanent solution we will simply rebase our change onto latest releases.

[manmal]

I'm getting the same error with TCA 0.28.1 and Xcode 13.1 (swift-tools-version:5.5), in certain SwiftUI previews, but not all of them.
However, the problem seems to be fixed as of Xcode 13.2 beta 2 🥳

[ferologics]

I tested on our project with Xcode 13.2 beta 2 (13C5081f) but no dice, same error as before 😕

[mfclarke-ts]

Looks like this affects test targets too. Not the ones auto created by defining them in the Package.swift, but where you create them in the Xcode project itself.

[stephencelis]

@mfclarke-ts Are you seeing this problem on Xcode 13.1? Or a more recent Xcode?

[mfclarke-ts]

Hey @stephencelis sorry for the late reply. I was seeing this on Xcode 13.1, and now on 13.2.1. I can send a minimal project reproducing the issue if you like.

[jefflewis]

We ran into this building an SDK that we intend deliver as an XCFramework. We need to build dynamic libraries so that we can generate the correct XCFramework in the end. We have an example app colocated with the libraries that was importing the local (non-XCFramework) version, and hit this error when building the example app.

To work around this, we just added a new library generated in the SDK's package.swift that was a static library with DEV appended to the library name. The example app can then link the static library while our CI pipeline can build the proper XCFramework for distribution.

[joelCHonor]

Hi all, is this issue been resolved?
I'm using TCA, I can compile and run on my computer, but when the code is compiling in our CI, it is failing with this issue