Skip to content

Commit c3da8c9

Browse files
Fix parallel build failures on platforms with Glibc
## Motivation Since adding `MemberImportVisibility`, when Vapor is compiled in highly parallel environments it fails with high probability: ```console % git rev-parse HEAD ac3aeb7 % rm -rf .build ~/.cache/org.swift.swiftpm/manifests/ && swift build -j 64 ... Building for debugging... /pwd/Sources/Vapor/Utilities/String+IsIPAddress.swift:10:24: error: initializer 'init()' is not available due to missing import of defining module 'CNIOLinux' [#Membe rImportVisibility] 1 | import Foundation 2 | import NIOCore 3 | #if canImport(Android) | `- note: add import of module 'CNIOLinux' 4 | import Android 5 | #endif : 8 | func isIPAddress() -> Bool { 9 | // We need some scratch space to let inet_pton write into. 10 | var ipv4Addr = in_addr() | `- error: initializer 'init()' is not available due to missing import of defining module 'CNIOLinux' [#MemberImportVisibility] 11 | var ipv6Addr = in6_addr() 12 | ---[ similar error for in6_addr too ]--- ``` Building with `-j 1` always succeeds. I've spent quite some time looking into this and it appears that we're hitting a combination of: 1. The Glibc module map is broken. It only declares a top-level `SwiftGlibc.h`, which then `#include`s many of the system headers—relevant to this this issue, it includes `in.h`. As a result of this investigation, @al45tair has filed swiftlang/swift#85427, but we'll need a solution in the interim. 2. The CNIOLinux non-product C target of Swift NIO also has an umbrella `CNIOLinux.h` header, which also `#include`s many headers, including a transitive include of `in.h`. 3. Vapor has started building with `MemberImportVisibility`. We can see in the above error that the Swift compiler is suggesting to add `import CNIOLinux`, which is a dubious suggestion, since you'd expect the fix to be to add an `import Glibc` or similar. However, if we poke around at the failed build output we can see that `in.h` is _only_ part of the `CNIOLinux` and _absent_ from `Glibc`: ```console % find .build/aarch64-unknown-linux-gnu/debug/ModuleCache/ -name '*.pcm' | grep -e NIOLinux -e Glibc | xargs -n 1 -t clang -cc1 -module-file-info 2>&1 | grep -e "clang -cc1" -e "Input file:.*/in\\.h" clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/SwiftGlibc-IGSGJKVMPVR1.pcm clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/CNIOLinux-1MNQXZXDCO5H5.pcm Input file: /usr/include/linux/in.h [System] clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/SwiftGlibc-IGSGJKVMPVR1.pcm clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/CNIOLinux-1MNQXZXDCO5H5.pcm Input file: /usr/include/linux/in.h [System] clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1O3SENRIBS9OC/SwiftGlibc-IGSGJKVMPVR1.pcm ``` Compare this to the build output of a successful build (`-j 1`), where we can see that `in.h` is part of `Glibc`: ```console % find .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/ -name '*.pcm' | grep Glibc | xargs -n 1 -t clang -cc1 -module-file-info 2>&1 | grep -e "clang -cc1" -e "Input file:.*/in\\.h" clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/SwiftGlibc-IGSGJKVMPVR1.pcm Input file: /usr/include/netinet/in.h [System] Input file: /usr/include/bits/in.h [System] clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/SwiftGlibc-IGSGJKVMPVR1.pcm Input file: /usr/include/netinet/in.h [System] Input file: /usr/include/bits/in.h [System] clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/1O3SENRIBS9OC/SwiftGlibc-IGSGJKVMPVR1.pcm Input file: /usr/include/netinet/in.h [System] Input file: /usr/include/bits/in.h [System] ``` What's happening is that a header file can only belong to exactly one Clang module and how header files are attributed to modules is non-deterministic in some cases. `CNIOLinux` is an implicit module——it does _not_ have a `.modulemap`——and is inferred from the presence and contents of `Sources/CNIOLinux/include/CNIOLinux.h`. `Glibc` is an explicit module——it _does_ have a `.modulemap`——but its module map does not explicitly list all the headers that belong to the module. It only lists `SwiftGlibc.h`. Header files that are explicitly listed in a module map are deterministically attributed to that module, but header files that are transitively included using `#include` are attributed to modules on a first-come-first-attributed basis. To illustrate this, I can make the `-j64` build succeed with the following hot-patch of the `Glibc.modulemap` in my container: ```diff module SwiftGlibc [system] { // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 23) link "m" // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 26) link "pthread" // FIXME: util contains rarely used functions and not usually needed. Unfortunately // link directive doesn't work in the submodule yet. // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 30) link "util" // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 33) // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 35) link "dl" // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 37) // ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 43) header "SwiftGlibc.h" + header "netinet/in.h" // <assert.h>'s use of NDEBUG requires textual inclusion. textual header "assert.h" export * } ``` We'll have to track swiftlang/swift#85427 for a real fix for this issue, but in the meantime, we'll need a workaround in Vapor if we want it to work with the `MemberImportVisibility` compiler setting. As it happens, it looks like the compiler hint——`import CNIOLinux`——might be the best we can do for today, and it turns out that there are other projects that are downstream of NIO that added this when they added `MemberImportVisibilty` too: - apple/swift-nio-ssl#497 - apple/swift-nio-extras#240 - swift-server/async-http-client#794 ## Modifications Add missing imports, with a link to the Swift issue to track. ## Result Vapor will now reliably build when built in parallel.
1 parent ac3aeb7 commit c3da8c9

File tree

1 file changed

+15
-3
lines changed

1 file changed

+15
-3
lines changed

Sources/Vapor/Utilities/String+IsIPAddress.swift

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,19 @@
1-
import Foundation
2-
import NIOCore
3-
#if canImport(Android)
1+
#if canImport(Glibc)
2+
import Glibc
3+
#if canImport(CNIOLinux)
4+
/// Until the Glibc Swift module is fixed, it's possible that the system header files on which the helpers in this file
5+
/// rely are misattributed to another module in a very parallel build, in this case CNIOLinux, which also imports
6+
/// `in.h`, which defines `in_addr` and `in6_addr`.
7+
///
8+
/// See: https://github.com/swiftlang/swift/issues/85427
9+
import CNIOLinux
10+
#endif
11+
#elseif canImport(Musl)
12+
import Musl
13+
#elseif canImport(Android)
414
import Android
15+
#else
16+
import Darwin
517
#endif
618

719
extension String {

0 commit comments

Comments
 (0)