Commit c3da8c9
committed
Fix parallel build failures on platforms with Glibc
## Motivation
Since adding `MemberImportVisibility`, when Vapor is compiled in highly
parallel environments it fails with high probability:
```console
% git rev-parse HEAD
ac3aeb7
% rm -rf .build ~/.cache/org.swift.swiftpm/manifests/ && swift build -j 64
...
Building for debugging...
/pwd/Sources/Vapor/Utilities/String+IsIPAddress.swift:10:24: error: initializer 'init()' is not available due to missing import of defining module 'CNIOLinux' [#Membe
rImportVisibility]
1 | import Foundation
2 | import NIOCore
3 | #if canImport(Android)
| `- note: add import of module 'CNIOLinux'
4 | import Android
5 | #endif
:
8 | func isIPAddress() -> Bool {
9 | // We need some scratch space to let inet_pton write into.
10 | var ipv4Addr = in_addr()
| `- error: initializer 'init()' is not available due to missing import of defining module 'CNIOLinux' [#MemberImportVisibility]
11 | var ipv6Addr = in6_addr()
12 |
---[ similar error for in6_addr too ]---
```
Building with `-j 1` always succeeds.
I've spent quite some time looking into this and it appears that we're hitting a combination of:
1. The Glibc module map is broken. It only declares a top-level `SwiftGlibc.h`,
which then `#include`s many of the system headers—relevant to this this
issue, it includes `in.h`. As a result of this investigation, @al45tair has
filed swiftlang/swift#85427, but we'll need
a solution in the interim.
2. The CNIOLinux non-product C target of Swift NIO also has an umbrella
`CNIOLinux.h` header, which also `#include`s many headers, including
a transitive include of `in.h`.
3. Vapor has started building with `MemberImportVisibility`.
We can see in the above error that the Swift compiler is suggesting to add
`import CNIOLinux`, which is a dubious suggestion, since you'd expect the fix
to be to add an `import Glibc` or similar.
However, if we poke around at the failed build output we can see that `in.h` is
_only_ part of the `CNIOLinux` and _absent_ from `Glibc`:
```console
% find .build/aarch64-unknown-linux-gnu/debug/ModuleCache/ -name '*.pcm' | grep -e NIOLinux -e Glibc | xargs -n 1 -t clang -cc1 -module-file-info 2>&1 | grep -e "clang -cc1" -e "Input file:.*/in\\.h"
clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/SwiftGlibc-IGSGJKVMPVR1.pcm
clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/CNIOLinux-1MNQXZXDCO5H5.pcm
Input file: /usr/include/linux/in.h [System]
clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/SwiftGlibc-IGSGJKVMPVR1.pcm
clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/CNIOLinux-1MNQXZXDCO5H5.pcm
Input file: /usr/include/linux/in.h [System]
clang -cc1 -module-file-info .build/aarch64-unknown-linux-gnu/debug/ModuleCache/1O3SENRIBS9OC/SwiftGlibc-IGSGJKVMPVR1.pcm
```
Compare this to the build output of a successful build (`-j 1`), where we can
see that `in.h` is part of `Glibc`:
```console
% find .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/ -name '*.pcm' | grep Glibc | xargs -n 1 -t clang -cc1 -module-file-info 2>&1 | grep -e "clang -cc1" -e "Input file:.*/in\\.h"
clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/29YJZD87ZLI67/SwiftGlibc-IGSGJKVMPVR1.pcm
Input file: /usr/include/netinet/in.h [System]
Input file: /usr/include/bits/in.h [System]
clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/1NHH1B2IBMMHE/SwiftGlibc-IGSGJKVMPVR1.pcm
Input file: /usr/include/netinet/in.h [System]
Input file: /usr/include/bits/in.h [System]
clang -cc1 -module-file-info .build-j1/aarch64-unknown-linux-gnu/debug/ModuleCache/1O3SENRIBS9OC/SwiftGlibc-IGSGJKVMPVR1.pcm
Input file: /usr/include/netinet/in.h [System]
Input file: /usr/include/bits/in.h [System]
```
What's happening is that a header file can only belong to exactly one Clang
module and how header files are attributed to modules is non-deterministic in
some cases.
`CNIOLinux` is an implicit module——it does _not_ have a `.modulemap`——and is
inferred from the presence and contents of
`Sources/CNIOLinux/include/CNIOLinux.h`.
`Glibc` is an explicit module——it _does_ have a `.modulemap`——but its module
map does not explicitly list all the headers that belong to the module. It only
lists `SwiftGlibc.h`.
Header files that are explicitly listed in a module map are deterministically
attributed to that module, but header files that are transitively included
using `#include` are attributed to modules on a first-come-first-attributed
basis.
To illustrate this, I can make the `-j64` build succeed with the following
hot-patch of the `Glibc.modulemap` in my container:
```diff
module SwiftGlibc [system] {
// ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 23)
link "m"
// ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 26)
link "pthread"
// FIXME: util contains rarely used functions and not usually needed. Unfortunately
// link directive doesn't work in the submodule yet.
// ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 30)
link "util"
// ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 33)
// ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 35)
link "dl"
// ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 37)
// ###sourceLocation(file: "/home/build-user/swift/stdlib/public/Platform/glibc.modulemap.gyb", line: 43)
header "SwiftGlibc.h"
+ header "netinet/in.h"
// <assert.h>'s use of NDEBUG requires textual inclusion.
textual header "assert.h"
export *
}
```
We'll have to track swiftlang/swift#85427 for a real
fix for this issue, but in the meantime, we'll need a workaround in Vapor if we
want it to work with the `MemberImportVisibility` compiler setting.
As it happens, it looks like the compiler hint——`import CNIOLinux`——might be the best we can do for
today, and it turns out that there are other projects that are downstream of
NIO that added this when they added `MemberImportVisibilty` too:
- apple/swift-nio-ssl#497
- apple/swift-nio-extras#240
- swift-server/async-http-client#794
## Modifications
Add missing imports, with a link to the Swift issue to track.
## Result
Vapor will now reliably build when built in parallel.1 parent ac3aeb7 commit c3da8c9
1 file changed
+15
-3
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
4 | 14 | | |
| 15 | + | |
| 16 | + | |
5 | 17 | | |
6 | 18 | | |
7 | 19 | | |
| |||
0 commit comments