You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/design/SwiftDriverIntegration.md
+9-7Lines changed: 9 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,19 +1,21 @@
1
1
## Swift Driver Integration
2
2
3
-
### Pre Swift Driver Integraiton
3
+
### Pre Swift Driver Integration
4
4
5
-
Historically (prior to Xcode 14), Swift compilation step was invoked by Xcode as a single external process. Xcode was calling `swiftc` and passed all required parameters (like all input files, output destinations, header paths etc.), and read its standard output to recognize the status of a compilation. Esentially, there were two build systems: "the big one" from Xcode and "small one" by Swift.
5
+
Historically (prior to Xcode 14), Swift compilation step was invoked by Xcode as a single external process. Xcode was calling `swiftc` and passing all required parameters (like all input files, output destinations, header paths etc.), and reading its standard output to recognize the status/state of a compilation. Essentially, there were two build systems: "the big one" from Xcode and "small one" by Swift.
6
6
7
-
That design was easy to mock using XCRemoteCache, where the `xcswiftc` wrapper was first inspecting if the cached artifact can be reused (e.g. no new input `.swift` files were added to the list of compilation files) and based on that either continued with the local compilation or mocking the compilation and existing early (cache hit).
7
+
That design was easy to mock in the XCRemoteCache, where the `xcswiftc` wrapper was first inspecting if the cached artifact can be reused (e.g. no new input `.swift` files were added to the list of compilation files) and based on that either continuing with the local compilation (cache miss) or mocking the compilation and existing early (cache hit).
8
8
9
9

10
10
11
11
12
12
### Swift Driver Integration Design
13
13
14
-
With the upgraded design, Xcode splits the work into `n` subprocesses (when `n` is ~CPU), each responsible to compile a subset of files/actions. To mitigate that, XCRemoteCache sets a single place to identify if the cached artifact is applicable - in a `swift-frontend` process responsible for module emitting. It has been found that this process is scheduled very early int the workflow timeline (with some aproximation, we could say it is scheduled as a first step) so it seems as best candidate for the "pre-work".
14
+
With the upgraded design (aka Swift Driver Integration), Xcode splits the work into `n` subprocesses (when `n` is ~CPU), each responsible to compile a subset of files/actions. To align with that, XCRemoteCache meeds to specify a single place to identify if the cached artifact is applicable. `swift-frontend`has been picked for that - process responsible for module emitting. By reviewing Xcode's behavior, it has been found that this process is scheduled very early in the workflow timeline (with some approximation, we could say it is scheduled as a first step) so it seems as best candidate for the "pre-work".
15
15
16
-
As the same executable is invoked multiple times for the same target (emit module, and multiple batches of compilation), XCRemoteCaches uses a file lock-based synchronization. Each `xcswift-frontend` (the wrapper for `swift-frontend`) tries to acquire a target-unique lock file (the file has a name equal to the `LLBUILD_BUILD_ID` ENV, which is unique for each build, and the file is placed in the `Intermediate` directory) and reads its content to find if the "pre-work" from the emit-module has been already done or not. As a lock file is unique per target and a build (it is actually unique per target compilation, as placed in `TARGET_TEMP_DIR`), initially the file is empty. If any of a non emit module invocation sees a file is empty (note that before reading, it acquires a shared lock), the lock is immediatelly released to quickly let the actual emit-module wrapper do the "pre-work". Emit module step holds a shared lock for the time of the entire process lifetime, so only once the "pre-work" is finished, all other `xcswift-frontend` processes can continue their job (wither noop or fallbacking to the `swift-frontent` in case a cache miss). Non emit-module steps (often, but not exclusive, compilation steps) acquire a lock only for a very short perdio - to read the content of that file so they could run in parallel.s
16
+
As the same executable `swift-frontend` is invoked multiple times for the same target (e.g. to emit module, multiple batches of compilation etc.), XCRemoteCaches uses a file lock-based synchronization. Each `xcswift-frontend` (the wrapper for `swift-frontend`) tries to acquire a unique lock file. The lock has a name `$LLBUILD_BUILD_ID.lock`, which is unique for each build, placed in the `Intermediate` directory. `xcswift-frontend` process reads its content to find if the "pre-work" from the emit-module has already been done - if not, it releases a lock a gives a way to other processes (presumably the "emit-module") to do the required work. As a lock file is unique per target and a build (it is actually unique per target compilation, placed in `TARGET_TEMP_DIR`), initially the file is empty.
17
+
18
+
Note the emit module step holds a shared lock for the time of the entire process lifetime, so only once the "pre-work" is finished, all other `xcswift-frontend` processes can continue their job (with either noop or fallbacking to the `swift-frontend` in case a cache miss). Non emit-module steps (compilation steps) acquire a lock only for a very short period - to read the content of that file, thus multiple batches of compilation can run in parallel.
17
19
18
20

19
21
@@ -31,5 +33,5 @@ As the same executable is invoked multiple times for the same target (emit modul
31
33
32
34
### Other considerations/open questions
33
35
34
-
* For mixed targets (ObjC&Swift), Xcode triggers `.m` compilation steps after the module emitting to ensure that the `-Swift.h` is available. That means, the synchronization algorithm will also postpone any `clang` invocations until the Swift "pre-work" is done - so mixed targets should behave the same way as in the non Swift Driver Integration flow
35
-
* For the WMO mode (Whole Module Optimization), all compilation steps are combined into a single `swift-frontend` process. As the emitmodule step is still invoked first, the WMO flow build builds down to a special case of the non-WMO. Therefore, the same algoritm should work for WMO builds too.
36
+
* For mixed targets (ObjC&Swift), Xcode triggers `.m` compilation steps **after** the module emitting to ensure that the `-Swift.h` is available for clang compilation. That means, the synchronization algorithm will postpone any `clang` invocations until the Swift "pre-work" is done. Therefore, mixed targets should behave the same way as in the non Swift Driver Integration flow
37
+
* For the WMO (Whole Module Optimization) mode, all compilation steps are combined into a single `swift-frontend` process. As the emit-module step is still invoked first, the WMO flow build can be considered as a special case of the algorithm described above (where there is only one compilation invocation). Therefore, the presented algorithm will work for the WMO mode out of the box.
0 commit comments