Skip to content

Commit 397105e

Browse files
authored
generator: Use a streaming pipeline to unpack host Swift toolchain (#148)
Currently, unpacking the host Swift toolchain happens in several steps connected by temporary files. Switching to a single concurrent pipeline reduces the time taken to extract the toolchain from around 30s to under 6s on my machine. ### Changes The `xar` manual page says that it is no longer under development: > DEPRECATION WARNING > xar is no longer under active development by Apple. Clients of > xar should pursue alternative archive formats. On Darwin, `tar` is able to unpack `xar` archives. Unlike `xar`, it can also extract a subset of the files from the archive and stream them to standard output. This lets us extract the Swift toolchain in a single pipeline, without writing temporary files to disk. The first `tar` process unpacks the .pkg file and extracts the payload, writing it out on standard output. The second `tar` process can now start in parallel instead of waiting for the payload to be fully unpacked to disk. This saves a couple of seconds. `tar` is also able to extract gzipped cpio archives. This lets us also extract the package payload as part of the same pipeline. Switching to `tar` also lets us use `--include` and `--exclude` patterns to control which files are extracted from the payload, instead of extracting the entire archive and copying out only the files we need. On my machine, the `cat | gunzip | cpio` pipeline takes about 27 seconds to unpack the entire archive; `tar`, without `--include` or `--exclude` patterns, takes about 8 seconds to extract everything. ### Result Overall, this PR reduces the time to build an SDK from Debian packages on an M3 MacBook Air from 1m30s to about 53s, excluding download time. The SDKs generated by the old and new methods are identical.
1 parent 9071f98 commit 397105e

File tree

1 file changed

+13
-5
lines changed

1 file changed

+13
-5
lines changed

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Unpack.swift

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,19 @@ extension SwiftSDKGenerator {
4141
logGenerationStep("Unpacking and copying Swift binaries for the host triple...")
4242
let pathsConfiguration = self.pathsConfiguration
4343

44-
try await inTemporaryDirectory { fileSystem, tmpDir in
45-
try await fileSystem.unpack(file: hostSwiftPackagePath, into: tmpDir)
46-
try await self.removeToolchainComponents(tmpDir)
47-
try await fileSystem.rsync(from: tmpDir.appending("usr"), to: pathsConfiguration.toolchainDirPath)
48-
}
44+
let excludes =
45+
unusedDarwinPlatforms.map { "--exclude usr/lib/swift/\($0)" } +
46+
unusedDarwinPlatforms.map { "--exclude usr/lib/swift_static/\($0)" } +
47+
unusedHostBinaries.map { "--exclude usr/bin/\($0)" } +
48+
unusedHostLibraries.map { "--exclude usr/lib/\($0)" }
49+
50+
try await Shell.run(
51+
#"""
52+
tar -x --to-stdout -f \#(hostSwiftPackagePath) \*.pkg/Payload |
53+
tar -C "\#(pathsConfiguration.toolchainDirPath)" -x \#(excludes.joined(separator: " ")) --include usr
54+
"""#,
55+
shouldLogCommands: isVerbose
56+
)
4957
}
5058

5159
func removeToolchainComponents(

0 commit comments

Comments
 (0)