Skip to content

Commit b3394ab

Browse files
committed
contrib: macdeploy: Correctly generate macOS SDK
Previously, we did not include the macOS SDK libc++ headers in our SDK creation process and instead used whichever libc++ headers shipped with the clang package we downloaded in depends. This change adds a script (which works on both GNU/Linux and macOS) to correctly generate the macOS SDK including the libc++ headers. This can be thought of as a simplified rewrite of tpoechtrager's script: https://github.com/tpoechtrager/osxcross/blob/d3392f4eae78f3fa3f1fd065fa423f2712825102/tools/gen_sdk_package.sh The location within the SDK where we place the libc++ headers is chosen such that clang's search path detection logic for sysroots would pick up the headers properly. We also document this change.
1 parent febe582 commit b3394ab

File tree

2 files changed

+119
-10
lines changed

2 files changed

+119
-10
lines changed

contrib/macdeploy/README.md

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,17 @@ When complete, it will have produced `Bitcoin-Qt.dmg`.
1414

1515
## SDK Extraction
1616

17-
Our current macOS SDK (`macOSX10.14.sdk`) can be extracted from
17+
### Step 1: Obtaining `Xcode.app`
18+
19+
Our current macOS SDK
20+
(`Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz`) can be
21+
extracted from
1822
[Xcode_10.2.1.xip](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
1923
An Apple ID is needed to download this.
2024

21-
`Xcode.app` is packaged in a `.xip` archive.
22-
This makes the SDK less-trivial to extract on non-macOS machines.
23-
One approach (tested on Debian Buster) is outlined below:
25+
After Xcode version 7.x, Apple started shipping the `Xcode.app` in a `.xip`
26+
archive. This makes the SDK less-trivial to extract on non-macOS machines. One
27+
approach (tested on Debian Buster) is outlined below:
2428

2529
```bash
2630

@@ -41,17 +45,28 @@ popd
4145
xar -xf Xcode_10.2.1.xip -C .
4246

4347
./pbzx/pbzx -n Content | cpio -i
44-
45-
find Xcode.app -type d -name MacOSX.sdk -exec sh -c 'tar --transform="s/MacOSX.sdk/MacOSX10.14.sdk/" -c -C$(dirname {}) MacOSX.sdk/ | gzip -9n > MacOSX10.14.sdk.tar.gz' \;
4648
```
4749

48-
on macOS the process is more straightforward:
50+
On macOS the process is more straightforward:
4951

5052
```bash
5153
xip -x Xcode_10.2.1.xip
52-
tar -s "/MacOSX.sdk/MacOSX10.14.sdk/" -C Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.14.sdk.tar.gz MacOSX.sdk
5354
```
5455

56+
### Step 2: Generating `Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz` from `Xcode.app`
57+
58+
To generate `Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz`, run
59+
the script [`gen-sdk`](./gen-sdk) with the path to `Xcode.app` (extracted in the
60+
previous stage) as the first argument.
61+
62+
```bash
63+
# Generate a Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz from
64+
# the supplied Xcode.app
65+
./contrib/macdeploy/gen-sdk '/path/to/Xcode.app'
66+
```
67+
68+
### Historial macOS SDK Extraction Notes
69+
5570
Our previously used macOS SDK (`MacOSX10.11.sdk`) can be extracted from
5671
[Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
5772
The script [`extract-osx-sdk.sh`](./extract-osx-sdk.sh) automates this. First
@@ -91,13 +106,13 @@ and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done
91106

92107
To complicate things further, all builds must target an Apple SDK. These SDKs are free to
93108
download, but not redistributable. To obtain it, register for an Apple Developer Account,
94-
then download [Xcode 10.2.1](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
109+
then download [Xcode_10.2.1](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
95110

96111
This file is many gigabytes in size, but most (but not all) of what we need is
97112
contained only in a single directory:
98113

99114
```bash
100-
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
115+
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
101116
```
102117

103118
See the SDK Extraction notes above for how to obtain it.

contrib/macdeploy/gen-sdk

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/usr/bin/env python3
2+
import argparse
3+
import plistlib
4+
import pathlib
5+
import sys
6+
import tarfile
7+
import gzip
8+
import os
9+
import contextlib
10+
11+
@contextlib.contextmanager
12+
def cd(path):
13+
"""Context manager that restores PWD even if an exception was raised."""
14+
old_pwd = os.getcwd()
15+
os.chdir(str(path))
16+
try:
17+
yield
18+
finally:
19+
os.chdir(old_pwd)
20+
21+
def run():
22+
parser = argparse.ArgumentParser(
23+
description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
24+
25+
parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
26+
parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
27+
28+
args = parser.parse_args()
29+
30+
xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
31+
assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
32+
33+
xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
34+
with xcode_app_plist.open('rb') as fp:
35+
pl = plistlib.load(fp)
36+
xcode_version = pl['CFBundleShortVersionString']
37+
xcode_build_id = pl['ProductBuildVersion']
38+
print("Found Xcode (version: {xcode_version}, build id: {xcode_build_id})".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id))
39+
40+
sdk_dir = xcode_app.joinpath("Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
41+
sdk_plist = sdk_dir.joinpath("System/Library/CoreServices/SystemVersion.plist")
42+
with sdk_plist.open('rb') as fp:
43+
pl = plistlib.load(fp)
44+
sdk_version = pl['ProductVersion']
45+
sdk_build_id = pl['ProductBuildVersion']
46+
print("Found MacOSX SDK (version: {sdk_version}, build id: {sdk_build_id})".format(sdk_version=sdk_version, sdk_build_id=sdk_build_id))
47+
48+
out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
49+
50+
xcode_libcxx_dir = xcode_app.joinpath("Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1")
51+
assert xcode_libcxx_dir.is_dir()
52+
53+
if args.out_sdktgz:
54+
out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
55+
else:
56+
# Construct our own out_sdktgz if not specified on the command line
57+
out_sdktgz_path = pathlib.Path("./{}.tar.gz".format(out_name))
58+
59+
def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
60+
"""Add all files in dir_to_add to tarfp, but prepent MEMBERPREFIX to the files'
61+
names
62+
63+
e.g. if the only file under /root/bazdir is /root/bazdir/qux, invoking:
64+
65+
tarfp_add_with_base_change(tarfp, "foo/bar", "/root/bazdir")
66+
67+
would result in the following members being added to tarfp:
68+
69+
foo/bar/ -> corresponding to /root/bazdir
70+
foo/bar/qux -> corresponding to /root/bazdir/qux
71+
72+
"""
73+
def change_tarinfo_base(tarinfo):
74+
if tarinfo.name and tarinfo.name.startswith("./"):
75+
tarinfo.name = str(pathlib.Path(alt_base_dir, tarinfo.name))
76+
if tarinfo.linkname and tarinfo.linkname.startswith("./"):
77+
tarinfo.linkname = str(pathlib.Path(alt_base_dir, tarinfo.linkname))
78+
return tarinfo
79+
with cd(dir_to_add):
80+
tarfp.add(".", recursive=True, filter=change_tarinfo_base)
81+
82+
print("Creating output .tar.gz file...")
83+
with out_sdktgz_path.open("wb") as fp:
84+
with gzip.GzipFile(fileobj=fp, compresslevel=9, mtime=0) as gzf:
85+
with tarfile.open(mode="w", fileobj=gzf) as tarfp:
86+
print("Adding MacOSX SDK {} files...".format(sdk_version))
87+
tarfp_add_with_base_change(tarfp, sdk_dir, out_name)
88+
print("Adding libc++ headers...")
89+
tarfp_add_with_base_change(tarfp, xcode_libcxx_dir, "{}/usr/include/c++/v1".format(out_name))
90+
print("Done! Find the resulting gzipped tarball at:")
91+
print(out_sdktgz_path.resolve())
92+
93+
if __name__ == '__main__':
94+
run()

0 commit comments

Comments
 (0)