Merge #19240: build: macOS toolchain simplification and bump

adf543d darwin: pass mlinker-version so that clang enables new features (Cory Fields)
2418f73 macos: Bump to xcode 11.3.1 and 10.15 SDK (Cory Fields)
5c2c835 depends: bump MacOS toolchain (Cory Fields)
85b5e42 contrib: macdeploy: Remove historical extraction notes (Carl Dong)
351beb5 contrib: macdeploy: Use apple-sdk-tools instead of xar+pbzx (Carl Dong)
fbcfcf6 native_cctools: Don't use libc++ from pinned clang (Carl Dong)
3381e4a Adapt rest of tooling to new SDK naming scheme (Carl Dong)
b3394ab contrib: macdeploy: Correctly generate macOS SDK (Carl Dong)

Pull request description:

  This PR achieves 3 main things:
  1. It simplifies the macOS SDK generation by putting the logic inside a (semi-)portable python3 script `gen-sdk`
  2. It transitions us to using `libc++` headers extracted from the `Xcode.app`, which is more correct as those headers better match the `.tbd` library stubs we use from the `MacOSX.sdk` (located under the same `Xcode.app`). Previously, we used `libc++` headers copied from our downloaded, pinned clang (see `native_cctools.mk`).
  3. It bumps the macOS toolchain in a way that fulfills all of the following constraints:
      1. The new SDK should support compiling with C++17 (our current one doesn't)
      2. The new toolchain should not change our minimum supported macOS version (`-mmacosx-version-min`)
      3. The new toolchain should expect to use a version of `cctools` that is supported by https://github.com/tpoechtrager/cctools-port

  For the constraints in (3), you can reference [this chart](https://en.wikipedia.org/wiki/Xcode#Xcode_7.0_-_11.x_(since_Free_On-Device_Development)) to see that the newest toolchain we can use with our `cctools-port` is `11.3.1`, and the rest of the constraints were tested with local builds.

  #### But [the other Wikipedia chart](https://en.wikipedia.org/wiki/Xcode#Xcode_11.x_(since_SwiftUI_framework)) says that the "min macOS to run" for Xcode 11.3.1 is 10.14.4, doesn't that violate constraint (ii)?

  This confused me at first too, but the "min macOS to run" is for the Xcode.app App itself. The SDK still supports 10.12, as evident in a few plist files and as proven through local builds.

  #### Why bundle all of this together in a single PR?

  We need (1) and (2) together, because if we don't, manually adding the `libc++` headers and writing that out in a `README.md` is going to result in a lot of user error, so it's great to have these together to be more correct and also make it easier on the user at the same time.

  We need (3) together with everything else because bumping (or in the case of (1), renaming) the SDK requires some human coordination and may break some builds. And since it's not that complicated a change, it makes sense to do it together with the rest.

ACKs for top commit:
  theuni:
    ACK adf543d.
  fanquake:
    ACK adf543d - I'll take a look at the linker issue.

Tree-SHA512: 3813b69ebfe9610bee14100f26296fb5438d9bf0dd184ea55e6c38f5ebd94f7c171d98b150fc9e52fde626533f347f7ec51a2b72b79859d946284f578c1084a3

Author: fanquake

ci/test/00_setup_env_mac.sh

@@ -9,7 +9,8 @@ export LC_ALL=C.UTF-8
 export CONTAINER_NAME=ci_macos_cross
 export HOST=x86_64-apple-darwin16
 export PACKAGES="cmake imagemagick libcap-dev librsvg2-bin libz-dev libbz2-dev libtiff-tools python3-dev python3-setuptools"
-export OSX_SDK=10.14
+export XCODE_VERSION=11.3.1
+export XCODE_BUILD_ID=11C505
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export GOAL="deploy"

ci/test/05_before_script.sh

@@ -15,11 +15,14 @@ fi
 
 DOCKER_EXEC mkdir -p ${DEPENDS_DIR}/SDKs ${DEPENDS_DIR}/sdk-sources
 
-if [ -n "$OSX_SDK" ] && [ ! -f ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz ]; then
-  curl --location --fail $SDK_URL/MacOSX${OSX_SDK}.sdk.tar.gz -o ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz
+OSX_SDK_BASENAME="Xcode-${XCODE_VERSION}-${XCODE_BUILD_ID}-extracted-SDK-with-libcxx-headers.tar.gz"
+OSX_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${OSX_SDK_BASENAME}"
+
+if [ -n "$XCODE_VERSION" ] && [ ! -f "$OSX_SDK_PATH" ]; then
+  curl --location --fail "${SDK_URL}/${OSX_SDK_BASENAME}" -o "$OSX_SDK_PATH"
 fi
-if [ -n "$OSX_SDK" ] && [ -f ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz ]; then
-  DOCKER_EXEC tar -C ${DEPENDS_DIR}/SDKs -xf ${DEPENDS_DIR}/sdk-sources/MacOSX${OSX_SDK}.sdk.tar.gz
+if [ -n "$XCODE_VERSION" ] && [ -f "$OSX_SDK_PATH" ]; then
+  DOCKER_EXEC tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
 fi
 if [[ $HOST = *-mingw32 ]]; then
   DOCKER_EXEC update-alternatives --set $HOST-g++ \$\(which $HOST-g++-posix\)

contrib/gitian-build.py

@@ -209,7 +209,7 @@ def main():
     args.macos = 'm' in args.os
 
     # Disable for MacOS if no SDK found
-    if args.macos and not os.path.isfile('gitian-builder/inputs/MacOSX10.14.sdk.tar.gz'):
+    if args.macos and not os.path.isfile('gitian-builder/inputs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz'):
         print('Cannot build for MacOS, SDK does not exist. Will build for other OSes')
         args.macos = False
 

contrib/gitian-descriptors/gitian-osx.yml

@@ -32,7 +32,7 @@ remotes:
 - "url": "https://github.com/bitcoin/bitcoin.git"
   "dir": "bitcoin"
 files:
-- "MacOSX10.14.sdk.tar.gz"
+- "Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz"
 script: |
   set -e -o pipefail
 
@@ -90,7 +90,7 @@ script: |
   BASEPREFIX="${PWD}/depends"
 
   mkdir -p ${BASEPREFIX}/SDKs
-  tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/MacOSX10.14.sdk.tar.gz
+  tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz
 
   # Build dependencies for each host
   for i in $HOSTS; do

contrib/macdeploy/README.md

@@ -14,55 +14,44 @@ When complete, it will have produced `Bitcoin-Qt.dmg`.
 
 ## SDK Extraction
 
-Our current macOS SDK (`macOSX10.14.sdk`) can be extracted from
-[Xcode_10.2.1.xip](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
+### Step 1: Obtaining `Xcode.app`
+
+Our current macOS SDK
+(`Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz`) can be
+extracted from
+[Xcode_11.3.1.xip](https://download.developer.apple.com/Developer_Tools/Xcode_11.3.1/Xcode_11.3.1.xip).
 An Apple ID is needed to download this.
 
-`Xcode.app` is packaged in a `.xip` archive.
-This makes the SDK less-trivial to extract on non-macOS machines.
-One approach (tested on Debian Buster) is outlined below:
+After Xcode version 7.x, Apple started shipping the `Xcode.app` in a `.xip`
+archive. This makes the SDK less-trivial to extract on non-macOS machines. One
+approach (tested on Debian Buster) is outlined below:
 
 ```bash
+# Install/clone tools needed for extracting Xcode.app
+apt install cpio
+git clone https://github.com/bitcoin-core/apple-sdk-tools.git
 
-apt install clang cpio git liblzma-dev libxml2-dev libssl-dev make
-
-git clone https://github.com/tpoechtrager/xar
-pushd xar/xar
-./configure
-make
-make install
-popd
-
-git clone https://github.com/NiklasRosenstein/pbzx
-pushd pbzx
-clang -llzma -lxar pbzx.c -o pbzx -Wl,-rpath=/usr/local/lib
-popd
-
-xar -xf Xcode_10.2.1.xip -C .
-
-./pbzx/pbzx -n Content | cpio -i
-
-find Xcode.app -type d -name MacOSX.sdk -exec sh -c 'tar --transform="s/MacOSX.sdk/MacOSX10.14.sdk/" -c -C$(dirname {}) MacOSX.sdk/ | gzip -9n > MacOSX10.14.sdk.tar.gz' \;
+# Unpack Xcode_11.3.1.xip and place the resulting Xcode.app in your current
+# working directory
+python3 apple-sdk-tools/extract_xcode.py -f Xcode_11.3.1.xip | cpio -d -i
 ```
 
-on macOS the process is more straightforward:
+On macOS the process is more straightforward:
 
 ```bash
-xip -x Xcode_10.2.1.xip
-tar -s "/MacOSX.sdk/MacOSX10.14.sdk/" -C Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.14.sdk.tar.gz MacOSX.sdk
+xip -x Xcode_11.3.1.xip
 ```
 
-Our previously used macOS SDK (`MacOSX10.11.sdk`) can be extracted from
-[Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
-The script [`extract-osx-sdk.sh`](./extract-osx-sdk.sh) automates this. First
-ensure the DMG file is in the current directory, and then run the script. You
-may wish to delete the `intermediate 5.hfs` file and `MacOSX10.11.sdk` (the
-directory) when you've confirmed the extraction succeeded.
+### Step 2: Generating `Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz` from `Xcode.app`
+
+To generate `Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz`, run
+the script [`gen-sdk`](./gen-sdk) with the path to `Xcode.app` (extracted in the
+previous stage) as the first argument.
 
 ```bash
-apt-get install p7zip-full sleuthkit
-contrib/macdeploy/extract-osx-sdk.sh
-rm -rf 5.hfs MacOSX10.11.sdk
+# Generate a Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers.tar.gz from
+# the supplied Xcode.app
+./contrib/macdeploy/gen-sdk '/path/to/Xcode.app'
 ```
 
 ## Deterministic macOS DMG Notes
@@ -91,13 +80,13 @@ and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done
 
 To complicate things further, all builds must target an Apple SDK. These SDKs are free to
 download, but not redistributable. To obtain it, register for an Apple Developer Account,
-then download [Xcode 10.2.1](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
+then download [Xcode_11.3.1](https://download.developer.apple.com/Developer_Tools/Xcode_11.3.1/Xcode_11.3.1.xip).
 
 This file is many gigabytes in size, but most (but not all) of what we need is
 contained only in a single directory:
 
 ```bash
-Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
+Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
 ```
 
 See the SDK Extraction notes above for how to obtain it.

contrib/macdeploy/gen-sdk

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+import argparse
+import plistlib
+import pathlib
+import sys
+import tarfile
+import gzip
+import os
+import contextlib
+
+@contextlib.contextmanager
+def cd(path):
+    """Context manager that restores PWD even if an exception was raised."""
+    old_pwd = os.getcwd()
+    os.chdir(str(path))
+    try:
+        yield
+    finally:
+        os.chdir(old_pwd)
+
+def run():
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
+
+    parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
+    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
+
+    args = parser.parse_args()
+
+    xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
+    assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
+
+    xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
+    with xcode_app_plist.open('rb') as fp:
+        pl = plistlib.load(fp)
+        xcode_version = pl['CFBundleShortVersionString']
+        xcode_build_id = pl['ProductBuildVersion']
+    print("Found Xcode (version: {xcode_version}, build id: {xcode_build_id})".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id))
+
+    sdk_dir = xcode_app.joinpath("Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
+    sdk_plist = sdk_dir.joinpath("System/Library/CoreServices/SystemVersion.plist")
+    with sdk_plist.open('rb') as fp:
+        pl = plistlib.load(fp)
+        sdk_version = pl['ProductVersion']
+        sdk_build_id = pl['ProductBuildVersion']
+    print("Found MacOSX SDK (version: {sdk_version}, build id: {sdk_build_id})".format(sdk_version=sdk_version, sdk_build_id=sdk_build_id))
+
+    out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
+
+    xcode_libcxx_dir = xcode_app.joinpath("Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1")
+    assert xcode_libcxx_dir.is_dir()
+
+    if args.out_sdktgz:
+        out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
+    else:
+        # Construct our own out_sdktgz if not specified on the command line
+        out_sdktgz_path = pathlib.Path("./{}.tar.gz".format(out_name))
+
+    def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
+        """Add all files in dir_to_add to tarfp, but prepent MEMBERPREFIX to the files'
+        names
+
+        e.g. if the only file under /root/bazdir is /root/bazdir/qux, invoking:
+
+            tarfp_add_with_base_change(tarfp, "foo/bar", "/root/bazdir")
+
+        would result in the following members being added to tarfp:
+
+            foo/bar/             -> corresponding to /root/bazdir
+            foo/bar/qux          -> corresponding to /root/bazdir/qux
+
+        """
+        def change_tarinfo_base(tarinfo):
+            if tarinfo.name and tarinfo.name.startswith("./"):
+                tarinfo.name = str(pathlib.Path(alt_base_dir, tarinfo.name))
+            if tarinfo.linkname and tarinfo.linkname.startswith("./"):
+                tarinfo.linkname = str(pathlib.Path(alt_base_dir, tarinfo.linkname))
+            return tarinfo
+        with cd(dir_to_add):
+            tarfp.add(".", recursive=True, filter=change_tarinfo_base)
+
+    print("Creating output .tar.gz file...")
+    with out_sdktgz_path.open("wb") as fp:
+        with gzip.GzipFile(fileobj=fp, compresslevel=9, mtime=0) as gzf:
+            with tarfile.open(mode="w", fileobj=gzf) as tarfp:
+                print("Adding MacOSX SDK {} files...".format(sdk_version))
+                tarfp_add_with_base_change(tarfp, sdk_dir, out_name)
+                print("Adding libc++ headers...")
+                tarfp_add_with_base_change(tarfp, xcode_libcxx_dir, "{}/usr/include/c++/v1".format(out_name))
+    print("Done! Find the resulting gzipped tarball at:")
+    print(out_sdktgz_path.resolve())
+
+if __name__ == '__main__':
+    run()

depends/hosts/darwin.mk

@@ -1,8 +1,12 @@
 OSX_MIN_VERSION=10.12
-OSX_SDK_VERSION=10.14
-OSX_SDK=$(SDK_PATH)/MacOSX$(OSX_SDK_VERSION).sdk
-darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK)
-darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++
+OSX_SDK_VERSION=10.15.1
+XCODE_VERSION=11.3.1
+XCODE_BUILD_ID=11C505
+LD64_VERSION=530
+
+OSX_SDK=$(SDK_PATH)/Xcode-$(XCODE_VERSION)-$(XCODE_BUILD_ID)-extracted-SDK-with-libcxx-headers
+darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION)
+darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++ -mlinker-version=$(LD64_VERSION)
 
 darwin_CFLAGS=-pipe
 darwin_CXXFLAGS=$(darwin_CFLAGS)

depends/packages/native_cctools.mk

@@ -1,14 +1,14 @@
 package=native_cctools
-$(package)_version=3764b223c011574971ee3ae09ce968ba5dc2f00f
+$(package)_version=4da2f3b485bcf4cef526f30c0b8c0bcda99cdbb4
 $(package)_download_path=https://github.com/tpoechtrager/cctools-port/archive
 $(package)_file_name=$($(package)_version).tar.gz
-$(package)_sha256_hash=3e35907bf376269a844df08e03cbb43e345c88125374f2228e03724b5f9a2a04
+$(package)_sha256_hash=a2d491c0981cef72fee2b833598f20f42a6c44a7614a61c439bda93d56446fec
 $(package)_build_subdir=cctools
-$(package)_clang_version=6.0.1
+$(package)_clang_version=8.0.0
 $(package)_clang_download_path=https://releases.llvm.org/$($(package)_clang_version)
 $(package)_clang_download_file=clang+llvm-$($(package)_clang_version)-x86_64-linux-gnu-ubuntu-14.04.tar.xz
 $(package)_clang_file_name=clang-llvm-$($(package)_clang_version)-x86_64-linux-gnu-ubuntu-14.04.tar.xz
-$(package)_clang_sha256_hash=fa5416553ca94a8c071a27134c094a5fb736fe1bd0ecc5ef2d9bc02754e1bef0
+$(package)_clang_sha256_hash=9ef854b71949f825362a119bf2597f744836cb571131ae6b721cd102ffea8cd0
 
 $(package)_libtapi_version=3efb201881e7a76a21e0554906cf306432539cef
 $(package)_libtapi_download_path=https://github.com/tpoechtrager/apple-libtapi/archive
@@ -72,7 +72,5 @@ define $(package)_stage_cmds
   cp -P bin/clang++ $($(package)_staging_prefix_dir)/bin/ &&\
   cp lib/libLTO.so $($(package)_staging_prefix_dir)/lib/ && \
   cp -rf lib/clang/$($(package)_clang_version)/include/* $($(package)_staging_prefix_dir)/lib/clang/$($(package)_clang_version)/include/ && \
-  cp bin/llvm-dsymutil $($(package)_staging_prefix_dir)/bin/$(host)-dsymutil && \
-  if `test -d include/c++/`; then cp -rf include/c++/ $($(package)_staging_prefix_dir)/include/; fi && \
-  if `test -d lib/c++/`; then cp -rf lib/c++/ $($(package)_staging_prefix_dir)/lib/; fi
+  cp bin/dsymutil $($(package)_staging_prefix_dir)/bin/$(host)-dsymutil
 endef