Merge bitcoin/bitcoin#21851: release: support cross-compiling for arm64-apple-darwin

edd70b5 doc: add arm macOS depends platform triplet (jarolrod)
0628815 guix: add arm64-apple-darwin triplet (fanquake)
ca47f2e guix: use autoconf 2.71 (fanquake)
6fe5516 contrib: support arm64 darwin in security checks (fanquake)
446e73c build: use macOS 11 SDK (Xcode 12.2) (fanquake)

Pull request description:

  With a few minor changes, our macOS toolchain, and Guix, supports building for `arm64-apple-darwin` (Apple M1). Newer [`config.guess`](https://git.savannah.gnu.org/cgit/config.git/plain/config.guess) and [`config.sub`](https://git.savannah.gnu.org/cgit/config.git/plain/config.sub) are required (via autoconf 2.71), as support for recognising `arm64-apple-darwin` [was only recently added](https://git.savannah.gnu.org/cgit/config.git/commit/?id=2593751ef276497e312d7c4ce7fd049614c7bf80). A newer version of the macOS SDK (11.0+) is also required; I've taken this from Xcode 12.2.

  Guix Build:
  ```bash
  bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
  c9c64c6252927c82d4a96eb2b1a13c557ac230afa3640a7f2feec87fabe2b7e5  guix-build-edd70b52fcd7/output/aarch64-linux-gnu/SHA256SUMS.part
  b57dd461ec6a0b22ce9614ab824bcbff4c3e6e935c22367aff9a718f46c3fd39  guix-build-edd70b52fcd7/output/aarch64-linux-gnu/bitcoin-edd70b52fcd7-aarch64-linux-gnu-debug.tar.gz
  66da9f57a6789108db3460d2b54a8bed68b7e147627404e51f66254d91500402  guix-build-edd70b52fcd7/output/aarch64-linux-gnu/bitcoin-edd70b52fcd7-aarch64-linux-gnu.tar.gz
  13213523984837fc66fe27cc4dac269f6ef554ec56316af92337f4c4845f6acf  guix-build-edd70b52fcd7/output/arm-linux-gnueabihf/SHA256SUMS.part
  e1a17b41c92879ccb6608a6a6f82aac22d999f10d798f40f96297d91ebd90d20  guix-build-edd70b52fcd7/output/arm-linux-gnueabihf/bitcoin-edd70b52fcd7-arm-linux-gnueabihf-debug.tar.gz
  6a6d54081c2dd8f9a44293275aea875587b5fa7cc31a30925fb5156bb1e55bf3  guix-build-edd70b52fcd7/output/arm-linux-gnueabihf/bitcoin-edd70b52fcd7-arm-linux-gnueabihf.tar.gz
  79f617a2f0a11e6a8fcf8058e05204a374ab5f689ccb15a2ff449ee2dd5e8216  guix-build-edd70b52fcd7/output/arm64-apple-darwin/SHA256SUMS.part
  6fb04a18b17812174ec35a6788dce5745d6c2dec558056dcfdd356b57d089b99  guix-build-edd70b52fcd7/output/arm64-apple-darwin/bitcoin-edd70b52fcd7-arm64-apple-darwin.tar.gz
  a67e415631bea940adab86adec9899593fa35e1773d4ac1149efd249adf89996  guix-build-edd70b52fcd7/output/arm64-apple-darwin/bitcoin-edd70b52fcd7-osx-unsigned.dmg
  2a41c273afb8bb621226de80302a7190eee057c2447d113999f40a9485430d44  guix-build-edd70b52fcd7/output/arm64-apple-darwin/bitcoin-edd70b52fcd7-osx-unsigned.tar.gz
  c8d9358471e1d72d56122ae6f53dc2a42ae0925628bf347acd1a3ae7110d8759  guix-build-edd70b52fcd7/output/dist-archive/bitcoin-edd70b52fcd7.tar.gz
  1985a4869307418ec2ee85f5f0ccfea0944eba0792318a8676ae213a9589ed97  guix-build-edd70b52fcd7/output/powerpc64-linux-gnu/SHA256SUMS.part
  6a1d3525011da84e24997c9ab39563b3597ae881b4d1af5a09a6e673cfc59808  guix-build-edd70b52fcd7/output/powerpc64-linux-gnu/bitcoin-edd70b52fcd7-powerpc64-linux-gnu-debug.tar.gz
  cda50c28b2920caef74de4b3e78fd4d81592d1aadc9758d1882f1cad16dc83ca  guix-build-edd70b52fcd7/output/powerpc64-linux-gnu/bitcoin-edd70b52fcd7-powerpc64-linux-gnu.tar.gz
  ca92936cf796543d13250f94175af772a9fc985e834017d6af2cfb36eea289e6  guix-build-edd70b52fcd7/output/powerpc64le-linux-gnu/SHA256SUMS.part
  64c46248065149a384c5710adeb4fd8e795ffee7d6441e4edb71468c6cf1478e  guix-build-edd70b52fcd7/output/powerpc64le-linux-gnu/bitcoin-edd70b52fcd7-powerpc64le-linux-gnu-debug.tar.gz
  bb9c3400a1dd88fa5fd26e8e22ba0f3a43263bdbb762233ed1670394d3d69ed1  guix-build-edd70b52fcd7/output/powerpc64le-linux-gnu/bitcoin-edd70b52fcd7-powerpc64le-linux-gnu.tar.gz
  926ef7a23ecdf02a6619f9dba0db814373c1a5105347e4c13b80a116910d1c8d  guix-build-edd70b52fcd7/output/riscv64-linux-gnu/SHA256SUMS.part
  cdf7ce78f206378a44122865150784c20bf1bec6943b48e35a41633c0f1f05ed  guix-build-edd70b52fcd7/output/riscv64-linux-gnu/bitcoin-edd70b52fcd7-riscv64-linux-gnu-debug.tar.gz
  026bbb6662f4074054457ce1678fc1d697bfbf87c80f1c3d5d560bcc18e3bea6  guix-build-edd70b52fcd7/output/riscv64-linux-gnu/bitcoin-edd70b52fcd7-riscv64-linux-gnu.tar.gz
  0190b93fc5d5202c6b252e51c34799fb153a284cc53b0e6f62d2521dc61866bd  guix-build-edd70b52fcd7/output/x86_64-apple-darwin/SHA256SUMS.part
  69aac1a962aeeec01c30015f79d93c57d68a484c3905b8a526f90e7e7382bc1b  guix-build-edd70b52fcd7/output/x86_64-apple-darwin/bitcoin-edd70b52fcd7-osx-unsigned.dmg
  ea37691f1df0389afedbb9eece45660dcb2ee867ffda1d26b244a471af2a6230  guix-build-edd70b52fcd7/output/x86_64-apple-darwin/bitcoin-edd70b52fcd7-osx-unsigned.tar.gz
  f56ff4035e3bc70c2cbb04f8a7f5ef23096088189e84fa172cec97f59336cf2b  guix-build-edd70b52fcd7/output/x86_64-apple-darwin/bitcoin-edd70b52fcd7-osx64.tar.gz
  a1c1bf47083c6f19bcc68761ea56bec666cfe128f4ba878a5655ae12874fe038  guix-build-edd70b52fcd7/output/x86_64-linux-gnu/SHA256SUMS.part
  cad26c66c764361b30c8204e7743dbb6c0e6dd1eb3af9418d77bd566e05e1470  guix-build-edd70b52fcd7/output/x86_64-linux-gnu/bitcoin-edd70b52fcd7-x86_64-linux-gnu-debug.tar.gz
  ed2b459a06d5bc0f895402323fc5cae543a387d0621a55214feb37513210324e  guix-build-edd70b52fcd7/output/x86_64-linux-gnu/bitcoin-edd70b52fcd7-x86_64-linux-gnu.tar.gz
  ```

ACKs for top commit:
  hebasto:
    ACK edd70b5, a Guix `bitcoin-edd70b52fcd7-osx-unsigned.dmg` tested on Mac mini (M1, 2020) + macOS Monterey 12.1 (21C52).

Tree-SHA512: 00c55e139ce60c89900ef9365c380beec9474d8a0459bb584fd966d17715344fcf95b208b7222f665a5e3989aed51d3161641d3737aa6122a41b0c9f0d35906d

Author: fanquake

.cirrus.yml

@@ -277,7 +277,7 @@ task:
   container:
     image: ubuntu:focal
   env:
-    MACOS_SDK: "Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers"
+    MACOS_SDK: "Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers"
     << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
     FILE_ENV: "./ci/test/00_setup_env_mac.sh"
 

ci/test/00_setup_env_mac.sh

@@ -10,8 +10,8 @@ export CONTAINER_NAME=ci_macos_cross
 export DOCKER_NAME_TAG=ubuntu:20.04  # Check that Focal can cross-compile to macos
 export HOST=x86_64-apple-darwin
 export PACKAGES="cmake libz-dev libtinfo5 python3-setuptools xorriso"
-export XCODE_VERSION=12.1
-export XCODE_BUILD_ID=12A7403
+export XCODE_VERSION=12.2
+export XCODE_BUILD_ID=12B45b
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export GOAL="deploy"

contrib/devtools/security-check.py

@@ -211,12 +211,9 @@ def check_MACHO_control_flow(binary) -> bool:
 ]
 
 BASE_MACHO = [
-    ('PIE', check_PIE),
     ('NOUNDEFS', check_MACHO_NOUNDEFS),
-    ('NX', check_NX),
     ('LAZY_BINDINGS', check_MACHO_LAZY_BINDINGS),
     ('Canary', check_MACHO_Canary),
-    ('CONTROL_FLOW', check_MACHO_control_flow),
 ]
 
 CHECKS = {
@@ -231,7 +228,10 @@ def check_MACHO_control_flow(binary) -> bool:
         lief.ARCHITECTURES.X86: BASE_PE,
     },
     lief.EXE_FORMATS.MACHO: {
-        lief.ARCHITECTURES.X86: BASE_MACHO,
+        lief.ARCHITECTURES.X86: BASE_MACHO + [('PIE', check_PIE),
+                                              ('NX', check_NX),
+                                              ('CONTROL_FLOW', check_MACHO_control_flow)],
+        lief.ARCHITECTURES.ARM64: BASE_MACHO,
     }
 }
 

contrib/devtools/symbol-check.py

@@ -229,7 +229,7 @@ def check_MACHO_min_os(binary) -> bool:
     return False
 
 def check_MACHO_sdk(binary) -> bool:
-    if binary.build_version.sdk == [10, 15, 6]:
+    if binary.build_version.sdk == [11, 0, 0]:
         return True
     return False
 

contrib/devtools/test-security-check.py

@@ -116,21 +116,34 @@ def test_MACHO(self):
         executable = 'test1'
         cc = determine_wellknown_cmd('CC', 'clang')
         write_testcode(source)
+        arch = get_arch(cc, source, executable)
+
+        if arch == lief.ARCHITECTURES.X86:
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS Canary PIE NX CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS PIE NX CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS PIE CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all']),
+                (1, executable+': failed LAZY_BINDINGS PIE CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all']),
+                (1, executable+': failed PIE CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
+                (1, executable+': failed PIE'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
+                (0, ''))
+        else:
+            # arm64 darwin doesn't support non-PIE binaries, control flow or executable stacks
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fno-stack-protector']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS Canary'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fstack-protector-all']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-fstack-protector-all']),
+                (1, executable+': failed LAZY_BINDINGS'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-bind_at_load','-fstack-protector-all']),
+                (0, ''))
 
-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector']),
-            (1, executable+': failed PIE NOUNDEFS NX LAZY_BINDINGS Canary CONTROL_FLOW'))
-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all']),
-            (1, executable+': failed PIE NOUNDEFS NX LAZY_BINDINGS CONTROL_FLOW'))
-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all']),
-            (1, executable+': failed PIE NOUNDEFS LAZY_BINDINGS CONTROL_FLOW'))
-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all']),
-            (1, executable+': failed PIE LAZY_BINDINGS CONTROL_FLOW'))
-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all']),
-            (1, executable+': failed PIE CONTROL_FLOW'))
-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
-            (1, executable+': failed PIE'))
-        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
-            (0, ''))
 
         clean_files(source, executable)
 

contrib/guix/README.md

@@ -224,7 +224,7 @@ details.
 
   _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu
   riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
-  x86\_64-w64-mingw32 x86\_64-apple-darwin")_
+  x86\_64-w64-mingw32 x86\_64-apple-darwin arm64-apple-darwin")_
 
 * _**SOURCES_PATH**_
 
@@ -249,7 +249,7 @@ details.
   Set the path where _extracted_ SDKs can be found. This is passed through to
   the depends tree. Note that this is should be set to the _parent_ directory of
   the actual SDK (e.g. `SDK_PATH=$HOME/Downloads/macOS-SDKs` instead of
-  `$HOME/Downloads/macOS-SDKs/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers`).
+  `$HOME/Downloads/macOS-SDKs/Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers`).
 
   The path that this environment variable points to **must be a directory**, and
   **NOT a symlink to a directory**.

contrib/guix/guix-build

@@ -76,7 +76,7 @@ mkdir -p "$VERSION_BASE"
 # Default to building for all supported HOSTs (overridable by environment)
 export HOSTS="${HOSTS:-x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
                        x86_64-w64-mingw32
-                       x86_64-apple-darwin}"
+                       x86_64-apple-darwin arm64-apple-darwin}"
 
 # Usage: distsrc_for_host HOST
 #

contrib/guix/guix-codesign

@@ -91,7 +91,7 @@ fi
 ################
 
 # Default to building for all supported HOSTs (overridable by environment)
-export HOSTS="${HOSTS:-x86_64-w64-mingw32 x86_64-apple-darwin}"
+export HOSTS="${HOSTS:-x86_64-w64-mingw32 x86_64-apple-darwin arm64-apple-darwin}"
 
 # Usage: distsrc_for_host HOST
 #

contrib/guix/manifest.scm

@@ -579,7 +579,7 @@ inspecting signatures in Mach-O binaries.")
         ;; Build tools
         gnu-make
         libtool
-        autoconf
+        autoconf-2.71
         automake
         pkg-config
         bison

contrib/macdeploy/README.md

@@ -13,13 +13,13 @@ When complete, it will have produced `Bitcoin-Core.dmg`.
 ### Step 1: Obtaining `Xcode.app`
 
 Our current macOS SDK
-(`Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz`) can be
+(`Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz`) can be
 extracted from
-[Xcode_12.1.xip](https://download.developer.apple.com/Developer_Tools/Xcode_12.1/Xcode_12.1.xip).
+[Xcode_12.2.xip](https://download.developer.apple.com/Developer_Tools/Xcode_12.2/Xcode_12.2.xip).
 Alternatively, after logging in to your account go to 'Downloads', then 'More'
-and look for [`Xcode_12.1`](https://download.developer.apple.com/Developer_Tools/Xcode_12.1/Xcode_12.1.xip).
+and look for [`Xcode_12.2`](https://download.developer.apple.com/Developer_Tools/Xcode_12.2/Xcode_12.2.xip).
 An Apple ID and cookies enabled for the hostname are needed to download this.
-The `sha256sum` of the archive should be `612443b1894b39368a596ea1607f30cbb0481ad44d5e29c75edb71a6d2cf050f`.
+The `sha256sum` of the archive should be `28d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0`.
 
 After Xcode version 7.x, Apple started shipping the `Xcode.app` in a `.xip`
 archive. This makes the SDK less-trivial to extract on non-macOS machines. One
@@ -30,25 +30,25 @@ approach (tested on Debian Buster) is outlined below:
 apt install cpio
 git clone https://github.com/bitcoin-core/apple-sdk-tools.git
 
-# Unpack Xcode_12.1.xip and place the resulting Xcode.app in your current
+# Unpack Xcode_12.2.xip and place the resulting Xcode.app in your current
 # working directory
-python3 apple-sdk-tools/extract_xcode.py -f Xcode_12.1.xip | cpio -d -i
+python3 apple-sdk-tools/extract_xcode.py -f Xcode_12.2.xip | cpio -d -i
 ```
 
 On macOS the process is more straightforward:
 
 ```bash
-xip -x Xcode_12.1.xip
+xip -x Xcode_12.2.xip
 ```
 
-### Step 2: Generating `Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz` from `Xcode.app`
+### Step 2: Generating `Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz` from `Xcode.app`
 
-To generate `Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz`, run
+To generate `Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz`, run
 the script [`gen-sdk`](./gen-sdk) with the path to `Xcode.app` (extracted in the
 previous stage) as the first argument.
 
 ```bash
-# Generate a Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz from
+# Generate a Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz from
 # the supplied Xcode.app
 ./contrib/macdeploy/gen-sdk '/path/to/Xcode.app'
 ```