add blog/2020-10-29-macos-arm64.md

Co-authored-by: isuruf <[email protected]>

Author: jaimergp

blog/2020-10-29-macos-arm64.md

@@ -0,0 +1,282 @@
+---
+authors:
+  - isuruf
+tags: [conda-forge]
+---
+
+# macOS ARM builds on conda-forge
+
+A new platform `osx-arm64` has been added to the build matrix of
+conda-forge. `osx-arm64` packages are built to run on upcoming macOS
+arm64 processors marketed as `Apple Silicon`. An installer for this
+platform can be found
+[here](https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh).
+
+<!--truncate-->
+
+This will install a conda environment with python and conda in it.
+Installed conda will be able to install packages like `numpy, scipy`.
+Currently there are about 100 packages out of 10000 packages pre-built
+for this platform.
+
+All these packages are built on conda-forge's current macOS `x86_64`
+infrastructure. In order to do so, we have made lots of changes to the
+infrastructure including,
+`conda, conda-build, conda-smithy, constructor, conda-forge-ci-setup` to
+facilitate cross-compiling which is the process of compiling a package
+that will run on a `host` platform (`osx-arm64` in our case), with the
+compilation done on a `build` platform (`osx-64` or `linux-64` in our
+case).
+
+`osx-arm64` is the first conda platform that is completely bootstrapped
+using conda-build's cross-compiling facility. Previously, when adding a
+new platform, conda-build was built with an existing python and pip
+environment on the new platform. With cross-compiling, when the
+compilers and a sysroot is set up on a different platform, an existing
+conda-build installation (on `osx-64` and `linux-64` in this case) will
+be able to start building packages right away.
+
+## Cross-compiling builds for `osx-arm64`
+
+In order to cross compile packages for `osx-arm64` we need compilers.
+So, we first built `clang=11.0.0.rc1` which has support for targetting
+`osx-arm64`. We also built `compiler-rt=11.0.0.rc1` as a universal build
+support both `osx-64` and `osx-arm64`.
+
+Linker, archiver, `otool`, `install_name_tool` was built using the
+[cctools-port project](https://github.com/tpoechtrager/cctools-port) by
+Thomas Pöchtrager.
+
+One issue we ran into was that the macOS 11.0 Big Sur Beta 7 required
+that all executables and shared libraries be ad-hoc signed which is
+signing without a verified signature. On suggestion of `cctools-port`
+developer we added support to `cctools-port` to sign these executables
+using `ldid` which can be used on Linux as well as macOS to sign.
+
+Using these, the first cross compiled package we built was `libcxx` to
+facilitate C++ builds. For the `osx-arm64` sysroot we used the
+`MacOSX11.0.sdk` already installed on Azure pipelines and Travis-CI. Due
+to licensing issues, we cannot distribute this, but it can be downloaded
+from the Apple developer website even on Linux.
+
+With clang we have a C/C++ compiler, but lack a Fortran compiler. We
+used the [GCC fork for
+darwin-arm64](https://github.com/iains/gcc-darwin-arm64). First, a cross
+compiler (`build == host != target`) was built. Using that compiler, we
+built a `cross-native` compiler (`build != host == target`) which gave
+use the shared libraries like `libgfortran.dylib`.
+
+We also added support for cross compiling rust programs to the rust
+packages in conda and installing `rust_osx-64` on Linux will give you a
+compiler that will build packages for `osx-64`.
+
+As we haven't done cross-compilation before, many packages needed to be
+updated. Most were trivial changes that we automated later on. These
+included getting a newer `config.sub` to identify the new autotools
+platform `arm64-apple-darwin20.0.0`, adding options to CMake with the
+environment variable `CMAKE_ARGS` to correctly set up the toolchain and
+recipes were update to use `cmake ${CMAKE_ARGS} ..`. Running tests when
+building were also disabled by guarding commands like `make check`,
+`make test`, `ctest` with the env variable
+`CONDA_BUILD_CROSS_COMPILATION`.
+
+Cross-compiling python extensions is quite tricky as `distutils` is not
+really setup to do this. Thanks to the project
+[crossenv](https://github.com/benfogle/crossenv) this is unofficially
+supported with a few quirks. With `crossenv`, we can run a python on the
+build machine (`osx-64` or `linux-64` in this case) that acts like it is
+on `osx-arm64`. `crossenv` monkey-patches a few functions like
+`os.uname` and sets up values like `_PYTHON_SYSCONFIG_DATA` to make
+python running on `osx-64` or `linux-64` behave like `osx-arm64`. One
+issue is that, monkey-patching `sys.platform` doesn't work and
+therefore if a python package in it's `setup.py` uses `sys.platform` to
+differentiate OSes this will lead to unintended consequences if you are
+cross-compiling from `linux-64`. Therefore, we have to use `osx-64` as
+our `build` system when cross-compiling for `osx-arm64`. Note that
+packages using `sysconfig.get_platform()` will get the correct platform.
+
+For creating an installer for conda, we needed a standalone conda
+executable to bootstrap the conda environment. For other platforms we
+relied on `conda-standalone` which is a standalone conda executable
+created using `pyinstaller`. Since `pyinstaller` does not support
+cross-compile, we decided to use `micromamba` as the bootstrapper and
+added features to `micromamba` so that it can function as the
+bootstrapper.
+
+## How to add a `osx-arm64` build to a feedstock
+
+All the below changes will be done by a bot and the packages the bot
+will send PRs to is determined by the list of packages at
+[conda-forge-pinning](https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/migrations/osx_arm64.txt)
+and their dependences. If you would like to add support, please send a
+PR adding the feedstock name to the above list. After that PR is merged,
+you can monitor the status at [conda-forge
+status-page](https://conda-forge.org/status/#armosxaddition) and if a
+particular PR is stalled you can send a PR to the feedstock to fix it.
+
+Following instructions are for when you want to add support manually.
+
+Add the following to `conda-forge.yml` (on Linux or OSX),
+
+```yaml
+build_platform:
+  osx_arm64: osx_64
+test: native_and_emulated
+```
+
+You can rerender using,
+
+```bash
+conda smithy rerender
+```
+
+For python packages, add one or more of the following to
+`recipe/meta.yaml` as needed, noting that you *must* only add
+`numpy`, `cython`, and/or `pybind11`
+if they are used in `host:` as well,
+
+```yaml
+requirements:
+  build:
+    - python                                 # [build_platform != target_platform]
+    - cross-python_{{ target_platform }}     # [build_platform != target_platform]
+    - cython                                 # [build_platform != target_platform]
+    - numpy                                  # [build_platform != target_platform]
+    - pybind11                               # [build_platform != target_platform]
+```
+
+For autotools package, add the following to `recipe/meta.yaml`,
+
+```yaml
+requirements:
+  build:
+    - gnuconfig   # [unix]
+```
+
+and to `recipe/build.sh`,
+
+```bash
+# Get an updated config.sub and config.guess
+cp $BUILD_PREFIX/share/gnuconfig/config.* .
+```
+
+For cmake packages, add the following to `recipe/build.sh`,
+
+```bash
+cmake ${CMAKE_ARGS} ..
+```
+
+For `meson` packages, add the following to `recipe/build.sh`,
+
+```bash
+meson ${MESON_ARGS} builddir/
+```
+
+:::note
+Conda automatically creates a [cross build definition
+file](https://mesonbuild.com/Cross-compilation.html) when
+cross-compiling, and adds the necessary argument to `${MESON_ARGS}` to
+point `meson` to that file. `${MESON_ARGS}` is only defined when
+cross-compiling, not for normal builds.
+:::
+
+For rust packages, add the following to `recipe/meta.yaml`,
+
+```bash
+requirements:
+  build:
+    - {{ compiler('rust') }}
+```
+
+If there's a line like `make check` in `recipe/build.sh` that cannot be
+run when cross-compiling, do the following,
+
+```bash
+if [[ "$CONDA_BUILD_CROSS_COMPILATION" != "1" ]]; then
+  make check
+fi
+```
+
+After these changes, another rerendering might be required.
+
+Some useful jinja variables,
+
+1.  `build_platform` - conda subdir for `BUILD_PREFIX`. eg:
+    `linux-64`
+2.  `target_platform` - conda subdir for `PREFIX`. eg: `osx-arm64`
+
+Some useful environment variables,
+
+1.  `build_platform`
+2.  `target_platform`
+3.  `CONDA_BUILD_CROSS_COMPILATION` - 1 if cross compiling
+4.  `CMAKE_ARGS` - arguments to pass to cmake
+5.  `CC_FOR_BUILD` - C compiler for build platform
+6.  `CXX_FOR_BUILD` - C++ compiler for build platform
+7.  `HOST` - a triplet for host passed to autoconf. eg:
+    `arm64-apple-darwin20.0.0`
+8.  `BUILD` - a triplet for build passed to autoconf. eg:
+    `x86_64-conda-linux-gnu`
+
+Some useful configure options in `conda-forge.yml`
+
+1.  `build_platform` - a dictionary mapping `build` subdir to `host` subdir. eg:
+
+    ```yaml
+    build_platform:
+      osx_arm64: osx_64
+      linux_ppc64le: linux_64
+      linux_aarch64: linux_64
+    ```
+
+2.  `test_on_native_only` - a boolean to turn off testing on cross
+    compiling. If the tests don't require emulation (for eg: check
+    that a file exists), then `test_on_native_only: false` will run
+    the tests even when cross compiling.
+
+## Building locally
+
+For building locally add the following in
+`$HOME/conda_build_config.yaml`.
+
+```yaml
+SDKROOT:
+  - /path/to/MacOSX11.0.sdk
+```
+
+After that, look for the config you want to run in `.ci_support` folder
+in the root of the feedstock For eg: `.ci_support/osx_arm64_.yaml`. Then
+run,
+
+```bash
+conda build recipe -m .ci_support/osx_arm64_.yaml -c conda-forge -c conda-forge/label/rust_dev
+```
+
+This should start a new build for `osx-arm64`.
+
+## Testing packages
+
+In order to test packages intended to run on future Apple Silicon
+hardware, Apple provides a machine called Developer Transition Kit
+(DTK). Jonathan Helmus and Eli Rykoff has helped with testing these
+packages on DTKs. Thanks to Eli Rykoff, we are now running tests for
+these packages as a daily cron job which has led to finding several bugs
+in our cross compiling infrastructure and also bugs in our recipes.
+
+To test cross compiled recipes, transfer the built conda package to the
+`host` and run,
+
+```bash
+conda build --test /path/to/package -c conda-forge
+```
+
+This work would not have been possible without the help of many people
+including the upstream maintainers of compiler infrastructure (which
+includes conda, conda-build, cctools, tapi, cctools-port, ldid, llvm,
+clang, compiler-rt, openmp, libcxx, crossenv, rust, gcc-darwin-arm64),
+`conda-forge/help-osx-arm64` team including Matt Becker, Eli Rykoff and
+Uwe Korn who sent PRs to fix recipes, `conda-forge/bot` team and also
+all the conda-forge maintainers of the 100 feedstocks who reviewed and
+fixed PRs.
+
+Isuru Fernando