[Docs] Clarify target vs host dependencies (#1068)

giordano · web-flow · commit f88bafdf672b · 2021-06-11T02:30:40.000+01:00
[skip azp]
diff --git a/docs/src/FAQ.md b/docs/src/FAQ.md
@@ -72,10 +72,6 @@ You can of course in turn build and deply this package with
 julia build_tarballs.jl --debug --verbose --deploy="MY_USERNAME/Quxlib_jll.jl"
 ```
 
-### Can I install packages in the build environment?
-
-Yes, but it's unlikely that you'll need to.  The build environment is based on Alpine Linux (triplet: `x86_64-linux-musl`) so you can use [`apk`](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management) to install packages in it.  However, if you need runtime libraries or programs for the target system these packages won't help you.  The package manager is useful only to install utilities, tools or libraries that are needed exclusively at compile time on the build system.
-
 ### What are those numbers in the list of sources?  How do I get them?
 
 The list of sources is a vector of [`BinaryBuilder.AbstractSource`](@ref)s.  What the hash is depends on what the source is:
diff --git a/docs/src/build_tips.md b/docs/src/build_tips.md
@@ -6,10 +6,6 @@ If your build fails with some errors, look at the [Build Troubleshooting](@ref)
 
 *If you have additional tips, please submit a PR with suggestions.*
 
-## Build strategy
-
-What BinaryBuilder does is to create a tarball containing all files that are found inside the `${prefix}` directory at the end of the build and which don't come from the dependencies listed in the build recipe.  Thus, what you want to do in a build script is to install the relevant files under the appropriate directories in `${prefix}` (see the [Automatic environment variables](@ref) section): the libraries in `${libdir}`, the binary executables in `${bindir}`, etc...  Most packages come with a build system to automate this process (GNU Autoconf, CMake, Meson, a plain Makefile, etc...), but sometimes you may need to manually move the files as appropriate.
-
 ## Initiating different shell commands based on target
 
 Sometimes, you need to adapt build scripts based on the target platform. This can be done within the shell script. Here is an example from [`OpenBLAS`](https://github.com/JuliaPackaging/Yggdrasil/blob/685cdcec9f0f0a16f7b90a1671af88326dcf5ab1/O/OpenBLAS/build_tarballs.jl):
@@ -120,6 +116,7 @@ The following variables are useful to control the build script over different ta
 * `libdir`: the path to the directory where the shared libraries should be installed.  This is `${prefix}/bin` when building for Windows, `${prefix}/lib` for all other platforms
 * `bindir`: the path to the directory where the executables should be installed.  This is equivalent to `${prefix}/bin`
 * `includedir`: the path to the directory where the header files should be installed.  This is equivalent to `${prefix}/include`
+* similar variables, with analogous meaning, exist for the host prefix (where [`HostBuildDependency`](@ref) are installed): `${host_prefix}`, `${host_bindir}`, `${host_libdir}`, `${host_includedir}`
 * `target`: the target platform
 * `bb_full_target`: the full target platform, containing things like libstdc++ string ABI platform tags, and libgfortran version
 * `MACHTYPE`: the triplet of the host platform
@@ -136,14 +133,31 @@ For these target systems Clang is the default compiler, however some programs ma
 
 For programs built with CMake (see the [CMake build](#CMake-builds-1) section) you can use the GCC toolchain file that is in `${CMAKE_TARGET_TOOLCHAIN%.*}_gcc.cmake`.
 
-If the project that you want to build uses the GNU Build System (also knows as the Autotools), there isn't an automatic switch to use GCC, but you have to set the appropriate variables.  For example, this setting can be used to build most C/C++ programs with GCC for FreeBSD and macOS:
+If the project that you want to build uses the GNU Build System (also known as the Autotools), there isn't an automatic switch to use GCC, but you have to set the appropriate variables.  For example, this setting can be used to build most C/C++ programs with GCC for FreeBSD and macOS:
 ```sh
 if [[ "${target}" == *-freebsd* ]] || [[ "${target}" == *-apple-* ]]; then
     CC=gcc
     CXX=g++
 fi
 ```
 
+## Dependencies for the target system vs host system
+
+BinaryBuilder provides a cross-compilation environment, which means that in general there is a distinction between the target platform (where the build binaries will eventually run) and the host platform (where compilation is currently happening).  In particular, inside the build environment in general you cannot run binary executables built for the target platform.
+
+For a build to work there may be different kinds of dependencies, for example:
+
+* binary libraries that the final product of the current build (binary executables or other libraries) will need to link to.  These libraries must have been built for the target platform.  You can install this type of dependency as [`Dependency`](@ref), which will also be a dependency of the generated JLL package.  This is the most common class of dependencies;
+* binary libraries or non-binary executables (usually shell scripts that can actually be run inside the build environment) for the target platform that are exclusively needed during the build process, but not for the final product of the build to run on the target system.  You can install this type of dependency as [`BuildDependency`](@ref).  Remember they will _not_ be added as dependency of the generated JLL package;
+* binary executables that are exclusively needed to be run during the build process.  They cannot generally have been built for the target platform, so they cannot be installed as `Dependency` or `BuildDependency`.  However you have two options:
+
+  * if they are available in a JLL package for the `x86_64-linux-musl` platform, you can install them as [`HostBuildDependency`](@ref).  In order to keep binaries for the target platform separated from those for the host system, these dependencies will be installed under `${host_prefix}`, in particular executables will be present under `${host_bindir}` which is automatically added to the `${PATH}` environment variable;
+  * if they are present in Alpine Linux repositories, you can install them with the system package manager [`apk`](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management).
+
+  Remember that this class of dependencies is built for the host platform: if the library you want to build for the target platform requires another binary library to link to, installing it as `HostBuildDepency` or with `apk` will not help.
+
+You need to understand the build process of package you want to compile in order to know what of these classes a dependency belongs to.
+
 ## Installing the license file
 
 Generated tarballs should come with the license of the library that you want to install.  If at the end of a successful build there is only one directory inside `${WORKSPACE}/srcdir`, BinaryBuilder will look into it for files with typical names for license (like `LICENSE`, `COPYRIGHT`, etc... with some combinations of extensions) and automatically install them to `${prefix}/share/licenses/${SRC_NAME}/`.  If in the final tarball there are no files in this directory a warning will be issued, to remind you to provide a license file.
diff --git a/docs/src/building.md b/docs/src/building.md
@@ -322,19 +322,25 @@ dependencies = [
 ```
 
 * [`Dependency`](@ref) specify a JLL package that is necessary to build and load
-  the current builder;
+  the current builder.  Binaries for the target platform will be installed;
 * [`BuildDependency`](@ref) is a JLL package necessary only to build the current
-  package, but not to load it.  This dependency will not be added to the list of
-  the dependencies of the generated JLL package.
-
-The argument of `Dependency` and `BuildDependency` can also be a
-`Pkg.PackageSpec`, with which you can specify more details about the dependency,
-like a version number, or also a non-registered package.  Note that in Yggdrasil
-only JLL packages in the [General
+  package, but not to load it.  This dependency will install binaries for the
+  target platforms and will not be added to the list of the dependencies of the
+  generated JLL package;
+* [`HostBuildDependency`](@ref): similar to `BuildDependency`, but it will
+  install binaries for the host system.  This kind of dependency is usually
+  added to provide some binary utilities to run during the build process.
+
+The argument of `Dependency`, `BuildDependency`, and `HostBuildDependency` can
+also be a `Pkg.PackageSpec`, with which you can specify more details about the
+dependency, like a version number, or also a non-registered package.  Note that
+in Yggdrasil only JLL packages in the [General
 registry](https://github.com/JuliaRegistries/General) can be accepted.
 
-The dependencies will be installed under `${prefix}` within the build
-environment.
+The dependencies for the target system (`Dependency` and `BuildDependency`) will
+be installed under `${prefix}` within the build environment, while the
+dependencies for the host system (`HostBuildDependency`) will be installed under
+`${host_prefix}`.
 
 In the wizard, dependencies can be specified with the prompt: *Do you require
 any (binary) dependencies?  [y/N]*.
