Skip to content

Commit 10df88d

Browse files
asbasb
committed
[docs] Rewrite HowToCrossCompileLLVM
The document has had a few minor tweaks over the years, but the last major piece of work on it was 2016, after first being introduced in 2013. My aim is to provide a clear and clean recipe for cross-compiling LLVM that: * Should be achievable for anyone on common variants of Linux (_including_ the step of acquiring a working sysroot). * I think I've kept the coverage of setting up acquiring a Debian sysroot minimal enough that it can reasonably be included. `debootstrap` is packaged for most common Linux distributions including non-Debian derived distributions like Arch Linux and Fedora. * Describes a setup that we can reasonably support within the community. * I realise with the ninja symlink canonicalisation issue I haven't completely avoided hacks, but I look particularly to point 2 under hacks in the current docs which talks about libraries on the host being found by CMake and adding `-L` and `-I` to try to hack around this. We've all been there and made these kind of temporary workarounds to see if we can get further, but it's very hard to support someone who has problems with a setup that's improperly leaking between the host and target like this. The approach I describe with a clean sysroot and setting appropriate `CMAKE_FIND_ROOT_PATH_MODE_*` settings doesn't have this issue. * Cuts down on extraneous / outdated information, especially where it is better covered elsewhere (e.g. detailed descriptions of CMake options not directly relevant to cross compilation). I've run through the instructions for AArch64, RISC-V (64-bit), and armhf.
1 parent 0751418 commit 10df88d

File tree

1 file changed

+155
-163
lines changed

1 file changed

+155
-163
lines changed

‎llvm/docs/HowToCrossCompileLLVM.rst‎

Lines changed: 155 additions & 163 deletions
Original file line numberDiff line numberDiff line change
@@ -6,210 +6,202 @@ Introduction
66
============
77

88
This document contains information about building LLVM and
9-
Clang on host machine, targeting another platform.
9+
Clang on a host machine, targeting another platform.
1010

1111
For more information on how to use Clang as a cross-compiler,
1212
please check https://clang.llvm.org/docs/CrossCompilation.html.
1313

14-
TODO: Add MIPS and other platforms to this document.
14+
This document describes cross-building a compiler in a single stage, using an
15+
existing ``clang`` install as the host compiler.
1516

16-
Cross-Compiling from x86_64 to ARM
17-
==================================
1817

19-
In this use case, we'll be using CMake and Ninja, on a Debian-based Linux
20-
system, cross-compiling from an x86_64 host (most Intel and AMD chips
21-
nowadays) to a hard-float ARM target (most ARM targets nowadays).
18+
Cross-Compiling from x86_64 to ARM, AArch64, or RISC-V
19+
======================================================
2220

23-
The packages you'll need are:
21+
These instructions have been tested for targeting 32-bit ARM, AArch64, or
22+
RISC-V but should be equally applicable to any other target.
2423

25-
* ``cmake``
26-
* ``ninja-build`` (from backports in Ubuntu)
27-
* ``gcc-4.7-arm-linux-gnueabihf``
28-
* ``gcc-4.7-multilib-arm-linux-gnueabihf``
29-
* ``binutils-arm-linux-gnueabihf``
30-
* ``libgcc1-armhf-cross``
31-
* ``libsfgcc1-armhf-cross``
32-
* ``libstdc++6-armhf-cross``
33-
* ``libstdc++6-4.7-dev-armhf-cross``
3424

35-
Configuring CMake
36-
-----------------
37-
38-
For more information on how to configure CMake for LLVM/Clang,
39-
see :doc:`CMake`.
40-
41-
The CMake options you need to add are:
42-
43-
* ``-DCMAKE_SYSTEM_NAME=<target-system>``
44-
* ``-DCMAKE_INSTALL_PREFIX=<install-dir>``
45-
* ``-DLLVM_HOST_TRIPLE=arm-linux-gnueabihf``
46-
* ``-DLLVM_TARGETS_TO_BUILD=ARM``
47-
48-
Note: ``CMAKE_CROSSCOMPILING`` is always set automatically when ``CMAKE_SYSTEM_NAME`` is set. Don't put ``-DCMAKE_CROSSCOMPILING=TRUE`` in your options.
49-
50-
Also note that ``LLVM_HOST_TRIPLE`` specifies the triple of the system
51-
that the cross built LLVM is going to run on - the flag is named based
52-
on the autoconf build/host/target nomenclature. (This flag implicitly sets
53-
other defaults, such as ``LLVM_DEFAULT_TARGET_TRIPLE``.)
54-
55-
If you're compiling with GCC, you can use architecture options for your target,
56-
and the compiler driver will detect everything that it needs:
57-
58-
* ``-DCMAKE_CXX_FLAGS='-march=armv7-a -mcpu=cortex-a9 -mfloat-abi=hard'``
59-
60-
However, if you're using Clang, the driver might not be up-to-date with your
61-
specific Linux distribution, version or GCC layout, so you'll need to fudge.
62-
63-
In addition to the ones above, you'll also need:
64-
65-
* ``--target=arm-linux-gnueabihf`` or whatever is the triple of your cross GCC.
66-
* ``'--sysroot=/usr/arm-linux-gnueabihf'``, ``'--sysroot=/opt/gcc/arm-linux-gnueabihf'``
67-
or whatever is the location of your GCC's sysroot (where /lib, /bin etc are).
68-
* Appropriate use of ``-I`` and ``-L``, depending on how the cross GCC is installed,
69-
and where are the libraries and headers.
70-
71-
You may also want to set the ``LLVM_NATIVE_TOOL_DIR`` option - pointing
72-
at a directory with prebuilt LLVM tools (``llvm-tblgen``, ``clang-tblgen``
73-
etc) for the build host, allowing you to them reuse them if available.
74-
E.g. ``-DLLVM_NATIVE_TOOL_DIR=<path-to-native-llvm-build>/bin``.
75-
If the option isn't set (or the directory doesn't contain all needed tools),
76-
the LLVM cross build will automatically launch a nested build to build the
77-
tools that are required.
78-
79-
The CXX flags define the target, cpu (which in this case
80-
defaults to ``fpu=VFP3`` with NEON), and forcing the hard-float ABI. If you're
81-
using Clang as a cross-compiler, you will *also* have to set ``--sysroot``
82-
to make sure it picks the correct linker.
83-
84-
When using Clang, it's important that you choose the triple to be *identical*
85-
to the GCC triple and the sysroot. This will make it easier for Clang to
86-
find the correct tools and include headers. But that won't mean all headers and
87-
libraries will be found. You'll still need to use ``-I`` and ``-L`` to locate
88-
those extra ones, depending on your distribution.
89-
90-
Most of the time, what you want is to have a native compiler to the
91-
platform itself, but not others. So there's rarely a point in compiling
92-
all back-ends. For that reason, you should also set the
93-
``TARGETS_TO_BUILD`` to only build the back-end you're targeting to.
25+
Setting up an appropriate sysroot
26+
---------------------------------
9427

95-
You must set the ``CMAKE_INSTALL_PREFIX``, otherwise a ``ninja install``
96-
will copy ARM binaries to your root filesystem, which is not what you
97-
want.
28+
You will need a sysroot that contains essential build dependencies compiled
29+
for the target architecture. In this case, we'll be using CMake and Ninja on a
30+
Linux host and compiling against a Debian sysroot. Detailed instructions on
31+
producing sysroots are outside of the scope of this documentation, but the
32+
following should work on any Linux distribution with the following
33+
pre-requisites:
9834

99-
Hacks
100-
-----
35+
* ``binfmt_misc`` configured to execute ``qemu-user`` for binaries of the
36+
target architecture. This is done by installing the ``qemu-user-static``
37+
and ``binfmt-support`` packages on Debian-derived distributions.
38+
* Root access (setups involving ``proot`` or other tools to avoid this
39+
requirement may be possible, but aren't described here).
40+
* The ``debootstrap`` tool. This is available in most distributions.
10141

102-
There are some bugs in current LLVM, which require some fiddling before
103-
running CMake:
104-
105-
#. If you're using Clang as the cross-compiler, there is a problem in
106-
the LLVM ARM back-end that is producing absolute relocations on
107-
position-independent code (``R_ARM_THM_MOVW_ABS_NC``), so for now, you
108-
should disable PIC:
42+
The following snippet will initialise sysroots for 32-bit Arm, AArch64, and
43+
RISC-V (you can of course just pick the target you're interested in):
10944

11045
.. code-block:: bash
11146
112-
-DLLVM_ENABLE_PIC=False
113-
114-
This is not a problem, since Clang/LLVM libraries are statically
115-
linked anyway, it shouldn't affect much.
47+
sudo debootstrap --arch=armhf --variant=minbase --include=build-essential,symlinks stable sysroot-deb-armhf-stable
48+
sudo debootstrap --arch=arm64 --variant=minbase --include=build-essential,symlinks stable sysroot-deb-arm64-stable
49+
sudo debootstrap --arch=riscv64 --variant=minbase --include=build-essential,symlinks unstable sysroot-deb-riscv64-unstable
11650
117-
#. The ARM libraries won't be installed in your system.
118-
But the CMake prepare step, which checks for
119-
dependencies, will check the *host* libraries, not the *target*
120-
ones. Below there's a list of some dependencies, but your project could
121-
have more, or this document could be outdated. You'll see the errors
122-
while linking as an indication of that.
51+
The created sysroot may contain absolute symlinks, which will resolve to a
52+
location within the host when accessed during compilation, so we must convert
53+
any absolute symlinks to relative ones:
12354

124-
Debian based distros have a way to add ``multiarch``, which adds
125-
a new architecture and allows you to install packages for those
126-
systems. See https://wiki.debian.org/Multiarch/HOWTO for more info.
127-
128-
But not all distros will have that, and possibly not an easy way to
129-
install them in any anyway, so you'll have to build/download
130-
them separately.
55+
.. code-block:: bash
13156
132-
A quick way of getting the libraries is to download them from
133-
a distribution repository, like Debian (http://packages.debian.org/jessie/),
134-
and download the missing libraries. Note that the ``libXXX``
135-
will have the shared objects (``.so``) and the ``libXXX-dev`` will
136-
give you the headers and the static (``.a``) library. Just in
137-
case, download both.
57+
sudo chroot sysroot-of-your-choice symlinks -cr .
13858
139-
The ones you need for ARM are: ``libtinfo``, ``zlib1g``,
140-
``libxml2`` and ``liblzma``. In the Debian repository you'll
141-
find downloads for all architectures.
14259
143-
After you download and unpack all ``.deb`` packages, copy all
144-
``.so`` and ``.a`` to a directory, make the appropriate
145-
symbolic links (if necessary), and add the relevant ``-L``
146-
and ``-I`` paths to ``-DCMAKE_CXX_FLAGS`` above.
60+
Configuring CMake
61+
-----------------
14762

63+
For more information on how to configure CMake for LLVM/Clang,
64+
see :doc:`CMake`. Following CMake's recommended practice, we will create a
65+
`toolchain file
66+
<https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html#toolchain-files>`_.
14867

149-
Running CMake and Building
150-
--------------------------
68+
The following assumes you have a system install of ``clang`` and ``lld`` that
69+
will be used for cross compiling and that the listed commands are executed
70+
from within the root of a checkout of the ``llvm-project`` git repository.
15171

152-
Finally, if you're using your platform compiler, run:
72+
First create a appropriate toolchain file:
15373

15474
.. code-block:: bash
15575
156-
$ cmake -G Ninja <source-dir> -DCMAKE_BUILD_TYPE=<type> <options above>
157-
158-
If you're using Clang as the cross-compiler, run:
76+
SYSROOT=$HOME/sysroot-deb-arm64-stable
77+
TARGET=aarch64-linux-gnu
78+
CFLAGS=""
79+
80+
cat - <<EOF > $TARGET-clang.cmake
81+
set(CMAKE_SYSTEM_NAME Linux)
82+
set(CMAKE_SYSROOT "$SYSROOT")
83+
set(CMAKE_C_COMPILER_TARGET $TARGET)
84+
set(CMAKE_CXX_COMPILER_TARGET $TARGET)
85+
set(CMAKE_C_FLAGS_INIT "$CFLAGS")
86+
set(CMAKE_CXX_FLAGS_INIT "$CFLAGS")
87+
set(CMAKE_LINKER_TYPE LLD)
88+
set(CMAKE_C_COMPILER clang)
89+
set(CMAKE_CXX_COMPILER clang++)
90+
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
91+
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
92+
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
93+
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
94+
EOF
95+
96+
To customise details of the compilation target or choose a different
97+
architecture altogether, just change the ``SYSROOT``,
98+
``TARGET``, and ``CFLAGS`` variables before copy and pasting the rest (e.g.
99+
for RISC-V you might set ``SYSROOT=$HOME/sysroot-deb-riscv64-unstable``,
100+
``TARGET=riscv64-linux-gnu``, ``CFLAGS="-march=rva20u64"``).
101+
102+
Then configure and build with ``cmake`` invocations such as the following:
159103
160104
.. code-block:: bash
161105
162-
$ CC='clang' CXX='clang++' cmake -G Ninja <source-dir> -DCMAKE_BUILD_TYPE=<type> <options above>
163-
164-
If you have ``clang``/``clang++`` on the path, it should just work, and special
165-
Ninja files will be created in the build directory. I strongly suggest
166-
you to run ``cmake`` on a separate build directory, *not* inside the
167-
source tree.
168-
169-
To build, simply type:
106+
cmake -G Ninja \
107+
-DCMAKE_BUILD_TYPE=Release \
108+
-DLLVM_ENABLE_PROJECTS="lld;clang" \
109+
-DCMAKE_TOOLCHAIN_FILE=$(pwd)/$TARGET-clang.cmake \
110+
-DLLVM_HOST_TRIPLE=$TARGET \
111+
-DCMAKE_INSTALL_PREFIX=$HOME/clang-$TARGET \
112+
-S llvm \
113+
-B build/$TARGET
114+
cmake --build build/$TARGET
115+
116+
The following options from the toolchain file and ``cmake`` invocation above
117+
are worth highlighting:
118+
119+
* ``CMAKE_SYSTEM_NAME``: Perhaps surprisingly, explicitly setting this
120+
variable `causes CMake to set
121+
CMAKE_CROSSCOMPIILING <https://cmake.org/cmake/help/latest/variable/CMAKE_CROSSCOMPILING.html#variable:CMAKE_CROSSCOMPILING>`_.
122+
* ``CMAKE_{C,CXX}_COMPILER_TARGET``: This will be used to set the
123+
``--target`` argument to ``clang``. The triple should match the triple used
124+
within the sysroot (i.e. ``$SYSROOT/usr/lib/$TARGET`` should exist).
125+
* ``CMAKE_FIND_ROOT_PATH_MODE_*``: These `control the search behaviour for
126+
finding libraries, includes or binaries
127+
<https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html#finding-external-libraries-programs-and-other-files>`_
128+
and are set so as to avoid files for the host being used in the build.
129+
* ``LLVM_HOST_TRIPLE``: Specifies the target triple the built LLVM will run
130+
on, which also implicitly sets other defaults such as
131+
``LLVM_DEFAULT_TARGET_TRIPLE``.
132+
* ``CMAKE_SYSROOT``: Gives the path to the sysroot containing libraries and
133+
headers for the target, as generated earlier in this document.
134+
* ``CMAKE_INSTALL_PREFIX``: Unnecessary if you don't intend to use the
135+
``install`` target. But setting it avoids installing binaries compiled for
136+
the target across system directories on the host system.
137+
138+
See `LLVM's build documentation
139+
<https://llvm.org/docs/CMake.html#frequently-used-cmake-variables>`_ for more
140+
guidance on CMake variables (e.g. ``LLVM_TARGETS_TO_BUILD`` may be useful if
141+
your cross-compiled binaries only need to support compiling for one target).
142+
143+
Assuming you have ``qemu-user`` installed you can test the produced target
144+
binaries either by relying on binfmt_misc (as was necessary for debootstrap)
145+
or invoking ``qemu-{tgt}-static`` directly. For instance, to first check the
146+
type of executable and then run ``clang --version`` both ways:
170147
171148
.. code-block:: bash
172149
173-
$ ninja
150+
$ file -L ./build/aarch64-linux-gnu/bin/clang
151+
./build/aarch64-linux-gnu/bin/clang: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=516b8b366a790fcd3563bee4aec0cdfcb90bb1c7, not stripped
152+
153+
$ qemu-aarch64-static -L $SYSROOT ./build/aarch64-linux-gnu/bin/clang --version
154+
clang version 21.0.0git (https://github.com/llvm/llvm-project cedfdc6e889c5c614a953ed1f44bcb45a405f8da)
155+
Target: aarch64-unknown-linux-gnu
156+
Thread model: posix
157+
InstalledDir: /home/asb/llvm-project/build/aarch64-linux-gnu/bin
158+
159+
$ export QEMU_LD_PREFIX=$SYSROOT; ./build/aarch64-linux-gnu/bin/clang --version
160+
clang version 21.0.0git (https://github.com/llvm/llvm-project cedfdc6e889c5c614a953ed1f44bcb45a405f8da)
161+
Target: aarch64-unknown-linux-gnu
162+
Thread model: posix
163+
InstalledDir: /home/asb/llvm-project/build/aarch64-linux-gnu/bin
164+
165+
Working around a ninja dependency issue
166+
---------------------------------------
167+
168+
If you followed the instructions above to create a sysroot, you may run into a
169+
`longstanding problem related to path canonicalization in ninja
170+
<https://github.com/ninja-build/ninja/issues/1330>_`. GCC canonicalizes system
171+
headers in dependency files, so when ninja reads them it doesn't need to do
172+
so. Clang doesn't, and unfortunately ninja doesn't implement the
173+
canonicalization logic at all, meaning for some system headers with symlinks
174+
in the paths, it can incorrectly compute a non-existing path and consider it
175+
as always dirty. If you are suffering from this issue, you will find at any
176+
attempt at an incremental build (including the suggested command to build the
177+
``install`` target in the next section) results in recompiling everything.
178+
``ninja -C build/$TARGET -t deps`` shows files in ``$SYSROOT/include/*`` that
179+
don't (as the ``$SYSROOT/include`` folder doesn't exist) and you can
180+
further confirm these files are causing the build to be seen as dirty with
181+
``ninja -C build/$TARGET -d deps``. Until such time as the issue is avoided
182+
due to a change in behaviour for Clang or for Ninja, a functional workaround
183+
is to create a symlink so that the incorrect ``$SYSROOT/include/*``
184+
dependencies resolve to files within ``$SYSROOT/usr/include/*``. This works in
185+
practice for the simple cross-compilation use case described here, but isn't
186+
a general solution.
174187
175-
It should automatically find out how many cores you have, what are
176-
the rules that needs building and will build the whole thing.
188+
.. code-block:: bash
177189
178-
You can't run ``ninja check-all`` on this tree because the created
179-
binaries are targeted to ARM, not x86_64.
190+
sudo ln -s usr/include $SYSROOT/include
180191
181-
Installing and Using
192+
Installing and using
182193
--------------------
183194
184-
After the LLVM/Clang has built successfully, you should install it
185-
via:
186-
187-
.. code-block:: bash
188-
189-
$ ninja install
190-
191-
which will create a sysroot on the install-dir. You can then tar
192-
that directory into a binary with the full triple name (for easy
193-
identification), like:
195+
After LLVM/Clang has built successfully, you can install it via:
194196
195197
.. code-block:: bash
196198
197-
$ ln -sf <install-dir> arm-linux-gnueabihf-clang
198-
$ tar zchf arm-linux-gnueabihf-clang.tar.gz arm-linux-gnueabihf-clang
199+
cmake --build build/$TARGET --target=install
199200
200-
If you copy that tarball to your target board, you'll be able to use
201-
it for running the test-suite, for example. Follow the guidelines at
202-
https://llvm.org/docs/lnt/quickstart.html, unpack the tarball in the
203-
test directory, and use options:
201+
This will install the LLVM/Clang headers, binaries, libraries, and other files
202+
to paths within ``CMAKE_INSTALL_PREFIX`` You can then tar that directory for
203+
transfer to a device that runs the target architecture natively:
204204
205205
.. code-block:: bash
206206
207-
$ ./sandbox/bin/python sandbox/bin/lnt runtest nt \
208-
--sandbox sandbox \
209-
--test-suite `pwd`/test-suite \
210-
--cc `pwd`/arm-linux-gnueabihf-clang/bin/clang \
211-
--cxx `pwd`/arm-linux-gnueabihf-clang/bin/clang++
212-
213-
Remember to add the ``-jN`` options to ``lnt`` to the number of CPUs
214-
on your board. Also, the path to your clang has to be absolute, so
215-
you'll need the `pwd` trick above.
207+
tar -czvf clang-$TARGET.tar.gz -C $HOME clang-$TARGET

0 commit comments

Comments
 (0)