Skip to content

Commit d27d82c

Browse files
committed
tc_build: Add basic support for building Rust
There a lot of options missing and it does not do any fancy kind of build. But it is a start, and it is already useful, e.g. Peter could have used it to test the new KCFI arity flag that requires LLVM 21 but upstream Rust still uses LLVM 20. I took the approach that the new script only takes care of building Rust provided an existing LLVM, which seemed simple and clear. Thus add the basic infrastructure, plus a bit of documentation. Add it to the CI, too. I successfully built it in a clean Debian 12 and Fedora 41. Cc: Peter Zijlstra <[email protected]> Signed-off-by: Miguel Ojeda <[email protected]>
1 parent 2e8cb39 commit d27d82c

File tree

6 files changed

+309
-1
lines changed

6 files changed

+309
-1
lines changed

.github/workflows/build.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ jobs:
1212
run: bash ci.sh deps
1313
- name: Build LLVM
1414
run: bash ci.sh llvm
15+
- name: Build Rust
16+
run: bash ci.sh rust
1517
- name: Build binutils
1618
run: bash ci.sh binutils
1719
- name: Build kernel

README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ These scripts have been tested in a Docker image of the following distributions
3737
lld \
3838
make \
3939
ninja-build \
40+
pkg-config \
4041
python3-dev \
4142
texinfo \
4243
u-boot-tools \
@@ -145,6 +146,19 @@ bfd plugin: LLVM gold plugin has failed to create LTO module: Unknown attribute
145146

146147
Having a standalone copy of binutils (ideally in the same folder at the LLVM toolchain so that only one `PATH` modification is needed) works around this without any adverse side effects. Another workaround is bind mounting the new `LLVMgold.so` to `/usr/lib/LLVMgold.so`.
147148

149+
## build-rust.py
150+
151+
By default, `./build-rust.py` will clone Rust and build it using an LLVM previously built by `./build-llvm.py`, e.g.:
152+
153+
```sh
154+
./build-llvm.py
155+
./build-rust.py
156+
```
157+
158+
This script does not apply any Rust-specific patches to LLVM.
159+
160+
Run `./build-rust.py -h` for more options and information.
161+
148162
## Contributing
149163

150164
This repository openly welcomes pull requests! There are a few presubmit checks that run to make sure the code stays consistently formatted and free of bugs.

build-rust.py

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
#!/usr/bin/env python3
2+
# pylint: disable=invalid-name
3+
4+
from argparse import ArgumentParser, RawTextHelpFormatter
5+
from pathlib import Path
6+
import textwrap
7+
import time
8+
9+
import tc_build.utils
10+
11+
from tc_build.rust import RustBuilder, RustSourceManager
12+
13+
# This is a known good revision of Rust for building the kernel
14+
GOOD_REVISION = '69b3959afec9b5468d5de15133b199553f6e55d2'
15+
16+
parser = ArgumentParser(formatter_class=RawTextHelpFormatter)
17+
clone_options = parser.add_mutually_exclusive_group()
18+
19+
parser.add_argument('--debug',
20+
help=textwrap.dedent('''\
21+
Build a debug compiler and standard library. This enables debug assertions,
22+
debug logging, overflow checks and debug info. The debug assertions and overflow
23+
checks can help catch issues when compiling.
24+
25+
'''),
26+
action='store_true')
27+
parser.add_argument('-b',
28+
'--build-folder',
29+
help=textwrap.dedent('''\
30+
By default, the script will create a "build/rust" folder in the same folder as this
31+
script and build each requested stage within that containing folder. To change the
32+
location of the containing build folder, pass it to this parameter. This can be either
33+
an absolute or relative path. If it is provided, then a custom LLVM install folder
34+
needs to be provided as well to prevent mistakes.
35+
36+
'''),
37+
type=str)
38+
parser.add_argument('-i',
39+
'--install-folder',
40+
help=textwrap.dedent('''\
41+
By default, the script will leave the toolchain in its build folder. To install it
42+
outside the build folder for persistent use, pass the installation location that you
43+
desire to this parameter. This can be either an absolute or relative path.
44+
45+
'''),
46+
type=str)
47+
parser.add_argument('-l',
48+
'--llvm-install-folder',
49+
help=textwrap.dedent('''\
50+
By default, the script will try to use a built LLVM by './build-llvm.py'. To use
51+
another LLVM installation (perhaps from './build-llvm.py --install-folder'), pass
52+
it to this parameter.
53+
54+
'''),
55+
type=str)
56+
parser.add_argument('-R',
57+
'--rust-folder',
58+
help=textwrap.dedent('''\
59+
By default, the script will clone the Rust project into the tc-build repo. If you have
60+
another Rust checkout that you would like to work out of, pass it to this parameter.
61+
This can either be an absolute or relative path. Implies '--no-update'. When this
62+
option is supplied, '--ref' and '--use-good-revision' do nothing, as the script does
63+
not manipulate a repository it does not own.
64+
65+
'''),
66+
type=str)
67+
parser.add_argument('-n',
68+
'--no-update',
69+
help=textwrap.dedent('''\
70+
By default, the script always updates the Rust repo before building. This prevents
71+
that, which can be helpful during something like bisecting or manually managing the
72+
repo to pin it to a particular revision.
73+
74+
'''),
75+
action='store_true')
76+
parser.add_argument('-r',
77+
'--ref',
78+
help=textwrap.dedent('''\
79+
By default, the script builds the main branch (tip of tree) of Rust. If you would
80+
like to build an older branch, use this parameter. This may be helpful in tracking
81+
down an older bug to properly bisect. This value is just passed along to 'git checkout'
82+
so it can be a branch name, tag name, or hash. This will have no effect if
83+
'--rust-folder' is provided, as the script does not manipulate a repository that it
84+
does not own.
85+
86+
'''),
87+
default='master',
88+
type=str)
89+
parser.add_argument('--show-build-commands',
90+
help=textwrap.dedent('''\
91+
By default, the script only shows the output of the comands it is running. When this option
92+
is enabled, the invocations of the build tools will be shown to help with reproducing
93+
issues outside of the script.
94+
95+
'''),
96+
action='store_true')
97+
clone_options.add_argument('--use-good-revision',
98+
help=textwrap.dedent('''\
99+
By default, the script updates Rust to the latest tip of tree revision, which may at times be
100+
broken or not work right. With this option, it will checkout a known good revision of Rust
101+
that builds and works properly. If you use this option often, please remember to update the
102+
script as the known good revision will change. This option may work best with a matching good
103+
revision used to build LLVM by './build-llvm.py'.
104+
105+
'''),
106+
action='store_const',
107+
const=GOOD_REVISION,
108+
dest='ref')
109+
parser.add_argument('--vendor-string',
110+
help=textwrap.dedent('''\
111+
Add this value to the Rust version string (like "rustc ... (ClangBuiltLinux)"). Useful when
112+
reverting or applying patches on top of upstream Rust to differentiate a toolchain built
113+
with this script from upstream Rust or to distinguish a toolchain built with this script
114+
from the system's Rust. Defaults to ClangBuiltLinux, can be set to an empty string to
115+
override this and have no vendor in the version string.
116+
117+
'''),
118+
type=str,
119+
default='ClangBuiltLinux')
120+
args = parser.parse_args()
121+
122+
# Start tracking time that the script takes
123+
script_start = time.time()
124+
125+
# Folder validation
126+
tc_build_folder = Path(__file__).resolve().parent
127+
src_folder = Path(tc_build_folder, 'src')
128+
129+
if args.build_folder:
130+
build_folder = Path(args.build_folder).resolve()
131+
132+
if not args.llvm_install_folder:
133+
raise RuntimeError(
134+
'Build folder customized, but no custom LLVM install folder provided -- this is likely a mistake. Provide both if you want to build in a custom folder?'
135+
)
136+
else:
137+
build_folder = Path(tc_build_folder, 'build/rust')
138+
139+
if args.llvm_install_folder:
140+
llvm_install_folder = Path(args.llvm_install_folder).resolve()
141+
else:
142+
llvm_install_folder = Path(tc_build_folder, 'build/llvm/final')
143+
144+
# Validate and configure Rust source
145+
if args.rust_folder:
146+
if not (rust_folder := Path(args.rust_folder).resolve()).exists():
147+
raise RuntimeError(f"Provided Rust folder ('{args.rust_folder}') does not exist?")
148+
else:
149+
rust_folder = Path(src_folder, 'rust')
150+
rust_source = RustSourceManager(rust_folder)
151+
rust_source.download(args.ref)
152+
if not (args.rust_folder or args.no_update):
153+
rust_source.update(args.ref)
154+
155+
# Build Rust
156+
tc_build.utils.print_header('Building Rust')
157+
158+
# Final build
159+
final = RustBuilder()
160+
final.folders.source = rust_folder
161+
final.folders.build = Path(build_folder, 'final')
162+
final.folders.install = Path(args.install_folder).resolve() if args.install_folder else None
163+
final.llvm_install_folder = llvm_install_folder
164+
final.debug = args.debug
165+
final.vendor_string = args.vendor_string
166+
final.show_commands = args.show_build_commands
167+
168+
final.configure()
169+
final.build()
170+
final.show_install_info()
171+
172+
print(f"Script duration: {tc_build.utils.get_duration(script_start)}")

ci.sh

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ set -eu
99
function parse_parameters() {
1010
while (($#)); do
1111
case $1 in
12-
all | binutils | deps | kernel | llvm) action=$1 ;;
12+
all | binutils | deps | kernel | llvm | rust) action=$1 ;;
1313
*) exit 33 ;;
1414
esac
1515
shift
@@ -19,6 +19,7 @@ function parse_parameters() {
1919
function do_all() {
2020
do_deps
2121
do_llvm
22+
do_rust
2223
do_binutils
2324
do_kernel
2425
}
@@ -54,6 +55,7 @@ function do_deps() {
5455
lld \
5556
make \
5657
ninja-build \
58+
pkg-config \
5759
python3 \
5860
texinfo \
5961
xz-utils \
@@ -111,5 +113,13 @@ function do_llvm() {
111113
"${extra_args[@]}"
112114
}
113115

116+
function do_rust() {
117+
"$base"/build-rust.py \
118+
--debug \
119+
--llvm-install-folder "$install" \
120+
--install-folder "$install" \
121+
--show-build-commands
122+
}
123+
114124
parse_parameters "$@"
115125
do_"${action:=all}"

tc_build/builder.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,12 @@ def clean_build_folder(self):
3232
else:
3333
self.folders.build.unlink()
3434

35+
def make_build_folder(self):
36+
if not self.folders.build:
37+
raise RuntimeError('No build folder set?')
38+
39+
self.folders.build.mkdir(parents=True)
40+
3541
def run_cmd(self, cmd, capture_output=False, cwd=None):
3642
if self.show_commands:
3743
# Acts sort of like 'set -x' in bash

tc_build/rust.py

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
#!/usr/bin/env python3
2+
3+
from pathlib import Path
4+
import subprocess
5+
import time
6+
7+
from tc_build.builder import Builder
8+
from tc_build.source import GitSourceManager
9+
import tc_build.utils
10+
11+
12+
class RustBuilder(Builder):
13+
14+
def __init__(self):
15+
super().__init__()
16+
17+
self.llvm_install_folder = None
18+
self.debug = False
19+
self.vendor_string = ''
20+
21+
def build(self):
22+
if not self.folders.build:
23+
raise RuntimeError('No build folder set for build()?')
24+
if not Path(self.folders.build, 'bootstrap.toml').exists():
25+
raise RuntimeError('No bootstrap.toml in build folder, run configure()?')
26+
27+
build_start = time.time()
28+
self.run_cmd([Path(self.folders.source, 'x.py'), 'install'], cwd=self.folders.build)
29+
30+
tc_build.utils.print_info(f"Build duration: {tc_build.utils.get_duration(build_start)}")
31+
32+
if self.folders.install:
33+
tc_build.utils.create_gitignore(self.folders.install)
34+
35+
def configure(self):
36+
if not self.llvm_install_folder:
37+
raise RuntimeError('No LLVM install folder set?')
38+
if not self.folders.source:
39+
raise RuntimeError('No source folder set?')
40+
if not self.folders.build:
41+
raise RuntimeError('No build folder set?')
42+
43+
# Configure the build
44+
#
45+
# 'codegen-tests' requires '-DLLVM_INSTALL_UTILS=ON'.
46+
install_folder = self.folders.install if self.folders.install else self.folders.build
47+
48+
# yapf: disable
49+
configure_cmd = [
50+
Path(self.folders.source, 'configure'),
51+
'--release-description', self.vendor_string,
52+
'--disable-docs',
53+
'--enable-locked-deps',
54+
'--tools', 'cargo,clippy,rustdoc,rustfmt,src',
55+
'--prefix', install_folder,
56+
'--sysconfdir', 'etc',
57+
'--disable-codegen-tests',
58+
'--disable-lld',
59+
'--disable-llvm-bitcode-linker',
60+
'--llvm-root', self.llvm_install_folder,
61+
]
62+
# yapf: enable
63+
64+
if self.debug:
65+
configure_cmd.append('--enable-debug')
66+
67+
self.clean_build_folder()
68+
self.make_build_folder()
69+
self.run_cmd(configure_cmd, cwd=self.folders.build)
70+
71+
def show_install_info(self):
72+
# Installation folder is optional, show build folder as the
73+
# installation location in that case.
74+
install_folder = self.folders.install if self.folders.install else self.folders.build
75+
if not install_folder:
76+
raise RuntimeError('Installation folder not set?')
77+
if not install_folder.exists():
78+
raise RuntimeError('Installation folder does not exist, run build()?')
79+
if not (bin_folder := Path(install_folder, 'bin')).exists():
80+
raise RuntimeError('bin folder does not exist in installation folder, run build()?')
81+
82+
tc_build.utils.print_header('Rust installation information')
83+
install_info = (f"Toolchain is available at: {install_folder}\n\n"
84+
'To use, either run:\n\n'
85+
f"\t$ export PATH={bin_folder}:$PATH\n\n"
86+
'or add:\n\n'
87+
f"\tPATH={bin_folder}:$PATH\n\n"
88+
'before the command you want to use this toolchain.\n')
89+
print(install_info)
90+
91+
for tool in ['rustc', 'rustdoc', 'rustfmt', 'clippy-driver', 'cargo']:
92+
if (binary := Path(bin_folder, tool)).exists():
93+
subprocess.run([binary, '--version', '--verbose'], check=True)
94+
print()
95+
tc_build.utils.flush_std_err_out()
96+
97+
98+
class RustSourceManager(GitSourceManager):
99+
100+
def __init__(self, repo):
101+
super().__init__(repo)
102+
103+
self._pretty_name = 'Rust'
104+
self._repo_url = 'https://github.com/rust-lang/rust.git'

0 commit comments

Comments
 (0)