Skip to content

Commit 3b4e793

Browse files
bd1976brisbd1976bris
authored
[DTLTO][LLD][ELF] Add support for Integrated Distributed ThinLTO (llvm#142757)
This patch introduces support for Integrated Distributed ThinLTO (DTLTO) in ELF LLD. DTLTO enables the distribution of ThinLTO backend compilations via external distribution systems, such as Incredibuild, during the traditional link step: https://llvm.org/docs/DTLTO.html. It is expected that users will invoke DTLTO through the compiler driver (e.g., Clang) rather than calling LLD directly. A Clang-side interface for DTLTO will be added in a follow-up patch. Note: Bitcode members of archives (thin or non-thin) are not currently supported. This will be addressed in a future change. As a consequence of this lack of support, this patch is not sufficient to allow for self-hosting an LLVM build with DTLTO. Theoretically, --start-lib/--end-lib could be used instead of archives in a self-host build. However, it's unclear how --start-lib/--end-lib can be easily used with the LLVM build system. Testing: - ELF LLD `lit` test coverage has been added, using a mock distributor to avoid requiring Clang. - Cross-project `lit` tests cover integration with Clang. For the design discussion of the DTLTO feature, see: llvm#126654.
1 parent e324392 commit 3b4e793

File tree

14 files changed

+306
-1
lines changed

14 files changed

+306
-1
lines changed

‎cross-project-tests/CMakeLists.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,13 @@ add_lit_testsuite(check-cross-amdgpu "Running AMDGPU cross-project tests"
9494
DEPENDS clang
9595
)
9696

97+
# DTLTO tests.
98+
add_lit_testsuite(check-cross-dtlto "Running DTLTO cross-project tests"
99+
${CMAKE_CURRENT_BINARY_DIR}/dtlto
100+
EXCLUDE_FROM_CHECK_ALL
101+
DEPENDS ${CROSS_PROJECT_TEST_DEPS}
102+
)
103+
97104
# Add check-cross-project-* targets.
98105
add_lit_testsuites(CROSS_PROJECT ${CMAKE_CURRENT_SOURCE_DIR}
99106
DEPENDS ${CROSS_PROJECT_TEST_DEPS}

‎cross-project-tests/dtlto/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Tests for DTLTO (Integrated Distributed ThinLTO) functionality.
2+
3+
These are integration tests as DTLTO invokes `clang` for code-generation.

‎cross-project-tests/dtlto/ld-dtlto.c‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// REQUIRES: ld.lld
2+
3+
/// Simple test that DTLTO works with a single input bitcode file and that
4+
/// --save-temps can be applied to the remote compilation.
5+
6+
// RUN: rm -rf %t && mkdir %t && cd %t
7+
8+
// RUN: %clang --target=x86_64-linux-gnu -c -flto=thin %s -o dtlto.o
9+
10+
// RUN: ld.lld dtlto.o \
11+
// RUN: --thinlto-distributor=%python \
12+
// RUN: --thinlto-distributor-arg=%llvm_src_root/utils/dtlto/local.py \
13+
// RUN: --thinlto-remote-compiler=%clang \
14+
// RUN: --thinlto-remote-compiler-arg=--save-temps
15+
16+
/// Check that the required output files have been created.
17+
// RUN: ls | sort | FileCheck %s
18+
19+
/// No files are expected before.
20+
// CHECK-NOT: {{.}}
21+
22+
/// Linked ELF.
23+
// CHECK: {{^}}a.out{{$}}
24+
25+
/// Produced by the bitcode compilation.
26+
// CHECK-NEXT: {{^}}dtlto.o{{$}}
27+
28+
/// --save-temps output for the backend compilation.
29+
// CHECK-NEXT: {{^}}dtlto.s{{$}}
30+
// CHECK-NEXT: {{^}}dtlto.s.0.preopt.bc{{$}}
31+
// CHECK-NEXT: {{^}}dtlto.s.1.promote.bc{{$}}
32+
// CHECK-NEXT: {{^}}dtlto.s.2.internalize.bc{{$}}
33+
// CHECK-NEXT: {{^}}dtlto.s.3.import.bc{{$}}
34+
// CHECK-NEXT: {{^}}dtlto.s.4.opt.bc{{$}}
35+
// CHECK-NEXT: {{^}}dtlto.s.5.precodegen.bc{{$}}
36+
// CHECK-NEXT: {{^}}dtlto.s.resolution.txt{{$}}
37+
38+
/// No files are expected after.
39+
// CHECK-NOT: {{.}}
40+
41+
int _start() { return 0; }

‎cross-project-tests/dtlto/lit.local.cfg‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
if any(
2+
f not in config.available_features
3+
for f in ("clang", "x86-registered-target")
4+
):
5+
config.unsupported = True

‎lld/ELF/Config.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -275,6 +275,10 @@ struct Config {
275275
llvm::SmallVector<llvm::StringRef, 0> searchPaths;
276276
llvm::SmallVector<llvm::StringRef, 0> symbolOrderingFile;
277277
llvm::SmallVector<llvm::StringRef, 0> thinLTOModulesToCompile;
278+
llvm::StringRef dtltoDistributor;
279+
llvm::SmallVector<llvm::StringRef, 0> dtltoDistributorArgs;
280+
llvm::StringRef dtltoCompiler;
281+
llvm::SmallVector<llvm::StringRef, 0> dtltoCompilerArgs;
278282
llvm::SmallVector<llvm::StringRef, 0> undefined;
279283
llvm::SmallVector<SymbolVersion, 0> dynamicList;
280284
llvm::SmallVector<uint8_t, 0> buildIdVector;

‎lld/ELF/Driver.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1396,6 +1396,11 @@ static void readConfigs(Ctx &ctx, opt::InputArgList &args) {
13961396
args.hasFlag(OPT_dependent_libraries, OPT_no_dependent_libraries, true);
13971397
ctx.arg.disableVerify = args.hasArg(OPT_disable_verify);
13981398
ctx.arg.discard = getDiscard(args);
1399+
ctx.arg.dtltoDistributor = args.getLastArgValue(OPT_thinlto_distributor_eq);
1400+
ctx.arg.dtltoDistributorArgs =
1401+
args::getStrings(args, OPT_thinlto_distributor_arg);
1402+
ctx.arg.dtltoCompiler = args.getLastArgValue(OPT_thinlto_compiler_eq);
1403+
ctx.arg.dtltoCompilerArgs = args::getStrings(args, OPT_thinlto_compiler_arg);
13991404
ctx.arg.dwoDir = args.getLastArgValue(OPT_plugin_opt_dwo_dir_eq);
14001405
ctx.arg.dynamicLinker = getDynamicLinker(ctx, args);
14011406
ctx.arg.ehFrameHdr =

‎lld/ELF/LTO.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,13 @@ BitcodeCompiler::BitcodeCompiler(Ctx &ctx) : ctx(ctx) {
180180
std::string(ctx.arg.thinLTOPrefixReplaceNew),
181181
std::string(ctx.arg.thinLTOPrefixReplaceNativeObject),
182182
ctx.arg.thinLTOEmitImportsFiles, indexFile.get(), onIndexWrite);
183+
} else if (!ctx.arg.dtltoDistributor.empty()) {
184+
backend = lto::createOutOfProcessThinBackend(
185+
llvm::hardware_concurrency(ctx.arg.thinLTOJobs), onIndexWrite,
186+
ctx.arg.thinLTOEmitIndexFiles, ctx.arg.thinLTOEmitImportsFiles,
187+
ctx.arg.outputFile, ctx.arg.dtltoDistributor,
188+
ctx.arg.dtltoDistributorArgs, ctx.arg.dtltoCompiler,
189+
ctx.arg.dtltoCompilerArgs, !ctx.arg.saveTempsArgs.empty());
183190
} else {
184191
backend = lto::createInProcessThinBackend(
185192
llvm::heavyweight_hardware_concurrency(ctx.arg.thinLTOJobs),

‎lld/ELF/Options.td‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -714,7 +714,17 @@ def thinlto_object_suffix_replace_eq: JJ<"thinlto-object-suffix-replace=">;
714714
def thinlto_prefix_replace_eq: JJ<"thinlto-prefix-replace=">;
715715
def thinlto_single_module_eq: JJ<"thinlto-single-module=">,
716716
HelpText<"Specify a single module to compile in ThinLTO mode, for debugging only">;
717-
717+
def thinlto_distributor_eq: JJ<"thinlto-distributor=">,
718+
HelpText<"Distributor to use for ThinLTO backend compilations. If specified, "
719+
"ThinLTO backend compilations will be distributed">;
720+
defm thinlto_distributor_arg: EEq<"thinlto-distributor-arg", "Arguments to "
721+
"pass to the ThinLTO distributor">;
722+
def thinlto_compiler_eq: JJ<"thinlto-remote-compiler=">,
723+
HelpText<"Compiler for the ThinLTO distributor to invoke for ThinLTO backend "
724+
"compilations">;
725+
defm thinlto_compiler_arg: EEq<"thinlto-remote-compiler-arg", "Compiler "
726+
"arguments for the ThinLTO distributor to pass for ThinLTO backend "
727+
"compilations">;
718728
defm fat_lto_objects: BB<"fat-lto-objects",
719729
"Use the .llvm.lto section, which contains LLVM bitcode, in fat LTO object files to perform LTO.",
720730
"Ignore the .llvm.lto section in relocatable object files (default).">;

‎lld/docs/DTLTO.rst‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
Integrated Distributed ThinLTO (DTLTO)
2+
======================================
3+
4+
Integrated Distributed ThinLTO (DTLTO) enables the distribution of backend
5+
ThinLTO compilations via external distribution systems, such as Incredibuild,
6+
during the traditional link step.
7+
8+
The implementation is documented here: https://llvm.org/docs/DTLTO.html.
9+
10+
Currently, DTLTO is only supported in ELF LLD. Support will be added to other
11+
LLD flavours in the future.
12+
13+
ELF LLD
14+
-------
15+
16+
The command-line interface is as follows:
17+
18+
- ``--thinlto-distributor=<path>``
19+
Specifies the file to execute as the distributor process. If specified,
20+
ThinLTO backend compilations will be distributed.
21+
22+
- ``--thinlto-remote-compiler=<path>``
23+
Specifies the path to the compiler that the distributor process will use for
24+
backend compilations. The compiler invoked must match the version of LLD.
25+
26+
- ``--thinlto-distributor-arg=<arg>``
27+
Specifies ``<arg>`` on the command line when invoking the distributor.
28+
Can be specified multiple times.
29+
30+
- ``--thinlto-remote-compiler-arg=<arg>``
31+
Appends ``<arg>`` to the remote compiler's command line.
32+
Can be specified multiple times.
33+
34+
Options that introduce extra input/output files may cause miscompilation if
35+
the distribution system does not automatically handle pushing/fetching them to
36+
remote nodes. In such cases, configure the distributor - possibly using
37+
``--thinlto-distributor-arg=`` - to manage these dependencies. See the
38+
distributor documentation for details.
39+
40+
Some LLD LTO options (e.g., ``--lto-sample-profile=<file>``) are supported.
41+
Currently, other options are silently accepted but do not have the intended
42+
effect. Support for such options will be expanded in the future.

‎lld/docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,3 +147,4 @@ document soon.
147147
ELF/start-stop-gc
148148
ELF/warn_backrefs
149149
MachO/index
150+
DTLTO

0 commit comments

Comments
 (0)