Skip to content

Commit 5ea21c5

Browse files
bd1976brisbd1976bris
committed
[DTLTO][LLD][ELF] Support bitcode members of thin archives
This patch adds support for bitcode members of thin archives to DTLTO (https://llvm.org/docs/DTLTO.html) in ELF LLD. For DTLTO, bitcode identifiers must be valid paths to bitcode files on disk. Clang does not support archive inputs for ThinLTO backend compilations. This patch adjusts the identifier for bitcode members of thin archives in DTLTO links so that it is the path to the member file on disk, allowing such members to be supported in DTLTO. This patch is sufficient to allow for self-hosting an LLVM build with DTLTO when thin archives are used. Note: Bitcode members of non-thin archives remain unsupported. This will be addressed in a future change. Testing: - LLD lit test coverage has been added to check that the identifier is adjusted appropriately. - A cross-project lit test has been added to show that a DTLTO link can succeed when linking bitcode members of thin archives. For the design discussion of the DTLTO feature, see: #126654.
1 parent edfec9c commit 5ea21c5

File tree

5 files changed

+207
-7
lines changed

5 files changed

+207
-7
lines changed

‎cross-project-tests/CMakeLists.txt‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,12 @@ set(CROSS_PROJECT_TEST_DEPS
1919
FileCheck
2020
check-gdb-llvm-support
2121
count
22-
llvm-dwarfdump
22+
llvm-ar
2323
llvm-config
24+
llvm-dwarfdump
2425
llvm-objdump
25-
split-file
2626
not
27+
split-file
2728
)
2829

2930
if ("clang" IN_LIST LLVM_ENABLE_PROJECTS)

‎cross-project-tests/dtlto/ld-archive-thin.test‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
REQUIRES: ld.lld,llvm-ar
2+
3+
## Test that a DTLTO link succeeds and outputs the expected set of files
4+
## correctly when thin archives are present.
5+
6+
RUN: rm -rf %t && split-file %s %t && cd %t
7+
8+
## Compile bitcode. -O2 is required for cross-module importing.
9+
RUN: %clang -O2 --target=x86_64-linux-gnu -flto=thin -c \
10+
RUN: foo.c bar.c dog.c cat.c start.c
11+
12+
## Generate thin archives.
13+
RUN: llvm-ar rcs foo.a foo.o --thin
14+
## Create this bitcode thin archive in a subdirectory to test the expansion of
15+
## the path to a bitcode file that is referenced using "..", e.g., in this case
16+
## "../bar.o".
17+
RUN: mkdir lib
18+
RUN: llvm-ar rcs lib/bar.a bar.o --thin
19+
## Create this bitcode thin archive with an absolute path entry containing "..".
20+
RUN: llvm-ar rcs dog.a %t/lib/../dog.o --thin
21+
## The bitcode member of cat.a will not be used in the link.
22+
RUN: llvm-ar rcs cat.a cat.o --thin
23+
RUN: llvm-ar rcs start.a start.o --thin
24+
25+
## Link from a different directory to ensure that thin archive member paths are
26+
## resolved correctly relative to the archive locations.
27+
RUN: mkdir %t/out && cd %t/out
28+
29+
RUN: %clang --target=x86_64-linux-gnu -flto=thin -fuse-ld=lld %t/foo.a %t/lib/bar.a ../start.a %t/cat.a \
30+
RUN: -Wl,--whole-archive ../dog.a \
31+
RUN: -fthinlto-distributor=%python \
32+
RUN: -Xthinlto-distributor=%llvm_src_root/utils/dtlto/local.py \
33+
RUN: -Wl,--save-temps -nostdlib -Werror
34+
35+
## Check that the required output files have been created.
36+
RUN: ls | sort | FileCheck %s
37+
38+
## No files are expected before.
39+
CHECK-NOT: {{.}}
40+
41+
## JSON jobs description.
42+
CHECK: {{^}}a.[[PID:[a-zA-Z0-9_]+]].dist-file.json{{$}}
43+
44+
## Native output object files and individual summary index files.
45+
CHECK: {{^}}bar.3.[[PID]].native.o{{$}}
46+
CHECK: {{^}}bar.3.[[PID]].native.o.thinlto.bc{{$}}
47+
CHECK: {{^}}dog.1.[[PID]].native.o{{$}}
48+
CHECK: {{^}}dog.1.[[PID]].native.o.thinlto.bc{{$}}
49+
CHECK: {{^}}foo.2.[[PID]].native.o{{$}}
50+
CHECK: {{^}}foo.2.[[PID]].native.o.thinlto.bc{{$}}
51+
CHECK: {{^}}start.4.[[PID]].native.o{{$}}
52+
CHECK: {{^}}start.4.[[PID]].native.o.thinlto.bc{{$}}
53+
54+
## No files are expected after.
55+
CHECK-NOT: {{.}}
56+
57+
58+
## It is important that cross-module inlining occurs for this test to show that Clang can
59+
## successfully load the bitcode file dependencies recorded in the summary indices.
60+
## Explicitly check that the expected importing has occurred.
61+
62+
RUN: llvm-dis start.4.*.native.o.thinlto.bc -o - | \
63+
RUN: FileCheck %s --check-prefixes=FOO,BAR,START
64+
65+
RUN: llvm-dis dog.1.*.native.o.thinlto.bc -o - | \
66+
RUN: FileCheck %s --check-prefixes=FOO,BAR,DOG,START
67+
68+
RUN: llvm-dis foo.2.*.native.o.thinlto.bc -o - | \
69+
RUN: FileCheck %s --check-prefixes=FOO,BAR,START
70+
71+
RUN: llvm-dis bar.3.*.native.o.thinlto.bc -o - | \
72+
RUN: FileCheck %s --check-prefixes=FOO,BAR,START
73+
74+
FOO-DAG: foo.o
75+
BAR-DAG: bar.o
76+
DOG-DAG: dog.o
77+
START-DAG: start.o
78+
79+
80+
#--- foo.c
81+
extern int bar(int), _start(int);
82+
__attribute__((retain)) int foo(int x) { return x + bar(x) + _start(x); }
83+
84+
#--- bar.c
85+
extern int foo(int), _start(int);
86+
__attribute__((retain)) int bar(int x) { return x + foo(x) + _start(x); }
87+
88+
#--- dog.c
89+
extern int foo(int), bar(int), _start(int);
90+
__attribute__((retain)) int dog(int x) { return x + foo(x) + bar(x) + _start(x); }
91+
92+
#--- cat.c
93+
__attribute__((retain)) void cat(int x) {}
94+
95+
#--- start.c
96+
extern int foo(int), bar(int);
97+
__attribute__((retain)) int _start(int x) { return x + foo(x) + bar(x); }

‎cross-project-tests/lit.cfg.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
2020

2121
# suffixes: A list of file extensions to treat as test files.
22-
config.suffixes = [".c", ".cl", ".cpp", ".m"]
22+
config.suffixes = [".c", ".cl", ".cpp", ".m", ".test"]
2323

2424
# excludes: A list of directories to exclude from the testsuite. The 'Inputs'
2525
# subdirectories contain auxiliary inputs for various tests in their parent
@@ -107,6 +107,8 @@ def get_required_attr(config, attr_name):
107107
if lldb_path is not None:
108108
config.available_features.add("lldb")
109109

110+
if llvm_config.use_llvm_tool("llvm-ar"):
111+
config.available_features.add("llvm-ar")
110112

111113
def configure_dexter_substitutions():
112114
"""Configure substitutions for host platform and return list of dependencies"""

‎lld/ELF/InputFiles.cpp‎

Lines changed: 38 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include "llvm/ADT/CachedHashString.h"
2121
#include "llvm/ADT/STLExtras.h"
2222
#include "llvm/LTO/LTO.h"
23+
#include "llvm/Object/Archive.h"
2324
#include "llvm/Object/IRObjectFile.h"
2425
#include "llvm/Support/ARMAttributeParser.h"
2526
#include "llvm/Support/ARMBuildAttributes.h"
@@ -1753,6 +1754,36 @@ static uint8_t getOsAbi(const Triple &t) {
17531754
}
17541755
}
17551756

1757+
// For DTLTO, bitcode member names must be valid paths to files on disk.
1758+
// For thin archives, resolve `memberPath` relative to the archive's location.
1759+
// Returns true if adjusted; false otherwise. Non-thin archives are unsupported.
1760+
static bool dtltoAdjustMemberPathIfThinArchive(Ctx &ctx, StringRef archivePath,
1761+
std::string &memberPath) {
1762+
assert(!archivePath.empty() && !ctx.arg.dtltoDistributor.empty());
1763+
1764+
// Read the archive header to determine if it's a thin archive.
1765+
auto bufferOrErr =
1766+
MemoryBuffer::getFileSlice(archivePath, sizeof(ThinArchiveMagic) - 1, 0);
1767+
if (std::error_code ec = bufferOrErr.getError()) {
1768+
ErrAlways(ctx) << "cannot open " << archivePath << ": " << ec.message();
1769+
return false;
1770+
}
1771+
1772+
if (!bufferOrErr->get()->getBuffer().starts_with(ThinArchiveMagic))
1773+
return false;
1774+
1775+
SmallString<64> resolvedPath;
1776+
if (path::is_relative(memberPath)) {
1777+
resolvedPath = path::parent_path(archivePath);
1778+
path::append(resolvedPath, memberPath);
1779+
} else
1780+
resolvedPath = memberPath;
1781+
1782+
path::remove_dots(resolvedPath, /*remove_dot_dot=*/true);
1783+
memberPath = resolvedPath.str();
1784+
return true;
1785+
}
1786+
17561787
BitcodeFile::BitcodeFile(Ctx &ctx, MemoryBufferRef mb, StringRef archiveName,
17571788
uint64_t offsetInArchive, bool lazy)
17581789
: InputFile(ctx, BitcodeKind, mb) {
@@ -1770,10 +1801,13 @@ BitcodeFile::BitcodeFile(Ctx &ctx, MemoryBufferRef mb, StringRef archiveName,
17701801
// symbols later in the link stage). So we append file offset to make
17711802
// filename unique.
17721803
StringSaver &ss = ctx.saver;
1773-
StringRef name = archiveName.empty()
1774-
? ss.save(path)
1775-
: ss.save(archiveName + "(" + path::filename(path) +
1776-
" at " + utostr(offsetInArchive) + ")");
1804+
StringRef name =
1805+
(archiveName.empty() ||
1806+
(!ctx.arg.dtltoDistributor.empty() &&
1807+
dtltoAdjustMemberPathIfThinArchive(ctx, archiveName, path)))
1808+
? ss.save(path)
1809+
: ss.save(archiveName + "(" + path::filename(path) + " at " +
1810+
utostr(offsetInArchive) + ")");
17771811
MemoryBufferRef mbref(mb.getBuffer(), name);
17781812

17791813
obj = CHECK2(lto::InputFile::create(mbref), this);

‎lld/test/ELF/dtlto/archive-thin.test‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
REQUIRES: x86
2+
3+
## Test that a DTLTO link assigns Module IDs to thin archive members as expected.
4+
5+
RUN: rm -rf %t && split-file %s %t && cd %t
6+
7+
RUN: sed 's/@t1/@t2/g' t1.ll > t2.ll
8+
RUN: sed 's/@t1/@t3/g' t1.ll > t3.ll
9+
10+
RUN: opt -thinlto-bc t1.ll -o t1.bc
11+
RUN: opt -thinlto-bc t2.ll -o t2.bc
12+
RUN: opt -thinlto-bc t3.ll -o t3.bc
13+
14+
RUN: llvm-ar rcs t1.a t1.bc --thin
15+
## Create this bitcode thin archive in a subdirectory to test the expansion of
16+
## the path to a bitcode file that is referenced using "..", e.g., in this case
17+
## "../t2.bc".
18+
RUN: mkdir lib
19+
RUN: llvm-ar rcs lib/t2.a t2.bc --thin
20+
## Create this bitcode thin archive with an absolute path entry containing "..".
21+
RUN: llvm-ar rcs t3.a %t/lib/../t3.bc --thin
22+
23+
## Link from a different directory to ensure that thin archive member paths are
24+
## resolved correctly relative to the archive locations.
25+
RUN: mkdir %t/out && cd %t/out
26+
27+
## Build a response file to share common linking arguments.
28+
## Note: validate.py does not perform any compilation. Instead, it validates the
29+
## received JSON, pretty-prints the JSON and the supplied arguments, and then
30+
## exits with an error. This allows FileCheck directives to verify the
31+
## distributor inputs.
32+
RUN: echo "%t/t1.a %t/lib/t2.a ../t3.a \
33+
RUN: --thinlto-distributor=\"%python\" \
34+
RUN: --thinlto-distributor-arg=\"%llvm_src_root/utils/dtlto/validate.py\" " > rsp
35+
36+
## Link thin archives using -u/--undefined.
37+
RUN: not ld.lld @rsp -u t1 -u t2 -u t3 2>&1 | FileCheck %s
38+
39+
## Link thin archives using --whole-archive.
40+
RUN: not ld.lld --whole-archive @rsp 2>&1 | FileCheck %s
41+
42+
## Check the module IDs in the JSON jobs description.
43+
CHECK: "jobs": [
44+
CHECK: "inputs": [
45+
CHECK-NEXT: "{{([a-zA-Z]:)|/}}
46+
CHECK-SAME: {{/|\\\\}}archive-thin.test.tmp{{/|\\\\}}t1.bc"
47+
48+
CHECK: "inputs": [
49+
CHECK-NEXT: "{{([a-zA-Z]\:)|/}}
50+
CHECK-SAME: {{/|\\\\}}archive-thin.test.tmp{{/|\\\\}}t2.bc"
51+
52+
CHECK: "inputs": [
53+
CHECK-NEXT: "{{([a-zA-Z]:)|/}}
54+
CHECK-SAME: {{/|\\\\}}archive-thin.test.tmp{{/|\\\\}}t3.bc"
55+
56+
## Ensure backend compilation fails as expected (due to validate.py dummy behavior).
57+
CHECK: error: DTLTO backend compilation: cannot open native object file:
58+
59+
#--- t1.ll
60+
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
61+
target triple = "x86_64-unknown-linux-gnu"
62+
63+
define void @t1() {
64+
ret void
65+
}
66+

0 commit comments

Comments
 (0)