You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[LLVM] Emit dwarf data for changed-signature and new functions
Add a new pass EmitChangedFuncDebugInfo which will add dwarf for
additional functions whose signatures are changed during compiler
transformations.
The original intention is for bpf-based linux kernel tracing.
The function signature is available in vmlinux BTF generated
from pahole/dwarf. Such signature is generated from dwarf
at the source level. But this is not ideal since some function
may have signatures changed. If user still used the source
level signature, users may not get correct results and may
need some efforts to workaround the issue.
So we want to encode the true signature (different
from the source one) in dwarf. With such additional information,
dwarf users can get these signature changed functions.
For example, pahole is able to process these signature
changed functions and encode them into vmlinux BTF properly.
History of multiple attempts
============================
Previously I have attempted a few tries ([1], [2] and [3]).
Initially I tried to modify debuginfo in passes like
ArgPromotion and DeadArgElim, but later on it is suggested
to have a central place to handle new signatures ([1]).
Later, I have another version of patch similar to this
one, but the recommendation is to modify debuginfo to
encode new signature within the same function,
either through inlinedAt or new signature overwriting
the old one. This seems working but it has some
side effect on lldb, some lldb output (e.g. back trace)
will be different from the previous one. The recommendation
is to avoid any behavior change for lldb ([2] and [3]).
So now, I came back to the solution discussed at the
end of [1]. Basically a special dwarf entry will be generated
to encode the new signature. The new signature will have
a reference to the old source-level signature.
So the tool can inspect dwarf to retrieve the related
info.
Examples and dwarf output
=========================
In below, a few examples will show how changed signatures
represented in dwarf:
Example 1
---------
Source:
$ cat test.c
struct t { int a; };
char *tar(struct t *a, struct t *d);
__attribute__((noinline)) static char * foo(struct t *a, int b, struct t *d)
{
return tar(a, d);
}
char *bar(struct t *a, struct t *d)
{
return foo(a, 1, d);
}
Compiled and dump dwarf with:
$ clang -O2 -c -g test.c -mllvm -enable-changed-func-dbinfo
$ llvm-dwarfdump test.o
0x0000000c: DW_TAG_compile_unit
...
0x0000005c: DW_TAG_subprogram
DW_AT_low_pc (0x0000000000000010)
DW_AT_high_pc (0x0000000000000015)
DW_AT_frame_base (DW_OP_reg7 RSP)
DW_AT_call_all_calls (true)
DW_AT_name ("foo")
DW_AT_decl_file ("/home/yhs/tests/sig-change/deadarg/test.c")
DW_AT_decl_line (3)
DW_AT_prototyped (true)
DW_AT_calling_convention (DW_CC_nocall)
DW_AT_type (0x000000b1 "char *")
0x0000006c: DW_TAG_formal_parameter
DW_AT_location (DW_OP_reg5 RDI)
DW_AT_name ("a")
DW_AT_decl_file ("/home/yhs/tests/sig-change/deadarg/test.c")
DW_AT_decl_line (3)
DW_AT_type (0x000000ba "t *")
0x00000076: DW_TAG_formal_parameter
DW_AT_name ("b")
DW_AT_decl_file ("/home/yhs/tests/sig-change/deadarg/test.c")
DW_AT_decl_line (3)
DW_AT_type (0x000000ce "int")
0x0000007e: DW_TAG_formal_parameter
DW_AT_location (DW_OP_reg4 RSI)
DW_AT_name ("d")
DW_AT_decl_file ("/home/yhs/tests/sig-change/deadarg/test.c")
DW_AT_decl_line (3)
DW_AT_type (0x000000ba "t *")
0x00000088: DW_TAG_call_site
...
0x0000009d: NULL
...
0x000000d2: DW_TAG_inlined_subroutine
DW_AT_name ("foo")
DW_AT_type (0x000000b1 "char *")
DW_AT_artificial (true)
DW_AT_specification (0x0000005c "foo")
0x000000dc: DW_TAG_formal_parameter
DW_AT_name ("a")
DW_AT_type (0x000000ba "t *")
0x000000e2: DW_TAG_formal_parameter
DW_AT_name ("d")
DW_AT_type (0x000000ba "t *")
0x000000e8: NULL
In the above, the DISubprogram 'foo' has the original signature but
since parameter 'b' does not have DW_AT_location, it is clear that
parameter will not be used. The actual function signature is represented
in DW_TAG_inlined_subroutine.
For the above case, it looks like DW_TAG_inlined_subroutine is not
necessary. Let us try a few other examples below.
Example 2
---------
Source:
$ cat test.c
struct t { long a; long b;};
__attribute__((noinline)) static long foo(struct t arg) {
return arg.b * 5;
}
long bar(struct t arg) {
return foo(arg);
}
Compiled and dump dwarf with:
$ clang -O2 -c -g test.c -mllvm -enable-changed-func-dbinfo
$ llvm-dwarfdump test.o
...
0x0000004e: DW_TAG_subprogram
DW_AT_low_pc (0x0000000000000010)
DW_AT_high_pc (0x0000000000000015)
DW_AT_frame_base (DW_OP_reg7 RSP)
DW_AT_call_all_calls (true)
DW_AT_name ("foo")
DW_AT_decl_file ("/home/yhs/tests/sig-change/struct/test.c")
DW_AT_decl_line (2)
DW_AT_prototyped (true)
DW_AT_calling_convention (DW_CC_nocall)
DW_AT_type (0x0000006d "long")
0x0000005e: DW_TAG_formal_parameter
DW_AT_location (DW_OP_piece 0x8, DW_OP_reg5 RDI, DW_OP_piece 0x8)
DW_AT_name ("arg")
DW_AT_decl_file ("/home/yhs/tests/sig-change/struct/test.c")
DW_AT_decl_line (2)
DW_AT_type (0x00000099 "t")
0x0000006c: NULL
...
0x00000088: DW_TAG_inlined_subroutine
DW_AT_name ("foo")
DW_AT_type (0x0000006d "long")
DW_AT_artificial (true)
DW_AT_specification (0x0000004e "foo")
0x00000092: DW_TAG_formal_parameter
DW_AT_name ("arg__coerce1")
DW_AT_type (0x0000006d "long")
0x00000098: NULL
In the above case for function foo(), the original argument is 'struct t',
but the final actual argument is a 'long' type. DW_TAG_inlined_subroutine
can clearly represent the signature type instead of doing DW_AT_location
thing. Note that the name 'arg__coerce1' presents the second long type
value of the struct 't'. The llvm may put 'arg.coerce1' as the func argument
name, we use 'arg__coerce1' so the argument name can be represented in C
code.
Example 3
---------
Source:
$ cat test2.c
struct t { long a; long b; long c;};
__attribute__((noinline)) static long foo(struct t arg, int a) {
return arg.a * arg.c;
}
long bar(struct t arg) {
return foo(arg, 1);
}
Compiled and dump dwarf with:
$ clang -O2 -c -g test2.c -mllvm -enable-changed-func-dbinfo
$ llvm-dwarfdump test2.o
...
0x0000003e: DW_TAG_subprogram
DW_AT_low_pc (0x0000000000000010)
DW_AT_high_pc (0x0000000000000015)
DW_AT_frame_base (DW_OP_reg7 RSP)
DW_AT_call_all_calls (true)
DW_AT_name ("bar")
DW_AT_decl_file ("/home/yhs/tests/sig-change/struct/test2.c")
DW_AT_decl_line (5)
DW_AT_prototyped (true)
DW_AT_type (0x0000005f "long")
DW_AT_external (true)
0x0000004d: DW_TAG_formal_parameter
DW_AT_location (DW_OP_fbreg +8)
DW_AT_name ("arg")
DW_AT_decl_file ("/home/yhs/tests/sig-change/struct/test2.c")
DW_AT_decl_line (5)
DW_AT_type (0x00000079 "t")
0x00000058: DW_TAG_call_site
DW_AT_call_origin (0x00000023 "foo")
DW_AT_call_tail_call (true)
DW_AT_call_pc (0x0000000000000010)
0x0000005e: NULL
...
0x00000063: DW_TAG_inlined_subroutine
DW_AT_name ("foo")
DW_AT_type (0x0000005f "long")
DW_AT_artificial (true)
DW_AT_specification (0x00000023 "foo")
0x0000006d: DW_TAG_formal_parameter
DW_AT_name ("arg")
DW_AT_type (0x00000074 "t")
0x00000073: NULL
In the above example, from DW_TAG_subprogram, it is not clear what kind
of type the parameter should be. But DW_TAG_inlined_subroutine can
clearly show what the type should be.
Example 4
---------
Source:
$ cat test.c
__attribute__((noinline)) static int callee(const int *p) { return *p + 42; }
int caller(void) {
int x = 100;
return callee(&x);
}
Compiled and dump dwarf with:
$ clang -O3 -c -g test.c -mllvm -enable-changed-func-dbinfo
$ llvm-dwarfdump test.o
...
0x0000004a: DW_TAG_subprogram
DW_AT_low_pc (0x0000000000000010)
DW_AT_high_pc (0x0000000000000014)
DW_AT_frame_base (DW_OP_reg7 RSP)
DW_AT_call_all_calls (true)
DW_AT_name ("callee")
DW_AT_decl_file ("/home/yhs/tests/sig-change/prom/test.c")
DW_AT_decl_line (1)
DW_AT_prototyped (true)
DW_AT_calling_convention (DW_CC_nocall)
DW_AT_type (0x00000063 "int")
0x0000005a: DW_TAG_formal_parameter
DW_AT_name ("p")
DW_AT_decl_file ("/home/yhs/tests/sig-change/prom/test.c")
DW_AT_decl_line (1)
DW_AT_type (0x00000078 "const int *")
0x00000062: NULL
...
0x00000067: DW_TAG_inlined_subroutine
DW_AT_name ("callee")
DW_AT_type (0x00000063 "int")
DW_AT_artificial (true)
DW_AT_specification (0x0000004a "callee")
0x00000071: DW_TAG_formal_parameter
DW_AT_name ("__0")
DW_AT_type (0x00000063 "int")
0x00000077: NULL
In the above, the function
static int callee(const int *p) { return *p + 42; }
is transformed to
static int callee(int p) { return p + 42; }
But the new signature is not reflected in DW_TAG_subprogram.
The DW_TAG_inlined_subroutine can precisely capture the
signature. Note that the parameter name is "__0" and "0" means
the first argument. The reason is due to the following IR:
define internal ... i32 @callee(i32 %0) unnamed_addr llvm#1 !dbg !23 {
#dbg_value(ptr poison, !29, !DIExpression(), !30)
%2 = add nsw i32 %0, 42, !dbg !31
ret i32 %2, !dbg !32
}
...
!29 = !DILocalVariable(name: "p", arg: 1, scope: !23, file: !1, line: 1, type: !26)
The reason is due to 'ptr poison' as 'ptr poison' mean the debug
value should not be used any more. This is also the reason that
the above DW_TAG_subprogram does not have location information.
DW_TAG_inlined_subroutine can provide correct signature though.
With additional option like
clang -O3 -c -g test.c -mllvm -enable-changed-func-dbinfo -fsave-optimization-record \
-foptimization-record-passes=emit-changed-func-debuginfo
a file test.opt.yaml is generated with the following remark:
$ cat test.opt.yaml
--- !Passed
Pass: emit-changed-func-debuginfo
Name: FindNoDIVariable
DebugLoc: { File: test.c, Line: 1, Column: 0 }
Function: callee
Args:
- String: 'create a new int type '
- ArgName: ''
- String: '('
- ArgIndex: '0'
- String: ')'
...
If we compile like below:
clang -O3 -c -g test.c -fno-discard-value-names -mllvm -enable-changed-func-dbinfo
The function argument name will be preserved
... i32 @callee(i32 %p.0.val) ...
and in such cases,
the DW_TAG_inlined_subroutine looks like below:
0x00000067: DW_TAG_inlined_subroutine
DW_AT_name ("callee")
DW_AT_type (0x00000063 "int")
DW_AT_artificial (true)
DW_AT_specification (0x0000004a "callee")
0x00000071: DW_TAG_formal_parameter
DW_AT_name ("p__0__val")
DW_AT_type (0x00000063 "int")
0x00000077: NULL
Note that the original argument name replaces '.' with "__"
so argument name has proper C standard.
Non-LTO vs. LTO
---------------
For thin-lto mode, we often see kernel symbols like
p9_req_cache.llvm.13472271643223911678
Even if this symbol has identical source level signature with p9_req_cache,
a special DW_TAG_inlined_subroutine will be generated with
name 'p9_req_cache.llvm.13472271643223911678'.
With this, some tool (e.g., pahole) may generate a BTF entry
for this name which could be used for bpf fentry/fexit tracing.
But if a symbol with "<foo>.llvm.<hash>" has different signatures
than the source level "<foo>", then a special DW_TAG_inlined_subroutine
will be generated like below:
0x10f0793f: DW_TAG_inlined_subroutine
DW_AT_name ("flow_offload_fill_route")
DW_AT_linkage_name ("flow_offload_fill_route.llvm.14555965973926298225")
DW_AT_artificial (true)
DW_AT_specification (0x10ee9e54 "flow_offload_fill_route")
0x10f07949: DW_TAG_formal_parameter
DW_AT_name ("flow")
DW_AT_type (0x10ee837a "flow_offload *")
0x10f07951: DW_TAG_formal_parameter
DW_AT_name ("route")
DW_AT_type (0x10eea4ef "nf_flow_route *")
0x10f07959: DW_TAG_formal_parameter
DW_AT_name ("dir")
DW_AT_type (0x10ecef15 "flow_offload_tuple_dir")
0x10f07961: NULL
In the above, function "flow_offload_fill_route" has return type
"int" at source level, but optimization eventually made the return
type as "void". The tools like pahole may choose to generate
two entries with DW_AT_name and DW_AT_linkage_name for vmlinux BTF.
Function specialization
-----------------------
LLVM has a pass FunctionSpecializer (FunctionSpecialization.cpp) which
is called by SCCP pass (Interprocedural Sparse Conditional Constant
Propagation). The FunctionSpecializer may clone functions and SCCP
pass is available for both non-LTO and LTO passes. For any function,
the default clones can be up to 3 and all these clones will have
different signatures than the source signature. This is rare but
it did happen. For example, for linux kernel thin lto mode, I found
the following in the kernel symbol table:
ffffffff812036d0 t print_cpu.specialized.1
In this particular case, after cloning, the original function
'print_cpu' is not used so it is removed. Here, the print_cpu()
call is a static function.
Basically, the compiler creates a specialized 'print_cpu.specialized.1'
function and the original funciton 'print_cpu' also exists. The dwarf
for the above two functions:
0x01484bea: DW_TAG_subprogram
DW_AT_low_pc (0xffffffff812036d0)
DW_AT_high_pc (0xffffffff8120400c)
DW_AT_frame_base (DW_OP_reg6 RBP)
DW_AT_call_all_calls (true)
DW_AT_name ("print_cpu")
DW_AT_decl_file ("/home/yhs/work/bpf-next/kernel/sched/debug.c")
DW_AT_decl_line (922)
DW_AT_prototyped (true)
DW_AT_calling_convention (DW_CC_nocall)
0x01484bfa: DW_TAG_formal_parameter
DW_AT_const_value (0)
DW_AT_name ("m")
DW_AT_decl_file ("/home/yhs/work/bpf-next/kernel/sched/debug.c")
DW_AT_decl_line (922)
DW_AT_type (0x0146fd21 "seq_file *")
0x01484c06: DW_TAG_formal_parameter
DW_AT_location (indexed (0x7ee) loclist = 0x0011ce6d:
[0xffffffff812036d5, 0xffffffff81203730): DW_OP_reg5 RDI
[0xffffffff81203730, 0xffffffff812039fa): DW_OP_reg3 RBX
[0xffffffff812039fa, 0xffffffff81203a89): DW_OP_entry_value(DW_OP_reg5 RDI), DW_OP_stack_value
[0xffffffff81203a89, 0xffffffff81203a8d): DW_OP_reg3 RBX
[0xffffffff81203a8d, 0xffffffff81203d58): DW_OP_breg7 RSP+12
[0xffffffff81203d7a, 0xffffffff81203ddd): DW_OP_breg7 RSP+12
[0xffffffff81203dfa, 0xffffffff81203f7b): DW_OP_breg7 RSP+12
[0xffffffff81203f7b, 0xffffffff81203f80): DW_OP_entry_value(DW_OP_reg5 RDI), DW_OP_stack_value
[0xffffffff81203f80, 0xffffffff8120400c): DW_OP_reg3 RBX)
DW_AT_name ("cpu")
DW_AT_decl_file ("/home/yhs/work/bpf-next/kernel/sched/debug.c")
DW_AT_decl_line (922)
DW_AT_type (0x01462560 "int")
......
0x014981fc: DW_TAG_inlined_subroutine
DW_AT_name ("print_cpu.specialized.1")
DW_AT_artificial (true)
DW_AT_specification (0x01484bea "print_cpu")
0x01498204: DW_TAG_formal_parameter
DW_AT_name ("cpu")
DW_AT_type (0x01462560 "int")
0x0149820c: NULL
The specailized function "print_cpu.specialized.1" has a signature different
from the original one "print_cpu" and its name directly encoded into
DW_AT_name.
Some restrictions
=================
There are some restrictions in the current implementation:
- Only C language is supported
- BPF target is excluded as one of main goals for this pull request
is to generate proper vmlinux BTF for arch's like x86_64/arm64 etc.
- Function must not be a intrinsic, decl only, return value size more
than arch register size and func with variable arguments.
- For arguments, only int/ptr types are supported.
- Some union type arguments (e.g., 8B < union_size <= 16B) may
have issue to pick which member so the related functions may be skipped.
Remarks
=======
A few remarks are available for debugging purpose including
- cannot handle union arguments (greater than 8B but less/equal to 16B).
- cannot find corresponding DILocalVariable for the argument.
- certain cases of dbg fragment handling.
Some statistics with linux kernel
=================================
I have tested this patch set by building latest bpf-next linux kernel.
For no-lto case:
66051 original number of functions
894 signature changed or new with-dot functions with this patch
For thin-lto case:
66227 original number of functions
2993 signature changed or new with-dot functions with this patch
Next step
=========
With this llvm change, we will be able to do some work in pahole.
For pahole, currently we will see the warning:
die__process_unit: DW_TAG_inlined_subroutine (0x1d) @ <0xf2db986> not handled in a c11 CU!
Basically these DW_TAG_inlined_subroutine are not inside the DISubprogram.
[1] llvm#127855
[2] llvm#157349
[3] https://discourse.llvm.org/t/rfc-identify-func-signature-change-in-llvm-compiled-kernel-image/82609
0 commit comments