Inlined frames are skipped

[bobrik]

Describe the bug

During symbolication inlined frames are skipped, resulting in confusing profiles.

There's lots of inlining in real world software (especially with LTO).

To Reproduce

Compile this:

#include <stdio.h>

int innie() {
  for (;;) {
    printf(".");
  }

  return 0;
}

int middle() {
  return innie();
}

int __attribute__ ((noinline)) outie() {
  return middle();
}

int main() {
  return outie();
}

And run it on a host covered by Grafana Alloy with pyroscope.ebpf enabled.

Observe profiles missing middle:

If you try running addr2line manually with inlines requested, you'd see this:

ivan@cube:~$ llvm-addr2line -ifa -e derp 0x1164
0x1164
middle
/home/ivan/./derp.c:12
outie
/home/ivan/./derp.c:16

Expected behavior

Inline frames are present in profiles.

Additional Context

You can see that multiple frames can be returned, but only one is used:

https://github.com/grafana/opentelemetry-ebpf-profiler/blob/f910fc4db23e6552ed79507de3788f660a9471bc/pyroscope/symb/irsymcache/table.go#L45-L49

[bobrik]

You can see that multiple frames can be returned, but only one is used.

While they can be returned, only one is in fact returned, so taking the first one doesn't matter.

The problem is deeper in the stack.

[korniltsev-grafanista]

You can see that multiple frames can be returned, but only one is used:

The returned value is the result of symbol resolution from .symtab section.

To resolve inlined frames we would need to support dwarf. We actually used to use the dwarf implementation from the rust-crates in the otel repo for a while, but it was impractical to use due to heavy resources consumption. We just need to implement dwarf on our server symbolyzer now at this point.

[bobrik]

What I ended up doing: libbpf/blazesym#1399 (comment).