Linux kernel version is unknown

[jakesteele]

OS: Linux Cachy 6.15.2-2
LLVM Version: 20.7.1
Perf Version: perf version 6.15-2

Command I am running:

~/llvm-project/build/bin tags/llvmorg-20.1.7^0 jacob@Server
❯ sudo ./perf2bolt -p perf.data -o bolt.fdata /usr/src/debug/linux-cachyos/vmlinux

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read branch events
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6146a88f60492b520a36f8f8f3231e15f3cc6082
BOLT-INFO: Linux kernel binary detected
BOLT-INFO: first alloc address is 0xffffffff81000000
BOLT-INFO: static input executable detected
BOLT-INFO: enabling lite mode
BOLT-WARNING: ignoring symbol __end_rodata at 0xffffffff834c6000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_hpage_align at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_aligned at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: split function detected on input : task_rq_lock.cold
BOLT-ERROR: symbol seen in the middle of the function srso_untrain_ret/1(*2). Skipping.
BOLT-ERROR: symbol seen in the middle of the function retbleed_untrain_ret/1(*2). Skipping.
BOLT-INFO: pre-processing profile using perf data aggregator
BOLT-INFO: binary build-id is:     873e93cae96bc10932d87f1da57dcee21256f08d
PERF2BOLT: spawning perf job to read buildid list
PERF2BOLT-WARNING: build-id matched a different file name
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 0 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parse branch events...
PERF2BOLT: read 264338 samples and 6877582 LBR entries
PERF2BOLT: 0 samples (0.0%) were ignored
PERF2BOLT: traces mismatching disassembled function contents: 6923 (0.1%)
PERF2BOLT: out of range traces involving unknown regions: 630520 (9.5%)
PERF2BOLT: waiting for perf mem events collection to finish...
BOLT-ERROR: while running linux-kernel-rewriter in pre-CFG state: Linux kernel version is unknown

However I clearly see it:

❯ strings /usr/src/debug/linux-cachyos/vmlinux | grep 'Linux version'
Linux version 6.15.2-2-cachyos (linux-cachyos@cachyos) (clang version 20.1.6, LLD 20.1.6) # SMP Thu, 19 Jun 2025 03:26:16 +0000
Linux version 6.15.2-2-cachyos (linux-cachyos@cachyos) (clang version 20.1.6, LLD 20.1.6) #1 SMP Thu, 19 Jun 2025 03:26:16 +0000

I believe it's failing to parse the string for some reason.

[llvmbot]

@llvm/issue-subscribers-bolt

Author: Jake Steele (jakesteele)

OS: Linux Cachy 6.15.2-2 LLVM Version: 20.7.1 Perf Version: `perf version 6.15-2`

Command I am running:

~/llvm-project/build/bin tags/llvmorg-20.1.7^0 jacob@<!-- -->Server
❯ sudo ./perf2bolt -p perf.data -o bolt.fdata /usr/src/debug/linux-cachyos/vmlinux

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read branch events
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6146a88f60492b520a36f8f8f3231e15f3cc6082
BOLT-INFO: Linux kernel binary detected
BOLT-INFO: first alloc address is 0xffffffff81000000
BOLT-INFO: static input executable detected
BOLT-INFO: enabling lite mode
BOLT-WARNING: ignoring symbol __end_rodata at 0xffffffff834c6000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_hpage_align at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_aligned at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: split function detected on input : task_rq_lock.cold
BOLT-ERROR: symbol seen in the middle of the function srso_untrain_ret/1(*2). Skipping.
BOLT-ERROR: symbol seen in the middle of the function retbleed_untrain_ret/1(*2). Skipping.
BOLT-INFO: pre-processing profile using perf data aggregator
BOLT-INFO: binary build-id is:     873e93cae96bc10932d87f1da57dcee21256f08d
PERF2BOLT: spawning perf job to read buildid list
PERF2BOLT-WARNING: build-id matched a different file name
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 0 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parse branch events...
PERF2BOLT: read 264338 samples and 6877582 LBR entries
PERF2BOLT: 0 samples (0.0%) were ignored
PERF2BOLT: traces mismatching disassembled function contents: 6923 (0.1%)
PERF2BOLT: out of range traces involving unknown regions: 630520 (9.5%)
PERF2BOLT: waiting for perf mem events collection to finish...
BOLT-ERROR: while running linux-kernel-rewriter in pre-CFG state: Linux kernel version is unknown

However I clearly see it:

❯ strings /usr/src/debug/linux-cachyos/vmlinux | grep 'Linux version'
Linux version 6.15.2-2-cachyos (linux-cachyos@<!-- -->cachyos) (clang version 20.1.6, LLD 20.1.6) # SMP Thu, 19 Jun 2025 03:26:16 +0000
Linux version 6.15.2-2-cachyos (linux-cachyos@<!-- -->cachyos) (clang version 20.1.6, LLD 20.1.6) #<!-- -->1 SMP Thu, 19 Jun 2025 03:26:16 +0000

I believe it's failing to parse the string for some reason.

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: Jake Steele (jakesteele)

OS: Linux Cachy 6.15.2-2 LLVM Version: 20.7.1 Perf Version: `perf version 6.15-2`

Command I am running:

~/llvm-project/build/bin tags/llvmorg-20.1.7^0 jacob@<!-- -->Server
❯ sudo ./perf2bolt -p perf.data -o bolt.fdata /usr/src/debug/linux-cachyos/vmlinux

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read branch events
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6146a88f60492b520a36f8f8f3231e15f3cc6082
BOLT-INFO: Linux kernel binary detected
BOLT-INFO: first alloc address is 0xffffffff81000000
BOLT-INFO: static input executable detected
BOLT-INFO: enabling lite mode
BOLT-WARNING: ignoring symbol __end_rodata at 0xffffffff834c6000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_hpage_align at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_aligned at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: split function detected on input : task_rq_lock.cold
BOLT-ERROR: symbol seen in the middle of the function srso_untrain_ret/1(*2). Skipping.
BOLT-ERROR: symbol seen in the middle of the function retbleed_untrain_ret/1(*2). Skipping.
BOLT-INFO: pre-processing profile using perf data aggregator
BOLT-INFO: binary build-id is:     873e93cae96bc10932d87f1da57dcee21256f08d
PERF2BOLT: spawning perf job to read buildid list
PERF2BOLT-WARNING: build-id matched a different file name
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 0 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parse branch events...
PERF2BOLT: read 264338 samples and 6877582 LBR entries
PERF2BOLT: 0 samples (0.0%) were ignored
PERF2BOLT: traces mismatching disassembled function contents: 6923 (0.1%)
PERF2BOLT: out of range traces involving unknown regions: 630520 (9.5%)
PERF2BOLT: waiting for perf mem events collection to finish...
BOLT-ERROR: while running linux-kernel-rewriter in pre-CFG state: Linux kernel version is unknown

However I clearly see it:

❯ strings /usr/src/debug/linux-cachyos/vmlinux | grep 'Linux version'
Linux version 6.15.2-2-cachyos (linux-cachyos@<!-- -->cachyos) (clang version 20.1.6, LLD 20.1.6) # SMP Thu, 19 Jun 2025 03:26:16 +0000
Linux version 6.15.2-2-cachyos (linux-cachyos@<!-- -->cachyos) (clang version 20.1.6, LLD 20.1.6) #<!-- -->1 SMP Thu, 19 Jun 2025 03:26:16 +0000

I believe it's failing to parse the string for some reason.

[DavidSpickett]

Zero bolt experience but here's how you could check some basics.

llvm-project/bolt/lib/Rewrite/LinuxKernelRewriter.cpp

Line 434 in 6273c5d

Error LinuxKernelRewriter::detectLinuxKernelVersion() {

Is where that message comes from. One test for it is:

llvm-project/bolt/test/X86/linux-alt-instruction.s

Line 145 in 6273c5d

## Linux kernel version

I have a kernel build locally and I see:

$ readelf --symbols ./arm64/vmlinux | grep linux_banner
242525: ffff80008181e1a0   284 OBJECT  GLOBAL DEFAULT    3 linux_banner

$ readelf -S ./arm64/vmlinux
<...>
  [ 3] .rodata           PROGBITS         ffff8000811c0000  011d0000
       0000000000bd4128  0000000000000000  WA       0     0     4096
<...>

$ readelf -x 3 ./arm64/vmlinux
  0xffff80008181e1a0 4c696e75 78207665 7273696f 6e20362e Linux version 6.
  0xffff80008181e1b0 31362e30 2d726331 2d303030 31362d67 16.0-rc1-00016-g
  0xffff80008181e1c0 30313736 33383163 34323935 20286461 0176381c4295 (da

Reading the Linux source, linux_banner does not appear to be optional.

Can you confirm you have that symbol?

[DavidSpickett]

Bolt's pattern does match the string you've shown. You can ignore the --- around it, this is just for https://en.cppreference.com/w/cpp/language/string_literal.html to work. Perhaps it cannot find the location of that string.

[jakesteele]

These are the results of those commands, the bottom one I stiched up a bit using grep, can get raw if you want but it was huge!

❯ readelf --symbols /usr/src/debug/linux-cachyos/vmlinux | grep linux_banner

265786: ffffffff82e00eb0   130 OBJECT  LOCAL  HIDDEN     3 linux_banner

^ I guess thats the issue, needs to be Global Default not Local Hidden.

❯ readelf -S /usr/src/debug/linux-cachyos/vmlinux

 [ 3] .rodata           PROGBITS         ffffffff82600000  01800000
       0000000000896b68  0000000000000000 WAMSo       0     0     4096

~/llvm-project/build/bin tags/llvmorg-20.1.7^0 jacob@Server
❯ readelf -x 3 /usr/src/debug/linux-cachyos/vmlinux

Hex dump of section '.rodata':
...
0xffffffff82c16f40 4c696e75 78207665 7273696f 6e20362e Linux version 6.
0xffffffff82b43b60 362e3135 2e322d32 2d636163 68796f73 6.15.2-2-cachyos
0xffffffff82e00eb0 4c696e75 78207665 7273696f 6e20362e Linux version 6.
0xffffffff82b43b60 362e3135 2e322d32 2d636163 68796f73 6.15.2-2-cachyos

[jakesteele]

I ended up patching LinuxKernelRewriter with

Error LinuxKernelRewriter::detectLinuxKernelVersion() {
  // ---- Patch Start ----
  LinuxKernelVersion = LKVersion(6, 15, 2);
  BC.outs() << "BOLT-INFO: (patched) Forcing Linux kernel version to 6.15.2\n";
  return Error::success();
  // ---- Patch End ----

  if (BinaryData *BD = BC.getBinaryDataByName("linux_banner")) {
    const BinarySection &Section = BD->getSection();
    const std::string S =
        Section.getContents().substr(BD->getOffset(), BD->getSize()).str();

    const std::regex Re(R"---(Linux version ((\d+)\.(\d+)(\.(\d+))?))---");
    std::smatch Match;
    if (std::regex_search(S, Match, Re)) {
      const unsigned Major = std::stoi(Match[2].str());
      const unsigned Minor = std::stoi(Match[3].str());
      const unsigned Rev = Match[5].matched ? std::stoi(Match[5].str()) : 0;
      LinuxKernelVersion = LKVersion(Major, Minor, Rev);
      BC.outs() << "BOLT-INFO: Linux kernel version is " << Match[1].str()
                << "\n";
      return Error::success();
    }
  }
  return createStringError(errc::executable_format_error,
                           "Linux kernel version is unknown");
}

and that solved my issue:

❯ sudo ./perf2bolt -p perf.data -o bolt.fdata /usr/src/debug/linux-cachyos/vmlinux
[sudo] password for jacob: 
PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read branch events
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 6146a88f60492b520a36f8f8f3231e15f3cc6082
BOLT-INFO: Linux kernel binary detected
BOLT-INFO: first alloc address is 0xffffffff81000000
BOLT-INFO: static input executable detected
BOLT-INFO: enabling lite mode
BOLT-WARNING: ignoring symbol __end_rodata at 0xffffffff834c6000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_hpage_align at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: ignoring symbol __end_rodata_aligned at 0xffffffff83600000, which lies outside .BTF_ids
BOLT-WARNING: split function detected on input : task_rq_lock.cold
BOLT-ERROR: symbol seen in the middle of the function srso_untrain_ret/1(*2). Skipping.
BOLT-ERROR: symbol seen in the middle of the function retbleed_untrain_ret/1(*2). Skipping.
BOLT-INFO: pre-processing profile using perf data aggregator
BOLT-INFO: binary build-id is:     873e93cae96bc10932d87f1da57dcee21256f08d
PERF2BOLT: spawning perf job to read buildid list
PERF2BOLT-WARNING: build-id matched a different file name
PERF2BOLT: waiting for perf task events collection to finish...
PERF2BOLT: parsing perf-script task events output
PERF2BOLT: input binary is associated with 0 PID(s)
PERF2BOLT: waiting for perf events collection to finish...
PERF2BOLT: parse branch events...
PERF2BOLT: read 264338 samples and 6877582 LBR entries
PERF2BOLT: 0 samples (0.0%) were ignored
PERF2BOLT: traces mismatching disassembled function contents: 6923 (0.1%)
PERF2BOLT: out of range traces involving unknown regions: 630520 (9.5%)
PERF2BOLT: waiting for perf mem events collection to finish...
BOLT-INFO: (patched) Forcing Linux kernel version to 6.15.2
BOLT-INFO: parsed 16384 SMP lock entries
BOLT-INFO: parsed 89 static call entries
BOLT-INFO: parsed 806 exception table entries
BOLT-INFO: parsed 14523 bug table entries
BOLT-INFO: setting --alt-inst-has-padlen=0
BOLT-INFO: setting --alt-inst-feature-size=4
BOLT-INFO: parsed 28491 alternative instruction entries
BOLT-INFO: parsed 545940 ORC entries
BOLT-INFO: parsed 966 PCI fixup entries
BOLT-INFO: parsed 14260 static keys jump entries
PERF2BOLT: processing branch events...
PERF2BOLT: wrote 24676 objects and 0 memory objects to bolt.fdata
BOLT-INFO: 745 out of 154161 functions in the binary (0.5%) have non-empty execution profile
BOLT-INFO: 2846 functions with profile could not be optimized
BOLT-WARNING: BOLT is estimated to optimize better with 3.2x more samples. Please consider increasing sampling rate or profiling for longer duration to get more samples.
BOLT-INFO: Functions with density >= 18.7 account for 99.00% total sample counts.

Thanks for your help!

[DavidSpickett]

That's good you solved your immediate problem but it would be good to make this work generally.

Can a Bolt contributor tell us whether the symbol visibility is in fact the problem and whether it's up to the user to configure this on the kernel side, or Bolt should be changed to handle it?