Bolt: BAT is forward-incompatible between bolt-18 and bolt-19

[itrofimow]

What I did:

1. Built the binary baseline_emit_relocs with -Wl,--emit-relocs
2. Gathered the profile: perf record -e cycles:u -j any,u -o perfdata/baseline_emit_relocs.perf.data -p $(pidof baseline_emit_relocs) sleep 20
3. Converted the profile into yaml-format with bolt-19 (to which perf2bolt resolves): perf2bolt baseline_emit_relocs --perfdata=perfdata/baseline_emit_relocs.perf.data --profile-format=yaml -w profiles/baseline_emit_relocs.bolt.yaml -o /dev/null
4. Bolt-ed the original binary with bolt-18: llvm-bolt-18 baseline_emit_relocs -o bolted_bat --data=profiles/baseline_emit_relocs.bolt.yaml --reorder-blocks=ext-tsp --reorder-functions=cdsort --split-functions --split-all-cold --split-eh --dyno-stats --update-debug-sections --infer-stale-profile --enable-bat
5. Gathered the profiled from bolt-ed binary: perf record -e cycles:u -j any,u -o perfdata/bolted_bat.perf.data -p $(pidof bolted_bat) sleep 20
6. Tried to convert the profile into yaml-format with bolt-19 (to which perf2bolt resolves): perf2bolt bolted_bat --perfdata=perfdata/bolted_bat.perf.data --profile-format=yaml -w profiles/bolted_bat.bolt.yaml -o /dev/null

which resulted in error BOLT-ERROR: failed to parse BOLT address translation table.

---

So I need

• yaml-format when converting the profile for --infer-stale-profile to work when bolting the binary later
• --enable-bat when bolting the binary for profile collection to work on bolt-ed binary
• bolt-19 for converting profiles of bolt-ed binaries into yaml-format (bolt-18 fails with BOLT-ERROR: unable to save profile in YAML format for input file processed by BOLT. Please remove -w option and use branch profile.)

to enable continuous-bolting in production, but forward-incompatibility of BAT gets in the way.

---

1. Is BAT intended to be compatible between bolt versions?
2. If not, are there any ways to convert perf.data of bolt-ed binary into .yaml profile somehow (maybe with additional hops through fdata?) with bolt-18?

[itrofimow]

@aaupov I see you've landed some changes to BAT between 18 and 19, could you please have a look?

[aaupov]

1. Is BAT intended to be compatible between bolt versions?

There are no guarantees, but my recent changes were the first breaking changes since BAT was introduced. When using BAT, please use matching versions of perf2bolt and llvm-bolt.

2. If not, are there any ways to convert perf.data of bolt-ed binary into .yaml profile somehow (maybe with additional hops through fdata?) with bolt-18?

You can convert perf.data from bolt-ed binary into fdata, and then use pre-bolt binary to convert fdata into yaml with hashes (pass -w yaml to llvm-bolt command-line).

[itrofimow]

When using BAT, please use matching versions of perf2bolt and llvm-bolt.

I think this poses severe practical challenges to anyone who doesn't do 2-stage release (a variant of which AFAIK Meta does w.r.t. Bolt), and my understanding of --enable-bat and --infer-stale-profile is that these were made to support exactly such cases?

In other words, when building (bolting included) and profile collection are separated systems, how does one update the toolchain?

and then use pre-bolt binary to convert fdata into yaml with hashes

That's some hops indeed! Thanks, will look into it

[aaupov]

I think this poses severe practical challenges to anyone who doesn't do 2-stage release (a variant of which AFAIK Meta does w.r.t. Bolt), and my understanding of --enable-bat and --infer-stale-profile is that these were made to support exactly such cases?

Practically speaking, there was no way to use BOLT in a continuous fashion before these breaking BAT changes. BAT and infer-stale-profile were introduced separately but they couldn't support a true continuous operation mode. That's until perf2bolt was extended to emit YAML profile with hashes necessary for infer-stale-profile from BOLTed binary.

[itrofimow]

Ah, I see.
Seeing that --infer-stale-profile was merged in 17 I incorrectly assumed that continuous bolting was meant to be supported already, but now I realize that a lot of glue needed (BAT <-> YAML) only recently landed in 19. Thank you for all the hard work you've put into this!

Given that BOLT is now usable in a continuous fashion, maybe that deserves a release note (alongside with the breaking BAT change)? The only thing there is in 19-notes is a line about --match-profile-with-function-hash.

[aaupov]

We plan to make a publication about the continuous mode, describing the setup and performance implications compared to normal (2-step mode), so stay tuned!

[itrofimow]

That's very cool, looking forward to it!