Improve tracing in bootstrap

[Kobzol]

I was annoyed that bootstrap had like 5 separate ways of debugging/tracing/profiling, and it was hard for me to understand how are individual steps executed. This PR tries to unify severla things behind BOOTSTRAP_TRACING, and improve tracing/profiling in general:

• All generated tracing outputs are now stored in a single directory to make it easier to examine them, plus bootstrap prepares a latest symlink to the latest generated tracing output directory for convenience.
• All executed spans are now logged automatically (without requiring usage of #[tracing::instrument]).
• A custom span/event formatter was implemented, to provide domain-specific output (like location of executed commands or spans) and hopefully also to reduce visual clutter.
  • tracing_forest was removed. While it did some useful postprocessing, it didn't expose enough information for making the dynamic step spans work.
• You can now explicitly log steps (STEP=info) and/or commands (COMMAND=info), to have more granular control over what gets logged.
• print-step-timings also show when a step starts its execution (not just when it ends it), so that when some step fails in CI, we can actually see what step it was (before we would only see the end of the previous step).
• The rustc-dev-guide page on debugging/profiling bootstrap was updated.

There are still some things that work outside of tracing (print-step-timings and dump-bootstrap-shims), but I think that for now this improvement is good enough.

I removed the > step, < step verbose output, because I found it unusable, as verbose bootstrap output also enables verbose Cargo output, and then you simply drown in too much data, and because I think that the new tracing system makes it obsolete (although it does require recompilation with the tracing feature). If you want to keep it, happy to revert 690c781. And the information about cached steps is now also shown in the Graphviz step dependency graph.

We can modify the tracing output however we want, as we now implement it ourselves. Notably, we could also show exit logs for step spans, currently I only show enter spans. Maybe creating indents for each executed nested command is also not needed. Happy to hear feedback!

Some further improvements could be to print step durations, if we decide to also log step exit events. We could also try to enable tracing in CI logs, but it might be too verbose.

Best reviewed commit-by-commit.

r? @jieyouxu

CC @Shourya742

[rustbot]

This PR changes how LLVM is built. Consider updating src/bootstrap/download-ci-llvm-stamp.

The rustc-dev-guide subtree was changed. If this PR only touches the dev guide consider submitting a PR directly to rust-lang/rustc-dev-guide otherwise thank you for updating the dev guide with your changes.

cc @BoxyUwU, @jieyouxu, @Kobzol, @tshepang