RDR: avoid rebuilding dependent crates after comment changes

[yaahc]

an initial step of Relink don't Rebuild

The goal is to avoid needing to rebuild downstream crates just because an upstream crate changed a comment.

Our current plan is to treat a hash of the rmeta as the indicator that "something that would require a rebuild has changed" and add an unstable compiler flag to start moving things that are "relink independent" (I don't know what to call this, "relink compatible" maybe? "relink safe"? /shrug) into a separate file alongside the rmeta.

In this PR I've taken the baby step of simply removing the data that is causing rebuilds from the rmeta (source map and span information). This has a significant detrimental impact on diagnostics and causes code that depends on libraries that depend on span information to disambiguate bindings to fail to compile (libraries that depend on thiserror such as sqlx-core for example).

This is fine because we don't necessarily care that this initial flag can build all code, as long as it can be used for basic cargo integration and a simple proof of concept cargo project with multiple dependencies where we can enable the flag, edit a comment, and show downstream crates not re-compiling. Once we have that cargo integration we should be able to work on the real impl that moves the data somewhere else rather than simply deleting it and use that with the already setup cargo integration to realize the same gains as the proof of concept against real world codebases.

This is broken down into two commits as follows:

1. Tests for current behavior and the impact of editing comments on the hash of the output rmeta file
2. The changes to remove sourcemap and span information alongside an updated version of the test to enable the associated flag and verify that the hashes no longer change.

[bjorn3]

The crate metadata contains a serialized version of the source map, which in turn contains hashes of all source files. In addition the crate hash also has all source files as one of the inputs.

[yaahc]

based on a comment from bjorn in https://rust-lang.zulipchat.com/#narrow/channel/182449-t-compiler.2Fhelp/topic/RDR.3A.20insensitivity.20to.20comment.20changes/with/527063126 I'm gonna go ahead and start with not even emitting a second file and just killing the sourcemap when the rdr flag is enabled

[rustbot]

r? @jieyouxu

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

This PR modifies run-make tests.

cc @jieyouxu

[jieyouxu]

r? @bjorn3 (as you're likely a better reviewer for the design)

[wesleywiser]

Hi @bjorn3, just a ping to remind you this is ready to review when you have a chance or we can re-roll for a different reviewer. Thanks!

[bjorn3]

I was under the impression that this didn't work yet due to the macro hygiene info getting destroyed and thus cross-crate macro invocations resolving identifiers to the wrong location.

[ConradIrwin]

@yaahc @bjorn3 I am really looking forward to having RDR.

It seems like this approach is probably a non-starter (too much stuff relies on having the source information right now).

Are there any smaller subsystems that we could start working on to remove that dependency? Not sure that "macros" by themselves is a good place to start, but wondering if there's another place to start making incremental changes.

I'm happy to help out with code changes, but I'd need some direction on knowing where to start.

[yaahc]

It seems like this approach is probably a non-starter (too much stuff relies on having the source information right now).

FWIW I don't agree with this assessment. It's definitely true that the biggest barrier to RDR right now is how much we depend on source information throughout the compiler but I don't really see how we could meaningfully achieve RDR without working to break down those dependencies as much as possible. I'm still convinced this is a fruitful, if challenging, avenue of exploration. If you're concerned about the lack of progress I can say that for my part that's mostly because I was reassigned to work on contributing to the Reference and I haven't been able to touch this in a few months.

Are there any smaller subsystems that we could start working on to remove that dependency? Not sure that "macros" by themselves is a good place to start, but wondering if there's another place to start making incremental changes.

As for pieces that could use work: https://github.com/rust-lang/rust-project-goals/blob/main/src/2025h2/relink-dont-rebuild.md is the most up to date piece of info, we made that right as I was getting reassigned so there's not been any significant progress or changes since that proposal was written.

There may also be some additional useful info in https://hackmd.io/@pineapple/rJeURFNXxl/edit which is an older document we used when I was more actively working on RDR prior to it being approved as a project goal.

As for the mention of macros, I'm not sure how macros might contribute to RDR, do you have something in mind?

[ConradIrwin]

I was maybe over-rotating to Bjorn's comments above about even this change causing problems with macro compilation. Glad to hear you don't think that's a deal breaker for this.

Agree there's a lot to get done! Thanks for the links, I'll take a look through and see if there's anything I can figure out enough to be useful with.