Project json compatibility improvements

[cormacrelf]

Adds better backwards- and forwards-compatibility to the project JSON deserializer. For example, today, if you use flycheck runnables from #18043 with an old rust-analyzer, it bails on deserializing the whole structure and you can't do anything. That's fixed (at least for new rust-analyzer). Otherwise adds and recommends more #[serde(default)].

Updated:

• added JSON deserialization testing for these newer pieces of the project model. These have been in fairly widespread use for about 18 months or so now. At least if someone is breaking these users, they'll know it when the tests fail.

[ChayimFriedman2]

We generally do not provide forward compatability for our config. However in this case I do support this, since rust-project.json is often committed and used by different r-a versions. I do think this requires a more wide agreement though, so: CC @rust-lang/rust-analyzer.

I have a concern, though: this could lead to an effective stabilization of format versions we don't officially support, but can't stop supporting because they're too widespread.

[cormacrelf]

I think it's warranted generally for the project JSON as opposed to config, not just because people commit their JSON to git, but also because the JSON is generated by other tools that have an independent release cycle. Especially if you're using workspace.discoverConfig. In contrast, RA config is usually done by humans and checked at the time of writing.

I recall, from the design discussions around the runnable kinds, Matklad describing some principles for JSON formats and pushing for a vec of runnables rather than a struct, so we would be more relaxed as to what's acceptable and more flexible to accept multiple of each kind, etc. Originally, it was a struct like { run: Option<Runnable>, test: Option<Runnable>, ... }, which actually is forwards compatible for new runnable kinds since Serde ignores unknown fields. The alternative that we landed on is not, and I think it should be brought back to parity.

As a practical example, facebook/buck2#767 will make the rust-project JSON generator incompatible with most users' copies of rust-analyzer unless we wait for RA's half of this functionality to make it to a stable rustup release so that we can point users somewhere. I wish I had made this compatibility PR a year ago so this would not be the case.

[Veykril]

I think it's fine for us to say we don't necessarily support all version of this format here but keep a supported window (like we do with rust versions). Either way sounds like we should merge this quickly (and maybe beta back port as well?)

[cormacrelf]

I'm not sure given the Meta people would be able to merge that PR quick enough for a backport to matter, but maybe it would be helpful to the rest of the community. You can do it if you have the time.

[Wilfred]

Overall this makes sense to me :)

I do think we should update https://rust-analyzer.github.io/book/non_cargo_based_projects.html to explain that unknown runnable types are permitted but ignored. It might also be nice to log a warning on unknown runnables.

[Wilfred]

Nit: it's not that they're newer tools, it's that they're tool that can output additional types of runnable.

[cormacrelf]

That is implied by this comment being attached to a serde(other) variant called "Unknown"

[Wilfred]

"most of this datastructure" is ambiguous: which parts of the datastructure? How about "everything should be optional except crates, so we can add support for new fields that old rust-project.json tools aren't generating yet".

[Wilfred]

Wearing my rust-project maintainer hat, we package r-a the same way as OSS does: including the binary in the VS Code extension. So if a user hasn't updated their extension, adding new types of runnables to rust-project.json is going to break their setup.

Given that, making r-a tolerant to unknown runnable types seems worthwhile.

[davidbarsky]

since rust-project.json is often committed and used by different r-a versions.

With my former-Meta engineer/current rust-analyzer team member hat on, I recommend not committing and instead generating a rust-project.json on-the-fly through the discoverProjectConfig, but I agree with Wilfred: not breaking people is good, but I think relying can dramatically shorten the migration periods where rust-analyzer needs to be tolerant of old/new versions of rust-project.json.

Wearing my rust-project maintainer hat, we package r-a the same way as OSS does: including the binary in the VS Code extension. So if a user hasn't updated their extension, adding new types of runnables to rust-project.json is going to break their setup.

The details are already fading from my memory: rust-project was packaged with dotslash, so the repo/dotslash version might move forward, but the extension might be not updated, right?