JSON transformer plugin API

[Oblarg]

I've been developing a rust units of measure library that does a lot of const generic wizardry to handle dimensional coherence checking and rescaling at compile time.

As part of this, I have a very substantial need to prettyprint highly-parameterized types in a human readable way. I've been doing this with a LSP plugin that wraps r-a, allowing me to intercept JSON messages and customize the appearance of my types within them. This has proven extremely useful:

The implementation here is dead simple - we search for Quantity<...> with the right number of commas, and when we find it we use our existing prettyprint integration for Quantity (e.g., the one hooked up to the display and debug traits) to handle rendering it to a human-readable form.

However, wrapping the r-a binary and intercepting the LSP messages is hacky and not very composable. It feels like this is a good opportunity for r-a to support a plugin API with a minimal json->json contract, so that integrations like this can be more easily used, and sensibly composed together.

I could write a draft of such a feature, but i figured i'd open an issue first and see what people think?

[nik-rev]

Idea 1: In Rust-analyzer's configuration there can be an array of plugins, which are just binaries to call. Each binary takes JSON and outputs JSON to the stdout, allowing these plugins to be stacked. Each plugin is executed serially on each JSON request before they are sent to the Client

Pros: The simplest approach
Cons: a lot of IO. For every 1 plugin, that's an extra IO operation everytime rust-analyzer sends something to the Client

Idea 2: it could be done the way that Rust currently does proc macros. Each proc macro is a DLL, Rust compiler loads this DLL into memory (IO) then directly executes its code without needing to do any IO (provided the proc macro is pure). Maybe something like this can be done for rust-analyzer.

Pros: IO is done just once to load the plugin, then the plugin's code resides in memory
Cons: It is much more complex. Also, it requires restarting rust-analyzer to use new plugins (minor)

[ChayimFriedman2]

This is an interesting idea (that I also had in the past), however r-a has a very large API surface, both in things that a plugin will want to customize and in things that plugins will want to use. Providing a stable API interface will be a great churn for us at this point in development of r-a.

So I think we should at least defer this to when r-a stabilizes more (which is an unknown number of years from now). But CC @rust-lang/rust-analyzer what are your thoughts?

[Oblarg]

@ChayimFriedman2 I think the churn can be minimized if the plugin architecture centers the LSP contract, instead of r-a specifically. It doesn't much matter what r-a does on the other end of that contract; the putative plugins are just JSON transformers, and won't need to be rewritten unless the LSP message framing changes.

To my mind, the major value comes not from offering rich r-a integrations, but from offering an easy way to leverage existing print functionality that the user has likely already written, to improve the appearance of their types. This depends on their contract, not r-a's.

[ChayimFriedman2]

If you don't want access to r-a information (e.g. verify that this is indeed the type you want, and not some type that has the same name) or stability (e.g. verify that the information shown on hover won't change), then I don't think this should be builtin to r-a, and post-processing LSP communication seems to do the work fine.

[Oblarg]

@ChayimFriedman2 why shouldn't this be built-into r-a, though? that would provide the best possible experience for developers in terms of accessibility and composability, at fairly low cost to r-a itself.

r-a is optimally positioned to do the work of passing the JSON to the transformers, and then emitting the transformed JSON to the IDE - it already has representation of the message framing protocol in rust, so it is very easy from the r-a source to expose an API that accepts rust functions that act on each of the message types (inlay hint, hover, etc). Any attempt to do this externally would involve re-engineering a lot of what r-a has already done; and would necessarily be less performant and less clean.

moreover, a minimal json-transformer API that does not expose any r-a internals leaves the door open to future expansion/stabilization of more substantive r-a integrations on top of the minimal contract that costs r-a nothing, as the internals become stable enough for that to make sense.

[ChayimFriedman2]

Mainly, because you're building on top of LSP, not r-a, so it's not r-a's responsibility.

For instance, other LSP clients could benefit from that as well. Also, I don't want to introduce code to r-a that is that different from what it does, introducing unneeded complexity.

[Oblarg]

@ChayimFriedman2 Other LSP clients could benefit, but the natural place for it to be done for other languages is, imo, in their respective r-a equivalents.

A big value-add for this, is that because it is uniformly in rust, it can be made to use the code from your library to prettify your types when they're detected. Libraries tend to already have prettyprinting code for values via display/debug trait impls - so, to get a minimal LSP plugin with a contract like this, a library only needs to write a type detector/parser that plumbs to display code they've already written.

A general LSP tool that tries to do this for multiple languages faces a lot more friction here; it has to either support JSON transformation in multiple languages, or else force libraries that want a LSP integration to maintain a type detector/parser in a language different than that of the library.

The amount of maintenance burden on r-a here in the minimal viable case is tiny - all r-a has to do is call a user-provided transformer on each message JSON before sending it to the IDE. The ask here is that r-a provide a way for users to easily leverage the work it is already doing in plumbing the LSP protocol, rather than forcing them to duplicate it.

[ChayimFriedman2]

I don't see how if, as you describe, all r-a does is call the plugin with the JSON, a general purpose plugin system will face harder challenges, or why it's better than just piping stdout of r-a to some process than postprocesses the JSON.

[Oblarg]

@ChayimFriedman2 i just explicitly stated why it faces harder challenges; it needs to be language-compatible if you're going to reuse first-class library code in the LSP integration, which means you have to duplicate the work r-a already does in representing LSP with rust types.

My quantity prettyprinter would be completely untenable if I were not able to leverage the code used for the ordinary debug trait directly in the LSP integration. Is a general LSP plugin system going to allow me to do this? How?

[ChayimFriedman2]

We are not going to pass Rust types to a dll, only JSON strings, as they're layout unstable, so this work will need to be done anyway.

And it's not a big deal either: you can just use libraries that represent LSP types.