API usage patterns for complex editor extensions

[andrewbranch]

TypeScript 6.0 and earlier allowed loading third-party plugin code directly into TS Server, exposing language service and project system internals for proxying and patching. This was often used in coordination with editor extensions, sometimes in tandem with an additional custom language server, to provide language support for languages that embed or transform into TypeScript.

In the Go implementation, we can't dynamically link third-party code into the server process. Instead, we provide an IPC-based API and a TypeScript client library, currently under development, for communicating with the server. The API will expose most of the read-only features of today's ts.Program and ts.TypeChecker APIs; I want to use this issue to discuss and explore what other features the API needs in order to replace TS Server plugins.

Case study: Vue

I'm going to present my very basic understanding of how Vue's language support integrates with TypeScript today. There are three components:

1. @vue/typescript-plugin, the TS Server plugin
2. @vue/language-server, an additional LSP server responsible mostly for serving requests in the template and CSS parts of .vue files
3. The VS Code extension that registers the TS plugin, spawns the Vue language server, and brokers communication between the two.

When the TS Server plugin initializes, it tells TypeScript to include the .vue extension in its projects and overrides TS Server's view of .vue file contents with valid TypeScript syntax. It then intercepts requests and responses for those files in order to map positions between locations in the original .vue file content and locations in the virtual TypeScript-only content. It also overrides module resolution to allow import {} from "./MyComponent.vue" to resolve, and readDirectory to include .vue files in path completions. Finally, it installs some custom request handlers that provide information from the TypeScript language service to @vue/typescript-plugin, proxied through the VS Code extension.

Proposal

I'd like to get feedback on a system where, instead of making the TypeScript LSP server serve requests for non-TypeScript files directly, we let an auxiliary language server, through API usage, compute or proxy whatever information it needs from the TypeScript language service. Roughly how I imagine that looking:

• The Vue language server would tell the TypeScript API to open / keep open any projects for which it wants to include .vue files, and would "open" the virtual files, then continue to manage their state in a similar way in textDocument/didChange and workspace/didChangeWatchedFiles handlers.

  const snapshot = await api.updateSnapshot({
    openProject: projectTsconfigUri,
    openFiles: projectVueFiles.map(file => ({
      uri: `virtual://@vue/language-server/${file.basePath}`,
      content: getServiceScriptContent(file),
      scriptKind: ScriptKind.TS,
      defaultProject: projectTsconfigUri,
      version: file.version,
    }))
  })

• Language service handlers in the Vue language server could access these virtual files through Program/TypeChecker APIs for semantic analysis as well as initiate standard LSP requests for them.

  connection.onCompletion(async (params) => {
    const { uri, position } = params.textDocument;
    using snapshot = await api.updateSnapshot();
    const { project, checker } = await snapshot.getDefaultProjectForFile(uri);
    const sourceFile = program.getSourceFile(uri);
    
    const virtualFilePosition = mapToVirtualFilePosition(uri, position);
    const node = getTokenAtPosition(sourceFile, virtualFilePosition);
    const tsCompletions = await snapshot.lspRequest("textDocument/completion", {
      textDocument: { uri },
      position: virtualFilePosition,
    });
  
    if (shouldProvideSpecialVueCompletions(node)) {
      const vueCompletions = await getVueCompletionsUsingSemanticAPIs(node, checker, params);
      return [...tsCompletions, ...vueCompletions];
    }
  });

• For module resolution, we'd likely want to provide some kind of static mapping API that can be updated as needed—having callbacks into client code for module resolution is something we want to avoid. Whatever the format, it should cover module resolution, auto imports, and path completions—it seems silly that today's TS Server plugins can specify extra file extensions and external files but still have to patch readDirectory to get them to show up in path completions.

This approach shifts the responsibility and control for servicing LSP requests for non-TypeScript files to the language servers that actually own those files. With the exception of module resolution, most of what other language servers need from TypeScript falls pretty cleanly out of existing LSP functionality (opening and updating a virtual file is the same thing as opening an untitled file in an editor; we just need to add a project association) or existing/planned API functionality.

[chriskrycho]

There’s one additional consideration not named here, which was a significant part of the Glint v1 design (I’m no longer working on it, but Glint v2 uses Volar, so its tradeoffs would apply there and to similar tools): the really important distinction between what the language server provides and what the CLI provides for type checking and emit. These kinds of tools generally need to provide that as well, and have had to use the same mechanics, mutatis mutants in terms of patching the tsc paths, rather than the tsserver paths, to accomplish the same set of goals. A component definition needs to be type checked and to be able to participate in declaration emit the same as it needs to be able to integrate with the language server.

The details that server plugins and compiler extensions have to do in those cases are basically the same, though: synthesize virtual modules, present them to TS, and then map them back to the original component file on disk. The challenge was mostly making sure that we covered all the different paths into and out of the APIs, which (as I don’t need to tell you!) had funny divergences over time.

(The real alternative here is to stop treating JSX/TSX as special and let everyone else play with the same toys. 😉 But I know you all drew a line on that over a decade ago, so I don’t expect you to change it. I do reserve the right to tease about it, though! 😝)

[auvred]

the really important distinction between what the language server provides and what the CLI provides for type checking and emit

+1. For me personally, proper CLI typechecking and emitting are even more important than LSP support (I can live without LSP features, but how can we be sure that the Vue code we commit is valid if we can't typecheck it in CI?)

But I know you all drew a line on that over a decade ago, so I don’t expect you to change it.

Yeah, it's probably too much to ask to add extension language support to the TS core, but it would be really cool if TS could coordinate extension languages itself. Otherwise, the ecosystem would have to deal with hacky approaches like:

• auvred/golar - extends tsgo to add support for extension languages
• sveltejs/language-tools, astralhpi/svelte-fast-check, pheuter/svelte-check-rs, KazariEX/vue-tsgo - emit files with rewritten extensions to /tmp, spawn tsgo, parse CLI output and map dianogstic locations
• ubugeeei/vize - uses the tsgo LSP to run CLI typechecking
• ... maybe some other projects I'm unaware of

Volar.js hacks can be considered moderate nowadays compared to the hacks these projects have had to use!

[andrewbranch]

It will be possible to use the API to load a project and get diagnostics, which would allow these kinds of tools to substitute in transformed files, then perform source mapping of the diagnostic positions themselves. Why would it be important for tsgo to be a plugin host rather than be wrapped as an API within another tool's CLI? Compilers for languages that embed TypeScript are always going to have steps that are totally unrelated and beyond the scope of the TypeScript compiler. It makes more sense to me to embed the TS API in an external compiler than to plug in external language support into the TS compiler.

[remcohaszing]

Hey, I’m glad to see this issue is getting attention. For context, I’m a Volar maintainer and the maintainer of the MDX language tooling. I also helped the Ember team onboard. :)

You can use Volar to create a language server. The basic concept is that you can process formats with embedded content into virtual files, and let existing language services handle the IntelliSense. Volar uses text range mappings to map virtual content back to the original file. This way for example Vue and Astro can offer HTML IntelliSense based on vscode-html-languageservice, and MDX can offer markdown based IntelliSense based on vscode-markdown-languageservice.

You can use the Volar Labs VSCode extension to visually inspect virtual files. I think this really helps to understand some of this.

Previously we also used this to add TypeScript support, but this caused some problems. However, this caused some problems. TypeScript needs a full program context. This means you need a TypeScript Program for Astro, one for Volar, one for MDX, etc. All of these process the same TypeScript files, but also none of these are aware of each other. The fix was to switch to a TypeScript plugin instead. Now TypeScript IntelliSense for Vue and MDX can interop. Astro still provides TypeScript support via the language server, because they had some problem converting to a TypeScript plugin. It could be interesting to see what issues they ran into.

As far as the TypeScript integration goes now, you can forget all about the language server. Only the TypeScript plugin matters.

Currently TypeScript has support for server plugins. This is nice, but not sufficient. TypeScript wouldn’t be nearly as useful without its CLI. I suggest to completely scrap the idea of server plugins, and think about a more generic plugin concept. If your editor shows type errors, then running tsc should show those same errors, even in .vue or .mdx files.

Another example that needs a TypeScript plugin is Vite. Vite suggests you import vite/client in your project, but that should really be a plugin. Those types declare that any import that ends with .css can be resolved, but that’s not true. Importing typescript/this-is-fine.css won’t yield a type error, but it will break your build. On the other hand, importing @wooorm/starry-night/style/dark resolves to a CSS file, but the Vite types don’t know this.

Yet another example that needs a TypeScript plugin is file-system based routing. Both React Router and Next.js have a typegen subcommand. So for a Next.js project you need to run next typegen before you can use tsc. It would be nice if their plugin could just inject their types into the TypeScript program, instead of generating those types on disk.

Back to virtual content: Volar uses text mappings. This works, but I think it’s fairly fragile. It’s easy to produce bad content. Ideally I would like it if a TypeScript plugin can just receive a string of content, and transform it into an AST. If nodes in the AST contain positional information, they exist in the real file. Otherwise it’s some virtual helper code that can’t be referenced or have type errors.

Also plugins may need configuration. For example MDX supports plugins, so the MDX plugin must be able to resolve those.

I’ll gladly answer any questions you have here, but you are also welcome to join the Volar Discord server or DM me.

[chriskrycho]

It will be possible to use the API to load a project and get diagnostics, which would allow these kinds of tools to substitute in transformed files, then perform source mapping of the diagnostic positions themselves. Why would it be important for tsgo to be a plugin host rather than be wrapped as an API within another tool's CLI? Compilers for languages that embed TypeScript are always going to have steps that are totally unrelated and beyond the scope of the TypeScript compiler. It makes more sense to me to embed the TS API in an external compiler than to plug in external language support into the TS compiler.

Basically, needing to reimplement --build, --watch, etc.—the way we did it with tsc was using the same “patch the CLI” approach we all ended up doing with the server as well. In principle, I could see ways that the API could support that; the point is that end users of these tools expect there to be a glint --build to Just Work™ as basically a drop-in replacement for tsc --build, and the same for --watch and so on.

The intuition here is: Svelte/Vue/Angular/Ember/etc. all want to do the exact same things that React/Solid/etc. do, but the latter get them for free with .tsx files.¹ So it’s less that “TS should be a plugin host” than “at least make it easy for the rest of the TS/JS ecosystem to do those same things”. Plugin host is one way you could express that, but not the only. The “constraint” for consumers like these is that the API “just” (!!!) be rich enough to be able to support the same level of support for --build, --watch, etc. as previous consumers have been able to do. (This is all deeply loaded up in my brain, so I may be Curse of Knowledge-ing—happy to try to clarify further, including via chat or even a short call.)

Footnotes

1. TSX/JSX is also a “language that embeds TypeScript”, after all, though it’s been this way for so long that it’s easy for the ecosystem at large to forget. ↩

[remcohaszing]

Why would it be important for tsgo to be a plugin host rather than be wrapped as an API within another tool's CLI?

Languages need to be able to interoperate. It’s not uncommon to have projects that use CSS + Astro + MDX, or CSS + MDX + Vue, or Astro + CSS + Vue.

I also think it’s good if the same tool that provides IntelliSense, also provides the CLI.

[andrewbranch]

Plugin host is one way you could express that, but not the only. The “constraint” for consumers like these is that the API “just” (!!!) be rich enough to be able to support the same level of support for --build, --watch, etc. as previous consumers have been able to do.

I think we're pretty on board with building out higher-level APIs that make it much easier for people to build CLI experiences on top of the API. That said, --build is an orchestrator for building TypeScript code, and there are lots of good options out there for more general-purpose orchestrators that can themselves instrument tsc or tsgo via CLI or API. It's a non-goal for TypeScript to become the top-level orchestrator for polyglot projects that happen to include multiple tsconfigs.

It would be nice if their plugin could just inject their types into the TypeScript program, instead of generating those types on disk.

I do not think we're on board with this or anything in this direction.

Previously we also used this to add TypeScript support, but this caused some problems. However, this caused some problems. TypeScript needs a full program context. This means you need a TypeScript Program for Astro, one for Volar, one for MDX, etc. All of these process the same TypeScript files, but also none of these are aware of each other. The fix was to switch to a TypeScript plugin instead. Now TypeScript IntelliSense for Vue and MDX can interop.

This problem is solved with the architecture in my proposal (in fact this much already exists in main today)—any number of extensions can connect to our LSP server and get an API connection served by the same underlying state. This would provide the primitives for a future Volar-like framework to coordinate interop between multiple extensions/languages.

We want to provide a really good API that lets people build rich experiences that can leverage TypeScript features. We're not looking to reimagine our role in the JavaScript ecosystem as an umbrella coordinating every possible language dialect and extension you can imagine.

[DanielRosenwasser]

It's a non-goal for TypeScript to become the top-level orchestrator for polyglot projects that happen to include multiple tsconfigs.

I just want to clarify that things like --build with .vue, .astro, etc. files is still fair game - it has parity with <=6.0 and we're interested in making it possible for people to build tsc-like experiences that understand those files. We're just not trying to understand every other language/ecosystem convention.

[andrewbranch]

Does the 6.0 CLI allow that, or do you mean running the solution builder APIs? It would be news to me if tsc could do that natively. My view on the new API is basically

• it should enable you to do basically whatever you could do with the Strada API, only easier
• because the whole thing is based on the project system, anything you can do with the API as an "LSP plugin" you can also do standalone

[DanielRosenwasser]

Does the 6.0 CLI allow that, or do you mean running the solution builder APIs?

Sorry, I'll clarify further - I don't mean running the CLI as-is, but rather, running the solution builder APIs. I feel like we should provide the tools to build something like the Vue CLI which I believe is already able to understand project references, insert TS files, change resolution, etc. Others can correct me if I'm wrong (or think differently).

[jasonlyu123]

I have a prototype using LSP as api a while ago. The idea is that the IPC client process runs typescript-go in LSP mode. Client "configured" a custom file extension. TypeScript add that to extraFileExtension for matching project files, and also uses the file extension for module resolution. Once the file is loaded, TypeScript can send a request to the client for virtual content and ScriptKind. The virtual content can contain a /// <reference path="..."> directive to load helper functions referenced in the virtual content.

The reason I think it should be TypeScript pulls virtual content is that assigning files to which project can be pretty complex. Svelte currently has a simplified version here. Hope we don't need to have it when using typescript-go.

For module resolution, we'd likely want to provide some kind of static mapping API that can be updated as needed—having callbacks into client code for module resolution is something we want to avoid

I don't quite get this part. Does the approach I mentioned match the thing you want to avoid? Would it be better if TypeScript treated it as an empty file first? TypeScript could then send a batch request for all the virtual content at once. Or have an API for the client to request these "pending" files, then the client uses the proposed api.updateSnapshot approach. One problem I could think of about this "batch" update approach is that the files in the batch might load more files through module resolution. It could be slow if TypeScript kept rebuilding the program on each of these batch requests.

In Svelte's use case of CLI, we could work with embedding LSP/API, but we still need an API to extract all project files and project references for a solution-style project. And optionally also a way to leverage --incremental

[remcohaszing]

I just want to clarify that things like --build with .vue, .astro, etc. files is still fair game

I’m not sure how the --build option relates to this. Since I use project references so often, My muscle memory pretty much adds this flag by default whenever I run tsc.

But this does remind me that in cases where you want to check these kind of files, you’re probably using noEmit anyway. The use of plugins might as well imply and require "noEmit": true.

• MDX files are syntactic sugar for content heavy JSX. I don’t think anyone is publishing that to npm.
• CSS, SVG, font, and image files in npm packages are more common, but the behaviour really depends on the bundler that the consumer uses. It’s best to let the user decide how their types behave, not the package author. For example Next.js proceses images different than Vite.
• Astro and Vue files are typically published to npm as-is I think. But please correct me if I’m wrong.