WASM-based language server support

[AlphaNecron]

• I (Necron) own the content in this Pull Request. Neither my employer
  or anyone else has rights to this content. I here by grant to Dave Halter
  (the owner of Zuban) a perpetual, worldwide, non-exclusive, no-charge,
  royalty-free, irrevocable copyright license to reproduce, prepare derivative
  works of, publicly display, publicly perform, sublicense, sell and distribute
  my contributions and such derivative works.

This work is derived from #137 (as well as Claude) so I'm not sure whether to check the one above xD. This PR implements a working language server for WASM conforming to LSP@3.17. I've tested internally with CodeMirror 6 and @codemirror/lsp-client, completions, hover definition, diagnostics, they are all functional as expected. All builtin stubs are embedded to the binary so the server itself should work out of the box.

import init, {LS} from './playground_wasm.js';
await init();

const server = new LS(false);
// we can even add 3rd stubs, as long as we prefix the path with `site-packages`
server.register_stubs(Object.fromEntries(
	Object.entries(stubs).map(([path, content]) => [`site-packages/${path}`, content])
));

// this takes a json-encoded string and return a json-encoded string as well, havent tested if a jsvalue would work but it should
const resp = server.handle_message(...);

Current capabilities

• initialize
• shutdown
• initialized
• exit
• textDocument/didClose
• textDocument/didSave
• textDocument/didOpen
• textDocument/didChange
• textDocument/completion
• textDocument/hover
• textDocument/definition
• textDocument/declaration
• textDocument/typeDefinition
• textDocument/implementation
• textDocument/references
• textDocument/signatureHelp
• textDocument/prepareRename
• textDocument/rename
• textDocument/documentSymbol
• textDocument/diagnostic
• textDocument/inlayHint

Haven't tested inlayHint and stuff which navigates to other files.

It also exposes set_unavailable which marks symbols as unavailable, useful for environments like Pyodide where most syscalls aren't implemented.

A playground is kinda out-of-scope here so there so it better be a dedicated project.

[AlphaNecron]

It now suggests all keywords based on the context, so thats probably why tests failed. Not sure if this is false negative /shrug

[davidhalter]

Thanks for trying to contribute.

I have only skimmed the code for a bit. There's a few problematic things in there.

1. This does not just do WASM, it's a mix of things. For example there's add_python_keyword_completions that adds keyword completions, which is totally fine, but that's a different PR.
2. There's random code changes, that have nothing to do with WASM or any other features. These should probably all go.
3. "A playground is kinda out-of-scope here so there so it better be a dedicated project." Then the playground related code should probably be removed.
4. Why is there a reimplementation of a language server (playground_wasm/src/ls.rs)? The language server already exists (obviously).

And please don't just give these prompts Claude, have a look at the stuff yourself. LLMs are incredibly bad at writing code for complex Rust codebases (they are great for other things though).

I need all code to be removed that has nothing to do with WASM. Otherwise it's just not reviewable. It's probably easy for me to add WASM with Claude Code, but I have a bit higher standards than what LLMs typically provide. You are probably pretty young and are getting better at programming with LLMs. I just really want to encourage you to trust them less and try to think about the programs critically. There's a lot of non-obvious issues here, that are typical for LLMs.

It also exposes set_unavailable which marks symbols as unavailable, useful for environments like Pyodide where most syscalls aren't implemented.

Why? This just sounds like a broken implementation of LSP.

[AlphaNecron]

Some environments like Pyodide does not fully support stuff like syscalls, and I think adding a method to "blacklist" unavailable symbols would debloat completion results. About the actual playground, "playground" here means a web version, this only provides the binary needed for powering a true playground. It would require something like monaco-editor or codemirror, haven't tried the former, and the latter is known to cause deps conflicts. Thats why a dedicated Vite + maybe Solid would be better (like other projects, pyright also do that). Also why a dedicated crate for language? Cuz I dont really wanna touch zubanls, that would increase the binary size as the project scales. This implementation also requires some arch-dependent stuff like custom FS stub, no parallelism, thus needing some "random" code change. This is literally my first time experiencing LLM/vibe-coding so I don't know how to do that properly. I'm experimenting a fully-fledged code editor, after going through python-lsp-server, jedi, pyright, pyrefly. Ive decided to settle down here, but its pretty early-stage and does not have proper WASM support, and I dont have the guts and time to experiment much with LSP and this (rlly overwhelmed by exams 😭). Tried my best to make it working, so the code may not be good, mb cuz its kinda bad even imo.

[AlphaNecron]

Also this alters the behavior of UTF-16 chars, lemme revert that as well.

[davidhalter]

(rlly overwhelmed by exams 😭)

No worries, take your time. I just wanted to make it clear that I want a clean solution, otherwise I will not merge.

Cuz I dont really wanna touch zubanls, that would increase the binary size as the project scales.

That's the wrong approach. The zubanls crate is mostly finished and will not grow much larger. And also besides: I would advice you to never duplicate code in an open source repo, it's generally a code smell. There are for sure better ways.

This implementation also requires some arch-dependent stuff like custom FS stub, no parallelism, thus needing some "random" code change.

That's ok. These are just parameters to zubanls then.

[AlphaNecron]

Ye, I'm refining the impl, rayon does fallback to noop on WASM so we can use that. We still have to refrain from FS accesses as they are not available on this env. Initially I wanted to use a dedicated crate containing a subset of features for binary size but imma try to make use of zubanls directly xD.

[AlphaNecron]

Imma try making a playground this week, it works with CodeMirror :D