Language Index support

[hamrik]

Rationale

Helix is perhaps best known for its out-of-the-box experience, such as being able to navigate codebases intelligently without any configuration if the appropriate language server is installed.

However, these language servers might not always be available, such as when editing remote files or checking out projects the tooling of which is not yet available.

There are currently two competing standards to provide code intelligence in the absence of language servers, LSIF and SCIP. Both rely on an indexer precomputing relationships and hover information for any position a cursor might be in a codebase, giving the editor an index file from which it can offer hover, go to definition, find references, etc.

LSIF

LSIF is made by Microsoft and it's basically a subset of LSP, using the same data types and commands as LSP. The indexer generates a JSON dump of every range for which LSP commands are available, and then the output of each of these commands.

The editor then instead of firing a request to the LSP consults the index file if a command output is available for the range the cursor sits in, and provides the output associated what that range-command pair in the index file.

Pros

• LSP types and command structs are already part of Helix codebase
• LSIF is pushed heavily by the LSP community

Cons

• Index files are large and slow to consume
• The implementation for Helix has to be done from scratch

SCIP

SCIP is developed by Sourcegraph and works similarly to LSIF, with two key differences:

• They use ProtoBuf instead on JSON
• They use human-readable IDs rather than graph edges to navigate between structures.

They claim their format is quicker to navigate and consumes less memory. As an example, a SCIP dump of the Helix codebase is 11MB while the LSIF dump is 47MB. They also claim that indexers are simpler to create for SCIP, and is already supported by major languages and even rust-analyzer.

Pros

• Lighter compared to LSIF
• Libraries exist for consulting SCIP indexes

Cons

• Not compatible with LSP and thus the types already implemented in Helix
• SCIP dumps can be converted to LSIF dumps but not vice versa

Support in Helix

It would be beneficial to support one or both of these index formats for when an index file is available in a repo but not the corresponding language server. I can envision two ways of implementing either:

a) Creating a thin LSP server that consumes these index files then communicates with Helix via the usual means
b) Creating a helix-lsif or helix-scip project coupled to helix-lsp to provide fallbacks when an LSP is unavailable.

I personally lean towards LSIF as it is starting to become an industry standard and SCIP dumps can be converted to LSIF dumps. Plus that way we don't have to maintain two almost-identical types for things like ranges and commands just to stay compliant with the ProtoBuf schema.

What do you guys think?