Builtin Library as document prelude

[kristiannotari]

Hello everyone,

I'm really enjoying using Langium as my first ever language engineering tool (I've briefly sketched the grammar in Xtext but then migrated to Langium). I'm developing a DSL where each file is independent from one another, so I'm really not using the workspace as a manager for multiple files feature. So each file can be checked, validated, scoped, linked and so on independently.

Then I have an option at the top of a file to include or not a prelude, which means a list of prebuilt definitions of values and references I want available in the file without redeclaring them (nor importing, since there's no import stuff going on between files).

This prelude is known and it's the same for everyone, if needed, so I thought about adding it directly to the language. My idea was to somehow prepend some text/nodes to the file/document I'm working on in a said moment, so that every check/validation/scope/linking process can then be executed as if the prelude was written originally in the file.

My questions/doubts are:

1. How could I implement something like this? The builtin libraries chapter on the docs site is basically adding a non-existing file to the ones checked by the workspace, but that's not exactly what I need cause I don't resolve cross-file scopes, but I do need to have "go to definitions" for things defined in the prelude as something written in a separate file and not the file the user is working on
2. At which point should I "insert" my prelude? I thought about adding it as soon as possible, directly when working with text files, but that way I'm messing up line numbers and stuff I guess (and with go to definitions as already said)
3. I'd like for this "prelude adding" system to work independently if it was run by the vscode extension, from a browser or from a cli application, so I need it in the core language files, that is something that works by changing the language module, am I right?
4. Moreover, I need to add or not add the prelude based on something I need to check after parsing the file I'm working on, which is also confusing me

For context: since I don't have imports/exports between files I reimplemented the computeExports method of the DefaultScopeComputation to export an empty array. Just so you know, and can, possibly, tell me to do otherwise if needed.

[msujew]

Hey @kristiannotari,

Glad to hear you're enjoying your time with Langium! Leave the GitHub repo a star ;)

I believe that modifying the text to-be-parsed in place is not a good idea. At the very least it will massively mess with all language services as you already mentioned, due to all offsets being incorrect compared to the original file that's displayed by the user. I think a best-practice solution involves the builtin libraries as well as some minor modification to the scoping mechanism.

For context: since I don't have imports/exports between files I reimplemented the computeExports method of the DefaultScopeComputation to export an empty array. Just so you know, and can, possibly, tell me to do otherwise if needed.

That's a good start! First off, define your different preludes as separate builtin library documents. The solution that I have in mind also does something similar as your current solution, but performs the normal computeExports method when the document in question is a builtin library. So the globally exported objects are only those from library files.

Next, you will need to override the getGlobalScope method. The override should filter the results of the global index depending on option that you mentioned you had at the top of the file. The context.container variable holds the AST node that owns the reference, so you can use that to find the options. That should actually be everything.

I believe that solves everything:

• Doesn't need to include the prelude into each and every file (which actually might incur quite a performance hit)
• Doesn't mess with any language services
• Is very much in line with how builtin libraries should be used, so there's likely no unintended consequences.
• Allows to jump to the referenced elements if you implement a file system provider for vscode (as outlined in the guide).

I'd like for this "prelude adding" system to work independently if it was run by the vscode extension, from a browser or from a cli application, so I need it in the core language files, that is something that works by changing the language module, am I right?

Yes, you can just include it into the createLanguageModule function. We do that for the langium language as well to add a configuration provider to read vscode preferences.

[kristiannotari]

There's still one thing I'm missing from the way managing documents work. I need to rely on vscode-uri and vscode-languageserver, even if I'm writing my core language functionalities which could have nothing to do with vscode, but could be used in other LSP-based editors and such, right? So why the dependency? I assumed I needed to create a "core package" which involves declaring how the language work, which would be editor agnostic (and even agnostic to wherever you're running it, node or browser for example). Am I missing something? Should I have multiple implementation of the WorkspaceManager for example?

This is the reason I'm saying this:

import type { LangiumDocument, LangiumDocumentFactory, LangiumSharedServices } from "langium"
import { DefaultWorkspaceManager } from "langium"
import type { WorkspaceFolder } from "vscode-languageserver" // <------------------- ?????
import { URI } from "vscode-uri" // <----------------------------------------------- ?????
import { prelude } from "./builtin/prelude"

export class KantWorkspaceManager extends DefaultWorkspaceManager {
    private readonly documentFactory: LangiumDocumentFactory

    constructor(services: LangiumSharedServices) {
        super(services)
        this.documentFactory = services.workspace.LangiumDocumentFactory
    }

    protected override async loadAdditionalDocuments(
        folders: WorkspaceFolder[],
        collector: (document: LangiumDocument) => void
    ): Promise<void> {
        // Load our library using the `builtin` URI schema
        collector(this.documentFactory.fromString(prelude.content, URI.parse(`builtin:///${prelude.name}`)))
    }
}

EDIT: It seems like following the builtin library documentation tells you to add a WorkspaceManager to the workspace config for the module but that property doesn't seems to exists with langium 1.0.1
EDIT 2: It's due to the service being a shared one, sorry for that

[msujew]

Both the vscode-languageserver and vscode-uri packages are IDE and runtime (node/browser) agnostic. They are not necessarily bound to vscode, but rather build by the vscode team. The only thing which is IDE related is the extension part which is using the vscode-languageclient package (which is indeed bound to the vscode runtime).

The actual runtime implementation of the language server is provided by the vscode-languageserver/node or vscode-languageserver/browser import respectively. Everything else is just language server related types which are completely runtime and IDE agnostic.

[kristiannotari]

That's what I was assuming/hoping 😄

Regarding the FileSystemProvider to actually have a goto definition for definitions exported by the builtin library file that's something I want to place in the extension, as it's interfering with vscode-related stuff so it's not agnostic I guess.

[msujew]

Yes, the FileSystemProvider is meant to be living in the extension code, as it has to be part of the vscode runtime in order to import from vscode. It is only there so vscode knows where to lookup documents for your special library uri scheme. It is not part of your DSL.

[kristiannotari]

Thanks for the help. I think I have everything setup for the builtin I need in my DSL!