Best way to "inject" a type system in a small expression language that lacks its own type definition mechanisms?

[haukepribnow]

I'm working on a formula language that supports simple expressions that can be compared to how one might write a Java Streams or a C# LINQ expression. Here's a flavor for (roughly) how the language looks like:

persons.filter(p => p.age > 18).map(p => {city: p.city, income: p.income})

The language does not have any mechanisms to define the types of the objects like persons or its collection members. It is limited to the formula-like expression. (The types and the members of the global scope are defined in a custom data format that is not meant to be parsable by Langium.)

Now I wonder what the architecturally most sound implementation approach could be for the following use case: I would like to provide a powerful VS Code extension for my language while making use of Langium's existing features as much as sensibly possible for this to keep implementation effort low and maintainability high. As a minimum, I would like the extension auto-complete:

1. the items available in the respective scope (e.g. persons in the global scope and p in the scope of the anonymous function) and
2. the member and types available in the objects (e.g. .filter as a member of persons or .age as a member of a Person-typed object).

I considered these options so far:

---

1) Treat this as a case of Multiple dependent languages
In this option, I would create an additional grammar meant specifically for defining types & global scope items. I would then convert the types and global scope items from the custom data format into the newly created grammar on-the-fly. Then I would make Langium parse the created file and go from there.

While this might work, it feels like trying to jump through too many hoops: When we read the custom data format, we basically already have the types & global scope items available in a parsed data format. So having to convert them first and then parse them again feels like too much work. Also, I fear that this might not be well-maintainable in the long run.

2) Hook into the ScopeProvider service to add global scope items and to make information about the object's members available
In this option, I would:
a) Use a cross-reference in my grammar that allows referencing some abstract named element - something like this:

grammar Sample

entry Entry:
    expression=Expression;

Expression:
    {infer Base} base=[NamedElement:ID] ({infer MemberAccess.previous=current} '.' member=[NamedElement:ID])*;

NamedElement:
    name=ID;

terminal ID: /[_a-zA-Z][\w_]*/;

b) Override the ScopeProvider service and have it i) provide the global scope items for the global scope, and ii) have it provide information about the object's members (like .filter, .age) as part of the MemberAccesses.

As a downside, it however looks like as if the result would be really hacky because the Scope interface require me to return a AstNodeDescription, i.e. an object that represents an AST node. But my global scope items would not actually live in any AST.

So in case this actually turns out to be a "suitable" option, I would love to get some guidance on how to populate my AstNodeDescription objects, e.g. what to put into the type field or the documentUri field?

3) Hook into the IndexManager or ScopeComputation service to add global scope items
I considered this option only to get the global scope items in place. The member accessing question would remain unresolved with this option. In this option, I would still use a cross-reference in my grammar like above and I would override the IndexManager or the ScopeComputation service to have it provide the global scope items that can be referenced.

Pretty much the same "hackiness" as before might bite me here: Both interfaces require me to return a AstNodeDescription. Also here, I'm not sure how to populate it properly.

4) "Give up" on cross-references completely and work only with the LSP capabilities
In this option, I would not make use of Langium's cross-references capability at all. I would rework my grammar to something like this:

grammar SampleWithoutCrossReferences

entry Entry:
    expression=Expression;

Expression:
    {infer Base} base=ID ({infer MemberAccess.previous=current} '.' member=ID)*;

terminal ID: /[_a-zA-Z][\w_]*/;

And I would need to live with the fact that all of Langium's nice scoping/linking/etc. capabilities would not support me anymore. Instead, I would need to override many of the existing default implementations for the LSP services, especially the CompletionProvider.

---

I'm not really happy with either of these options so far. So I would really love to get some guidance, both on further options that I haven't considered yet, and on aspects that I should watch out for when going for any of the given options.

[msujew]

Hey @haukepribnow,

these are some good ideas. I would personally go with (1) usually, since it has the added benefit of being able to provide users with the actual type definitions, so they have a way of looking up what the types are that they're working with. See also this guide on builtin libraries.

(2) and (3) are mostly the same, in the sense that they simply enrich the scope with purely virtual objects that don't actually exist in documents. You would still need to declare interfaces for these types though in the Langium grammar, even if they don't have an implementing parser rule. See our documentation on declared types. You can then create instances of these interfaces in memory based on your custom data format. You then create a virtual document for these instances using LangiumDocumentFactory#fromModel, put it into the LangiumDocuments and let the ScopeComputation and ScopeProvider do the rest by itself (i.e. creation of AstNodeDescriptions etc.).

It's rather clear that all three of these approaches are essentially the same:

1. Create an in-memory document
2. Populate it (either with an in-memory model, or some generated text)
3. Put it through the document lifecycle, where it is handled like every other document.

I wouldn't recommend (4), you basically lose 80% of Langium's features that way.

[haukepribnow]

Thanks a lot. You convinced me that I will select option 1. 👍