Scoping, typings, references, autocompletion structure guidance

[kristiannotari]

Hello everyone,

I've already received a lot of help for my DSL regarding builtin library here.

I'm new to both DSL creation and Langium and I'd like to better understand the document lifecycle, and how I should deal with scoping, typing, references and such. I've read the document lifecycle guide but I'm missing how to properly structure my code given my DSL constraints.

I'd like not to bother you all by creating a discussion uniquely to guide me creating my DSL, but I wanted to get a clearer idea on how to structure things in Langium.

Given this DSL example file (limited example, pretty straightforward to read):

function f (arg1: number, arg2: string)

entity A {
    a1 = f(42, "hello") 
}

entity B {
   b1 = f(42, "hello from b")
}

entity A {
   a2 = f(43, "world")
}

compare a1, b1

and given these requirements:

1. Each file is isolated with its own definitions, so no references to other files
2. I want to keep track of function definitions (file scoped, no nesting, accessible to every point in the file)
3. I want to track entity definitions which can be repeated later in the same file (and refer to the same entity) and the named values (a1, a2, etc) inside them (each entity can refer to its own named values only known at a given point in the file, so for example the second entity block of A can access the a1 named value defined earlier)
4. compare can access every named value from every entity, ignoring entity scoping
5. I want to create a type system for function definitions so that their application can be checked

Solutions/structures/questions I see/have:

1. global export for each file is empty (so no cross document referencing can happen) --> ScopeComputation.computeExports returns an empty array
2. functions definitions should be put in a "file global scope" so: ScopeComputation.computeLocalScopes should create a PrecomputedScopes with a bunch of function definitions ASTDescriptions with the "AST root node" as the containing node and then in the ScopeProvider.getScope how can I get those functions definitions? In the DefaultScopeProvider there's a "global scope" created with the indexManager.allElements method, but I can't understand how's that working since it's filtering with the ReferenceType???
3. entity definitions in the ScopeComputation.computeLocalScopes should be scoped under each entity definition block node, adding all the named values inside them. Doing so I create N scopes where N is the amount of entity definition blocks I encounter, even if repeated for the same entity. How can I then implement in the ScopeProvider the ability to access every named value know UP to the point I am? (example: in the first entity A definition block I should not see a2 since it's defined later)
4. in the ScopeProvider, when retrieving the scope for a compare node I should get every named value (and function) basically. Should I retrieve the "file global scope" (see point 2) and add to that each precomputed scope from each entity definition block in the entire document/file, going upwards in the ast node from where I am?
5. This is almost unknown territory, since I've read there's no "default" implementation for a type system and one has to come with one himself. I can follow along what's has been done so far (looking at the lox example or similar) but what's worrying me is where to put "validation/checks" since I need some precomputed stuff (function definitions typings for example) in advance to actually compute if a function application is well-typed. Is the "validator" the right place to add these checks?

I'd like to know if my understanding is correct or if I'm mistaken somewhere. Thanks in advance!

EDIT: bonus question: is there some way to visually grasp what's being built by the various "passes" from text to scopes and stuff? Like some diagrams populating based on cursor position in a DSL file or something like that? Cause that would help greatly to see what's being built behind the scene. Or is there another way to properly debug these "internals" without having to rely on editor hints to know what's available in scope?

[msujew]

Hey @kristiannotari,

1. Yep, that seems reasonable

how can I get those functions definitions?

You can always do it like the linker does.

but I can't understand how's that working since it's filtering with the ReferenceType???

Since every cross reference has a special cross reference type, we don't want to have all objects in scope which are visible in the index, but only those which fit the cross reference type. That's why we apply a filtering there.

How can I then implement in the ScopeProvider the ability to access every named value know UP to the point I am?

This is kind of a weird requirement, since normal language don't work that way. If a nested element is visible to a certain scope (i.e. in your case the root scope which contains the compare statement), it's also visible to all elements within that scope. I'm sure you could filter these elements in the getScope method, but it seems semantically unsound to design a language that way.

Should I retrieve the "file global scope" (see point 2) and add to that each precomputed scope from each entity definition block in the entire document/file, going upwards in the ast node from where I am?

Yes, that's what I would have recommended 👍

in advance to actually compute if a function application is well-typed. Is the "validator" the right place to add these checks?

In advance compared to what? A code generator? Yes, the validator is usually always the right place to add these checks when you have a language which could contain semantic errors. There's no other intended way to make users aware of semantic errors in their documents other than the validator infrastructure. The Lox language implementation does the same.

Bonus Question:

Not really, or at least nothing that Langium provides as a builtin feature. To debug the scoping at a certain position in the document, I usually just run the code completion via Ctrl+Space. For everything else, live debugging is a bit more complicated. I recommend building a test suite for your language, so you can test parts of your language in isolation.

[kristiannotari]

Not sure I've understood how following qualified names and member call scoping techniques could help me with the multiple entity definition in a single block but I think I managed to get it working by simply computing scopes for each entity and then when I use the entity somewhere in the file it gets a nice clickable hover "go to definition"-like behavior which tells me everything's working correctly I guess, since the reference is resolved comparing the name I give in the description and not the name retrieved by the name provider for the scoped node (solving the "multiple names in one block" problem).

Now I have another problem which is: since I can "define" my entities twice as in typescript interfaces, I occasionally see two hints for the same entity name in the editor, since I create N descriptions for the same name basically. What's the best solution to deal with this? Attach an additional data structure to the document and "save" what I've already defined there so that I can double check if I'm adding another description for the same thing and avoid doing so?

Right now I'm wondering that maybe I did not understand what the ScopeProvider does. The getScope method only works on cross references if I'm not mistaken, so if I don't setup cross references correctly in my grammar I'm basically opting out from using the ScopeProvider to help me building the language tools. As long as I'm dealing with entities and function declarations (which happen to be defined globally at the root level only) I have no problem adding cross references for them to the grammar, since it's simple to add a [Entity:ID] or [Function:ID] on some grammar rule. But when it comes to variables/members/named values definition inside entities, it becomes a little bit trickier, since I can access them in various ways:

• directly (eg. a)
• fully qualified (eg. A.a)
• nested properties accessing for some kind of values (eg. a.propertyAA.propertyAAA, like objects in typescript)

And I can define them in various ways:

• simple way (eg. a = 42)
• destructuring (eg. [a, ...] = [1,2,3])
• renamed destructuring (eg. { a3: a1 } = { a1: 42, a2: "hello" } see this as destructuring + renaming)

And I have a "ValueRef" rule which is the following (I know it's not beautiful):

ValueRef:
    ref=ID ('.' access+=ID)*;

And I don't see how can I define a NamedElement to use as [NamedElement:ID] as you're advising in the class member scoping guide for example.

[kristiannotari]

On top of that, I have some other blocks in my DSL which can implicitly define new entities if you put a name which is not used for already defined entities but I also want to have a cross reference there. The problem is that I can't have a cross reference mixed with a normal "ID" usage as in:

[msujew]

Not sure I've understood how following qualified names and member call scoping techniques could help me with the multiple entity definition

I probably misunderstood your problem.

which is: since I can "define" my entities twice as in typescript interfaces, I occasionally see two hints for the same entity name in the editor, since I create N descriptions for the same name basically. What's the best solution to deal with this?

Probably by overriding the CompletionProvider and filtering the results to prevent duplicates.

The getScope method only works on cross references if I'm not mistaken, so if I don't setup cross references correctly in my grammar I'm basically opting out from using the ScopeProvider to help me building the language tools.

Yep, that's correct. The cross reference grammar syntax works closely together with the scope system to provide the whole cross referencing feature. Without it, Langium isn't able to know whether your ID is supposed to be just a plain id or a reference to another element.

And I don't see how can I define a NamedElement to use as [NamedElement:ID] as you're advising in the class member scoping guide for example.

What is your issue here exactly? You just need to put all possible elements which can be referenced in a union type and use that in your cross reference.

On top of that, I have some other blocks in my DSL which can implicitly define new entities if you put a name which is not used for already defined entities but I also want to have a cross reference there. The problem is that I can't have a cross reference mixed with a normal "ID" usage as in

Yeah, that won't work. Although it might seem reasonable semantically, this doesn't work syntactically, since the parser can't identify whether you mean the ID that's supposed to reference a Principal element or the ID that's just a plain id.

One way you could accomplish this feature is by creating AstNodeDescription objects for those inferred elements, so that the scope can still resolve to a description, even though there's no real underlying element for that.

Alternatively, you can return some special scope object for those inferred elements, something like ANY_SCOPE, which gets special handling in the Linker implementation. Instead of creating a referencing error, it will just create an undefined ref value. That effectively simulates the any behavior in TypeScript.

[kristiannotari]

Probably by overriding the CompletionProvider and filtering the results to prevent duplicates.

Thanks!

What is your issue here exactly? You just need to put all possible elements which can be referenced in a union type and use that in your cross reference.

The problem being since I can have multiple types of access (direct, qualified, object nested properties, and so on), should I use the cross reference only for the "first" bit of the "potentially dot separated list of references for access" (eg. a.b.c) so that the first occurrence is something I actually have in scope? For example (I only have 1 level deep entity accessing values for fully qualified names, ever):

Reference: EntityDef | ValueDef; // yes I can reference entities

ValueRef:
    ref=[Reference:ID] ('.' access+=ID)*; // here I need [Reference:ID ('.' ID)?] really

From what I've understood I'd then supply the direct and qualified ways of accessing using a solution where I compute two descriptions for the same node, one with the brief name and one with the fully qualified name. For resolving object properties access I should rely on some autocompletion feature given a type system instead. Correct?

Yeah, that won't work. Although it might seem reasonable semantically, this doesn't work syntactically, since the parser can't identify whether you mean the ID that's supposed to reference a Principal element or the ID that's just a plain id.

So for this bit no ScopeProvider special implementation is needed I guess. Is the ScopeProvider useful for something else other than cross-reference resolution? Everything else I need for autocompletion stuff is based on specific work on validation/autocompletion IDE parts I guess. Correct?

One way you could accomplish this feature is by creating AstNodeDescription objects for those inferred elements, so that the scope can still resolve to a description, even though there's no real underlying element for that.

Yeah I'm already doing so. Basically, when I compute scopes, I treat both the definition blocks and my other blocks where an ID can define a new entity as nodes from where I create description for such IDs (names).

[msujew]

so that the first occurrence is something I actually have in scope?

At the bottom of the member scoping guide there is a call graph which shows how the resolution is supposed to work. The first element is always an element of the local scope, while every nested call is an element of the previous call. That's why we transform the AST using the {MemberCall.previous+=current} action. To make it easier on the scoping algorithm to figure out what came before.

From what I've understood I'd then supply the direct and qualified ways of accessing using a solution where I compute two descriptions for the same node, one with the brief name and one with the fully qualified name. For resolving object properties access I should rely on some autocompletion feature given a type system instead. Correct?

Note that qualified name access and nested object property access are somewhat in conflict. Looking at an expression like a.b.c is ambiguous from a parser perspective, as both are just IDs separated by .. The member call mechanism is generally better suited for a language like yours. The qualified name stuff is usually less powerful, and I wouldn't recommend using it in your case. So yes, combining member calls + a type system is the way to go.

Is the ScopeProvider useful for something else other than cross-reference resolution?

Well, only in completion when the next (or current) element at the cursor position is a cross reference. Otherwise, yes, it's mainly used for the cross reference resolution mechanism.

[snarkipus]

Hello @kristiannotari ,

Here's some things I found helpful - I'll defer to the Langium team for deep technical advice, but this is what I use to try and understand what's going on under the hood.

DISCLAIMER: IMHO The best resource by far is the Langium implementation itself and I treat it as the 'reference implementation' for Langium. Anything provided below is probably 'almost correct'ish but occasionally not'.

I'd like not to bother you all by creating a discussion uniquely to guide me creating my DSL, but I wanted to get a clearer idea on how to structure things in Langium.

I have found the Langium team to be incredibly supportive and helpful. There's certainly a precedent for creating a dedicated thread for your DSL, so I wouldn't hesitate to do so (my thread, older thread). I just try to return the favor by contributing to the project if and where I'm able.

One thing I found incredibly helpful for non-specific DSL design questions was the following book (based on Xtext):

Implementing Domain-Specific Languages with Xtext and Xtend - Second Edition (L. Bettini 2016).

• Xtext Source Code (Book)

Note: I attempted to follow along with the book using Langium and had relatively decent success - unfortunately, I've been too busy to finish the scoping portion. However, you're free to take a look as there's a pretty large amount of validation examples one would typically need.

• Example DSL Repo

is there some way to visually grasp what's being built by the various "passes" from text to scopes and stuff? Like some diagrams populating based on cursor position in a DSL file or something like that? Cause that would help greatly to see what's being built behind the scene. Or is there another way to properly debug these "internals" without having to rely on editor hints to know what's available in scope?

I'm a picture person and not a SW dev by trade, so I drew this:

• Document Life Cycle

• For inspecting various data structures during different phases of the document build process, I added a test file like this.

[kristiannotari]

Thanks for the advices! I have found them extremely supportive and fast at it too! Great job btw.

I'm not able to open your Document Life Cycle excalidraw link, even after registration, it keeps telling me to create a paid workspace (sorry, new to excalidraw, I'm probably doing something wrong on my side).

For your test case, the important bit is the console.log with some of the computed stuff for your document at line 149, right? I'll give it a try to actually see how data structures are evolving and have quick feedback for quick changes and exploration stuff while trying langium. Thanks!

[snarkipus]

I'm not able to open your Document Life Cycle excalidraw link, even after registration, it keeps telling me to create a paid workspace (sorry, new to excalidraw, I'm probably doing something wrong on my side).

How strange, I'll just embed it:

For your test case, the important bit is the console.log with some of the computed stuff for your document at line 149, right?

That particular console.log() allows you to see the parsed document. What was more helpful for me was being able to peek at some of the various internal structures/services at various phases of the document lifecycle.

[kristiannotari]

Thanks a lot for your diagram, I'll try to associate the various steps with what I should do in my own DSL.

As of now I wanted to see the result of scope computation (for example) in a graphical way and I'm actually using your console.log (more or less) as a cli to debug files to see what I'm building with langium. Thanks!

[snarkipus]

As of now I wanted to see the result of scope computation (for example) in a graphical way and I'm actually using your console.log (more or else) as a cli to debug files to see what I'm building with langium.

I'm sure there's a more clever way of doing it, but that's the approach I landed on. Same challenge - working through scope computation.

[kristiannotari]

Yeah... you can see up here I'm a little bit lost to be honest. Trying to picture what each step does to ease my work on building up validation for the language basically. I mean, I think I know how to write the scope computation, and I think to currently have a semi-working partial implementation of it, but I'm not sure what I should do with that.