Data rules are not context dependent

[svallory]

I read in the documentation that data rules are context-dependent but that does not seem to be the case.

Langium version: 3.5 (I also tested locally

Link to code example:

https://langium.org/playground/

Grammar

grammar SampleGram

/** A textual representation of a state machine */
entry Sample:
    (phrases+=Phrase | cmds+=Command)+;

terminal YES: /\byes\b/;
terminal NO: /\bno\b/;

terminal W: /[\w]+/;

hidden terminal WS: /[ \t\r\n]+/;

Word returns string: W;

Phrase returns string: 'phrase' Word+;

Command returns string: 'cmd' Word (YES|NO);

Content:

phrase Please answer yes
cmd answer yes

The current behavior

The parser seems to try all rules and in an order I cannot control

See image

The expected behavior

I would the parser behavior for the rule Command returns string: 'cmd' Word (YES|NO); to either match the entire rule at once them break it down into tokens, or to match and tag the tokens respecting the rule I created. If it matches a 'cmd' string, it then tries to match a single Word, then it tries to match either YES or NO and sets the appropriate token types at each step

[msujew]

This is working as intended, the terminals YES and NO do not use the token category Word and therefore cannot be consumed as such. You can adjust the TokenBuilder to fix this.

Where does the documentation say that data type rules parse context dependent text? That's definitely not correct. Langium can only parse context free grammars, with some exceptions, depending on your changes to the lexer.

[svallory]

This is working as intended, the terminals YES and NO do not use the token category Word and therefore cannot be consumed as such. You can adjust the TokenBuilder to fix this.

Where does the documentation say that data type rules parse context dependent text? That's definitely not correct. Langium can only parse context free grammars, with some exceptions, depending on your changes to the lexer.

Thank you for the quick response! Here's where the document states it: https://langium.org/docs/reference/grammar-language/#data-type-rules

If you don't mind, I have 2 questions:

1. Is there a way to rewrite the example grammar above to match a value, "yes" or "no" in the command value and not inside match them independently when parsing a phrase?

2. Where can I learn how to adjust the TokenBuilder to fix this?

[svallory]

Langium can only parse context free grammars, with some exceptions, depending on your changes to the lexer.

One more thing... What are the exceptions? Is it possible to, embed a grammar inside another? Or to parse different segments of a document using different grammars?

[msujew]

Thank you for the quick response! Here's where the document states it: https://langium.org/docs/reference/grammar-language/#data-type-rules

I see, it probably makes sense to change that, since this is not correct - the existence of a parser rule doesn't make the lexer suddenly create different kinds of tokens (this is impossible, since the parser cannot have this kind of feedback effect on the lexer).

Where can I learn how to adjust the TokenBuilder to fix this?
Or to parse different segments of a document using different grammars?

See this WIP guide on multi-mode lexing, which explains how to add state to lexers and also how to override the token builder in general.

[svallory]

Thanks, @msujew !

I'm also going to allow embedding another language I'm also creating inside code blocks.

I was checking the multi-language setup like explained here: https://eclipse-langium.github.io/langium-previews/pr-previews/pr-132/guides/multiple-languages/

Would It be possible to combine the grammars in different parts of the document instead of different files?

[msujew]

Would It be possible to combine the grammars in different parts of the document instead of different files?

I'm not sure I can follow. What is the difference between having multiple grammars or one big grammar for this purpose?

[svallory]

Would It be possible to combine the grammars in different parts of the document instead of different files?

I'm not sure I can follow. What is the difference between having multiple grammars or one big grammar for this purpose?

Think about how we can embed other languages inside markdown. Except, in my case, the outer scope would provide context for autocomplete of the language in the inner scope.

I'm creating two languages: Rcl and RclScript. They will be used to create RCS (the new SMS) agents and message flows. The first is similar to yaml, indentation sensitive and focused on defining data objects. The second allows writing logical conditions for the message flow.

They are really different and having them in separate grammar would make them more manageable

[msujew]

I've converted this into a discussion, since there's nothing actionable to do in Langium to fix this.

Langium does not support switching between grammars within a file. That would require being able to lex and parse parts of the file individually, which would be simply way to complicated to offer on a framework level. However, I still don't see why this would be necessary. You can essentially just have a rcl.langium file like this:

import './rcl-script';

...
RclScriptContainer: '```' script=RclScript '```';

Where RclScript is a rule defined in ./rcl-script.langium which acts as the entry point into your scripting language. You can then use the ``` token (or similar, you can use whatever you want) to act as a state switch between the token list for normal Rcl and RclScript. See the linked guide for more info.

[svallory]

I was trying to simplify and ended up hiding the point :/

Besides RclScript I would like to support embedding other languages like JavaScript, typescript etc. At least the ones that have public xtext grammars written for that I can convert. So I can't simply merge all grammars inside a single grammar.

Even between RCL and RclScript I believe I will already have some issues with token clashing. Although for these I think that guide you sent earlier may be the solution

[msujew]

Besides RclScript I would like to support embedding other languages like JavaScript, typescript etc. At least the ones that have public xtext grammars written for that I can convert. So I can't simply merge all grammars inside a single grammar.

I see. I would probably say that this is beyond Langiums ability as a framework. Perhaps volar might be more interesting for your use case?