How to correctly handle imported rules in overriden files

[Yokozuna59]

Hi there, thanks for your work!

We have been discussing for a while in both mermaid-js/mermaid#4799 and mermaid-js/mermaid#4910 how we should handle imported rules for languages that have overriden rules.

So to summarize the discussions, we have a language that imports some common rules from a file, and this language has a greedy overriden rule:

• The rule in the grammar:

  https://github.com/Yokozuna59/mermaid/blob/bf95a305bc7c8257a13b174368f9952f4513876f/packages/parser/src/language/sankey/sankey.langium#L23

• The rule overriden regex:

  https://github.com/Yokozuna59/mermaid/blob/bf95a305bc7c8257a13b174368f9952f4513876f/packages/parser/src/language/sankey/matcher.ts#L9

The issue we're facing is that imported rules are sorted at the end of the rules list, and this greedy rule will tokenize everything without reaching the imported rules:

[
// language rules
  {
	name: 'sankey-beta', // tokenized
  },
  // ...,
  {
    name: 'SANKEY_LINK_NODE', // tokenized (the greedy rule)
  },
// imported rules
  {
    name: 'SINGLE_LINE_COMMENT',  // never tokenized
  }
  // ...,
]

How can we handle it effectively?

We have three solutions in mind:

1. Unimport the common rules and place them in the grammar itself:

   https://github.com/mermaid-js/mermaid/blob/c364715fc782a06b013c6651dbc632da9059e812/packages/parser/src/language/sankey/sankey.langium#L3-L23

   But this approach requires us to modify more than one place.

2. Rearrange the rules array using a rule as a pivot, which in this case it's TitleAndAccessibilities, see: feat: add pie langium parser mermaid-js/mermaid#4751 (comment).

   But this approach requires us to modify the pivot rule once we add another rule before the pivot one:

   https://github.com/Yokozuna59/mermaid/blob/bf95a305bc7c8257a13b174368f9952f4513876f/packages/parser/src/language/common/common.langium#L7

3. Create a custom function that lookahead and make sure that all rules inside the common grammar aren't matched:

   https://github.com/Yokozuna59/mermaid/blob/add-sankey-langium-parser/packages/parser/src/language/sankey/matcher.ts#L42-L53

   But this approach requires us to update this rule once a new rule added to the common grammar.

---

What would you guys recommend? Is there another, better approach?

[msujew]

Hey @Yokozuna59,

I would recommend a modified approach of no. 2. You could annotate a terminal rule with something like:

/**
 * @greedy
 */

And ask the CommentProvider about the annotated content of your grammar. In your token builder, you can then move the terminal to the back of the array, when it has the @greedy annotation.

[Yokozuna59]

Hey @msujew, thanks for your fast answer.

I'll discuss this approach with the team and see their thoughts about it.

[Yokozuna59]

Hey @msujew, Sorry to disturb, but how can we ask the CommentProvider about the annotated content in the grammar?

After trying to implement this approach, it appears that the CommentProvider#getComment requires AstNode as an argument, which indicates that we need to parse the document to get the AstNode, but the main idea is to sort rules before parsing.

[msujew]

You can just use the Grammar service of your respective language. Each TerminalRule is also an AstNode.