match unlimited unordered group

[Yokozuna59]

If I have the following information: age, name, and address (Don't worry about the parse rule itself). Which occurs at a specific location in the language, let's say at the top of the file. For example:

age ...
name ...
address ...

There is no specific order required. age can occurs before and after name and address and so on.

Each one of those information can occur more than once, but only the last value will be recorded. For example, here age value will be 30:

age 20
age 30

But how can I parse unordered info that could occurs more than once unlimited times?

For example:

name ...
age ...
address ...
age ...

---

I've created a fragment that can parse repeated information of one type. For example, age only or name only.

fragment INFORMATIONS:
    age=AGE+ & name=NAME+ & address=ADDRESS+
;

Can parse:

age 20
age 30
age 40

[msujew]

The whole idea behind unordered groups is that they only parse an element once, never twice. Using unordered groups for this doesn't really make sense. You should just use alternatives instead:

fragment INFORMATIONS:
    (age=AGE | name=NAME | address=ADDRESS)*
;

[Yokozuna59]

The example in the chevrotain documentation wasn't complex, which is why I thought of removing the offset argument. Then it appears that I need to make major changes to those matching functions according to the new grammar.

Thank you so much for your help!

[Yokozuna59]

One more thing: how can I create a custom hidden rule? Since matching functions should return tokens, hidden rules should remove stuff from the entire text.

[msujew]

You should usually just use normal hidden terminals for that. They are just normal Chevrotain token types with the group property set to hidden. Langium has no concept of hidden rules. If you don't need the result of a rule, you can just assign it to a variable you don't need.

[Yokozuna59]

Continuing from this conversation, I modified the matching function to be:

const titleRegex = /title(?:[\t ]+([^\n]+)?|$)/y;
export const matchTitle: CustomPatternMatcherFunc = (
    text: string,
    startOffset: number,
) => {
    titleRegex.lastIndex = startOffset;
    let match = titleRegex.exec(text);
    if (match !== null && match[1] !== undefined) {
        match.payload = match[1].trim();
    } else {
        match = null;
    }
    return match;
};

But it's appear that payload has no effect on the output, from chevrotain documentation:

Sometimes we want to collect additional properties on an IToken object, for example:

• Save RegExp capturing groups on the token object.
• Subsets of the matched text, e.g: strip away the quotes from string literals.
• ...

This can be done by attaching a payload property to our custom token matcher returned value...

So assuming that title sample as passed, the parseResult would has title sample as it is with out the modification, instead of sample only.

[Yokozuna59]

It appears that the problem is on Langium's side. I tracked the stack and found the line that caused the unexpected behavior:

https://github.com/langium/langium/blob/9dcd2416da265f488f84b369b0c3b8f134513a11/packages/langium/src/parser/langium-parser.ts#L195

There is no validation to determine if token.payload exists or not and use it if it does instead of token.image. And it's appear the whole method doesn't validate.

I have created a PR here, #1067, that resolves my issue. Although I didn't resolve that issue where the payload might be an object rather than just a string since I'm not sure what the best approach is..