Terminal rule shadowing another - longer_alt

[vasilache-radu]

Hi guys,

I spent some time on this problem I have, but I couldn't solve it myself, so I am asking the community.
Relevant for my question are the following terminal definitions:

terminal PRIORITY: 'ALARM'|'A'|'B'|'C'|'D'|'E'|'I'|'P';
terminal ID: '^'?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;

The set of words following the terminal rule PRIORITY is included in the set of words following the terminal rule ID, so if PRIORITY was after ID, I would get the error in my dsl code: Expected PRIORITY, but found ID.
Looking on the internet, on the Chevrotain website I found out that the solution is that PRIORITY should go before ID and I should set longer_alt of PRIORITY to ID, just like in the following example: https://github.com/Chevrotain/chevrotain/blob/master/examples/lexer/keywords_vs_identifiers/keywords_vs_identifiers.js

Unfortunately I can't find the file where to do this. I think on the website they use a manual parser, while Langium creates the Parser depending on the grammar one wrote. I don't know exactly how the whole parsing/ lexing works, but I tried to do a CustomTokenBuilder class, overriding some methods in the DefaultTokenBuilder, but the language server fails when starting it, the error being: "Errors detected in definition of Lexer: A MultiMode Lexer cannot be initialized with a longer_alt on token outside of mode ". I should mention that by commenting/deleting the line: "token.LONGER_ALT = super.buildTerminalToken(this.idRule as GrammarAST.TerminalRule); ", the server starts, but the problem is not solved.

import { DefaultTokenBuilder, Grammar, GrammarAST, Stream, TokenBuilderOptions, getAllReachableRules, isWhitespaceRegExp, stream } from "langium";
import {TokenType, Lexer, TokenVocabulary} from "chevrotain"
import { terminalRegex} from "langium/internal";
//import {isTerminalRule} from '../../node_modules/langium/src/grammar/generated/ast.js'
export class CustomTokenBuilder extends DefaultTokenBuilder{

terminalRules: Stream<GrammarAST.AbstractRule> | undefined;
idRule: GrammarAST.TerminalRule | undefined;

override buildTokens(grammar: Grammar, options?: TokenBuilderOptions): TokenVocabulary {
    const allReachableRules = getAllReachableRules(grammar, false);
    const reachableRules = stream(allReachableRules);

    for(let rule of allReachableRules){
        if(rule.$type == 'TerminalRule' && rule.name == 'ID'){
            this.idRule = rule;
            break;
        }
    }

    const terminalTokens: TokenType[] = this.buildTerminalTokens(reachableRules);
    const tokens: TokenType[] = this.buildKeywordTokens(reachableRules, terminalTokens, options);

    terminalTokens.forEach(terminalToken => {
        const pattern = terminalToken.PATTERN;
        if (typeof pattern === 'object' && pattern && 'test' in pattern && isWhitespaceRegExp(pattern)) {
            tokens.unshift(terminalToken);
        } else {
            tokens.push(terminalToken);
        }
    });
    return tokens;
}

protected override buildTerminalToken(terminal: GrammarAST.TerminalRule): TokenType {
    const regex = terminalRegex(terminal);
    const pattern = regex.flags.includes('u') ? this.regexPatternFunction(regex) : regex;
    const token: TokenType = {
        name: terminal.name,
        PATTERN: pattern,
        LINE_BREAKS: true,
    };
    if (terminal.hidden) {
        // Only skip tokens that are able to accept whitespace
        token.GROUP = isWhitespaceRegExp(regex) ? Lexer.SKIPPED : 'hidden';
    }
    if(token.name == 'PRIORITY'){        
        token.LONGER_ALT = super.buildTerminalToken(this.idRule as GrammarAST.TerminalRule);
        console.log(token.name + ":" + token.LONGER_ALT + " -> " + token.PATTERN);
    }
    return token;
}

}

[msujew]

Hey @vasilache-radu,

without going into the details of Chevrotain+Langium, I would recommend you to make PRIORITY a data type rule, that should fix the issue by itself:

Priority returns string: 'ALARM' | 'A' | 'B' | 'C' | 'D' | 'E' | 'I' | 'P';

[vasilache-radu]

Hey @msujew ,

Thank you a lot for the answer, it solved the problem. I never declared a data type in Langium, only terminals.
Could you explain how the lexer differentiates between the data types and the terminal rules? And what exactly is the difference between them? ID and PRIORITY are both strings at the end.

[msujew]

Basically, the data type rule keeps the contained keywords as they are (as strings), while each terminal is transformed into a regular expression. Determining the longer_alt property of a keyword is fairly simple, but doing so for a Regex is pretty non-trivial. That's why we want to keep everything as a keyword using the data type rule.