case-insensitive behavior in Langium?

[maxir269]

hier is simple grammar

grammar HelloWorld

entry Model:
    (persons+=Person | greetings+=Greeting)*;

Person:
    FUNC name=ID;

Greeting:
    'Hello' person=[Person:ID] '!';



terminal FUNC: /func/i;   // i want the func case-insensitive


hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;

hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

even when no space between keyword and ID it see the keyword Func

Funcdsd // that should be ID but it parser it as Func and the rest a ID.

How can I change the behavior?

[msujew]

Hey @maxir269,

our internal lexer engine requires us to explicitly specify whether one token is a possible longer alternative of another token. See:

langium/packages/langium/src/parser/token-builder.ts

Lines 151 to 159 in e9d0c27

	protected findLongerAlt(keyword: Keyword, terminalTokens: TokenType[]): TokenType[] {
	return terminalTokens.reduce((longerAlts: TokenType[], token) => {
	const pattern = token?.PATTERN as RegExp;
	if (pattern?.source && partialMatches('^' + pattern.source + '$', keyword.value)) {
	longerAlts.push(token);
	}
	return longerAlts;
	}, []);
	}

While this is easily possible to identify for keyword tokens, this is quite difficult to accomplish for arbitrary terminal/regex definitions. In order to fix this you can:

1. Either override the DefaultTokenBuilder service to ensure that the longer_alt property on the FUNC token type is correctly set.
2. Use the caseInsensitive feature of the langium-config file, but that makes all keywords case insensitive. See
   

   langium/examples/arithmetics/langium-config.json

   Line 7 in e9d0c27

   "caseInsensitive": true,

[maxir269]

Hallo @msujew

yes i want to make alle keywords in my grammar case-Insensitive.

when i use "caseInsensitive": true,

is that make the Identifer also caseInsensitive.

VAR
var

i want to both (VAR , var ) to refer to the same ID

how should i define a keywords in langium?

inside ' ' like in lok grammar or as terminal?

[msujew]

To enable case insensitive linking, adjust your scope provider:

langium/examples/arithmetics/src/language-server/arithmetics-scope-provider.ts

Lines 16 to 22 in e9d0c27

	protected override createScope(elements: Iterable<AstNodeDescription>, outerScope: Scope, options?: ScopeOptions): Scope {
	return new StreamScope(stream(elements), outerScope, { ...options, caseInsensitive: true });
	}
	protected override getGlobalScope(referenceType: string): Scope {
	return this.globalScopeCache.get(referenceType, () => new MapScope(this.indexManager.allElements(referenceType), undefined, { caseInsensitive: true }));
	}

how should i define a keywords in langium?

Ideally as a keyword '', and not as a terminal. Note that we have a whole documentation page for that over here.