New Indentation functionality not working correctly

[mnaderhirn]

I am trying out the new identation-aware function.
this is my grammar:

grammar IndentGrammar

entry IntendTest:
    '{{{' NEWLINE
    test=Indent_part?
    '}}}';

Indent_part:
    INDENT
    (test+='text' NEWLINE)*
    Indent_part?
    DEDENT;

terminal NEWLINE: /[\r?\n]+/;
terminal DEDENT: 'synthetic:indent';
terminal INDENT: 'synthetic:dedent';
terminal SPACES: /[ \t]+/; // whitespace and newlines now have to be separated

when i enter this:

{{{
    text
        text
    text
}}}

I get the following errors:

[{
	"resource": "/home/easysafe/test/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "Expecting token of type 'DEDENT' but found `text`.",
	"source": "easysafe",
	"startLineNumber": 4,
	"startColumn": 5,
	"endLineNumber": 4,
	"endColumn": 9
},{
	"resource": "/home/easysafe/test/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "Expecting token of type '}}}' but found `text`.",
	"source": "easysafe",
	"startLineNumber": 4,
	"startColumn": 5,
	"endLineNumber": 4,
	"endColumn": 9
},{
	"resource": "/home/easysafe/test/test.easy",
	"owner": "_generated_diagnostic_collection_name_#0",
	"severity": 8,
	"message": "Expecting end of file but found `text`.",
	"source": "easysafe",
	"startLineNumber": 4,
	"startColumn": 5,
	"endLineNumber": 4,
	"endColumn": 9
}]

This are the tokens i get

expect(tokenNames).toEqual([
            '{{{',
            '\n',
            '\t',
            'text',
            '\n',
            '\t\t',
            'text',
            '\n',
            '\t',
            'text',
            '\n',
            'DEDENT',
            '}}}'
        ]);

shouldn't there be an 'DEDENT' after the '\n' after the send 'text' token?

[msujew]

Hey @mnaderhirn,

the lexer is working correctly, and the test you're using via expect(tokenNames) is not correct - you're actually looking at the IMAGE (i.e. the lexed content) and not the names of the token (indentified via tokenType.name).

The parser error you're seeing is due to a grammar issue. Your grammar doesn't actually allow to parse more text after dedenting once. After leaving any Indent_part rule, no more text tokens can be consumed (which is exactly what the parser errors are trying to tell you). See the following fixed grammar:

grammar IndentGrammar

entry IntendTest:
    '{{{' NEWLINE
    test=Indent_part?
    '}}}';

Indent_part:
    INDENT
    (test+='text' NEWLINE | Indent_part)* // <-- Allows to parse more `text` tokens after leaving the `Indent_part` rule
    DEDENT;

terminal NEWLINE: /[\r?\n]+/;
terminal DEDENT: 'synthetic:indent';
terminal INDENT: 'synthetic:dedent';
hidden terminal WS: /[ \t]+/;

[blewis13]

Should one expect these examples to work in the Langium Playground? They don't appear to be working in the current playground as of 8/19/2024

[msujew]

@blewis13 The feature hasn't been released yet and will be included in 3.2. See also our milestone.

[blewis13]

These examples use this style of terminal declaration:

terminal INDENT: 'synthetic:indent';
terminal DEDENT: 'synthetic:dedent';

Whereas the code comments indicate this style in the indentation-aware.ts src/ file:

terminal INDENT: ':synthetic-indent:';
terminal DEDENT: ':synthetic-dedent:';

Is there a material difference? The playground chokes on the first syntax and at least doesn't lock up loading... on the second syntax

[msujew]

Note that the playground will not feature support for the indentation aware tokenization, since it isn't something that can be activated via the grammar.