Grammar for Template Strings

[JanikNex]

Hey guys,
I'm currently trying to implement template strings in a Langium language.

Basically, the syntax should be similar to template strings in JavaScript, although I want to use different quotes and don't need tag functions. Within the embedded expressions, it should then be possible to use other expressions from my grammar.

So the following strings should be possible:

''

'string text'

'string text line 1
 string text line 2'

'string text ${expression} string text'

'string text ${expression} string text ${another expression} string text'

My first attempt at a corresponding grammar looked like this, but then I failed because of the terminal symbols...

grammar HelloWorld

entry Model:
    (stmt+=InfoStatement)*;
    
InfoStatement:
    'info' elm=TemplatedString;

TemplatedString:
    "'" (elements+=TemplatedStringElement)* "'";

TemplatedStringElement:
    var=TemplateExpression | str=TEMPLATED_STRING_BODY;

TemplateExpression:
    '$' '{' expr=TestObj '}';

/// Placeholder for rule of the actual language
TestObj:
    name=DOUBLE;

hidden terminal WS: /\s+/;
terminal DOUBLE returns number: /-?\d+\.\d+/;
terminal NUM returns number: /-?\d+/;
terminal STRING returns string: /"[^"]*"/;
terminal TEMPLATED_STRING_BODY returns string: ???;
hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /[^:]\/\/[^\n\r]*/;

Could someone help me how I can fix/define a grammar for template strings?

[montymxb]

Hi @JanikNex . You'll probably want to look into using multi-mode lexing to handle this one, since template strings are not regular. There's a test we put together some time ago that verifies multi-mode lexing behavior, but it also serves as a good example of how you can push/pop state to change what tokens are acceptable.

In your case it would probably make sense to push/pop based on the ${ and } tokens, and then handle the expression in-between. Back to the point before, if you do nest template strings, this will work as expected, since you'll just have additional states on your stack to track how deeply nested you are.

[msujew]

In addition to what @montymxb said, we have an unfinished guide on multi-mode lexing that uses template strings as an example. See here for a preview.

[JanikNex]

we have an unfinished guide on multi-mode lexing that uses template strings as an example. See here for a preview.

This guide was exactly what I needed to get it working, thank you!