How to implement a line breaks and indentations aware language? Any example?

[jindong-zhannng]

Hi team,

I am going to implement a DSL that recognizes statements via line breaks and indentations (similar to yaml or python), is there any example for reference? Thanks.

[msujew]

Hey @jindong-zhannng,

I don't think there's a public example for this available right now. Though you might want to take a look at how these languages work in ANTLR4 to see how one would implement such a thing in Langium. See also this example from Chevrotain to see how one implements indentation based lexing.

Generally, white space aware languages work well from a parser perspective, but break pretty spectacularly in the presence of parser errors due to the amount of tokens the parser needs to keep track of. Whenever we had to deal with a whitespace aware language, we tried to make it as non-whitespace aware as possible (not possible with some languages) and only add some whitespace awareness to special terminal tokens.

[jindong-zhannng]

Whenever we had to deal with a whitespace aware language, we tried to make it as non-whitespace aware as possible (not possible with some languages) and only add some whitespace awareness to special terminal tokens.

In my scenario whitespace awareness is only needed in few specific context. I am not sure is it possible that I can keep the language to be non-whitespace aware in most cases, but be whitespace aware in some special cases.

Here is my language design:

form ExampleForm {
  num numberFieldA                      // separate field definitions with **line break**
  num numberFieldB = numberFieldA * 2   // can add an optional expression (one line)

  // **The problem**: I also want to support multiple lines expression
  // without any special delimiter to make it as clean as possible.
  // So in current draft I plan to use indentation to scope the expression (similar to multiline strings in yaml).
  // The expression grammar will be extensible, so I can't scope it by analyzing semantics
  num numberFieldC =
    IF(
      numberFieldA > 0,
      numberFieldA,
      0
    )
}

Any suggestions for my goal?
My current idea is implement it with langium completely through a whitespace aware way, similar to implement yaml. But it may not be the easiest way.
Another idea is do automatic semicolon insertion in some step, just like javascript does. But I am not sure is it possible and how.

[msujew]

I'm a bit confused. Usually whitespace aware languages require line breaks/indent/whitespace to parse semantically correctly. I.e. in Python:

if a:
  print(b)
  print(c)

and

if a:
  print(b)
print(c)

generate different ASTs, since the last print(c) call can be either part of the if (1) or not (2). Your language doesn't seem to require that, since it has clear delimiters as far as I can tell. For example, from a parser perspective, num is a delimiter for a new variable, since it cannot appear as part of a previous statement, so it automatically indicates the start of a new variable declaration. No line break necessary to make that clear.

What I want to say is: Maybe your language doesn't need to be whitespace sensitive at all? Only a lack of delimiters in a language really necessitate whitespace aware parsers, everything else can be handled by formatters/linters.

[jindong-zhannng]

Sorry I left out some necessary context above:

Problem 1: For field type, it can be one of built-in types e.g. num, it's ok as a keyword for delimitation. However it can also be a custom type:

form Example {
  TypeA fieldNameA
  TypeAnything fieldNameB
}

type TypeA { ... }
type TypeAnything { ... }

It is also valid. In this case things may get a little more complicated. But I guess langium can also handle this perfectly by its cross reference feature. Whitespace awareness is still not necessary at this stage.

However, Problem 2: For expressions, their grammar will be purely unknown in my dsl's perspective. Their content won't be parsed by the parser but will be evaluated later in runtime.
That means the only thing the parser knows is there can be a set of characters (allows almost any character, except for a few reserved words based on convention) after = sign of the current statement and before next statement.

For example:

form Example {
  TypeA fieldNameA = something + TypeA(a) + TypeB(b)  // unfortunately, an identifier with same name as TypeA but a different meaning appears in the expression
  TypeA fieldNameB
}

type TypeA { ... }

So in this case there are not enough clear delimiters. It might be parsed like this:

form Example {
  TypeA fieldNameA = something + 
  TypeA(a) + TypeB(b)
  //   ^ error here
  TypeA fieldNameB
}

type TypeA { ... }

[msujew]

Ok, I see, that makes sense. Well, I guess you won't be able to get around using some whitespace sensitivity. Something like:

VariableDecl: type=TypeRef name=ID ('=' value=Expression)? NEWLINE+; // Expect one or more newlines after the variable declaration
terminal NEWLINE = /\r?\n/;

Note that this has the major drawback of explicitly using the NEWLINE everywhere you expect a newline character to appear, as it is now no longer hidden.

[jindong-zhannng]

Thanks for your confirmation.
How about indentations? Can they be implemented in langium language? I noticed the above example uses custom token pattern feature from Chevrotain.

[msujew]

Yes, since we use Chevrotain under the hood, the idea is to override the TokenBuilder service to change the pattern of the token to a custom one that handles indentation.