Language Server implementation for the structurizr dsl

[torsten-systemticks]

Just in case someone is interested in a generic way of integrating the structurizr dsl into any editor/IDE that supports Language Server Protocol (short: LSP):

For my C4 DSL VS Code Extension I have meanwhile split the software stack into two major components:
• The pure VS Code Extension (i.e. the UI and client-specific part)
• A general C4 DSL language server, which is triggered from the VS Code extension and is doing the "real work" like code completion, cross-referencing, validation, text decorations, etc.

The C4 DSL language server fully supports the LSP. and can then be adopted by any editor/IDE that supports LSP on the client side. Here is the current complete list: https://microsoft.github.io/language-server-protocol/implementors/tools/.

I have also implemented a first version of an Eclipse plugin. Only very rudimentary, just to proof that it works.

[jillesmc]

@torsten-systemticks You have done a great job!

I saw that you are using a TextMate Grammar to describe the syntax.
Have you thought about using something like ANTLR before?

I just started a journey to learn how to use ANTLR to create a parser and then a Prettier formatter plugin or perhaps something like you did.
But I'm not sure if I'm going in a wrong way. 🤔

[torsten-systemticks]

Here are my thoughts regarding structurizr dsl and Xtext:

If you would start from scratch (i.e. the dsl is not there yet) I would highly recommend using any kind of language engineering workbench like Xtext, ANTLR or Langium.
The benefits are clear. The grammar is specified in textual rules (BNF-like rules) and nearly everything is derived from that and generated: The lexer, the parser, the data model, model persistency, etc.
If you change/extend the grammar all artifacts are re-generated again consistently. You don’t need to care about fixing the parser or the data model, since it is just there again.
Furthermore you usually have some good default implementations for editor features like content assist, validation, navigating, formatting, etc. And you can hook into all those features and replace it with your own implementations. And Xtext has a native support for LSP.

But, there are downsides as well. As always. 
Xtext hast a steep learning curve and documentation is rare. Xtext depends on EMF-based data models, i.e. you cannot just simply re-use the existing structurizr model Pojos.
And as mentioned before the existing DSL models around might be incompatible with an Xtext-based structurizr grammar.
And what is still a challenge is the import feature. With Xtext, it is like in Java, where an Import is refering to a fully self-contained valid other file. With structurizr dsl it is currently a text substitution…

[jillesmc]

@torsten-systemticks, thank you for sharing your knowledge with us. 🙏🏻

[dirkgroot]

I'm developing a Structurizr DSL plugin for IntelliJ, which uses a combination of JFlex and ANTLR-like BNF to create a lexer and parser. Syntax highlighting can be based on both the tokens from the lexer and the AST from the parser. I can confirm that dealing with the DSL's flexibility makes this a challenging undertaking (at least for me, with limited experience creating lexers/parsers). Even more so, because for editor support, the lexer and parser should be very fault tolerant and be able to gracefully handle incomplete and (partially) incorrect code.

Ideally, the lexer would be able to distinguish different types of tokens without needing any context. In reality, this is not possible, because for example softwareSystem could be a keyword, an identifier or an unquoted string, depending on where it is used.

If the language would be a little bit more strict, lexing/parsing it would be a lot easier. For example, if strings (names, descriptions) must always be quoted and keywords cannot be used as identifiers, a lexer could easily tokenize all keywords, without needing any context. Many editors provide syntax highlighting based on lexing the source code using simple string comparison and/or regexes. The way the DSL is specified right now, makes it impossible to correctly highlight everything using this method.

Take this simple piece of valid Structurizr DSL for example:

workspace {
    model {
        softwareSystem = softwareSystem softwareSystem
        system2 = softwareSystem "System 2"
    }
}

VIM highlights this as follows:

The VSCode plugin has the same issue:

This is how my IntelliJ plugin highlights this code:

This is only doable by either creating a pretty complicated lexer or by using the AST to annotate the code after it has been parsed (I've tried both, and both have their downsides).

Both from a tooling-support perspective, and from my own perspective as a software developer, I prefer a language to be stricter than the Structurizr DSL in it's current state. Ideally, IMHO, there should be one, and only one way to do something.

To be fair, I'm fully aware that this is my personal preference and YMMV, so @simonbrowndotje I'd love to see your thoughts on this.

[simonbrowndotje]

Languages such as PlantUML have the same characteristics; for example this is valid:

@startuml
component component
@enduml

There are likely thousands of people with existing investment in the DSL, so it's probably too late to make such major changes unfortunately. Of course there's nothing preventing you from forcing users to quote strings, as it will still be valid DSL.

[dirkgroot]

Thanks for your quick reply! Funny enough, the IntelliJ plugin for PlantUML highlights your example code incorrectly 😄:

Making my IntelliJ plugin stricter than the actual DSL is a tempting idea, I'm going to seriously consider doing this.

A possible strategy for such a major change in the DSL could be to introduce an optional "strict mode", which is disabled by default. Over the course of a longer period of time and multiple major versions, this option could become enabled by default, and ultimately non-strict mode could be removed. This strategy should give everyone enough time to migrate their models to the stricter version of the language.