hidden terminal question

[snarkipus]

This is probably easy for somebody savvy, but I'm stuck. Freely admit that this is completely new to me, so I appreciate any help provided (even if it's "go read this").

I'm just trying to create a terminal for a comment string that begins with '$' followed by at least one space using the HelloWorld example grammar.

• Langium Version: 0.4.0

I've tested the regex 100x times and it seems to work: \$ [^\n\r]* test

hello-world.langium:

grammar HelloWorld

entry Model:
    (persons+=Person | greetings+=Greeting)*;

Person:
    'person' name=ID;

Greeting:
    'Hello' person=[Person:ID] '!';

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal INT returns number: /[0-9]+/;
terminal STRING: /"[^"]*"|'[^']*'/;

hidden terminal SL_COMMENT: /\$ [^\n\r]*/;
// hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
// hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

test.hello:

person Bob 
Hello Bob!

person Alice 
Hello Alice! 

$ failed comment

Lint and Errors:

• unexpected character: ->$<- at offset: 53, skipped 1 characters.
• Expecting token of type 'EOF' but found failed.

[msujew]

Hi @snarkipus,

I can't seem to reproduce this issue. The terminal parses successfully for me (I modified our arithmetrics example for it):

Did you rerun the langium generate command after modifying the grammar?

[snarkipus]

Ah, that was exactly it. Thank you very much.

Just to recap, the workflow for initializing, editing, and regenerating is as follows (in this case, for the arithmetic grammar):

• npm install from the langium root directory
• Start Run Arithmetics Extension (langium) to open client VSCode instance
• Start Attach to Language Server (langium) to attach the debugger
• Verify that the example/example.calc file works as expected
• while changing arithmetics grammar:
  • Modify the arithmetics.langium file
  • Regenerate TypeScript files via npm run langium:generate from the example root directory
  • Compile TypeScript via npm run build from the example root directory
  • Restart the debugger and reattach the extension

Couple of novice questions (although not particularly germane to the original question):

1. I see a watch script in the root package.json and a langium:watch script in the workspace package.json. Is there a way to 'automagically' regenerate the TypeScript and then compile it when making updates to the example.langium grammar file?
2. What does the Attach to Language Server step do and what should I be focussing on while debugging? Admittedly a broad question, so even just a reference/source to study would be helpful.

Thanks again - I apologize in advance as I'm absolutely not a TypeScript (or JS for that matter) developer, nor am I a language designer.

[msujew]

I see a watch script in the root package.json and a langium:watch script in the workspace package.json.

Right, you can call the langium generate command using the -w/--watch argument, which will then recompile your grammar changes as soon as the file changes on disk. Running tsc -w (the TypeScript compiler in watch mode) additionally in another shell (or combined in one using something like concurrently should simplify your dev workflow.

What does the Attach to Language Server step do and what should I be focussing on while debugging?

When starting the generated vscode extension, vscode will only try to debug the extension itself. However, the language server (the provider for the language smartness in vscode) is running in a separate process from the extension. Fortunately, we can easily expose a debug instruction port (6009 by default) which we then attach to using the Attach to Language Server launch config. It enables us to debug our actual language server.

You'll usually debug in two scenarios: You've modified some service implementation (such as validation or scoping), but it isn't working as expected. In this case you'll focus on your own code, setting breakpoints in there after calling the Attach to Language Server config. On the other hand, some default behavior of Langium might seem weird to you and you want to debug into library code. In this case you'll need to set a breakpoint in the node_modules/langium/src directory and also add another outFiles entry to your launch config. See #572 for more info.

I'm absolutely not a TypeScript (or JS for that matter) developer, nor am I a language designer.

Now I'm curious :) Most users of Langium are either TypeScript devs who want to try themselves at creating a language or language designers who want to leverage a more modern stack, compared to existing language framework solutions. What is your usecase?

[snarkipus]

Thanks for the detailed response!

I sort of figured I'm an enduser outlier based on the types of rabbit holes that Google has led me down ...

I recently transitioned roles and one of the analysis suites I need to use is an incredibly mature (and powerful) legacy FORTRAN application. While some may scoff, there's actually a lot of precedent for preserving some of these legacy tools and this is one of those cases. Regardless, the current developer tooling to support this is limited to notepad++, vim, or Emacs if you're fancy.

Since its inception, this application relies on a strongly typed, rich scripting syntax to describe analysis setup, options, and execution across various input files. However, the only attempt at tooling to support this was an Emacs plugin written >15 years ago that provides basic syntax highlighting. While studying the documentation it occurred to me that I was really working with an external domain-specific language (I might be mashing up buzzwords here). At that point, having recently used VSCode to learn rust for funsies, I was genuinely curious about what modern tooling for an existing DSL would or should look like. And then the rabbit hole opened up ...

Given I have reams of documentation ( ad nauseam) describing the syntax, types, and format/structure of the input files I thought it might be reasonable to at least make a cursory attempt at documenting the grammar. Initially, I thought just basic syntax highlighting would be handy, but then I realized that if you could just successfully document the grammar it exposes a metric ton of functionality with only minor additional effort. This led to ANTLR, then to Xtext, and finally to Langium which is incredibly appealing since the entire effort is native to VSCode itself.

So, hopefully this is a reasonable (albeit non-typical) use case. If not, please don't hesitate to let me know this isn't the rabbit hole I'm looking for. If so, please don't hesitate to point me in the direction of anything that would be helpful or informative for documenting the grammar of an existing DSL. That would spare everyone the pain of endless questions on regex patterns or questions like "How do I capture the fact that the first line has to be 'EXECUTE' (nothing else) followed by 'INSTRUCTIONS-FOR:' (nothing else) on the second line?"

Note: that's a real question btw :)

[msujew]

Thanks for the elaborate explanation. This actually sounds like a really good use case for Langium :D

external domain-specific language (I might be mashing up buzzwords here)

Yeah, it definitely sounds like one 👍

How do I capture the fact that the first line has to be 'EXECUTE' (nothing else) followed by 'INSTRUCTIONS-FOR:' (nothing else) on the second line?

Since we use JS regexp syntax in Langium, that could probably simply be represented by terminal ExecuteInstruction: /EXECUTE\nINSTRUCTIONS-FOR:/;. Perhaps some \s+ (multiple whitespaces) in between. Then you would start your actual parser rule like Execute: ExecuteInstruction ...;. Dealing with whitespace/newline sensitive languages is always a somewhat painful experience.

[snarkipus]

Thanks @msujew!

I'm glad to hear that my intuition was correct. I see hundreds of terminal definitions in my future I think.