Extend or customize syntax highlighting

[EDKarlsson]

I only started using this yesterday so I may have not stumbled on it. Is there a way to customize or extend the generated syntax highlighting? I'd prefer to not have everything go under "keyword.control". I know I could add semantic highlighting. do a embedded language or injection. I tried injection but I'm new to using that feature of vscode extensions. My use case is having 2 different styles of comments.

Comment belongs to the type it is being referred to while a note should be ignored.
Note starts with //* and comment has only 1 /

terminal COMMENT: /\/\*[\s\S]*\*\//
hidden terminal NOTE: /\/\/*[\s\S]*?\*\//;

This results in anything within /* .... */ not having any highlighting, well it does have highlighting that follows the patterns of the language and but no specific rule in the "repository". I tried adding a return to the COMMENT terminal but didn't do anything. It would be nice to be able to define the scope of a rule within the grammar.

[msujew]

Hi @EDKarlsson, the generated textMate syntax highlighting file is basically just a starting help for new grammars. Anyone who wants more fine-grained control over their syntactical highlighting should deactivate the textMate option in their langium-config.json and write/modify their textMate file by hand. It's basically impossible to infer accurate syntactical highlighting just based on a grammar. That's also what we did for the Langium syntax highlighting.

Regarding your comments: I'm pretty sure that this style of commenting will lead to a lot of problems down the line. Your COMMENT terminal contains a greedy [\s\S]* regex (compare to the non-greedy [\s\S]*? regex), which basically parses/lexes as "read everything in the file until the end of time itself". Have you confirmed that this doesn't introduce any parsing errors? Additionally, it will definitely override the NOTE terminal, since they are ambiguous, so that one will never be lexed. I would recommend you to remove the COMMENT terminal. Just use the NOTE terminal and call the findCommentNode function to find comments for a CST node (you can use astNode.$cstNode to get the CST node of an AST node).

It would be nice to be able to define the scope of a rule within the grammar.

I think this is (*drum roll*) out of scope for Langium. Our grammar language is already complicated enough as it is, and syntax highlighting can be easily done manually, so I don't really see a reason to include this into the grammar language.

[AviCraimer]

Hi,

You mentioned disabling the textMate generation, but you didn't specify how to do this using langium-config.json settings. I cannot find documentation for those settings on the Langium website. At the Langium language link you sent it appears that you are not using langium:generate at all (at least there is no script for it in your package.json). Please let me know if there is a way to use langium:generate without generating the textMate grammar. Thank you.

[msujew]

Hey @AviCraimer, you just need to remove the textMate entry in the langium-config.json. Here an example without a textmate entry:

langium/packages/langium/langium-config.json

Lines 3 to 12 in d2eda28

	"languages": [{
	"id": "langium",
	"grammar": "src/grammar/langium-grammar.langium",
	"fileExtensions": [
	"langium"
	],
	"chevrotainParserConfig": {
	"maxLookahead": 3
	}
	}],

At the Langium language link you sent it appears that you are not using langium:generate at all

We do, it's just a different package that does the generate call:

langium/packages/langium/package.json

Line 62 in d2eda28

"langium:generate": "langium generate",

[AviCraimer]

Thank you!

[EDKarlsson]

That works as well. I couldn't find any documentation on deactivating the tmLanguage generation. Thanks for answering my question!

I did try it and it does not overwrite the NOTE terminal. Unfortunately I do not have control of the grammar. This is a fun side project implementing SysML v2. I got decently far with my first attempt using textX (https://github.com/textX/textX) but it doesn't have any concept of fragments and rules started becoming unwieldy just to avoid creating an element representation of a rule

[EDKarlsson]

Here is an example of it working. I deactivated the tmLanguage generation which is a perfectly acceptable solution. There isn't documentation about doing so being okay. Though I will probably have to generate it until I've finished with the entire language spec since it would be a hassle to constantly have to add each rule to the the syntax grammar.

[EDKarlsson]

Found an easy workaround to keep auto generated syntax highlighting and having a custom one by using https://www.npmjs.com/package/json-merger and adding a build step. I have Langium write to sysml.generated.tmlLanguage.json then use this template.tmLanguage.json and finally write it to sysml.tmLanguage.json

{
    "$merge": {
        "source": {
            "$import": "syntaxes/sysml.generated.tmLanguage.json"
        },
        "with": {
            "patterns": {
                    "$concat": [{
                         "include": "#sysml-comments"
                    }
                ],
            },
            "repository": {
                "sysml-comments": {
                    "patterns": [
                        {
                            "name": "string.unquoted.sysml",
                            "begin": "/\\*",
                            "beginCaptures": {
                                "0": {
                                    "name": "punctuation.definition.string.begin.sysml"
                                }
                            },
                            "end": "\\*/",
                            "endCaptures": {
                                "0": {
                                    "name": "punctuation.definition.string.end.sysml"
                                }
                            }
                        }
                    ]
                }
            }
        }
    }
}

[msujew]

Here is an example of it working.

Oh I see now, I didn't notice that the NOTE and COMMENT terminals started differently /* and //*. Yeah, now it all makes sense.

Found an easy workaround to keep auto generated syntax highlighting and having a custom one

Cool, that's good to know. We might want to include that in the documentation on textMate grammars that's supposed to come eventually.

[ponyatov]

@msujew Can I highlight language elements based on some primitive semantic analysis?

For example, can I use different colors for different type constants (type-dependent coloring):

const TGRAM = '@dponyatov';
const TAB = 0x09`u8;
const PI = 3.1415926535;

or differently highlight names created by const, let and mut:

const TGRAM = '@dponyatov';
let dialect = 'FORTH';
mut compile = false;

[cdietrich]

This would be done by implementing semantic highlighting as eg in

langium/packages/langium/src/grammar/lsp/grammar-semantic-tokens.ts

Line 8 in f53d8c7

import type { SemanticTokenAcceptor } from '../../lsp/semantic-token-provider.js';