Refactor GDScriptSyntaxHighlighter to not require editor settings to work.

[weedleeedle]

Currently GDScriptSyntaxHighlighter, which is used in TextEdit nodes to enable syntax highlighting, only works in the editor. It's my understanding that this is due to GDScriptSyntaxHighlighter having hard dependencies on certain settings for the editor for things like colors.

My expectation is that Godot users can create TextEdit nodes, instantiate a GDScriptSyntaxHighlighter in it and get GDScript syntax highlighting. Currently, while the editor will completely allow you to do this, when you attempt to run the game you get the following error:

E 0:00:00:352 _instantiate_internal: Class 'GDScriptSyntaxHighlighter' can only be instantiated by editor.
<C++ Source> core/object/class_db.cpp:556 @ _instantiate_internal()

This feels like an unnecessary restriction. I understand that my use case for wanting specifically GDScript syntax highlighting in-game is very niche, but it still feels like it would be nice if GDScriptSyntaxHighlighter didn't have a hard-coded dependency on the editor settings.

I think it would be nice if GDScriptSyntaxHighlighter had a set of default colors or could otherwise be customized if the editor doesn't exist (i.e during game deployment). Here is my proposal for how this could be done:

Could we add the various syntax highlighting colors (and other settings needed by GDSyntaxScriptHighlighter) to the theme resource? GDSyntaxScriptHighlighter already uses the TextEdit node to get the font color, could we add the other options to the settings?

As an example, lines 621-624

symbol_color = EDITOR_GET("text_editor/theme/highlighting/symbol_color");
function_color = EDITOR_GET("text_editor/theme/highlighting/function_color");
number_color = EDITOR_GET("text_editor/theme/highlighting/number_color");
member_variable_color = EDITOR_GET("text_editor/theme/highlighting/member_variable_color");

Instead, we could set the various colors and settings of the syntax highlighter to the TextEdit theme, like how the font color is already done on line 620

font_color = text_edit->get_theme_color(SceneStringName(font_color));

Since the built-in editor is already a TextEdit and is styled via Godot's default theme, this should be more or less a drop-in replacement for the existing behavior. I do not know for sure what changes this would require in the settings UI. The expected workflow is that game developers can set the color for functions, keywords, etc., inside the TextEdit's theme, which would then be used by the GDSyntaxScriptHighlighter.

There are a few downsides to this. One obvious one is that there are a lot of colors associated with syntax highlighting, and it could be an overwhelming addition to the TextEdit theme for anyone who isn't trying to get extensive syntax highlighting. One workaround could be to implement the change for CodeEdit nodes instead. However, syntax highlighting is a feature on any TextEdit node, not just CodeEdit nodes, leading to a discrepancy in the intended responsibilities of TextEdit vs CodeEdit. Another is the aforementioned changes to the settings UI. Again I don't know the details of how the engine theme works, but my hope is that the transition would be trivial and unify the already-existing behavior for themes and TextEdit nodes. Another downside is that the added functionality really doesn't serve a whole lot of people other than me. I imagine this change could be useful if someone wanted to make a game in Godot about learning Godot/GDScript, but other than that I can't imagine it's a very commonly requested feature.

I have not previously contributed to Godot before but I would like to take this on if it seems like a good first issue and it's a change people want to see.

[AThousandShips]

This is not the only dependency so simply separating it from the settings won't make it usable in general

Also enabling using this highlighter in projects is essentially already tracked in:

• Enable using GDScriptSyntaxHighlighter in exported games for TextEdit #12470

[weedleeedle]

Oh cool! This is an excellent resource that I missed when looking for this, thanks!

[dalexeev]

In my opinion, the desire to have GDScript syntax highlighting in game is a rather niche need. In most cases, users will not need it, and including the syntax highlighter for everyone will increase the build size. Also note that running untrusted code is potentially unsafe. GDScript does not have a sandbox mode, and almost any simple security checks are easy to bypass.

I do not mean that we should not strive to isolate syntax highlighters. In my opinion, it makes sense to move the GDScript and C# highlighters outside of their modules, since code samples are used in documentation, and the engine can be compiled without one or both of the modules. However, isolating the highlighters somewhat conflicts with the intelligent highlighting that some users want (unused variables, local variables vs class members, etc.). But again, this is only valuable for the editor, most projects do not need GDScript highlighting in game.

Also, in my opinion, the more important issue is refactoring the syntax highlighters. Currently, the GDScriptSyntaxHighlighter code is not in the best state. It uses a very primitive approach with character-by-character traversal, which is quite difficult to maintain. As a result, regressions often occur when trying to change something. I think it would be better to rewrite it using a language grammar or context-based approach. Perhaps this can be extended to other highlighters (C#, JSON, Markdown, etc.).

[AThousandShips]

I agree it would be very desirable to make the syntax highlighting system more flexible and make it possible to just use a standard system for the GDScript one, using a more complex callback system could allow it to be more flexible while still reusing some of the existing functionality

[weedleeedle]

I agree, learning that including the syntax highlighter would increase the size of the binary for a very niche need is a performance cost not worth paying. It sounds like if I really need it I have a few other options I can try. In the absolute worst case I can write my own syntax highlighter code, I was just hoping to avoid having to rewrite the exact same code in a different language. I'll consider my own proposal withdrawn due to those limitations.

The other issues you mentioned do draw my attention and I would be interested in seeing if it's something I can help with, though I imagine it may not be the best issue for a first time contributor to tackle. I'm not a complete stranger to language parsing but it's not something I have a ton of experience in. I agree with the sentiment of the GDScriptSyntaxHighlighter code, I think it would be better to tokenize the input or otherwise reduce the amount of character-by-character traversal and state management all done in one place. I don't have any clear visions of what the end product of that would look like or how it could potentially be used by other syntax highlighters.

Other than that, is there any other changes this would encompass? Like I said I don't really have a clear picture of how the syntax highlighter fits into the larger whole. It might be best to consider this as two proposals, the immediate issue of reworking the GDScriptSyntaxHighlighter main logic and the secondary issue of refactoring where the syntax highlighters fit into the code as a whole.

As such, would it be best if I opened up a follow-up proposal to discuss this further? Since this problem is somewhat divorced from the original proposal I had in mind, I don't know if this thread is helpful for future reference.