A space between method name and arguments causes incorrect tokenization and syntax highlighting

[OwlVision]

Godot version

v4.6.1.stable.official [1419694e]

VS Code version

1.114.0

Godot Tools VS Code extension version

2.6.1

System information

Windows 10

Issue description

When a "space" is typed between a method name and its arguments, the tokenizer recognizes the name as a "variable" rather than a "function", which causes incorrect syntax highlighting.

Here's a tokens inspection:

1. Without a "space", the method name is correctly marked as a "function"

2. With a "space", the method name is marked as a "variable"

---

Despite correct highlighting even for plain function calls (i.e., outside of an object), the tokenizer shows a somewhat strange difference in the information:

1. Without a "space", the call is properly identified as a "function call"

2. With a space, the call is identified as a "function" without "meta.function-call". Parentheses are processed separately

---

It seems like there is an omission in the tokenization rules, which miss the semantics. Also, the Godot script editor itself highlights functions and methods correctly even when a space is present

Steps to reproduce

1. Take any object that has methods
2. Write an expression that calls a method
3. Place a space between the arguments and the method name

[DaelonSuzuka]

Did you test if this breaks the formatter, too?

[OwlVision]

If I understood you correctly, then it seems the formatter itself is working fine.
I applied the command 'Format Document' from the palette and got the following:

Before:

After:

---

Or did you mean something else?
The rest of the document is highlighted correctly

[OwlVision]

I noticed another trait of token recognition in this situation: if you type a space between the object name and the dot before the method call, the method is recognized again as a 'function'

Even like that:

[DaelonSuzuka]

My formatter reuses the syntax highlighting grammar to tokenize the files, so big problems with highlighting will also break the formatter, so I was just curious if this was causing an issue there. Thanks for checking!

None of these wrong scopes are totally unexpected just due to the way textmate grammars work (nested regex matching rules, and there's necessarily a lot of overlap).

I will try to tighten these definitions up a little, but please don't put a ton of work into tracking down every edge case because making it perfect would probably require starting a new grammar from scratch (and this grammar has to support BOTH gdscript 3 and 4, so there will be some imperfections in one or both no matter what).

[OwlVision]

Thank you!

Yes, I understand what you're saying. I looked at the grammar description too, and realized that even small changes can cause a bunch of painful consequences. Moreover, several situations could be imagined that are conceptually unresolvable by grammars alone. But maybe, there is a way to make the tool a bit better. I provided some edge cases in hope of being helpful and supplying as much information as possible.