Semantic highlighting does not distinguish escaped characters in string literals

[Steffeeen]

Swift version

swift-6.2.3-RELEASE

Platform

macOS Sequoia 15.4.1

Editor

Visual Studio Code with Swift extension 2.16.0

Description

When using semantic highlighting, escaped characters in string literals are not highlighted in a separate color. The (default) TextMate grammar-based syntax highlighting creates separate tokens for escaped characters.

With semantic highlighting:

Without semantic highlighting, using the TextMate grammar-based highlighting:

VSCode token debug hover showing the different token types:

I’m aware that sourcekit-lsp currently only forwards semantic tokens provided by sourcekitd / swift-syntax, and does not perform additional tokenization itself.

That said, it seems possible for this behavior to be implemented at the LSP level by augmenting or post-processing semantic tokens, even if the upstream tokens do not currently distinguish escaped characters.

However, I’m unsure whether this is the correct layer to address this issue. It may be more appropriate to implement this upstream. This would avoid splitting the tokenization logic across multiple projects. But I don't know which other projects consume the tokens and if it is thus possible / feasible to change this upstream.

I am open to working on this and creating a PR.

Steps to Reproduce

1. Ensure semantic highlighting is turned on in VSCode: "editor.semanticHighlighting.enabled": true
2. Open a Swift file with a string containing escaped characters, for example let a = "test \n \r \\ \' \""

Logging

No response

[RedCMD]

Semantic highlighting should only be used as an enhancement, for when TextMate falls short, not a replacement

so basic tokens like strings, comments etc should be dropped
which sourcekit-lsp could prob easily do?
and maybe add a setting to enable/disable it

[ahoppen]

We kind of misuse the semantic tokens request to also report syntactic coloring to account for the fact that Swift’s grammar often is more complicated than what’s usually covered in TextMate grammars (for example, these grammars regularly struggle with multi-line string literals, especially when interpolation is involved). This is why I don’t think we should stop reporting string literals.

We could detect the escape sequences by running the lexer on the string literal again, similar to what we do in representedLiteralValue but I’m not sure what would be the best way to report them. We could add a custom token modifier but then themes would need to adopt them, which they likely won’t. That is to say: I don’t see a great solution here at the moment.

[Steffeeen]

Could we maybe "invert" this to always primarily use the TextMate grammar and only if we detect that the TextMate grammar is not sufficient create the semantic tokens?

Do you have concrete examples of string literals that fail with just the TextMate grammar? I was only able to come up with two examples:

Example 1

let nested = ####" The value is \####( ####" Nested raw string "#### + "test" ) "####

However, even with semantic highlighting enabled the result is still incorrect:

This example is also easily fixed by using a smaller number of # for the inner string literal. In general this case seems correct as long as the inner string literals uses less # than the outer one.

Example 2

let threeDoubleQuotationMarks = """
Escaping the first quotation mark \"""
"""

Again the semantic highlighting does not fix the issue entirely:

This example can also easily be fixed by escaping at least two of the quotation marks.

There may be more trivial examples that I missed, but I would argue that having escaped characters shown in a different color would be beneficial enough to warrant accepting the few cases where the highlighting is wrong. Furthermore, in its current form the semantic highlighting does not even fix the issue entirely.

[RedCMD]

I do have a fix for the multi-escaped strings
jtbandes/swift-tmlanguage#41
I should revise it

[ahoppen]

The problem is that there isn’t a single TextMate grammar that we can compare against because different editors may be using different grammars or different formatting technologies (eg. Tree-sitter). I don’t have concrete examples at the moment, it’s more a general feeling of editors getting confused by multi-line string literals over the years. I think your two examples illustrate this pretty well. You do have a fair point that these issues tend to mess up syntactic highlighting beyond the scope of the string literal though. Taking that into account, maybe the best solution would indeed be to not report any syntax highlighting for string literal segments. I’d like to explore this in a PR if you want to open one.

[Steffeeen]

I created #2485 to explore this.