Raw string literals (more than one `#`, grammar limitations prevent us from supporting escapes)

[RedCMD]

swift-tmlanguage/Swift.tmLanguage.yaml

Line 2339 in 4028d24

# SE-0168: raw string literals (more than one #, grammar limitations prevent us from supporting escapes)

What limitation are you facing?
syntax examples?

https://github.com/apple/swift-evolution/blob/main/proposals/0168-multi-line-string-literals.md
doesn't seem to say anything about hashtags #

is it this section?
https://github.com/apple/swift-evolution/blob/main/proposals/0200-raw-string-escaping.md#customized-escape-delimiters

[RedCMD]

I see the problem now

could possibly add a single line raw string literal that supports all the delimiters

but then you probably couldn't support \#( 'meta.embedded.line.swift' ) 😔

[RedCMD]

I think I've managed to get something to work
and is probably most cursed textmate I've ever written (worse than ((?!(?<!\\\\)\\\\{2}*\").)*?)

just had "comment" as a place holder to visualize whats being tokenized

it works by chaining begin/end groups one after another
the end rule doesn't capture anything. allowing the begin rule to retarget the escape delimiter and "literals-string-raw-string-guts" to tokenize

annoying I had to make an exception for the first set of non-escape-delimiters
(?>\\\\(?!\\1(?!#))#+)*
otherwise the very first \# and \### would be passed into "literals-string-raw-string-guts" and be marked as invalid (for \## string delimiters)

{
	"name": "swift-raw-strings",
	"scopeName": "source.redcmd.syntax.swift-raw-strings",
	"patterns": [ { "include": "#swift-raw-strings" } ],
	"repository": {
		"swift-raw-strings": {
			"begin": "(##+)\"",
			"end": "\"\\1(#*)",
			"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.raw.swift" } },
			"endCaptures": {
				"0": { "name": "punctuation.definition.string.end.raw.swift" },
				"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
			},
			"patterns": [
				{
					"begin": "(?<=(?>[^#]|^)(#+)\")\\G(?>\\\\(?!\\1(?!#))#+)*",
					"end": "(?!\\G)(?=\\\\\\1(?!#))|(?=\"\\1)",
					"name": "comment",
					"patterns": [ { "include": "#literals-string-raw-string-guts" } ]
				},
				{
					"begin": "(?=\\\\(#+)(?!#))",
					"end": "(?!\\G)(?=\\\\\\1(?!#))|(?=\"\\1)",
					"name": "comment",
					"patterns": [ { "include": "#literals-string-raw-string-guts" } ]
				}
			]
		},
		"literals-string-raw-string-guts": {
			"comment": "the same as #string-guts but with # in escapes",
			"patterns": [
				{
					"name": "constant.character.escape.swift",
					"match": "\\G\\\\#+[0\\\\tnr\"']"
				},
				{
					"name": "constant.character.escape.unicode.swift",
					"match": "\\G\\\\#+u\\{[0-9a-fA-F]{1,8}\\}"
				},
				{
					"contentName": "source.swift",
					"name": "meta.embedded.line.swift",
					"begin": "\\G\\\\#+\\(",
					"end": "(\\))",
					"beginCaptures": { "0": { "name": "punctuation.section.embedded.begin.swift" } },
					"endCaptures": {
						"0": { "name": "punctuation.section.embedded.end.swift" },
						"1": { "name": "source.swift" }
					},
					"patterns": [
						{
							"comment": " --> $self <-- ",
							"include": "source.swift"
						},
						{
							"comment": "Nested parens",
							"begin": "\\(",
							"end": "\\)"
						}
					]
				},
				{
					"name": "invalid.illegal.escape-not-recognized",
					"match": "\\G\\\\#+."
				}
			]
		}
	}
}

a

b##"\#\###\##t\###\##z\#

c
\##t
d
\##z
e
\##(
func foo<T>(x: T) async throws -> Int where T: Equatable {}
)
f
\#
g
\##u{0123}
h
\###
i


j"##

k

[jtbandes]

Wow that's quite cursed, congrats 🫠 It's gonna take me a while to wrap my head around this. It's also been 4 years since I added raw string support so I've probably forgotten some context.

Sounds like you've answered your question yourself but I believe the limitations I was referring to was that I didn't think it was possible to reference groups from begin inside the scope content pattern itself (so that begin=####" would require escapes to be like \####(...) inside the scope)

Does your solution handle things like ##" \##(this is interpolated) \#(but this is not interpolated) "## ?

[RedCMD]

Does your solution handle things like...

\#( & \###( will be treated as raw string
\##( will be recognized as the source.swift

I'm assuming that delimiters are not recognized inside \##( ... )?

also
should be able to remove the exception from the first pattern
as long as that include can be removed

"begin": "(?<=(?>[^#]|^)(#+)\")\\G",
"end": "(?=\\\\\\1(?!#))|(?=\"\\1)",
"name": "comment"

[RedCMD]

better exception handler

"swift-raw-strings": {
	"begin": "(#+)\"",
	"end": "\"\\1(#*)",
	"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.raw.swift" } },
	"endCaptures": {
		"0": { "name": "punctuation.definition.string.end.raw.swift" },
		"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
	},
	"name": "string.quoted.double.single-line.raw.swift",
	"patterns": [
		{
			"begin": "(?<=(?>[^#]|^)(#+)\")\\G(?!\\\\\\1(?!#))",
			"end": "(?=\\\\\\1(?!#))|(?=\"\\1)",
			"patterns": [
				{
					"name": "invalid.illegal.returns-not-allowed.swift",
					"match": "\\r|\\n"
				}
			]
		},
		{
			"begin": "(?=\\\\(#+)(?!#))",
			"end": "(?!\\G)(?=\\\\\\1(?!#))|(?=\"\\1)",
			"patterns": [
				{
					"name": "invalid.illegal.returns-not-allowed.swift",
					"match": "\\r|\\n"
				},
				{ "include": "#literals-string-raw-string-guts" }
			]
		}
	]
}

[RedCMD]

took another crack at this

I swapped from using lookbehinds (which aren't compatible with TextMate/GitHub)
to not capturing the initial beginning delimiter (straight away) and instead using just lookaheads

couple of changes:

• support for any number of # in raw strings ###"foo\###nbar"###
• mark last character in single-line raw string as invalid if no ending delimited found #"invalid newline
• fixed begin raw string scope missing raw subscope

grammar half kilobyte smaller

literals-string

"literals-string": {
	"patterns": [
		{
			"comment": "SE-0168: Multi-Line String Literals",
			"begin": "\"\"\"",
			"end": "\"\"\"(#*)",
			"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.swift" } },
			"endCaptures": {
				"0": { "name": "punctuation.definition.string.end.swift" },
				"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
			},
			"name": "string.quoted.double.block.swift",
			"patterns": [
				{
					"match": "\\G.+(?=\"\"\")|\\G.+",
					"name": "invalid.illegal.content-after-opening-delimiter.swift"
				},
				{
					"match": "\\\\\\s*\\n",
					"name": "constant.character.escape.newline.swift"
				},
				{ "include": "#literals-string-string-guts" },
				{
					"comment": "Allow \\(\"\"\"...\"\"\") to appear inside a block string",
					"match": "\\S((?!\\\\\\().)*(?=\"\"\")",
					"name": "invalid.illegal.content-before-closing-delimiter.swift"
				}
			]
		},
		{
			"comment": "SE-0168: Multi-Line Raw String Literals (one or more #, escapes supported)",
			"begin": "(?=(#+)\"\"\")",
			"end": "\"\"\"\\1(#*)",
			"endCaptures": {
				"0": { "name": "punctuation.definition.string.end.swift" },
				"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
			},
			"name": "string.quoted.double.block.raw.swift",
			"patterns": [
				{
					"begin": "\\G(#+)\"\"\"",
					"end": "\\\\\\1#+.|(?=.*\"\"\"\\1|\\\\\\1)",
					"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.raw.swift" } },
					"endCaptures": { "0": { "name": "invalid.illegal.escape-not-recognized" } },
					"patterns": [
						{
							"match": "\\G.+(?=\"\"\"#+)|\\G.+",
							"name": "invalid.illegal.content-after-opening-delimiter.swift"
						}
					]
				},
				{
					"begin": "(?=\\\\(#+))",
					"end": "\\\\\\1#+.|(?!\\G)(?=.*\"\"\"\\1|\\\\\\1)",
					"endCaptures": { "0": { "name": "invalid.illegal.escape-not-recognized" } },
					"patterns": [
						{
							"match": "\\G\\\\#+\\s*\\n",
							"name": "constant.character.escape.newline.swift"
						},
						{ "include": "#literals-string-raw-string-guts" }
					]
				},
				{
					"comment": "Allow \\(\"\"\"...\"\"\") to appear inside a block string",
					"match": "\\S.*?(?=\"\"\"#+)",
					"name": "invalid.illegal.content-before-closing-delimiter.swift"
				}
			]
		},
		{
			"comment": "SE-0168: Single-Line String Literals",
			"begin": "\"",
			"end": "\"(#*)",
			"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.swift" } },
			"endCaptures": {
				"0": { "name": "punctuation.definition.string.end.swift" },
				"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
			},
			"name": "string.quoted.double.single-line.swift",
			"patterns": [
				{
					"match": "\\r|\\n",
					"name": "invalid.illegal.returns-not-allowed.swift"
				},
				{ "include": "#literals-string-string-guts" }
			]
		},
		{
			"comment": "SE-0168: Single-Line Raw String Literals (one or more #, escapes supported)",
			"begin": "(?=(#+)\")",
			"end": "\"\\1(#*)",
			"endCaptures": {
				"0": { "name": "punctuation.definition.string.end.raw.swift" },
				"1": { "name": "invalid.illegal.extra-closing-delimiter.swift" }
			},
			"name": "string.quoted.double.single-line.raw.swift",
			"patterns": [
				{
					"begin": "\\G(#+)\"",
					"end": "\\\\\\1#+.|(?=[\\\\\"]\\1)",
					"beginCaptures": { "0": { "name": "punctuation.definition.string.begin.raw.swift" } },
					"endCaptures": { "0": { "name": "invalid.illegal.escape-not-recognized" } },
					"patterns": [ { "include": "#invalid-returns" } ]
				},
				{
					"begin": "(?=\\\\(#+))",
					"end": "\\\\\\1#+.|(?!\\G)(?=[\\\\\"]\\1)",
					"endCaptures": { "0": { "name": "invalid.illegal.escape-not-recognized" } },
					"patterns": [
						{ "include": "#literals-string-raw-string-guts" },
						{ "include": "#invalid-returns" }
					]
				}
			]
		}
	],
	"repository": {
		"invalid-returns": {
			"match": ".?[\\r\\n]",
			"name": "invalid.illegal.returns-not-allowed.swift"
		},
		"literals-string-string-guts": {
			"patterns": [
				{
					"match": "\\\\[0\\\\tnr\"']",
					"name": "constant.character.escape.swift"
				},
				{
					"match": "\\\\u{[0-9a-fA-F]{1,8}}",
					"name": "constant.character.escape.unicode.swift"
				},
				{
					"begin": "\\\\\\(",
					"end": "\\)",
					"beginCaptures": { "0": { "name": "punctuation.section.embedded.begin.swift" } },
					"endCaptures": { "0": { "name": "punctuation.section.embedded.end.swift" } },
					"name": "meta.embedded.line.swift",
					"contentName": "source.swift",
					"patterns": [
						{ "include": "$self" },
						{
							"comment": "Nested parens",
							"begin": "\\(",
							"end": "\\)"
						}
					]
				},
				{
					"match": "\\\\.",
					"name": "invalid.illegal.escape-not-recognized"
				}
			]
		},
		"literals-string-raw-string-guts": {
			"comment": "the same as #string-guts but with # in escapes",
			"patterns": [
				{
					"match": "\\G\\\\#+[0\\\\tnr\"']",
					"name": "constant.character.escape.swift"
				},
				{
					"match": "\\G\\\\#+u{[0-9a-fA-F]{1,8}}",
					"name": "constant.character.escape.unicode.swift"
				},
				{
					"begin": "\\G\\\\#+\\(",
					"end": "\\)",
					"beginCaptures": { "0": { "name": "punctuation.section.embedded.begin.swift" } },
					"endCaptures": { "0": { "name": "punctuation.section.embedded.end.swift" } },
					"name": "meta.embedded.line.swift",
					"contentName": "source.swift",
					"patterns": [
						{ "include": "$self" },
						{
							"comment": "Nested parens",
							"begin": "\\(",
							"end": "\\)"
						}
					]
				},
				{
					"match": "\\G\\\\#+.",
					"name": "invalid.illegal.escape-not-recognized"
				}
			]
		}
	}
},

print(###"raw: a\##0b\##nc\##u{1}d \##(interpolation) a \##(1 + foo(x: 4)) nested: \##(1+"string"+2) x"###)
print(###"raw: a\###0b\###nc\###u{1}d \###(interpolation) a \###(1 + foo(x: 4)) nested: \###(1+"string"+2) x"####)
print(####"raw: a\###0b\###nc\###u{1}d \###(interpolation) a \###(1 + foo(x: 4)) nested: \###(1+"string"+2) x"####)
print(####"raw: a\####0b\####nc\####u{1}d \####(interpolation) a \####(1 + foo(x: 4)) nested: \####(1+"string"+2) x"#####)

let SE0168 = ##"""   illegal
        my, what a large…
    \##(1 + foo(x: 4))
    \##(##"""
      more \##( ##"""
        s
      """##)
    """##)
        …string you have!
    illegal"""##