Compatibility with github-linguist

[DecimalTurn]

As you know, there are currently multiple issues with Atom's YAML syntax highlighting grammar. For that reason, I was looking to see if there was a better alternative to use for github-linguist and found this repo which seems to be of high quality and would be a great improvement for YAML's syntax highlighting on GitHub.

However, for the grammar in this repo to be used with Linguist, it would have to pass a certain number of checks and based on the early tests I've done, there would be some adjustments to be made to the tmLanguage.json files for it to be used by github-linguist (see Linguist's Error Report below).

Obviously, you have no obligation to make any adjustments if having your grammar used by Linguist is not something you'd be interested in. However, if you are interested, let me know and I might also be able to help make the needed changes, which hopefully won't have any major impact on the behavior of the grammar.

Linguist's Error Report

(Note from me: some \t were replaced by spaces when generating the ouput)

37 errors found in new grammar 'repository vendor/grammars/YAML-Syntax-Highlighter (from https://github.com/RedCMD/YAML-Syntax-Highlighter.git)':

yaml-1.1.tmLanguage.json

• Grammar conversion failed. File syntaxes/yaml-1.1.tmLanguage.json failed to parse: 1 error(s) decoding:

• 'Repository[directives].Patterns[4].Patterns[0].BeginCaptures[comment]' expected a map, got 'string'

yaml-1.2.tmLanguage.json

• Grammar conversion failed. File syntaxes/yaml-1.2.tmLanguage.json failed to parse: 1 error(s) decoding:

• 'Repository[directives].Patterns[4].Patterns[0].BeginCaptures[comment]' expected a map, got 'string'

yaml-embedded.tmLanguage.json

• Grammar conversion failed. File syntaxes/yaml-embedded.tmLanguage.json failed to parse: 1 error(s) decoding:

• 'Repository[directives].Patterns[1].BeginCaptures[comment]' expected a map, got 'string'

yaml-1.0.tmLanguage.json

• Invali ,[]{...": non-hex character in \x{} (closing brace missing?) (at offset 11))ontains a malformed regex (regex "?(?=[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?<=[\t ,\[{]|^)(?=(?>[^-?:,\[\]{...": unknown property name after \P or \p (at offset 53))
• Invali ])": non-hex character in \x{} (closing brace missing?) (at offset 12))son) contains a malformed regex (regex "`---(?=[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "\G(?!(?>\.{3}|---)[\x{85 2028 20...": non-hex character in \x{} (closing brace missing?) (at offset 24))
• Invali ])": non-hex character in \x{} (closing brace missing?) (at offset 14))son) contains a malformed regex (regex "`(?=.{3}[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "\G(?!(?>\.{3}|---)[\x{85 2028 20...": non-hex character in \x{} (closing brace missing?) (at offset 24))
• Invali ]...": non-hex character in \x{} (closing brace missing?) (at offset 16))n) contains a malformed regex (regex "`\G(?!%|---[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(%)([^: \p{Cntrl}\p{Surrogate}\x...": unknown property name after \P or \p (at offset 16))
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "\G(:)([^ \p{Cntrl}\p{Surrogate}\...": unknown property name after \P or \p (at offset 17))
• Invali ]...": non-hex character in \x{} (closing brace missing?) (at offset 16))n) contains a malformed regex (regex "`\G.{3}(?=[\x{85 2028 2029}
• Invali -\x{10FFFF...": non-hex character in \x{} (closing brace missing?) (at offset 7))ains a malformed regex (regex "[^\x{85 2028 2029}
• Invali ,[]{...": non-hex character in \x{} (closing brace missing?) (at offset 11))ontains a malformed regex (regex "?(?=[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?=(?>[^-?:,\[\]{}#&*!|>'"%@ \p`...": unknown property name after \P or \p (at offset 38))
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?=%YAML:1\.0(?=[\x{85 2028 2029...": non-hex character in \x{} (closing brace missing?) (at offset 22))
• Invali ])": non-hex character in \x{} (closing brace missing?) (at offset 14))son) contains a malformed regex (regex "`\G(?!---[\x{85 2028 2029}
• Invali ])": non-hex character in \x{} (closing brace missing?) (at offset 14))son) contains a malformed regex (regex "`\G(?=---[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?=((?<=[-?:]) )?+)(?<![^ ][ ]...": non-hex character in \x{} (closing brace missing?) (at offset 71))
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?<=(?>["'\]}]|^)[ ]*+):": lookbehind assertion is not fixed length (at offset 23))
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?=((?<=[-?:]) )?+)\G( *+)(\?)(?...": non-hex character in \x{} (closing brace missing?) (at offset 39))
• Invali ])": non-hex character in \x{} (closing brace missing?) (at offset 11))son) contains a malformed regex (regex "`(?=[\x{2028 2029}
• Invali ])": non-hex character in \x{} (closing brace missing?) (at offset 12))son) contains a malformed regex (regex "`\G!(?=[\x{85 2028 2029}
• Invali ]{2,0}": non-hex character in \x{} (closing brace missing?) (at offset 8))) contains a malformed regex (regex "`%[^\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?=((?<=[-?:]) )?+)(?<![^ ][ ]...": non-hex character in \x{} (closing brace missing?) (at offset 59))
• Invali ,[]{}...": non-hex character in \x{} (closing brace missing?) (at offset 10))ntains a malformed regex (regex ":(?=[\x{85 2028 2029}
  0abtnvfre "...": non-hex character in \x{} (closing brace missing?) (at offset 8))n) contains a malformed regex (regex "`\[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "\\(?>x[^"]{2,0}|u[^"]{4,0}|U[^"]...": numbers out of order in {} quantifier (at offset 14))
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?<=\[)\G(?=[\x{85 2028 2029} ...": non-hex character in \x{} (closing brace missing?) (at offset 18))
• Invali ,#])": non-hex character in \x{} (closing brace missing?) (at offset 10))n) contains a malformed regex (regex "`(?=[^\x{85 2028 2029}
• Invali ...": non-hex character in \x{} (closing brace missing?) (at offset 17))on) contains a malformed regex (regex "`(?<={)\G(?=[\x{85 2028 2029}
• Invali ,#])": non-hex character in \x{} (closing brace missing?) (at offset 10))n) contains a malformed regex (regex "`(?=[^\x{85 2028 2029}
• Invali ])|(?<=...": non-hex character in \x{} (closing brace missing?) (at offset 10))ntains a malformed regex (regex ":(?=[\x{85 2028 2029}
• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "\G(?![?:!"'0-9A-Za-z$()+./;<=\\^...": non-hex character in \x{} (closing brace missing?) (at offset 101))

yaml.tmLanguage.json

• Invalid regex in grammar: source.yaml (in syntaxes/yaml.tmLanguage.json) contains a malformed regex (regex "(?<=^-{3,}\s*+)\G$": lookbehind assertion is not fixed length (at offset 14))
• Invalid regex in grammar: source.yaml (in syntaxes/yaml.tmLanguage.json) contains a malformed regex (regex "^(?! {3,0}-{3,}[ \t]*+$|[ ]*+\.{...": numbers out of order in {} quantifier (at offset 9))

Compilation failed. Aborting

[RedCMD]

thank you
I was going to look into updating Linguists grammar
there doesn't seem to be any more bug reports from VSCode
so I'm happy for it to be updated

doing some tests with Apple's TextMate 2.0 Application

• 'Repository[directives].Patterns[4].Patterns[0].BeginCaptures[comment]' expected a map, got 'string'
  TextMate 2.0 allows string based comments in captures
  so this is a Linguist bug

• Invali ,[]{...": non-hex character in \x{} (closing brace missing?) (at offset 11))ontains a malformed regex (regex "?(?=[\x{85 2028 2029}
  TextMate 2.0 uses an old version of oniguruma which doesn't support Hexadecimal and Octal Code point sequences
  so it instead just treats this as literal text
  Linguist should only provide a warning about it (as it doesn't error in TextMate 2.0)
  if Linguist claims its invalid, then I'd say its a bug with Linguist
  I would still need to change it regardless

• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?<=[\t ,\[{]|^)(?=(?>[^-?:,\[\]{...": unknown property name after \P or \p (at offset 53))
  \p{Cntrl} and \p{Surrogate} are valid and do work
  another Linguist bug

• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "(?<=(?>["'\]}]|^)[ ]*+):": lookbehind assertion is not fixed length (at offset 23))
  This is unfortunate, VSCode TextMate uses a later version of oniguruma which does support variable length lookbehind
  TextMate 2.0 and Linguist use the older version
  I guess I could just change it to capture the whitespace, prob wont have side effects(?)

• Invalid regex in grammar: source.yaml.1.0 (in syntaxes/yaml-1.0.tmLanguage.json) contains a malformed regex (regex "\\(?>x[^"]{2,0}|u[^"]{4,0}|U[^"]...": numbers out of order in {} quantifier (at offset 14))
  Again possessive quantifiers are a new oniguruma addition
  I can change it back to just a normal quantifier

• The last thing I'm slightly afraid about is while
  VSCode TextMate and TextMate 2.0 (and thus Linguist) treat while's interaction with begin/end slightly differently
  which may cause problems
  I think it will only maybe affect block-scalars | and versions %YAML 1.2
  so shouldn't be too bad

I need to test the grammar in TextMate 2.0, but I haven't figured out what data format Apple uses
it doesn't seem to be CSON as it uses semicolons ; and =

I can fix all these if need be
but I think Linguist should fix their bugs as well

do you know what YAML files they use in Linguist for testing?

[DecimalTurn]

I was going to look into updating Linguists grammar
there doesn't seem to be any more bug reports from VSCode
so I'm happy for it to be updated

Awesome!

Regarding "bugs" on the side of Linguist, those could be actually due to limitations of the internal library that GitHub is using to actually perform the syntax highlighting. Linguist is just the visible part of the iceberg in this case and it's there to aggregate grammars that are considered valid to be used elsewhere in the GitHub stack.

Looking at the code from the directory for the grammar compiler and seing pcre.go tells me that it propably has to do with PCRE.

If you'd like more details it might be better to open an issue or a discussion on the Linguist repo to confirm if those limitations are actually justified. Let me know if you want me to do that.

do you know what YAML files they use in Linguist for testing?

What kind of testing? There are a few YAML samples in the Linguist repo, but they are used to test language identification, not syntax highligting.

[DecimalTurn]

This actually confirms what I wrote in my previous post: github-linguist/linguist#5844
The internal library is named Prettylights and it uses PCRE.

So, to deal with elements that you labeled as "Linguist bugs", I see 2 options. First option is to modify the TM files to fit within Prettylights' limitations. Second option would be to leave the TM files in this repo untouched and write a script inside another repo that downloads the TM files and makes the automated changes there to get compatibility with Prettylights.

Thoughts?

EDIT: Option 2 would also allow to leave yaml-embedded.tmLanguage.json alone since it's not needed by Linguist from what I understand.

[ValouBambou]

While writing a tool around TextMate grammar files, I also encounter the same issue with the "comment" key in the beginCaptures table. I can't find anything in TextMate docs about comment outside a Rule.

This could be fixed by deleting the comment or moving it inside a rule, in the inner or the outer one.

I "fixed the bug" on my side by adding a special case for comment in captures. However, I found this a bit sad since Yaml is the only example I found using this among the grammars present in VS Code official repo. Could be nicer to just don't have to handle this special case, but I understand that you might have your reasons to not change the grammar files.

[RedCMD]

technically comments don't actually exist
they are just keys that are ignored just like all other ignored keys
Linguist just happens to have a too strict schema that doesn't like non-object pairs

VSCode and TextMate 2.0 ignore all non-numeric keys inside captures
and I'm pretty sure GitHub does too

I will change it at some point
I've just been busy getting all the tools ready so I can overhaul the grammar