Add Tree-sitter Query language

[mkatychev]

Description

This PR adds support for the Tree-sitter Query language and attempts to resolve #5746

Checklist:

• I am adding a new language.
  • The extension of the new language is used in hundreds of repositories on GitHub.com.
    • Search results for each extension:
      • https://github.com/search?type=code&q=NOT+is%3Afork+path%3A*.scm+tree-sitter
  • I have included a real-world usage sample for all extensions added in this PR:
  • Sample source(s):
    • https://github.com/tree-sitter/tree-sitter-rust/blob/master/queries/highlights.scm
    • https://github.com/tree-sitter/tree-sitter-rust/blob/master/queries/injections.scm
    • https://github.com/tree-sitter/tree-sitter-rust/blob/master/queries/tags.scm
  • Sample license(s):
    • https://github.com/tree-sitter/tree-sitter-rust?tab=MIT-1-ov-file
  • I have included a syntax highlighting grammar: https://github.com/jrieken/vscode-tree-sitter-query
  • I have added a color
    • Hex value: #8ea64c
    • Rationale: Most prominent colour at the base of the Tree-sitter logo
      
  • I have updated the heuristics to distinguish my language from others using the same extension.

[mkatychev]

I recall @lildude mentioned that the decision to do tree-sitter syntax highlighting is handled internally however the grammar in https://github.com/tree-sitter-grammars/tree-sitter-query is one of the most performant grammars out there (and is what the highlighting is based off of).

Tree-sitter Query

$ tree-sitter test --stat total-only --wasm

Total parses: 18; successful parses: 18; failed parses: 0; success percentage: 100.00%; average speed: 9755 bytes/ms

Rust:

$ tree-sitter test --stat total-only --wasm

Total parses: 136; successful parses: 136; failed parses: 0; success percentage: 100.00%; average speed: 7669 bytes/ms

[lildude]

Please also address the test failures.

[lildude]

See inline comments.

The classifier failure is because it is determining Tree-sitter Query/injections.scm is Scheme:

Tree-sitter Query/injections.scm BAD (Scheme)

This will be because it's quite a small sample and because there is only one very small Scheme .scm sample being used to train the classifier. I recommend experimenting (locally) with adding a few more larger more representative Scheme samples to better train the classifier... the more different they look from Tree-sitter Query files, the better.

[lildude]

This is vulnerable to ReDoS attack. Please fix it, and ensure the new regex runs linearly and is a Re2-style regex. See #7242 for other regexes we're slowly cleaning up.

[mkatychev]

Are all the pattern entries joined by a | or evaluated separately? Does pattern: ['a*', 'b*'] become pattern: '(a*|b*)'?

[lildude]

You can find the magic here.

My finding was based purely on testing the first regex in isolation 😉

[lildude]

Could I add those file suffixes to the test case as well?

No. That has no effect on the classifier tests. The classifier is only influenced by the samples themselves. More samples == better training.

EDIT: changed "*.scm" to "*" for Scheme in test_scm_by_heuristics

You don't want to do that. This is about testing a specific heuristic which is tied to an extension.

[mkatychev]

@lildude are we waiting on any feedback to be addressed?

[lildude]

LGTM. Thanks.

Important

The changes in this PR will not appear on GitHub until the next release has been made and deployed. See here for more details.