Why are Helix’s tree-sitter queries different from those defined in the grammars themselves?

[aral]

I noticed that the tree-sitter-svelte queries in Helix Editor (found at runtime/queries/svelte are different from the ones defined in the tree-sitter-svelte grammar itself. Using the ones in the grammar does not appear to work.

I’m wondering if the queries in the grammar itself are incorrect or if there is a difference in syntax for some reason (different version of tree-sitter used by Helix, perhaps?)

To illustrate, here are the injection definitions from Helix (which work):

; injections.scm
; --------------
((style_element
  (raw_text) @injection.content)
  (#set! injection.language "css"))

((attribute
   (attribute_name) @_attr
   (quoted_attribute_value (attribute_value) @css))
 (#eq? @_attr "style"))

((script_element
  (raw_text) @injection.content)
  (#set! injection.language "javascript"))

((raw_text_expr) @injection.content
 (#set! injection.language "javascript"))

(
  (script_element
    (start_tag
      (attribute
        (quoted_attribute_value (attribute_value) @_lang)))
    (raw_text) @injection.content)
  (#match? @_lang "(ts|typescript)")
  (#set! injection.language "typescript")
)

((comment) @injection.content
 (#set! injection.language "comment"))

And here are the ones defined in the original grammar (which don’t work):

; injections.scm
; --------------
((style_element
  (raw_text) @css))

((attribute
   (attribute_name) @_attr
   (quoted_attribute_value (attribute_value) @css))
 (#eq? @_attr "style"))

((script_element
  (raw_text) @javascript))

((raw_text_expr) @javascript)

(
  (script_element
    (start_tag
      (attribute
        (quoted_attribute_value (attribute_value) @_lang)))
    (raw_text) @typescript)
  (#match? @_lang "(ts|typescript)")
)

(comment) @comment

This is related to an issue I opened asking for Helix Editor to automatically copy tree-sitter queries from a tree-sitter grammar when running hx --grammar build my_lang and now I’m wondering if the reason this isn’t implemented currently is because Helix requires some special dialect of the queries and/or whether different versions of tree-sitter have syntax differences, etc.

[aral]

To somewhat answer my own question. Based on the Tree Sitter documentation, it would appear that the injection syntax in tree-sitter-svelte is incorrect (?)

https://tree-sitter.github.io/tree-sitter/syntax-highlighting#language-injection

If someone can confirm this, I’ll go ahead and prepare a pull request for them to fix it based on the queries in Helix.

[kirawi]

These queries look to match nvim-treesitter, which has an order of precedence reverse of tree-sitter cli (which Helix follows).

[aral]

Ah, that’s odd that they have their own incompatible query syntax. (Unless I’m mistaken, a design goal of tree sitter is that any editor can use a grammar once it’s created. And as far as most of the grammar goes, that seems to be the case. Odd to break that consistency just for the query syntax.)

(And thanks for the info, by the way) :)

[the-mikedavis]

The parsers can be re-used between editors but afaik it isn't a goal to have all editors use the same queries for a language

[archseer]

To add to the previous answers, usually the grammar itself provides some simple queries, but we have our own list of scopes based on sublime/textmate grammars to offer a lot of granularity for styling. Some query files also use editor-specific predicates, for example neovim uses vim-match? instead of match? which uses vim's regex engine that's not compatible with regex used by us/rust/tree-sitter-cli.

[Keats]

Would it make sense to have the queries be in a separate repo? I'd like to re-use them for a HTML highlighter since they are better than the built-in but I guess it would depend of the grammar version as well? Not super deep in tree-sitter yet so I don't know if it makes sense.

[the-mikedavis]

Queries do end up depending on the version of the grammar: certain changes to a grammar (for example removing named nodes) will make them incompatible with old queries. Storing queries with grammars could work though, as long as the queries are tied to certain versions of the grammars

[Keats]

I'm happy to copy the queries for my needs (as the result is nicer from my quick test with one language: getzola/zola#1787 (comment)) but it would be nicer (imo) to have a single repository. We have the issue currently with Syntect where several projects have their own syntaxes folders, kinda duplicating work. Is the plan to upstream them when possible?

[the-mikedavis]

Definitely feel free to copy them. For now it's more convenient for us to keep the queries in-repo here so that we can version them along-side the parsers in the languages.toml in the repo root.

We've upstreamed some highlighting changes but there are some (C# in particular) where we haven't yet. Some tree-sitter grammar repositories don't contain any queries themselves and those that do usually have scopes built for the tree-sitter-cli highlight command (these scopes). Each editor uses their own custom set of scopes (we use these, nvim another, emacs another, difftastic another) so it's hard to centralize the queries and re-use them between projects.

[Keats]

Thanks, will do that then!