Syntax highlighting breaking when switching from user-level to system-level install

[ksf]

Summary

Echoing the "after close" reports in #3634, Helix failed to highlight syntax for me, relevant log attached.

Installed today, 23.03, via the nix flake, on nixos. Note that that installs grammars system-wide. hx -g fetch and hx -g build did not change a thing.

The culprit, as it turns out, were stray grammar and query directories in ~/.config/helix/runtime, presumably from a previous install via cargo.

Maybe of note: While the grammars had today's mtime, the queries were ancient. Should -g also update those? Though failing with a .so load error seems strange for a grammar/query mismatch (not that I looked at the code, though :). The .sos themselves were linked for nixos, nothing suspicious there.

Deleting those directories fixed things for me but it doesn't seem like helix should fail here in the first place.

Reproduction Steps

No response

Helix log

~/.cache/helix/helix.log

2023-05-10T23:07:42.301 helix_core::syntax [ERROR] Failed to load tree-sitter parser for language "rust": Error opening
dynamic library "/home/ksf/.config/helix/runtime/grammars/rust.so"
2023-05-10T23:12:30.756 helix_core::syntax [ERROR] Could not parse queries for language "rust". Are your grammars out of
 sync? Try running 'hx --grammar fetch' and 'hx --grammar build'. This query could not be parsed: QueryError { row: 266,
 column: 1, offset: 4336, message: "meta_item", kind: NodeType }

Platform

Linux / NixOS

Terminal Emulator

alacritty 0.10.1 ()

Helix Version

helix 23.03

[archseer]

I'm not sure what the ask is here, especially on NixOS grammars tend to get built with an glibc then as the system updates they become outdated. There's nothing we can do on the editor side to load these

[ksf]

But there's probably something you can do to not load them and use system grammars instead, or at the very least warn that there's multiple around.

And, as said, it wasn't about the grammars being outdated: Completely rebuilding them didn't change a thing.

[ksf]

Just did a little experiment: With everything already working again, do hx -g build; hx - fetch: And, indeed things are still working, and the query directory in $HOME is empty. My best guess is that helix can't deal with mismatched grammars and queries.

[pascalkuthe]

queries are like user config files. They don't ship with the grammars (they are part of the helix repo) and can't be automatically updated. If your query files are wrong then that's a user/installation error. The query files must match the grammar version (which is why grammars are fixed to specific commit in languages.toml)

[ksf]

Then something like the following must have happened:

• Back in the ancient days I installed helix via cargo, which IIRC copies the queries over to $HOME/.config
• I then switched to a nix install and at first nothing happened because the queries in $HOME were still compatible
• Then they ceased to be, and stuff broke (with patently unhelpful error message)

In the nixos case, or generally when grammars are installed on a system-level it doesn't seem to make sense to have hx -g, all it would do is re-download already installed grammars, not even updated ones. Helix probably shouldn't even try to read user-installed grammars and queries in that case, or rather strictly prefer anything it can find system-wide over the user installs (that might be useful for grammar authors). With that behaviour my install wouldn't have broken as it still would have used the up to date, system-wide, queries instead of the ancient ones in $HOME.

For the cargo-install case -- well, on its own it's not broken, either, up-to date queries will be copied over.

Thus the issue is an incompatibility in how those two install methods install and prioritise loading, and should be able to be fixed by ignoring data in $HOME when system-wide data is available.

Side note: The runtime data dir should probably be XDG_DATA_HOME, not XDG_CONFIG_HOME. $HOME/.local/share, that is.

[pascalkuthe]

the current behavior is quite intentional because the expectation is that package managers place their files at the system level and user customize their queries in $HOME. Queries are config files. The user config should overwrite the system config same as for any other configuration.

[ksf]

I respectfully disagree about the "queries are config files" part. I see things like

[
  "break"
  "continue"
  "return"
  "await"
] @keyword.control.return

in there, I don't think any user will ever customize their rust keywords. They look like semantic annotations for a grammar to me, translating tree sitter semantics into helix theme semantics (I suppose), I wouldn't expect anyone not actively developing a grammar to touch those files. I certainly didn't, they bit-rotted away in a directory called runtime, not syntax-config.

Though if you insist another idea would be to annotate the query files with the grammar they're written against, then hx -g can fetch the right grammar for the query.

[pascalkuthe]

users often add their own grammars, update existing grammars and yes customizing makes a lot of sense because while some cases might be quite clear it can be subjective what highlight group you want to apply to a specific object in a language.

Though if you insist another idea would be to annotate the query files with the grammar they're written against, then hx -g can fetch the right grammar for the query.

that's impossible there is no way to associate a grammar with a query.

[jficz]

I stumbled upon this ticket when my highlight was broken so let me add my $0.02:

I just built the grammars once (not adding custom ones) when I was debugging an issue to a specific grammar. Then I completely forgot about this fact and after a few weeks my syntax highlight broke for certain formats.

Of course, custom grammars and user-level settings overriding system settings is a good practice in general, but in this specific case Helix should imho at least print out a warning when a user grammar is outdated compared to an existing system grammar.

I mean, how many people really use customized grammars for which an upstream version is already maintained? And would those people benefit from such warning message as well? Imho yes - for two reasons:

1. the custom grammar may have not been in upstream before, or
2. the custom grammar is (most likely) based on upstream, in which case upstream updates are probably still relevant

[britter]

Just for the record if anybody ends up in this thread as well. I ended up having helix in a state where none of the highlighting worked anymore. I inspected ~/.config/helix/runtime/grammars had *.so files for all the languages. Refetching and Rebuilding did not fix the problem for me. So I removed the grammars folder and built-in highlighting started working again. But obviously my custom highlighting stopped working. So I ran hx -g fetch && hx -g build and voila, highlighting now works again for built-in languages and custom ones.