treesitter: attempt to adapt to (breaking) Nixpkgs changes

[thamenato]

Here's an attempt to fix the issues caused by NixOS/nixpkgs@71f1b82.

I tested with my dotfiles pointing to my fork and the error went away but I am not 100% sure things are working as expected so take this PR with a grain of salt 🤔

Fixes #1312

Sanity Checking

• I have updated the changelog as per my changes
• I have tested, and self-reviewed my code
• My changes fit guidelines found in hacking nvf
• Style and consistency
  • I ran Alejandra to format my code (nix fmt)
  • My code conforms to the editorconfig configuration of the project
  • My changes are consistent with the rest of the codebase
• If new changes are particularly complex:
  • My code includes comments in particularly complex areas
  • I have added a section in the manual
  • (For breaking changes) I have included a migration guide
• Package(s) built:
  • .#nix (default package)
  • .#maximal
  • .#docs-html (manual, must build)
  • .#docs-linkcheck (optional, please build if adding links)
• Tested on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin

---

Add a 👍 reaction to pull requests you find important.

[github-actions]

🚀 Live preview deployed from 1bf7576

View it here:

Debug Information

Triggered by: NotAShelf

HEAD at: fix-nvim-treesitter

Reruns: 1774

[thamenato]

I compared checkhealth of my previous working config with my code here and it seems that the "Parsers/Features" are empty now so this isn't correct and I'm not sure what is missing - my knowledge if fairly limited here.

[NotAShelf]

I'm getting really sick of Nixpkgs at this point. "Yeah let's break everything without communication, the end-users can go fuck themselves! Oh my project? Yeah it's already updated."

Eventually we should look into dropping Nixpkgs entirely for plugins and Treesitter grammars. Both of those have been managed terribly. Viper had a repository, but I imagine it's abandoned. If it isn't, we can look at a migration path soon. Anyway.

My question about this PR is whether the updated grammar lines are all that broke. You seem to have changed a very small subset of languages, are those the ones you use or are those the ones you've confirmed to be broken? Might be worth giving the maximal configuration a try after enabling all language modules to test.

[Keyruu]

Tested with my nvf config and it works like a charm! Thanks!

[thamenato]

I'm getting really sick of Nixpkgs at this point. "Yeah let's break everything without communication, the end-users can go fuck themselves! Oh my project? Yeah it's already updated."

Eventually we should look into dropping Nixpkgs entirely for plugins and Treesitter grammars. Both of those have been managed terribly. Viper had a repository, but I imagine it's abandoned. If it isn't, we can look at a migration path soon. Anyway.

My question about this PR is whether the updated grammar lines are all that broke. You seem to have changed a very small subset of languages, are those the ones you use or are those the ones you've confirmed to be broken? Might be worth giving the maximal configuration a try after enabling all language modules to test.

Yeah, I feel you - the sad part is that I first assumed that something broke within nvf itself but then seeing the already opened issue and all the changes from nixpkgs I was a bit disappointed with the breaking changes w/o any type of warning/deprecated messages to give folks time to adjust.

Now for this PR - I naively thought that changing from builtGrammars to grammarPlugins would be the fix, and it did indeed stop the error messages and that's when I opened the PR.

But as I went back to a project I've been working on I noticed that the highlight wasn't showing up. Then I had the "brilliant" idea to check with checkhealth nvim-tree.

Installed languages     H L F I J ~

That was completely empty - here's the output I have before the changes:

Parser/Features         H L F I J
  - bash                ✓ ✓ ✓ . ✓
  - c                   ✓ ✓ ✓ ✓ ✓
  - cue                 ✓ ✓ ✓ ✓ ✓
  - go                  ✓ ✓ ✓ ✓ ✓
  - html                ✓ ✓ ✓ ✓ ✓
  - just                ✓ ✓ ✓ ✓ ✓
  - lua                 ✓ ✓ ✓ ✓ ✓
  - markdown            ✓ . ✓ ✓ ✓
  - markdown_inline     ✓ . . . ✓
  - python              ✓ ✓ ✓ ✓ ✓
  - query               ✓ ✓ ✓ ✓ ✓
  - regex               ✓ . . . .
  - terraform           ✓ . ✓ ✓ ✓
  - vim                 ✓ ✓ ✓ . ✓
  - vimdoc              ✓ . . . ✓

I also noticed that when I apply the changes nh says it's removing all of vimplugin-treesitter-grammar-* packages but added a nvim-treesitter-grammars.

I tried updating this part of my nvf in my dotfiles >

      treesitter.grammars = with pkgs.vimPlugins.nvim-treesitter-parsers; [
        just
      ];

I tried using Copilot to help me figure this out as I'm not well-versed in nix at all 😞 and it said that something changed about the .so files location in the way the new packages are defined but I'm way out of my league here.

Anyway, I think something is missing and I don't know how to debug/move forward. I also don't want to hold back the fix as you all know what you're doiwantng and I don't lol, definitely makes me properly learn nix :)

--- adding a bit more

The files for markdown and csharp the name of those packages seem to have changed so I had to adjust there. For python and sql I simply updated from using builtGrammars to grammarPlugins.

[thamenato]

Tested with my nvf config and it works like a charm! Thanks!

Are you actually getting highlights and whatnot? Did you change anything else?

[Keyruu]

Tested with my nvf config and it works like a charm! Thanks!

Are you actually getting highlights and whatnot? Did you change anything else?

I just pointed the input to your branch and everything worked like before. At least as far as I can see. I can test stuff if you want.

[NotAShelf]

I'm going to assume this is all, as I lack both the mental and the physical bandwidth to test all changes. Code looks fine, nothing out of the ordinary. For now I'm more concerned about getting the fix to the people, and then looking into ditching Nixpkgs' Neovim ecosystem as it is handled with the grace of a dead fish in a desert.

Before I merge though, this needs a changelog entry in the 0.9 changelog. Both as an user entry under your own name, and as a breaking change. See previous entries for how the breaking changes section operates.

[thamenato]

@NotAShelf just updated it.

[NotAShelf]

I've got some documentation nitpicks, but I'm going to hold that back to avoid keeping the fix from the main branch any longer. Thank you for your effort, I'm going to fix the merge conflict and merge this. Docs update can come later.

[suicide]

I just updated and it does not work for me. I don't get proper highlights

checkhealth nvim-treesitter says that tree-sitter-cli is missing

Installed languages H L F I J ~ is also empty

If I add tree-sitter I can install languages via :TSInstall and they show up in Installed languages, but that does also not fix highlighting as before.

However, even without installing a language manually or adding the tree-sitter package :InspectTree seems to work. I am kind of cofused

[Keyruu]

One thing that doesn't work is the inline languages, for example this:

  environment.etc."stacks/home-assistant/config/configuration.yaml".text = # yaml
    ''
      # Loads default set of integrations. Do not remove.
      default_config:

      automation: !include automations.yaml
      script: !include scripts.yaml
      scene: !include scenes.yaml
    '';

That is not syntax highlighted anymore.

[ethanh20009]

checkhealth nvim-treesitter says that tree-sitter-cli is missing

Installed languages H L F I J ~ is also empty

Same, using the latest commit, the error disappears but I get no syntax highlighting anymore.
Also had to disable treesitter textobjects, which throws errors otherwise.

[NotAShelf]

I just updated and it does not work for me. I don't get proper highlights

checkhealth nvim-treesitter says that tree-sitter-cli is missing

Installed languages H L F I J ~ is also empty

If I add tree-sitter I can install languages via :TSInstall and they show up in Installed languages, but that does also not fix highlighting as before.

However, even without installing a language manually or adding the tree-sitter package :InspectTree seems to work. I am kind of cofused

TS CLI missing is not really an issue; we're managing grammars with Nix, so the correct way of installing them is using runtime paths and not the CLI. Adding it to packages and using :TSInstall does not help with this issue.

The main issue, of course, is the non-communicated changes. According to the PR description, "nvim-treesitter now defaults to the main branch, which have many breaking changes, including not installing queries by default, so we need to install them, when specified." Do you know what "not installing queries by default" mean? Not installing the queries required for highlighting. A good additional change would be explaining this in the changelog itself or linking the relevant pull request, but no, all we get is "it's now incompatible, lol, use old if no work". Bravo.

At this point I can't say I'm disappointed, but I am irritated. For now I'm also quite busy, but I'll be investigating what the fix is. The PR makes a very poor job at explaining how to migrate, so I'm going to have to do some "light" reading before I can submit a fix. Of course, my idea to stop relying on Nixpkgs for plugins AND Treesitter is developing at a somewhat decent pace, so it might also be the next step here. Either way I must defer to my co-maintainers @Soliprem and @horriblename to investigate in case they can respond faster than I can.

[uorbe001]

This got merged an hour ago and seems to be the quickest way to fix this for now NixOS/nixpkgs#472119, in the long term someone will nee to go through the pain of figuring out how to migrate though, which I have to agree they've done a pretty poor job documenting.

[NotAShelf]

I think it would be a poor solution to go back at this point. I can tell why the change has happened, and I see the motive---but I condemn how it was handled in Nixpkgs.

That said I think we have managed to mostly complete our side of migration. I will probably pin v0.8's nixpkgs to nixos stable and clarify that users may wish to stay on v0.8, as critical changes are planned to be backported.

[HeitorAugustoLN]

Hello everyone! I am really sorry for the headache after merging the master -> main update, I assumed someone from the neovim team would have merged nvim-treesitter-legacy afterwards, so that would make it easy to revert to the previous version. I opened a PR migrating nvim-treesitter in #1318, so it should be mostly fixed, besides missing some features previously available such as incremental selection, since I have merged nvim-treesitter-legacy, after it gets on unstable, someone could add support for it later on

[NotAShelf]

Thank you for your comment @HeitorAugustoLN. I appreciate you chiming in, and please understand that my somewhat aggressive comments are not directed to you specifically and generally to how the Neovim ecosystem is handled in Nixpkgs. Treesitter is not the first example, and you are of course not to blame for updating it.

What I would've appreciated more is more details in the changelog (as this is a major change) on what the update entails, and how end-users should handle it. As a downstream consumer, I expect to be able to find relevant information in either the changelog or the manual. As it stands the manual doesn't cover builtGrammars and the such, and the changelog did not communicate the query changes. I happen to be outside of the loop, unlike Nixvim maintainers, and it is a little annoying that I've had to find the appropriate and read the diff to understand what needs to be done.

That said, thank you for your PR. I have not been able to review it but I will once I'm on my system. I appreciate that you took the time to send a PR :)

[HeitorAugustoLN]

please understand that my somewhat aggressive comments are not directed to you specifically and generally to how the Neovim ecosystem is handled in Nixpkgs.

I know, it is really not the best as of now, don't worry about it.

What I would've appreciated more is more details in the changelog (as this is a major change) on what the update entails, and how end-users should handle it.

I wanted to include something from upstream, but they don't have anything specific about it besides the README, which most people can find by searching for the plugin, so I did not include it in the changelog.

As a downstream consumer, I expect to be able to find relevant information in either the changelog or the manual. As it stands the manual doesn't cover builtGrammars and the such, and the changelog did not communicate the query changes.

Well, builtGrammars are only the parsers, the only difference between it and grammarPlugins and parsers, is that it currently includes an associatedQuery passthru, so that withPlugins still works same as before, installing parsers and queries. I thought builtGrammars didn't work, because I changed it on my config (maybe it actually didn't some time I was testing, but I probably made it work to not be as breaking as it already were, not sure), but looking at it again, builtGrammars are only the parsers, but in a way that it doesn't work in Neovim, it will still install queries, but they shouldn't be used as the plan is to in the future deprecate withPlugins and queries and parsers will be manually installed via pkgs.vimPlugins.nvim-treesitter.{parser,queries}.<language>, so it would break the configurations. This also makes it possible to have treesitter highlighting and folding without installing nvim-treesitter itself, as it is now handled neovim, only indenting still requires nvim-treesitter for now.

See: NixOS/nixpkgs#470883 (comment)

That said, thank you for your PR. I have not been able to review it but I will once I'm on my system. I appreciate that you took the time to send a PR :)

No problem :)

[rice-cracker-dev]

One thing that doesn't work is the inline languages, for example this:

  environment.etc."stacks/home-assistant/config/configuration.yaml".text = # yaml
    ''
      # Loads default set of integrations. Do not remove.
      default_config:

      automation: !include automations.yaml
      script: !include scripts.yaml
      scene: !include scenes.yaml
    '';

That is not syntax highlighted anymore.

I am having the same problem, JSX elements arent highlighted and .svelte files are straight up not highlighted except for a few svelte-specific features.

@HeitorAugustoLN is there any solutions for this?

[HeitorAugustoLN]

You need to install some queries manually:

{
  vim.startPlugins = with pkgs.vimPlugins.nvim-treesitter.queries; [ ecma html_tags jsx ];
}

I will try to fix this asap, by tracking the requires field for parsers

[HeitorAugustoLN]

Perchun opened a PR for it NixOS/nixpkgs#478853

[HeitorAugustoLN]

@rice-cracker-dev It is now fixed after: NixOS/nixpkgs#480516