Just for reference

[tjdevries]

Hey, I've got this PR here: nvim-treesitter/tree-sitter-lua#1

It might provide some inspiration on how we might do this.

I know @vigoux was thinking about making an injection language for parsing document style highlights. Just a thought (and maybe we can chat about it here).

[vigoux]

Great idea! I think that I will start working on language injection soon, and once that is working, on how this comment parser should work, and what it should do.

Here is a list of things this comment/documentation parser should do :

• parse @param and such, doxygen-like comments
• maybe parse and highlight TODO and such
• maybe trigger the markdown parser for wome languages (rust for example).

Somme difficulties I see though : comment nodes will have different form depending on the source language, while I know that @theHamsta is working on a strip! directive, I am not sure if we will be able to apply it.
I think that this comment form problem can be solved using a custom scanner for this future comment parser, and ignore somme spécial patterns at the start of the line.

If this comment parser comes to life, then the problem becomes a transpilation problem, that is, converting a part of the syntax tree in a given program to another syntax tree, that will correspond to a comment syntax tree.
The tricky part here is that we don't have an easy way:

• to do this conversion
• to convert a syntax tree back to the textual form

Thus I think that this project will be a first experiment of transpiling part of the tree, and seeing the approach we would take.
And my opinion is that the easy way to do it is through a kind of template mechanism to generate the output, and queries to extract the information from the source file.

Hopefully my comment is clear, and thanks again @steelsojka for this work!

[steelsojka]

@vigoux From my understanding this plugin just reads the tree and collects signature data, property data, etc... to generate doc comments, like JSDocs or JavaDocs. Is there anything else this should be doing that we would want to add?

I also think transpilation might be a rabbit hole we don't want to go down... unless there is something supported by treesitter to do such a thing. I looked around and didn't see anything, but I could have missed it.

[vigoux]

I am actually really pleased by the direction the plugin has! I just wanted to have my thoughts written down somewhere 😉

[theHamsta]

I added a doc-query for python locals. Maybe we can host parts of the documentation extraction or some utiliy functions in the main repo?

Years ago I tried to implement a macro engine in tree-sitter in Rust. Should be fairly possible to do something similar with a lua template engine.

[vigoux]

I rather like having all the doc extraction thing centralized here.
First of all because that is @steelsojka 's work, and helping him going further with it is how we should do.
Then because this plugin is the first of its kind and having it complete will be a great achievement.

[theHamsta]

doc-extraction should go to locals, right? Adjacent to definitions since we're taking our local queries mostly from treesitter's tag queries. If you need some inspiration how tree-sitter does doc-extraction: https://github.com/tree-sitter/tree-sitter/tree/master/tags

[steelsojka]

@theHamsta Thanks for the links! Does treesitter have an api for already doing doc extraction (that would make it easier)? There will most likely be some things that could get merged back in to the main repo. I plan on making the api extremely flexible. The user could even use a different template engine IF they really wanted to. The tag/doc queries overlap the locals a bit but there is some more information captured. I'll just get an MVP working and then we can see how it should move forward.

@vigoux I think getting comment/doc parsing (with highlights!) would take this to the next level. We could have navigation to corresponding documentation (think navigation from a @param doc to the parameter it corresponds with and vice versa)

[theHamsta]

@steelsojka yes, just use the crate I pointed you to if you have no problems compiling the rust crate and linking it with parsers you can directly use it. But you probably want to have some kind of lua thing.

[steelsojka]

@theHamsta Cool! I also found some of the tag queries for javascript https://github.com/tree-sitter/tree-sitter-javascript/blob/master/queries/tags.scm So I may try and base it off of these (minus any unsupported predicates for now).

[theHamsta]

We already started to add this doc-queries to definitions. So feel free to add the @definition.doc to the js locals.