Unable to go to definition if variable declared by macro.

[ShiinaManatsu]

As image shows below, also same for cbuffer member.

[antaalt]

Yes, this require to resolve the macro and there is currently no way to do this. Call expression macro in region directive are ignored as well for the same reason. Will need to find a way to solve macros efficiently without having to re implement a compiler ideally

[ShiinaManatsu]

btw, I found there is a SemanticTokensProvider can works better then TextMate grammars from page Syntax Highlight Guide, currently variables are just white color coded, will there be a future plan on this?

[antaalt]

Yes, that's what macros are using for getting highlights currently, we could probably extend this to params as well

[ShiinaManatsu]

btw, I found there is a SemanticTokensProvider can works better then TextMate grammars from page Syntax Highlight Guide, currently variables are just white color coded, will there be a future plan on this?

I tried to implement SemanticTokensProvider using tree-sitter-hlsl found in Cargo.toml in typescript hopes to get some highlights for my variables so they don't looks like plain text, but check the docs I found I have to get higher score for my provider so vscode can take it as semantic token result, it's so hard😂😂

[antaalt]

You will probably need to plug it into existing semantic token provider which already does most of the heavy work. But currently there is no way to check if a variable is a local variable or a function parameter to mark them.
I was thinking of a way to represent the shader symbol list into a tree to enable some new features with vscode. This will probably ease some things for this

[antaalt]

Last version added parameters highlighting :)

[ShiinaManatsu]

Thats huge improvement! Will try tomorrow. btw, I tried to modify tree-sitter-hlsl to support our specified grammars, and that works fine, but analysis needs to be process on the server side, so I tried to create rider plugin but the PSI(Program Structure Interface) thing just not fully documented, hlsltools looks like a way to do this which i am working on. Will add new features be more easier in shader-sense?

[repelliuss]

I am also encountering this. Any macro with a parameter is not being parsed. Any update on this? I am open to work on this if some insight is given about what needs to be done. I can also confirm that hlsltools support this

[antaalt]

Yes, thats a limitation with tree-sitter ast on which I am relying, macros are considered as raw string and are not part of the Abstract Syntax Tree. So we cant go with the standard way to inspect the content. Right now, it just ignore it. I have thought of eventually using some kind of proxy tree with tree sitter to parse these in real time, but might be hitting the performances quite hard.

Here its handling call expression

shader-sense/shader-sense/src/symbols/hlsl/hlsl_regions.rs

Line 329 in 2ceb500

"call_expression" => {

And here trying to parse macro to value

shader-sense/shader-sense/src/symbols/hlsl/hlsl_regions.rs

Line 147 in 2ceb500

// Here we try to detect expression that cannot be parsed, such as expression (#define MACRO (MACRO0 + MACRO1)).

[repelliuss]

Thank you for the detailed description. I have some experience with tree sitter. I will try to look into this in my free time

[ShiinaManatsu]

After learn some rust, I am starting to modify the code to fit our project specified syntax, first step is to add new syntax to tree-sitter-c, the tree-sitter-hlsl lack lot of syntax on hlsl, so I start to rewrite them, and I found that tree-sitter lack of ability to expand macros, that means, although dxil can validate the shader, but the ast can not been parsered. By looking at hlsltools, they write their own parser, and expand them, create some fake element on symbol, Will we change the way to parse the ast?

[antaalt]

Yes I would like to avoid writing our own parser, which would mean rewriting the entire project...
The best fit for solving this is probably using a cached tree sitter ast that would be updated at runtime to solve it's content, but I did not get the time to look at this yet, and I am a bit sceptical concerning the performances.

Maybe by batching all macros into a runtime tree it would be enough for that.

[ShiinaManatsu]

I do find some comments under tree-sitter-cpp repo issues says we can inject and merge ast, before we do the expansion, we have to collect all macro definitions, that means we have to do two parse on all referenced files, sounds pretty hurt haha. rewrite tree-sitter-hlsl based on tree-sitter-c maybe needed. tree-sitter-hlsl parser sizes around 26MB, my partial hlsl syntax version just 7MB. Also optimized the parse speed a lot. I may still use tree-sitter-hlsl for now since it passed all my shaders, and dig into how we can do the expand.