[clang] [clangd] Missing information about explicit instantiation of function template

[16bit-ykiko]

namespace test {
template <typename T>
void foo() {}
}

struct X {};

template void test::f^oo<X>();

Currently, if trigger go-to-definition at the location of ^, clangd will give no response. And the tokens in the explicit instantiation are also not highlighted. After some investigation, I found that the underlying cause is that the Clang frontend doesn't store the information about explicit instantiation of function templates.

Dump the AST of above code

|-NamespaceDecl 0xd083678 <<source>:1:1, line:4:1> line:1:11 test
| `-FunctionTemplateDecl 0xd0838e8 <line:2:1, line:3:13> col:6 foo
|   |-TemplateTypeParmDecl 0xd083708 <line:2:11, col:20> col:20 typename depth 0 index 0 T
|   |-FunctionDecl 0xd083838 <line:3:1, col:13> col:6 foo 'void ()'
|   | `-CompoundStmt 0xd0839c8 <col:12, col:13>
|   `-FunctionDecl 0xd083d18 <col:1, col:13> col:6 foo 'void ()' explicit_instantiation_definition
|     |-TemplateArgument type 'X'
|     | `-RecordType 0xd083aa0 'X'
|     |   `-CXXRecord 0xd0839f8 'X'
|     `-CompoundStmt 0xd0839c8 <col:12, col:13>
`-CXXRecordDecl 0xd0839f8 <line:6:1, col:11> col:8 referenced struct X definition

As you can see, the decl from explicit instantiation is only added to its semantic context but not to its lexical context.

If use a visitor to further explore:

bool VisitFunctionTemplateDecl(clang::FunctionTemplateDecl* decl) {
    for(auto spec: decl->specializations()) {
        spec->getLocation().dump(srcMgr);
        spec->getFunctionTypeLoc().getReturnLoc().getLocalSourceRange().dump(srcMgr)
        spec->getQualifierLoc().getSourceRange().dump(srcMgr);
        auto info = spec->getTemplateSpecializationInfo();
        info->PointOfInstantiation.dump(srcMgr);
        llvm::outs() << info->TemplateArguments << "\n";
        llvm::outs() << info->TemplateArgumentsAsWritten << "\n";
    }
    return true;
}

The output is

test.cpp:3:6
<test.cpp:3:1>
<<invalid sloc>>
test.cpp:8:21
0x564cf04d7560
0x0

Only instantiation point is recorded correctly, all other information is from primary template.

But for explicit specializations, the information is recorded properly.

namespace test {
template <typename T>
void foo() {}
}

struct X {};

template <>
void test::foo<X>() {}

The output is

test.cpp:9:12
<test.cpp:9:1>
<test.cpp:9:6, col:10>
<invalid loc>
0x56162dc7f6f8
0x56162dc7f718

---

Explicit instantiation of variable template has the same problem.

And I unexpectedly discovered that the location of TypeAliasTemplate incorrectly uses the position of the using keyword instead of the position of the identifier."

[llvmbot]

@llvm/issue-subscribers-clangd

Author: ykiko (16bit-ykiko)

```cpp namespace test { template <typename T> void foo() {} }

struct X {};

template void test::f^oo<X>();

Currently, if trigger go-to-definition at the location of `^`, clangd will give no response. And the tokens in the explicit instantiation are 
 also not highlighted.  After some investigation, I found that the underlying cause is that the Clang frontend doesn't store the information about explicit instantiation of function templates.
 
 Dump the AST of above code

 ```cpp
 |-NamespaceDecl 0xd083678 <<source>:1:1, line:4:1> line:1:11 test
| `-FunctionTemplateDecl 0xd0838e8 <line:2:1, line:3:13> col:6 foo
|   |-TemplateTypeParmDecl 0xd083708 <line:2:11, col:20> col:20 typename depth 0 index 0 T
|   |-FunctionDecl 0xd083838 <line:3:1, col:13> col:6 foo 'void ()'
|   | `-CompoundStmt 0xd0839c8 <col:12, col:13>
|   `-FunctionDecl 0xd083d18 <col:1, col:13> col:6 foo 'void ()' explicit_instantiation_definition
|     |-TemplateArgument type 'X'
|     | `-RecordType 0xd083aa0 'X'
|     |   `-CXXRecord 0xd0839f8 'X'
|     `-CompoundStmt 0xd0839c8 <col:12, col:13>
`-CXXRecordDecl 0xd0839f8 <line:6:1, col:11> col:8 referenced struct X definition

As you can see, the decl from explicit instantiation is only added to its semantic context but not to its lexical context.

If use a visitor to further explore:

bool VisitFunctionTemplateDecl(clang::FunctionTemplateDecl* decl) {
    for(auto spec: decl->specializations()) {
        spec->getLocation().dump(srcMgr);
        spec->getFunctionTypeLoc().getReturnLoc().getLocalSourceRange().dump(srcMgr)
        spec->getQualifierLoc().getSourceRange().dump(srcMgr);
        auto info = spec->getTemplateSpecializationInfo();
        info->PointOfInstantiation.dump(srcMgr);
        llvm::outs() << info->TemplateArguments << "\n";
        llvm::outs() << info->TemplateArgumentsAsWritten << "\n";
    }
    return true;
}

The output is

test.cpp:3:6
<test.cpp:3:1>
<<invalid sloc>>
test.cpp:8:21
0x564cf04d7560
0x0

Only instantiation point is recorded correctly, all other information is from primary template. But for explicit specializations, the information is recorded properly.

[llvmbot]

@llvm/issue-subscribers-clang-frontend

Author: ykiko (16bit-ykiko)

```cpp namespace test { template <typename T> void foo() {} }

struct X {};

template void test::f^oo<X>();

Currently, if trigger go-to-definition at the location of `^`, clangd will give no response. And the tokens in the explicit instantiation are 
 also not highlighted.  After some investigation, I found that the underlying cause is that the Clang frontend doesn't store the information about explicit instantiation of function templates.
 
 Dump the AST of above code

 ```cpp
 |-NamespaceDecl 0xd083678 <<source>:1:1, line:4:1> line:1:11 test
| `-FunctionTemplateDecl 0xd0838e8 <line:2:1, line:3:13> col:6 foo
|   |-TemplateTypeParmDecl 0xd083708 <line:2:11, col:20> col:20 typename depth 0 index 0 T
|   |-FunctionDecl 0xd083838 <line:3:1, col:13> col:6 foo 'void ()'
|   | `-CompoundStmt 0xd0839c8 <col:12, col:13>
|   `-FunctionDecl 0xd083d18 <col:1, col:13> col:6 foo 'void ()' explicit_instantiation_definition
|     |-TemplateArgument type 'X'
|     | `-RecordType 0xd083aa0 'X'
|     |   `-CXXRecord 0xd0839f8 'X'
|     `-CompoundStmt 0xd0839c8 <col:12, col:13>
`-CXXRecordDecl 0xd0839f8 <line:6:1, col:11> col:8 referenced struct X definition

As you can see, the decl from explicit instantiation is only added to its semantic context but not to its lexical context.

If use a visitor to further explore:

bool VisitFunctionTemplateDecl(clang::FunctionTemplateDecl* decl) {
    for(auto spec: decl->specializations()) {
        spec->getLocation().dump(srcMgr);
        spec->getFunctionTypeLoc().getReturnLoc().getLocalSourceRange().dump(srcMgr)
        spec->getQualifierLoc().getSourceRange().dump(srcMgr);
        auto info = spec->getTemplateSpecializationInfo();
        info->PointOfInstantiation.dump(srcMgr);
        llvm::outs() << info->TemplateArguments << "\n";
        llvm::outs() << info->TemplateArgumentsAsWritten << "\n";
    }
    return true;
}

The output is

test.cpp:3:6
<test.cpp:3:1>
<<invalid sloc>>
test.cpp:8:21
0x564cf04d7560
0x0

Only instantiation point is recorded correctly, all other information is from primary template. But for explicit specializations, the information is recorded properly.

[HighCommander4]

See also clangd/clangd#358

[16bit-ykiko]

Sorry to bother, but ping @mizvekov. Do you have any thoughts on this issue? Or any suggestions on how to fix it? I might give it a try. :)

[mizvekov]

The instantiated declaration is the result of substitution of the template arguments on the template pattern, and that has no relationship with the point of instantiation, so changing the lexical context would break that model, and would be ambiguous with friend declarations.

The issue @HighCommander4 linked (clangd/clangd#358) is most helpful here, the right solution would be to create an AST node which models explicit instantiations.

[16bit-ykiko]

The issue @HighCommander4 linked (clangd/clangd#358) is most helpful here, the right solution would be to create an AST node which models explicit instantiations.

What about variable template? It seems that it already has a corresponding AST node -- VarTemplateSpecializationDecl, but still get wrong or incomplete source information:

namespace test {
template <typename T>
int foo = 2;
}

template <>
int test::foo<int> = 3;

template int test::foo<char>;

Using following code to dump source information:

bool VisitVarTemplateSpecializationDecl(clang::VarTemplateSpecializationDecl* decl) {
    llvm::outs() << "----------------------------------------------\n";
    decl->getLocation().dump(srcMgr);
    decl->getTypeSourceInfo()->getTypeLoc().getSourceRange().dump(srcMgr);
    decl->getQualifierLoc().getBeginLoc().dump(srcMgr);
    if(auto args = decl->getTemplateArgsAsWritten()) {
        for(auto arg: args->arguments()) {
            arg.getLocation().dump(srcMgr);
        }
    }
    return true;
}

The output is

---------------------------------------------- explicit instantiation
test.cpp:3:5
<test.cpp:3:1>
<invalid loc>
test.cpp:9:24
---------------------------------------------- explicit specialization
test.cpp:7:11
<test.cpp:7:1>
test.cpp:7:5
test.cpp:7:15

For explicit specialization, everything is correct. But for explicit instantiation, only template argument information is correct.

Currently in clang only information about explicit instantiation of class template is recorded correctly.

[mizvekov]

The VarTemplateSpecializationDecl is in the same situation as the ClassTemplateSpecializationDecl, it's meant to model the specialization per se, not the explicit instantiation as it appears in source code.

The fact it carries source locations just for the specialization arguments is a little weird, and even that doesn't model the fact that explicit instantiations can appear multiple times. It's something we could certainly rethink once we make progress into the new AST nodes.

[16bit-ykiko]

Oh I understand it. So we need to create a new node to store source location information.

I think we can create something like ExplicitInstantiationDecl to store the location of qualifier, name, template arguments and angles. And it is just a wrapper to current specialization decl.

In this way, we keep compatible with the old implementation.

[mizvekov]

Yep, that sounds right, the ExplicitInstantiation node should just link to the *SpecializationDecl, and it should probably be redeclarable as well, so that we can easily iterate all explicit instantiations of a given template.

If you are interested in pursuing that, I'd encourage you to seek @hokein and @sam-mccall, as they would have more insights into clangd's needs in this area.

[16bit-ykiko]

In fact, I am writing a new language server for C++. I have been diving into clangd for months, so I am very familiar with it now. I found this problem when I was testing my new indexer, so I will try to fix it to make my indexer work better. :)