clangd 17 regression: response files are not expanded when using command inference

[drcxd]

Hello,

I have upgraded to 17.0.X, but I found that clangd reports that some of the header files cannot be found. However, downgrade back to 16.0.6, the error disappears.

All the other settings of the project are the same. I only replaced the clangd binary from 16.0.6 to 17.0.2 or 17.0.3, then the server reports that the header files are missing, which actually, of course, are not.

I have read 17.0.1 release notes but found nothing helpful. Any help would be appreciated. Otherwise, I can only stay in clangd 16.0.6 forever. That would be really sad for me.

[llvmbot]

@llvm/issue-subscribers-clangd

Author: None (drcxd)

Hello,

I have upgraded to 17.0.X, but I found that clangd reports that some of the header files cannot be found. However, downgrade back to 16.0.6, the error disappears.

All the other settings of the project are the same. I only replaced the clangd binary from 16.0.6 to 17.0.2 or 17.0.3, then the server reports that the header files are missing, which actually, of course, are not.

I have read 17.0.1 release notes but found nothing helpful. Any help would be appreciated. Otherwise, I can only stay in clangd 16.0.6 forever. That would be really sad for me.

[kadircet]

hi @drcxd can you provide some more details so that we can investigate the issue?

https://clangd.llvm.org/troubleshooting#gathering-logs explains how to gather some bare minimum. It'd be great if you can also mention which header is not found, and its location on your system.

[drcxd]

Hello @kadircet

Here is the log:

I[09:53:33.581] clangd version 17.0.3
I[09:53:33.582] Features: windows
I[09:53:33.582] PID: 10904
I[09:53:33.582] Working directory: d:\workplace\MyProject
I[09:53:33.582] argv[0]: c:\Program Files\LLVM\bin\clangd.exe
I[09:53:33.582] argv[1]: --completion-style=detailed
I[09:53:33.582] argv[2]: --header-insertion=never
I[09:53:33.582] argv[3]: --function-arg-placeholders=0
I[09:53:33.582] argv[4]: -j=32
I[09:53:33.582] argv[5]: --background-index
I[09:53:33.582] argv[6]: --pch-storage=memory
I[09:53:33.586] Starting LSP over stdin/stdout
I[09:53:33.586] <-- initialize(1)
I[09:53:33.588] --> reply:initialize(1) 1 ms
I[09:53:33.684] <-- initialized
I[09:53:33.684] <-- textDocument/didOpen
I[09:53:33.684] <-- textDocument/semanticTokens/full(2)
I[09:53:33.685] --> textDocument/publishDiagnostics
I[09:53:33.716] <-- $/cancelRequest
I[09:53:33.716] <-- textDocument/semanticTokens/full(3)
I[09:53:33.717] <-- textDocument/codeAction(4)
I[09:53:33.717] <-- textDocument/documentLink(5)
I[09:53:33.717] <-- textDocument/documentHighlight(6)
I[09:53:33.745] Loaded compilation database from d:\workplace\MyProject\compile_commands.json
I[09:53:33.746] ASTWorker building file d:\workplace\MyProject\Source\MyProject\MyProjectCharacter.h version 0 with command inferred from D:\workplace\MyProject\Source\MyProject\MyProjectCharacter.cpp
[D:\workplace\UnrealEngine-release\Engine\Source]
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\Hostx64\\x64\\cl.exe" --driver-mode=cl /TP -D__INTELLISENSE__ -ferror-limit=0 "-resource-dir=c:\\Program Files\\LLVM\\lib\\clang\\17" -- "d:\\workplace\\MyProject\\Source\\MyProject\\MyProjectCharacter.h"
I[09:53:33.748] --> textDocument/publishDiagnostics
I[09:53:33.759] Built preamble of size 233064 for file d:\workplace\MyProject\Source\MyProject\MyProjectCharacter.h version 0 in 0.01 seconds
I[09:53:33.761] Indexing c++14 standard library in the context of d:\workplace\MyProject\Source\MyProject\MyProjectCharacter.h
E[09:53:33.791] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
E[09:53:33.792] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
E[09:53:33.792] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
E[09:53:33.792] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
I[09:53:33.795] --> textDocument/publishDiagnostics
I[09:53:33.888] --> textDocument/publishDiagnostics
I[09:53:33.914] --> reply:textDocument/semanticTokens/full(2) 230 ms, error: Task was cancelled.
I[09:53:33.915] --> reply:textDocument/semanticTokens/full(3) 198 ms
I[09:53:33.915] --> reply:textDocument/codeAction(4) 197 ms
I[09:53:33.915] --> reply:textDocument/documentLink(5) 197 ms
I[09:53:33.915] --> reply:textDocument/documentHighlight(6) 197 ms
I[09:53:34.080] Loaded compilation database from D:\workplace\UnrealEngine-release\compile_commands.json
I[09:53:34.188] <-- textDocument/codeAction(7)
I[09:53:34.188] <-- textDocument/documentLink(8)
I[09:53:34.188] <-- textDocument/documentHighlight(9)
I[09:53:34.188] --> reply:textDocument/codeAction(7) 0 ms
I[09:53:34.188] --> reply:textDocument/documentLink(8) 0 ms
I[09:53:34.188] --> reply:textDocument/documentHighlight(9) 0 ms
I[09:53:34.335] --> window/workDoneProgress/create(0)
I[09:53:34.335] Enqueueing 6 commands for indexing
I[09:53:34.336] <-- reply(0)
I[09:53:34.336] --> $/progress
I[09:53:34.336] --> $/progress
I[09:53:34.610] Indexed c++14 standard library (incomplete due to errors): 12161 symbols, 1163 filtered
I[09:53:35.363] --> $/progress
I[09:53:35.363] --> $/progress
I[09:53:36.386] <-- textDocument/hover(10)
I[09:53:36.386] <-- textDocument/codeAction(11)
I[09:53:36.386] <-- textDocument/documentLink(12)
I[09:53:36.387] <-- textDocument/documentHighlight(13)
I[09:53:36.390] --> reply:textDocument/hover(10) 4 ms
I[09:53:36.390] --> reply:textDocument/codeAction(11) 4 ms
I[09:53:36.390] --> reply:textDocument/documentLink(12) 3 ms
I[09:53:36.390] --> reply:textDocument/documentHighlight(13) 3 ms

The first missing header file is CoreMinimal.h which is a header file provided by Unreal Engine. Its location on my computer is:

d:/workplace/UnrealEngine-release/Engine/Source/Runtime/Core/Public/

The compile instruction for the file I opened is:

	{
		"file": "D:\\workplace\\MyProject\\Source\\MyProject\\MyProjectCharacter.cpp",
		"command": "\"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX64\\x64\\cl.exe\" @D:\\workplace\\MyProject\\Intermediate\\Build\\Win64\\UnrealEditor\\Development\\MyProject\\MyProjectCharacter.cpp.obj.response.gcd",
		"directory": "D:\\workplace\\UnrealEngine-release\\Engine\\Source"
	},

In MyProjectCharacter.cpp.obj.response.gcd, it includes all the including path and other compiler switches, I confirm the directory that contains the CoreMinimal.h is included. Here are the first several lines of the gcd file:

D:/workplace/MyProject/Source/MyProject/MyProjectCharacter.cpp
/I D:/workplace/UnrealEngine-release/Engine/Source
/I D:/workplace/MyProject/Intermediate/Build/Win64/UnrealEditor/Inc/MyProject/UHT
/I D:/workplace/MyProject/Intermediate/Build/Win64/UnrealEditor/Inc/MyProject/VNI
/I D:/workplace/MyProject/Source
/I D:/workplace/UnrealEngine-release/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Core/UHT
/I D:/workplace/UnrealEngine-release/Engine/Intermediate/Build/Win64/UnrealEditor/Inc/Core/VNI
/I D:/workplace/UnrealEngine-release/Engine/Source/Runtime
/I D:/workplace/UnrealEngine-release/Engine/Source/Runtime/TraceLog/Public
/I D:/workplace/UnrealEngine-release/Engine/Source/Runtime/Core/Public

Notice that the last line is the including path which contains the CoreMinimal.h.

[drcxd]

I noticed that the problem is gone in release 17.0.4, so I'll just close this issue.

[drcxd]

I was wrong. The issue persists in 17.0.4. Sorry.

[drcxd]

A new update: with clangd and clang 16.0.6, when I try to complete at a certain point, the completion failed, I have the following error messages:

clang failed with error 1:
c:/Program Files/LLVM/bin/clang.exe -fsyntax-only -Xclang -code-completion-macros -x c++ -Xclang -code-completion-at=-:1962:28 -

In file included from <stdin>:4:
./PlayerSystem.h:5:10 fatal error: 'CoreMinimal.h' file not found
#include "CoreMinimal.h"

However, in my editor, clangd does not report such a file not found error.

[drcxd]

New update: I find that if I open X.h first, then the server reports that a header file included in X.h is missing. If I open X.h after I have opened X.cpp, then the header file missing error is gone. Is that when only the header file is opened, the server does not know which .cpp file includes it and thus does not know the corresponding compile instruction, which contains the including directory? But how does this problem not manifest in previous versions? I have used clangd on the same project for about 2 years and have only experienced such issue after upgrade to 17.0.X.

[HighCommander4]

Is that when only the header file is opened, the server does not know which .cpp file includes it and thus do not know the corresponding compile instruction, which contains the including directory?

Yes, that seems likely to be the explanation.

When a header is opened, clangd first checks to see if it's included by an open file, if so the open file's compile command is used to process the header. If not, clangd will pick some other compile command from compile_commands.json. You can see in the logs which file it picks, and what its compile command is.

[drcxd]

Hello, @HighCommander4 , I have posted the log in a previous reply. I excerpt part of the log I assume that is related to this problem here:

I[09:53:33.745] Loaded compilation database from d:\workplace\MyProject\compile_commands.json
I[09:53:33.746] ASTWorker building file d:\workplace\MyProject\Source\MyProject\MyProjectCharacter.h version 0 with command inferred from D:\workplace\MyProject\Source\MyProject\MyProjectCharacter.cpp
[D:\workplace\UnrealEngine-release\Engine\Source]
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\Hostx64\\x64\\cl.exe" --driver-mode=cl /TP -D__INTELLISENSE__ -ferror-limit=0 "-resource-dir=c:\\Program Files\\LLVM\\lib\\clang\\17" -- "d:\\workplace\\MyProject\\Source\\MyProject\\MyProjectCharacter.h"
I[09:53:33.748] --> textDocument/publishDiagnostics
I[09:53:33.759] Built preamble of size 233064 for file d:\workplace\MyProject\Source\MyProject\MyProjectCharacter.h version 0 in 0.01 seconds
I[09:53:33.761] Indexing c++14 standard library in the context of d:\workplace\MyProject\Source\MyProject\MyProjectCharacter.h
E[09:53:33.791] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
E[09:53:33.792] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
E[09:53:33.792] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
E[09:53:33.792] IncludeCleaner: Failed to get an entry for resolved path : no such file or directory
I[09:53:33.795] --> textDocument/publishDiagnostics

In the log file, I can not see clangd picked any compile command from compile_commands.json. Instead, it seems that clangd just compiled the header file directly. Then the missing header file error is generated.

[HighCommander4]

This line:

I[09:53:33.746] ASTWorker building file d:\workplace\MyProject\Source\MyProject\MyProjectCharacter.h version 0 with command inferred from D:\workplace\MyProject\Source\MyProject\MyProjectCharacter.cpp

is saying that clangd is using the command of MyProjectCharacter.cpp to parse MyProjectCharacter.h. (More specifically, it is using the flags from that command. The input file is indeed just the header file directly.)