[ClangScanDeps] Improve getTranslationUnitDependencies #159472
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Created using spr 1.3.6
|
@llvm/pr-subscribers-clang Author: Steven Wu (cachemeifyoucan) ChangesMake Full diff: https://github.com/llvm/llvm-project/pull/159472.diff 7 Files Affected:
diff --git a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
index c3601a4e73e1f..2974e67f363f5 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
@@ -139,9 +139,9 @@ class DependencyScanningTool {
/// \param LookupModuleOutput This function is called to fill in
/// "-fmodule-file=", "-o" and other output
/// arguments for dependencies.
- /// \param TUBuffer Optional memory buffer for translation unit input. If
- /// TUBuffer is nullopt, the input should be included in the
- /// Commandline already.
+ /// \param OverlayFS An optional VFS that overlays on top of WorkerFS to
+ /// provide additional virtual files to the scanning of the
+ /// translation unit.
///
/// \returns a \c StringError with the diagnostic output if clang errors
/// occurred, \c TranslationUnitDeps otherwise.
@@ -149,7 +149,7 @@ class DependencyScanningTool {
const std::vector<std::string> &CommandLine, StringRef CWD,
const llvm::DenseSet<ModuleID> &AlreadySeen,
LookupModuleOutputCallback LookupModuleOutput,
- std::optional<llvm::MemoryBufferRef> TUBuffer = std::nullopt);
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> OverlayFS = nullptr);
/// Given a compilation context specified via the Clang driver command-line,
/// gather modular dependencies of module with the given name, and return the
diff --git a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
index 6060e4b43312e..8ef81e518172a 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
@@ -90,10 +90,9 @@ class DependencyScanningWorker {
llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> FS);
/// Run the dependency scanning tool for a given clang driver command-line,
- /// and report the discovered dependencies to the provided consumer. If
- /// TUBuffer is not nullopt, it is used as TU input for the dependency
- /// scanning. Otherwise, the input should be included as part of the
- /// command-line.
+ /// and report the discovered dependencies to the provided consumer. An
+ /// additional OverlayFS can be provided to add additional virtual files to
+ /// the dependency scanning.
///
/// \returns false if clang errors occurred (with diagnostics reported to
/// \c DiagConsumer), true otherwise.
@@ -101,7 +100,7 @@ class DependencyScanningWorker {
StringRef WorkingDirectory, const std::vector<std::string> &CommandLine,
DependencyConsumer &DepConsumer, DependencyActionController &Controller,
DiagnosticConsumer &DiagConsumer,
- std::optional<llvm::MemoryBufferRef> TUBuffer = std::nullopt);
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> OverlayFS = nullptr);
/// Run the dependency scanning tool for a given clang driver command-line
/// for a specific module.
@@ -123,7 +122,7 @@ class DependencyScanningWorker {
llvm::Error computeDependencies(
StringRef WorkingDirectory, const std::vector<std::string> &CommandLine,
DependencyConsumer &Consumer, DependencyActionController &Controller,
- std::optional<llvm::MemoryBufferRef> TUBuffer = std::nullopt);
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> OverlayFS = nullptr);
/// Run the dependency scanning tool for a given clang driver command-line
/// for a specific module.
diff --git a/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp b/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
index 27734ffd0e20b..bd66ad66ea6df 100644
--- a/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
+++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
@@ -144,11 +144,11 @@ DependencyScanningTool::getTranslationUnitDependencies(
const std::vector<std::string> &CommandLine, StringRef CWD,
const llvm::DenseSet<ModuleID> &AlreadySeen,
LookupModuleOutputCallback LookupModuleOutput,
- std::optional<llvm::MemoryBufferRef> TUBuffer) {
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> OverlayFS) {
FullDependencyConsumer Consumer(AlreadySeen);
CallbackActionController Controller(LookupModuleOutput);
- llvm::Error Result = Worker.computeDependencies(CWD, CommandLine, Consumer,
- Controller, TUBuffer);
+ llvm::Error Result = Worker.computeDependencies(
+ CWD, CommandLine, Consumer, Controller, std::move(OverlayFS));
if (Result)
return std::move(Result);
diff --git a/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp b/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
index 0a12c479bf8e3..7aed691c8e6bf 100644
--- a/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
+++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
@@ -627,7 +627,7 @@ createDiagOptions(const std::vector<std::string> &CommandLine) {
llvm::Error DependencyScanningWorker::computeDependencies(
StringRef WorkingDirectory, const std::vector<std::string> &CommandLine,
DependencyConsumer &Consumer, DependencyActionController &Controller,
- std::optional<llvm::MemoryBufferRef> TUBuffer) {
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> OverlayFS) {
// Capture the emitted diagnostics and report them to the client
// in the case of a failure.
std::string DiagnosticOutput;
@@ -636,7 +636,7 @@ llvm::Error DependencyScanningWorker::computeDependencies(
TextDiagnosticPrinter DiagPrinter(DiagnosticsOS, *DiagOpts);
if (computeDependencies(WorkingDirectory, CommandLine, Consumer, Controller,
- DiagPrinter, TUBuffer))
+ DiagPrinter, std::move(OverlayFS)))
return llvm::Error::success();
return llvm::make_error<llvm::StringError>(DiagnosticsOS.str(),
llvm::inconvertibleErrorCode());
@@ -788,41 +788,26 @@ bool DependencyScanningWorker::scanDependencies(
bool DependencyScanningWorker::computeDependencies(
StringRef WorkingDirectory, const std::vector<std::string> &CommandLine,
DependencyConsumer &Consumer, DependencyActionController &Controller,
- DiagnosticConsumer &DC, std::optional<llvm::MemoryBufferRef> TUBuffer) {
+ DiagnosticConsumer &DC,
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> OverlayFS) {
// Reset what might have been modified in the previous worker invocation.
BaseFS->setCurrentWorkingDirectory(WorkingDirectory);
- std::optional<std::vector<std::string>> ModifiedCommandLine;
llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> ModifiedFS;
-
- // If we're scanning based on a module name alone, we don't expect the client
- // to provide us with an input file. However, the driver really wants to have
- // one. Let's just make it up to make the driver happy.
- if (TUBuffer) {
- auto OverlayFS =
+ // If an OverlayFS is provided, create a new FS that has the overlay on the
+ // top layer.
+ if (OverlayFS) {
+ auto NewFS =
llvm::makeIntrusiveRefCnt<llvm::vfs::OverlayFileSystem>(BaseFS);
- auto InMemoryFS =
- llvm::makeIntrusiveRefCnt<llvm::vfs::InMemoryFileSystem>();
- InMemoryFS->setCurrentWorkingDirectory(WorkingDirectory);
- auto InputPath = TUBuffer->getBufferIdentifier();
- InMemoryFS->addFile(
- InputPath, 0,
- llvm::MemoryBuffer::getMemBufferCopy(TUBuffer->getBuffer()));
- llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> InMemoryOverlay =
- InMemoryFS;
-
- OverlayFS->pushOverlay(InMemoryOverlay);
- ModifiedFS = OverlayFS;
- ModifiedCommandLine = CommandLine;
- ModifiedCommandLine->emplace_back(InputPath);
+ NewFS->pushOverlay(std::move(OverlayFS));
+ NewFS->setCurrentWorkingDirectory(WorkingDirectory);
+ ModifiedFS = NewFS;
}
- const std::vector<std::string> &FinalCommandLine =
- ModifiedCommandLine ? *ModifiedCommandLine : CommandLine;
auto &FinalFS = ModifiedFS ? ModifiedFS : BaseFS;
- return scanDependencies(WorkingDirectory, FinalCommandLine, Consumer,
- Controller, DC, FinalFS, /*ModuleName=*/std::nullopt);
+ return scanDependencies(WorkingDirectory, CommandLine, Consumer, Controller,
+ DC, FinalFS, /*ModuleName=*/std::nullopt);
}
bool DependencyScanningWorker::computeDependencies(
diff --git a/clang/test/ClangScanDeps/tu-buffer.c b/clang/test/ClangScanDeps/tu-vfs.c
similarity index 63%
rename from clang/test/ClangScanDeps/tu-buffer.c
rename to clang/test/ClangScanDeps/tu-vfs.c
index b450b13ff434b..5082293295bd7 100644
--- a/clang/test/ClangScanDeps/tu-buffer.c
+++ b/clang/test/ClangScanDeps/tu-vfs.c
@@ -19,26 +19,72 @@ module addition { header "addition.h" }
//--- addition.h
// empty
-//--- tu.c
+//--- hidden/tu.c
#include "root.h"
+//--- hidden/additional.h
+#include "addition.h"
+
//--- root/textual.h
// This is here to verify that the "root" directory doesn't clash with name of
// the "root" module.
//--- cdb.json.template
[{
- "file": "",
+ "file": "DIR/test.c",
"directory": "DIR",
- "command": "clang -fmodules -fmodules-cache-path=DIR/cache -I DIR -x c -c"
+ "command": "clang -fmodules -fmodules-cache-path=DIR/cache -I DIR -x c DIR/test.c -c"
}]
+//--- cdb2.json.template
+[{
+ "file": "DIR/test.c",
+ "directory": "DIR",
+ "command": "clang -fmodules -fmodules-cache-path=DIR/cache -include DIR/pre.h -I DIR -x c DIR/test.c -c"
+}]
+
+//--- vfs.yaml.template
+{
+ 'version': 0,
+ 'roots': [
+ { 'name': 'DIR', 'type': 'directory',
+ 'contents': [
+ { 'name': 'test.c', 'type': 'file',
+ 'use-external-name': false,
+ 'external-contents': 'DIR/hidden/tu.c'
+ },
+ { 'name': 'pre.h', 'type': 'file',
+ 'use-external-name': false,
+ 'external-contents': 'DIR/hidden/additional.h'
+ },
+ ]
+ }
+ ]
+}
+
// RUN: sed "s|DIR|%/t|g" %t/cdb.json.template > %t/cdb.json
-// RUN: clang-scan-deps -compilation-database %t/cdb.json -format experimental-full -tu-buffer-path %t/tu.c > %t/result.json
+// RUN: sed "s|DIR|%/t|g" %t/vfs.yaml.template > %t/vfs.yaml
+// RUN: clang-scan-deps -compilation-database %t/cdb.json -format experimental-full -vfs-overlay-path %t/vfs.yaml > %t/result.json
// RUN: cat %t/result.json | sed 's:\\\\\?:/:g' | FileCheck -DPREFIX=%/t %s --check-prefix=CHECK
+// RUN: sed "s|DIR|%/t|g" %t/cdb2.json.template > %t/cdb2.json
+// RUN: clang-scan-deps -compilation-database %t/cdb2.json -format experimental-full -vfs-overlay-path %t/vfs.yaml > %t/result2.json
+// RUN: cat %t/result2.json | sed 's:\\\\\?:/:g' | FileCheck -DPREFIX=%/t %s --check-prefix=CHECK --check-prefix=ADDITION
+
// CHECK: {
// CHECK-NEXT: "modules": [
+// ADDITION-NEXT: {
+// ADDITION-NEXT: "clang-module-deps": [],
+// ADDITION-NEXT: "clang-modulemap-file": "[[PREFIX]]/module.modulemap",
+// ADDITION-NEXT: "command-line": [
+// ADDITION: ],
+// ADDITION-NEXT: "context-hash": "{{.*}}",
+// ADDITION-NEXT: "file-deps": [
+// ADDITION-NEXT: "[[PREFIX]]/module.modulemap"
+// ADDITION-NEXT: "[[PREFIX]]/addition.h"
+// ADDITION-NEXT: ],
+// ADDITION: "name": "addition"
+// ADDITION-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: "clang-module-deps": [
// CHECK-NEXT: {
@@ -93,6 +139,10 @@ module addition { header "addition.h" }
// CHECK-NEXT: {
// CHECK-NEXT: "clang-context-hash": "{{.*}}",
// CHECK-NEXT: "clang-module-deps": [
+// ADDITION-NEXT: {
+// ADDITION-NEXT: "context-hash": "{{.*}}",
+// ADDITION-NEXT: "module-name": "addition"
+// ADDITION-NEXT: },
// CHECK-NEXT: {
// CHECK-NEXT: "context-hash": "{{.*}}",
// CHECK-NEXT: "module-name": "root"
@@ -101,9 +151,10 @@ module addition { header "addition.h" }
// CHECK-NEXT: "command-line": [
// CHECK: ],
// CHECK: "file-deps": [
-// CHECK-NEXT: [[PREFIX]]/tu.c
+// CHECK-NEXT: [[PREFIX]]/test.c
+// ADDITION-NEXT: [[PREFIX]]/pre.h
// CHECK-NEXT: ],
-// CHECK-NEXT: "input-file": "[[PREFIX]]/tu.c"
+// CHECK-NEXT: "input-file": "[[PREFIX]]/test.c"
// CHECK-NEXT: }
// CHECK-NEXT: ]
// CHECK-NEXT: }
diff --git a/clang/tools/clang-scan-deps/ClangScanDeps.cpp b/clang/tools/clang-scan-deps/ClangScanDeps.cpp
index 0e2758d123edc..63990f1966e6b 100644
--- a/clang/tools/clang-scan-deps/ClangScanDeps.cpp
+++ b/clang/tools/clang-scan-deps/ClangScanDeps.cpp
@@ -89,7 +89,7 @@ static unsigned NumThreads = 0;
static std::string CompilationDB;
static std::optional<std::string> ModuleName;
static std::vector<std::string> ModuleDepTargets;
-static std::string TranslationUnitFile;
+static std::string VFSOverlayFile;
static bool DeprecatedDriverCommand;
static ResourceDirRecipeKind ResourceDirRecipe;
static bool Verbose;
@@ -210,8 +210,8 @@ static void ParseArgs(int argc, char **argv) {
for (const llvm::opt::Arg *A : Args.filtered(OPT_dependency_target_EQ))
ModuleDepTargets.emplace_back(A->getValue());
- if (const llvm::opt::Arg *A = Args.getLastArg(OPT_tu_buffer_path_EQ))
- TranslationUnitFile = A->getValue();
+ if (const llvm::opt::Arg *A = Args.getLastArg(OPT_vfs_overlay_path_EQ))
+ VFSOverlayFile = A->getValue();
DeprecatedDriverCommand = Args.hasArg(OPT_deprecated_driver_command);
@@ -1082,24 +1082,23 @@ int clang_scan_deps_main(int argc, char **argv, const llvm::ToolContext &) {
LocalIndex, DependencyOS, Errs))
HadErrors = true;
} else {
- std::unique_ptr<llvm::MemoryBuffer> TU;
- std::optional<llvm::MemoryBufferRef> TUBuffer;
- if (!TranslationUnitFile.empty()) {
- auto MaybeTU =
- llvm::MemoryBuffer::getFile(TranslationUnitFile, /*IsText=*/true);
- if (!MaybeTU) {
- llvm::errs() << "cannot open input translation unit: "
- << MaybeTU.getError().message() << "\n";
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> OverlayFS;
+ if (!VFSOverlayFile.empty()) {
+ auto YAMLFile =
+ llvm::MemoryBuffer::getFile(VFSOverlayFile, /*IsText=*/true);
+ if (!YAMLFile) {
+ llvm::errs() << "cannot open vfs overlay yaml: "
+ << YAMLFile.getError().message() << "\n";
HadErrors = true;
continue;
}
- TU = std::move(*MaybeTU);
- TUBuffer = TU->getMemBufferRef();
- Filename = TU->getBufferIdentifier();
+ OverlayFS = llvm::vfs::RedirectingFileSystem::create(
+ std::move(*YAMLFile), nullptr, "", nullptr,
+ llvm::vfs::getRealFileSystem());
}
auto MaybeTUDeps = WorkerTool.getTranslationUnitDependencies(
Input->CommandLine, CWD, AlreadySeenModules, LookupOutput,
- TUBuffer);
+ OverlayFS);
if (handleTranslationUnitResult(Filename, MaybeTUDeps, *FD, LocalIndex,
DependencyOS, Errs))
HadErrors = true;
diff --git a/clang/tools/clang-scan-deps/Opts.td b/clang/tools/clang-scan-deps/Opts.td
index 03011f9ae1f75..909e311f9d824 100644
--- a/clang/tools/clang-scan-deps/Opts.td
+++ b/clang/tools/clang-scan-deps/Opts.td
@@ -29,7 +29,7 @@ defm compilation_database : Eq<"compilation-database", "Compilation database">;
defm module_name : Eq<"module-name", "the module of which the dependencies are to be computed">;
defm dependency_target : Eq<"dependency-target", "The names of dependency targets for the dependency file">;
-defm tu_buffer_path: Eq<"tu-buffer-path", "The path to the translation unit for depscan. Not compatible with -module-name">;
+defm vfs_overlay_path: Eq<"vfs-overlay-path", "The path to a VFS overlay file that is used to scan the translation unit. Not compatible with -module-name">;
def deprecated_driver_command : F<"deprecated-driver-command", "use a single driver command to build the tu (deprecated)">;
|
|
What's the use-case for this? Isn't this equivalent to adding an |
This is for a better swift dependency scanning. It is not the same as vfsoverlay since the file will be virtual and not existing on disk. |
|
So Swift will provide an in-memory VFS and a command-line whose input points into the VFS? |
Yes. That is the plan. |
jansvoboda11
approved these changes
Sep 18, 2025
|
Abandon as I decide to go the other direction. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make
getTranslationUnitDependenciesin clang depedency scanner moreflexible. Instead of taking an optional buffer as translation unit, take
an optional overlay VFS instead that can provide any kind of virtual
file buffer to the depedency scanning tool.