Support: Add vfs::OutputBackend and OutputFile to virtualize compiler outputs #113363
+3,215
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Created using spr 1.3.5 [skip ci]
Created using spr 1.3.5
Member
|
@llvm/pr-subscribers-llvm-support Author: Steven Wu (cachemeifyoucan) ChangesAdd OutputBackend and OutputFile to the The headers are:
OutputFile is moveable and owns an OutputFileImpl, which is provided by
OutputBackend can be stored in IntrusiveRefCntPtr.
OutputConfig has flags to configure the output. Backends may ignore or
This patch intentionally lacks a number of important features that have
Patch originally by dexonsmith Patch is 108.08 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/113363.diff 18 Files Affected:
diff --git a/llvm/include/llvm/Support/HashingOutputBackend.h b/llvm/include/llvm/Support/HashingOutputBackend.h
new file mode 100644
index 00000000000000..d2e79663f55266
--- /dev/null
+++ b/llvm/include/llvm/Support/HashingOutputBackend.h
@@ -0,0 +1,112 @@
+//===- HashingOutputBackends.h - Hashing output backends --------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_HASHINGOUTPUTBACKEND_H
+#define LLVM_SUPPORT_HASHINGOUTPUTBACKEND_H
+
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/HashBuilder.h"
+#include "llvm/Support/VirtualOutputBackend.h"
+#include "llvm/Support/VirtualOutputConfig.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace llvm::vfs {
+
+/// raw_pwrite_stream that writes to a hasher.
+template <typename HasherT>
+class HashingStream : public llvm::raw_pwrite_stream {
+private:
+ SmallVector<char> Buffer;
+ raw_svector_ostream OS;
+
+ using HashBuilderT = HashBuilder<HasherT, support::endianness::native>;
+ HashBuilderT Builder;
+
+ void write_impl(const char *Ptr, size_t Size) override {
+ OS.write(Ptr, Size);
+ }
+
+ void pwrite_impl(const char *Ptr, size_t Size, uint64_t Offset) override {
+ OS.pwrite(Ptr, Size, Offset);
+ }
+
+ uint64_t current_pos() const override { return OS.str().size(); }
+
+public:
+ HashingStream() : OS(Buffer) { SetUnbuffered(); }
+
+ auto final() {
+ Builder.update(OS.str());
+ return Builder.final();
+ }
+};
+
+template <typename HasherT> class HashingOutputFile;
+
+/// An output backend that only generates the hash for outputs.
+template <typename HasherT> class HashingOutputBackend : public OutputBackend {
+private:
+ friend class HashingOutputFile<HasherT>;
+ void addOutputFile(StringRef Path, StringRef Hash) {
+ OutputHashes[Path] = std::string(Hash);
+ }
+
+protected:
+ IntrusiveRefCntPtr<OutputBackend> cloneImpl() const override {
+ return const_cast<HashingOutputBackend<HasherT> *>(this);
+ }
+
+ Expected<std::unique_ptr<OutputFileImpl>>
+ createFileImpl(StringRef Path, std::optional<OutputConfig> Config) override {
+ return std::make_unique<HashingOutputFile<HasherT>>(Path, *this);
+ }
+
+public:
+ /// Iterator for all the output file names.
+ auto outputFiles() const { return OutputHashes.keys(); }
+
+ /// Get hash value for the output files in hex representation.
+ /// Return None if the requested path is not generated.
+ std::optional<std::string> getHashValueForFile(StringRef Path) {
+ auto F = OutputHashes.find(Path);
+ if (F == OutputHashes.end())
+ return std::nullopt;
+ return toHex(F->second);
+ }
+
+private:
+ StringMap<std::string> OutputHashes;
+};
+
+/// HashingOutputFile.
+template <typename HasherT>
+class HashingOutputFile final : public OutputFileImpl {
+public:
+ Error keep() override {
+ auto Result = OS.final();
+ Backend.addOutputFile(OutputPath, toStringRef(Result));
+ return Error::success();
+ }
+ Error discard() override { return Error::success(); }
+ raw_pwrite_stream &getOS() override { return OS; }
+
+ HashingOutputFile(StringRef OutputPath,
+ HashingOutputBackend<HasherT> &Backend)
+ : OutputPath(OutputPath.str()), Backend(Backend) {}
+
+private:
+ const std::string OutputPath;
+ HashingStream<HasherT> OS;
+ HashingOutputBackend<HasherT> &Backend;
+};
+
+} // namespace llvm::vfs
+
+#endif // LLVM_SUPPORT_HASHINGOUTPUTBACKEND_H
diff --git a/llvm/include/llvm/Support/VirtualOutputBackend.h b/llvm/include/llvm/Support/VirtualOutputBackend.h
new file mode 100644
index 00000000000000..2328252c7054fc
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualOutputBackend.h
@@ -0,0 +1,62 @@
+//===- VirtualOutputBackend.h - Output virtualization -----------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_VIRTUALOUTPUTBACKEND_H
+#define LLVM_SUPPORT_VIRTUALOUTPUTBACKEND_H
+
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualOutputConfig.h"
+#include "llvm/Support/VirtualOutputFile.h"
+
+namespace llvm::vfs {
+
+/// Interface for virtualized outputs.
+///
+/// If virtual functions are added here, also add them to \a
+/// ProxyOutputBackend.
+class OutputBackend : public RefCountedBase<OutputBackend> {
+ virtual void anchor();
+
+public:
+ /// Get a backend that points to the same destination as this one but that
+ /// has independent settings.
+ ///
+ /// Not thread-safe, but all operations are thread-safe when performed on
+ /// separate clones of the same backend.
+ IntrusiveRefCntPtr<OutputBackend> clone() const { return cloneImpl(); }
+
+ /// Create a file. If \p Config is \c std::nullopt, uses the backend's default
+ /// OutputConfig (may match \a OutputConfig::OutputConfig(), or may
+ /// have been customized).
+ ///
+ /// Thread-safe.
+ Expected<OutputFile>
+ createFile(const Twine &Path,
+ std::optional<OutputConfig> Config = std::nullopt);
+
+protected:
+ /// Must be thread-safe. Virtual function has a different name than \a
+ /// clone() so that implementations can override the return value.
+ virtual IntrusiveRefCntPtr<OutputBackend> cloneImpl() const = 0;
+
+ /// Create a file for \p Path. Must be thread-safe.
+ ///
+ /// \pre \p Config is valid or std::nullopt.
+ virtual Expected<std::unique_ptr<OutputFileImpl>>
+ createFileImpl(StringRef Path, std::optional<OutputConfig> Config) = 0;
+
+ OutputBackend() = default;
+
+public:
+ virtual ~OutputBackend() = default;
+};
+
+} // namespace llvm::vfs
+
+#endif // LLVM_SUPPORT_VIRTUALOUTPUTBACKEND_H
diff --git a/llvm/include/llvm/Support/VirtualOutputBackends.h b/llvm/include/llvm/Support/VirtualOutputBackends.h
new file mode 100644
index 00000000000000..6f702000d77b3e
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualOutputBackends.h
@@ -0,0 +1,110 @@
+//===- VirtualOutputBackends.h - Virtual output backends --------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_VIRTUALOUTPUTBACKENDS_H
+#define LLVM_SUPPORT_VIRTUALOUTPUTBACKENDS_H
+
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+#include "llvm/Support/VirtualOutputBackend.h"
+#include "llvm/Support/VirtualOutputConfig.h"
+
+namespace llvm::vfs {
+
+/// Create a backend that ignores all output.
+IntrusiveRefCntPtr<OutputBackend> makeNullOutputBackend();
+
+/// Make a backend where \a OutputBackend::createFile() forwards to
+/// \p UnderlyingBackend when \p Filter is true, and otherwise returns a
+/// \a NullOutput.
+IntrusiveRefCntPtr<OutputBackend> makeFilteringOutputBackend(
+ IntrusiveRefCntPtr<OutputBackend> UnderlyingBackend,
+ std::function<bool(StringRef, std::optional<OutputConfig>)> Filter);
+
+/// Create a backend that forwards \a OutputBackend::createFile() to both \p
+/// Backend1 and \p Backend2 and sends content to both places.
+IntrusiveRefCntPtr<OutputBackend>
+makeMirroringOutputBackend(IntrusiveRefCntPtr<OutputBackend> Backend1,
+ IntrusiveRefCntPtr<OutputBackend> Backend2);
+
+/// A helper class for proxying another backend, with the default
+/// implementation to forward to the underlying backend.
+class ProxyOutputBackend : public OutputBackend {
+ void anchor() override;
+
+protected:
+ // Require subclass to implement cloneImpl().
+ //
+ // IntrusiveRefCntPtr<OutputBackend> cloneImpl() const override;
+
+ Expected<std::unique_ptr<OutputFileImpl>>
+ createFileImpl(StringRef Path, std::optional<OutputConfig> Config) override {
+ OutputFile File;
+ if (Error E = UnderlyingBackend->createFile(Path, Config).moveInto(File))
+ return std::move(E);
+ return File.takeImpl();
+ }
+
+ OutputBackend &getUnderlyingBackend() const { return *UnderlyingBackend; }
+
+public:
+ ProxyOutputBackend(IntrusiveRefCntPtr<OutputBackend> UnderlyingBackend)
+ : UnderlyingBackend(std::move(UnderlyingBackend)) {
+ assert(this->UnderlyingBackend && "Expected non-null backend");
+ }
+
+private:
+ IntrusiveRefCntPtr<OutputBackend> UnderlyingBackend;
+};
+
+/// An output backend that creates files on disk, wrapping APIs in sys::fs.
+class OnDiskOutputBackend : public OutputBackend {
+ void anchor() override;
+
+protected:
+ IntrusiveRefCntPtr<OutputBackend> cloneImpl() const override {
+ return clone();
+ }
+
+ Expected<std::unique_ptr<OutputFileImpl>>
+ createFileImpl(StringRef Path, std::optional<OutputConfig> Config) override;
+
+public:
+ /// Resolve an absolute path.
+ Error makeAbsolute(SmallVectorImpl<char> &Path) const;
+
+ /// On disk output settings.
+ struct OutputSettings {
+ /// Register output files to be deleted if a signal is received. Also
+ /// disabled for outputs with \a OutputConfig::getNoDiscardOnSignal().
+ bool DisableRemoveOnSignal = false;
+
+ /// Disable temporary files. Also disabled for outputs with \a
+ /// OutputConfig::getNoAtomicWrite().
+ bool DisableTemporaries = false;
+
+ // Default configuration for this backend.
+ OutputConfig DefaultConfig;
+ };
+
+ IntrusiveRefCntPtr<OnDiskOutputBackend> clone() const {
+ auto Clone = makeIntrusiveRefCnt<OnDiskOutputBackend>();
+ Clone->Settings = Settings;
+ return Clone;
+ }
+
+ OnDiskOutputBackend() = default;
+
+ /// Settings for this backend.
+ ///
+ /// Access is not thread-safe.
+ OutputSettings Settings;
+};
+
+} // namespace llvm::vfs
+
+#endif // LLVM_SUPPORT_VIRTUALOUTPUTBACKENDS_H
diff --git a/llvm/include/llvm/Support/VirtualOutputConfig.def b/llvm/include/llvm/Support/VirtualOutputConfig.def
new file mode 100644
index 00000000000000..0b6a765cbd4c00
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualOutputConfig.def
@@ -0,0 +1,26 @@
+//===- VirtualOutputConfig.def - Virtual output config defs -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef HANDLE_OUTPUT_CONFIG_FLAG
+#error "Missing macro definition of HANDLE_OUTPUT_CONFIG_FLAG"
+#endif
+
+// Define HANDLE_OUTPUT_CONFIG_FLAG before including.
+//
+// #define HANDLE_OUTPUT_CONFIG_FLAG(NAME, DEFAULT)
+
+HANDLE_OUTPUT_CONFIG_FLAG(Text, false) // OF_Text.
+HANDLE_OUTPUT_CONFIG_FLAG(CRLF, false) // OF_CRLF.
+HANDLE_OUTPUT_CONFIG_FLAG(Append, false) // OF_Append.
+HANDLE_OUTPUT_CONFIG_FLAG(DiscardOnSignal, true) // E.g., RemoveFileOnSignal.
+HANDLE_OUTPUT_CONFIG_FLAG(AtomicWrite, true) // E.g., use temporaries.
+HANDLE_OUTPUT_CONFIG_FLAG(ImplyCreateDirectories, true)
+// Skip atomic write if existing file content is the same
+HANDLE_OUTPUT_CONFIG_FLAG(OnlyIfDifferent, false)
+
+#undef HANDLE_OUTPUT_CONFIG_FLAG
diff --git a/llvm/include/llvm/Support/VirtualOutputConfig.h b/llvm/include/llvm/Support/VirtualOutputConfig.h
new file mode 100644
index 00000000000000..d93bbf5ca63a03
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualOutputConfig.h
@@ -0,0 +1,91 @@
+//===- VirtualOutputConfig.h - Virtual output configuration -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_VIRTUALOUTPUTCONFIG_H
+#define LLVM_SUPPORT_VIRTUALOUTPUTCONFIG_H
+
+#include "llvm/Support/Compiler.h"
+#include <initializer_list>
+
+namespace llvm {
+
+class raw_ostream;
+
+namespace sys {
+namespace fs {
+enum OpenFlags : unsigned;
+} // end namespace fs
+} // end namespace sys
+
+namespace vfs {
+
+namespace detail {
+/// Unused and empty base class to allow OutputConfig constructor to be
+/// constexpr, with commas before every field's initializer.
+struct EmptyBaseClass {};
+} // namespace detail
+
+/// Full configuration for an output for use by the \a OutputBackend. Each
+/// configuration flag is either \c true or \c false.
+struct OutputConfig : detail::EmptyBaseClass {
+public:
+ void print(raw_ostream &OS) const;
+ void dump() const;
+
+#define HANDLE_OUTPUT_CONFIG_FLAG(NAME, DEFAULT) \
+ constexpr bool get##NAME() const { return NAME; } \
+ constexpr bool getNo##NAME() const { return !NAME; } \
+ constexpr OutputConfig &set##NAME(bool Value) { \
+ NAME = Value; \
+ return *this; \
+ } \
+ constexpr OutputConfig &set##NAME() { return set##NAME(true); } \
+ constexpr OutputConfig &setNo##NAME() { return set##NAME(false); }
+#include "llvm/Support/VirtualOutputConfig.def"
+
+ constexpr OutputConfig &setBinary() { return setNoText().setNoCRLF(); }
+ constexpr OutputConfig &setTextWithCRLF() { return setText().setCRLF(); }
+ constexpr OutputConfig &setTextWithCRLF(bool Value) {
+ return Value ? setText().setCRLF() : setBinary();
+ }
+ constexpr bool getTextWithCRLF() const { return getText() && getCRLF(); }
+ constexpr bool getBinary() const { return !getText(); }
+
+ /// Updates Text and CRLF flags based on \a sys::fs::OF_Text and \a
+ /// sys::fs::OF_CRLF in \p Flags. Rejects CRLF without Text (calling
+ /// \a setBinary()).
+ OutputConfig &setOpenFlags(const sys::fs::OpenFlags &Flags);
+
+ constexpr OutputConfig()
+ : EmptyBaseClass()
+#define HANDLE_OUTPUT_CONFIG_FLAG(NAME, DEFAULT) , NAME(DEFAULT)
+#include "llvm/Support/VirtualOutputConfig.def"
+ {
+ }
+
+ constexpr bool operator==(OutputConfig RHS) const {
+#define HANDLE_OUTPUT_CONFIG_FLAG(NAME, DEFAULT) \
+ if (NAME != RHS.NAME) \
+ return false;
+#include "llvm/Support/VirtualOutputConfig.def"
+ return true;
+ }
+ constexpr bool operator!=(OutputConfig RHS) const { return !operator==(RHS); }
+
+private:
+#define HANDLE_OUTPUT_CONFIG_FLAG(NAME, DEFAULT) bool NAME : 1;
+#include "llvm/Support/VirtualOutputConfig.def"
+};
+
+} // namespace vfs
+
+raw_ostream &operator<<(raw_ostream &OS, vfs::OutputConfig Config);
+
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_VIRTUALOUTPUTCONFIG_H
diff --git a/llvm/include/llvm/Support/VirtualOutputError.h b/llvm/include/llvm/Support/VirtualOutputError.h
new file mode 100644
index 00000000000000..5459fae4552d51
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualOutputError.h
@@ -0,0 +1,134 @@
+//===- VirtualOutputError.h - Errors for output virtualization --*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_VIRTUALOUTPUTERROR_H
+#define LLVM_SUPPORT_VIRTUALOUTPUTERROR_H
+
+#include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualOutputConfig.h"
+
+namespace llvm::vfs {
+
+const std::error_category &output_category();
+
+enum class OutputErrorCode {
+ // Error code 0 is absent. Use std::error_code() instead.
+ not_closed = 1,
+ invalid_config,
+ already_closed,
+ has_open_proxy,
+};
+
+inline std::error_code make_error_code(OutputErrorCode EV) {
+ return std::error_code(static_cast<int>(EV), output_category());
+}
+
+/// Error related to an \a OutputFile. Derives from \a ECError and adds \a
+/// getOutputPath().
+class OutputError : public ErrorInfo<OutputError, ECError> {
+ void anchor() override;
+
+public:
+ StringRef getOutputPath() const { return OutputPath; }
+ void log(raw_ostream &OS) const override {
+ OS << getOutputPath() << ": ";
+ ECError::log(OS);
+ }
+
+ // Used by ErrorInfo::classID.
+ static char ID;
+
+ OutputError(const Twine &OutputPath, std::error_code EC)
+ : ErrorInfo<OutputError, ECError>(EC), OutputPath(OutputPath.str()) {
+ assert(EC && "Cannot create OutputError from success EC");
+ }
+
+ OutputError(const Twine &OutputPath, OutputErrorCode EV)
+ : ErrorInfo<OutputError, ECError>(make_error_code(EV)),
+ OutputPath(OutputPath.str()) {
+ assert(EC && "Cannot create OutputError from success EC");
+ }
+
+private:
+ std::string OutputPath;
+};
+
+/// Return \a Error::success() or use \p OutputPath to create an \a
+/// OutputError, depending on \p EC.
+inline Error convertToOutputError(const Twine &OutputPath, std::error_code EC) {
+ if (EC)
+ return make_error<OutputError>(OutputPath, EC);
+ return Error::success();
+}
+
+/// Error related to an OutputConfig for an \a OutputFile. Derives from \a
+/// OutputError and adds \a getConfig().
+class OutputConfigError : public ErrorInfo<OutputConfigError, OutputError> {
+ void anchor() override;
+
+public:
+ OutputConfig getConfig() const { return Config; }
+ void log(raw_ostream &OS) const override {
+ OutputError::log(OS);
+ OS << ": " << Config;
+ }
+
+ // Used by ErrorInfo::classID.
+ static char ID;
+
+ OutputConfigError(OutputConfig Config, const Twine &OutputPath)
+ : ErrorInfo<OutputConfigError, OutputError>(
+ OutputPath, OutputErrorCode::invalid_config),
+ Config(Config) {}
+
+private:
+ OutputConfig Config;
+};
+
+/// Error related to a temporary file for an \a OutputFile. Derives from \a
+/// OutputError and adds \a getTempPath().
+class TempFileOutputError : public ErrorInfo<TempFileOutputError, OutputError> {
+ void anchor() override;
+
+public:
+ StringRef getTempPath() const { return TempPath; }
+ void log(raw_ostream &OS) const override {
+ OS << getTempPath() << " => ";
+ OutputError::log(OS);
+ }
+
+ // Used by ErrorInfo::classID.
+ static char ID;
+
+ TempFileOutputError(const Twine &TempPath, const Twine &OutputPath,
+ std::error_code EC)
+ : ErrorInfo<TempFileOutputError, OutputError>(OutputPath, EC),
+ TempPath(TempPath.str()) {}
+
+ TempFileOutputError(const Twine &TempPath, const Twine &OutputPath,
+ OutputErrorCode EV)
+ : ErrorInfo<TempFileOutputError, OutputError>(OutputPath, EV),
+ TempPath(TempPath.str()) {}
+
+private:
+ std::string TempPath;
+};
+
+/// Return \a Error::success() or use \p TempPath and \p OutputPath to create a
+/// \a TempFileOutputError, depending on \p EC.
+inline Error convertToTempFileOutputError(const Twine &TempPath,
+ const Twine &OutputPath,
+ std::error_code EC) {
+ if (EC)
+ return make_error<TempFileOutputError>(TempPath, OutputPath, EC);
+ return Error::success();
+}
+
+} // namespace llvm::vfs
+
+#endif // LLVM_SUPPORT_VIRTUALOUTPUTERROR_H
diff --git a/llvm/include/llvm/Support/VirtualOutputFile.h b/llvm/include/llvm/Support/VirtualOutputFile.h
new file mode 100644
index 00000000000000..0bf6c58f30484d
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualOutputFile.h
@@ -0,0 +1,162 @@
+//===- VirtualOutputFile.h - Output file virtualization ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===------------------------------------------------------------...
[truncated]
|
Created using spr 1.3.6 [skip ci]
Created using spr 1.3.6
cachemeifyoucan
changed the base branch from
users/cachemeifyoucan/spr/main.support-add-vfsoutputbackend-and-outputfile-to-virtualize-compiler-outputs
to
main
cachemeifyoucan
requested review from
adrian-prantl,
benlangmuir,
bogner and
ilovepi
Created using spr 1.3.6
Collaborator
Author
|
ping |
aganea
reviewed
Sep 2, 2025
| //===----------------------------------------------------------------------===// | ||
| /// | ||
| /// \file | ||
| /// This file contains the declarations of the VirtualOutputBackend class. |
Member
There was a problem hiding this comment.
I would rather have a short paragraph here about what the purpose of this file is, not what classes it declares? Same for the other files. The goal would be to create a quick mental model before actually browsing through the code.
Collaborator
Author
There was a problem hiding this comment.
I added more explanations. Please review and let me know if needs more improvements.
| struct ProxyOutputBackend2 : public ProxyOutputBackend { | ||
| using ProxyOutputBackend::ProxyOutputBackend; | ||
| }; | ||
| struct MirroringOutput final : public OutputFileImpl, raw_pwrite_stream { |
Member
There was a problem hiding this comment.
Can you please comment this class or bikeshed the naming a bit? It is seems it's more about demultiplexing, or writing concurently (from the API perspective) to several files.
Collaborator
Author
There was a problem hiding this comment.
This is not really demultiplexing, but duplicating. I clarify a bit in the header file, not sure if there is a better name but open to suggestions.
| }); | ||
| } | ||
|
|
||
| Error OnDiskOutputFile::initializeFD(std::optional<int> &FD) { |
Member
There was a problem hiding this comment.
nit: FD is a Unix nomenclature, the Windows equivalent is a Handle. If ever you have better ideas for a more broad/inclusive API naming.
Collaborator
Author
There was a problem hiding this comment.
Rename to initializeFile but keep the variable name since we still convert fileHandle to FD in LLVM sys::fs APIs.
aganea
reviewed
Sep 2, 2025
| }); | ||
| } | ||
|
|
||
| Error OnDiskOutputFile::tryToCreateTemporary(std::optional<int> &FD) { |
Member
There was a problem hiding this comment.
Ah I see, it wasn't clear for me at first that a lot of code here is a copy/rework of CompilerInstance::createOutputFileImpl()
Collaborator
Author
There was a problem hiding this comment.
Yes, correct. This re-implements the clang output file logics. The follow up is to adopt clang to use it. Other tools can use a unified output file logics in LLVM as well.
Created using spr 1.3.6
adrian-prantl
approved these changes
Sep 2, 2025
| template <typename HasherT> class HashingOutputBackend : public OutputBackend { | ||
| private: | ||
| friend class HashingOutputFile<HasherT>; | ||
| void addOutputFile(StringRef Path, StringRef Hash) { |
Collaborator
There was a problem hiding this comment.
can we move this implementation into the cpp file?
Collaborator
Author
There was a problem hiding this comment.
This is a template class on HasherType.
Created using spr 1.3.6
cachemeifyoucan
added a commit
to cachemeifyoucan/llvm-project
that referenced
this pull request
Sep 3, 2025
… outputs
Add OutputBackend and OutputFile to the `llvm::vfs` namespace for
virtualizing compiler outputs. This is intended for use in Clang,
The headers are:
- llvm/Support/VirtualOutputConfig.h
- llvm/Support/VirtualOutputError.h
- llvm/Support/VirtualOutputFile.h
- llvm/Support/VirtualOutputBackend.h
OutputFile is moveable and owns an OutputFileImpl, which is provided by
the derived OutputBackend.
- OutputFileImpl::keep() and OutputFileImpl::discard() should keep or
discard the output. OutputFile guarantees that exactly one of these
will be called before destruction.
- OutputFile::keep() and OutputFile::discard() wrap OutputFileImpl
and catch usage errors such as double-close.
- OutputFile::discardOnDestroy() installs an error handler for the
destructor to use if the file is still open. The handler will be
called if discard() fails.
- OutputFile::~OutputFile() calls report_fatal_error() if none of
keep(), discard(), or discardOnDestroy() has been called. It still
calls OutputFileImpl::discard().
- getOS() returns the wrapped raw_pwrite_stream. For convenience,
OutputFile has an implicit conversion to `raw_ostream` and
`raw_ostream &operator<<(OutputFile&, T&&)`.
OutputBackend can be stored in IntrusiveRefCntPtr.
- Most operations are thread-safe.
- clone() returns a backend that targets the same destination.
All operations are thread-safe when done on different clones.
- createFile() takes a path and an OutputConfig (see below) and returns
an OutputFile. Backends implement createFileImpl().
OutputConfig has flags to configure the output. Backends may ignore or
override flags that aren't relevant or implementable.
- The initial flags are:
- AtomicWrite: whether the output should appear atomically (e.g., by
using a temporary file and renaming it).
- CrashCleanup: whether the output should be cleaned up if there's a
crash (e.g., with RemoveFileOnSignal).
- ImplyCreateDirectories: whether to implicitly create missing
directories in the path to the file.
- Text: matches sys::fs::OF_Text.
- CRLF: matches sys::fs::OF_CRLF.
- Append: matches sys::fs::OF_Append and can use with AtomicWrite
for atomic append.
- OnlyIfDifferent: skip writting the output file if the existing file
at the output path is identical to the content to be written.
- Each "Flag" has `setFlag(bool)` and `bool getFlag()` and shortcuts
`setFlag()` and `setNoFlag()`. The setters are `constexpr` and return
`OutputConfig&` to make it easy to declare a default value for a filed
in a class or struct.
- Setters and getters for Binary and TextWithCRLF are derived from Text
and CRLF. For convenience, sys::fs::OpenFlags can be passed
directly to setOpenFlags().
This patch intentionally lacks a number of important features that have
been left for follow-ups:
- Set a (virtual) current working directory.
- Create a directory.
- Create a file or directory with a unique name (avoiding collisions
with existing filenames).
Patch originally by dexonsmith
Reviewers:
Pull Request: llvm#113363
cachemeifyoucan
deleted the
users/cachemeifyoucan/spr/support-add-vfsoutputbackend-and-outputfile-to-virtualize-compiler-outputs
branch
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/140/builds/30194 Here is the relevant piece of the build log for the reference |
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/168/builds/15920 Here is the relevant piece of the build log for the reference |
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/95/builds/17944 Here is the relevant piece of the build log for the reference |
cachemeifyoucan
added a commit
that referenced
this pull request
Sep 5, 2025
Drop few `constexpr` from the change that causes internal compiler error in some GCC versions used in some buildbot.
Collaborator
Author
|
Looks like GCC ICE on some If that doesn't work, I will revert. |
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/76/builds/12686 Here is the relevant piece of the build log for the reference |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add OutputBackend and OutputFile to the
llvm::vfsnamespace forvirtualizing compiler outputs. This is intended for use in Clang,
The headers are:
OutputFile is moveable and owns an OutputFileImpl, which is provided by
the derived OutputBackend.
discard the output. OutputFile guarantees that exactly one of these
will be called before destruction.
and catch usage errors such as double-close.
destructor to use if the file is still open. The handler will be
called if discard() fails.
keep(), discard(), or discardOnDestroy() has been called. It still
calls OutputFileImpl::discard().
OutputFile has an implicit conversion to
raw_ostreamandraw_ostream &operator<<(OutputFile&, T&&).OutputBackend can be stored in IntrusiveRefCntPtr.
All operations are thread-safe when done on different clones.
an OutputFile. Backends implement createFileImpl().
OutputConfig has flags to configure the output. Backends may ignore or
override flags that aren't relevant or implementable.
using a temporary file and renaming it).
crash (e.g., with RemoveFileOnSignal).
directories in the path to the file.
for atomic append.
at the output path is identical to the content to be written.
setFlag(bool)andbool getFlag()and shortcutssetFlag()andsetNoFlag(). The setters areconstexprand returnOutputConfig&to make it easy to declare a default value for a filedin a class or struct.
and CRLF. For convenience, sys::fs::OpenFlags can be passed
directly to setOpenFlags().
This patch intentionally lacks a number of important features that have
been left for follow-ups:
with existing filenames).
Patch originally by dexonsmith