Skip to content

Commit 42d2976

Browse files
chandlercdanakj
andauthored
Introduce a custom filesystem library (#5888)
The standard filesystem API lacks significant functionality, ranging from correct and secure creation of directories and files within them by using `openat` and avoiding [TOCTOU] issues, to support for filesystem locking. [TOCTOU]: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use The LLVM filesystem library has more functionality, but uses an API that is increasingly diverging from the standard, and also fails to defend against TOCTOU. This library is designed to carefully model the Unix or POSIX filesystem concepts of `openat` to avoid TOCTOU. However, it also tries to limit itself to an API subset that LLVM's filesystem library has also implemneted and so we have a strong reason to expect to be possible to port to Windows reasonably. This PR included several benchmarks that show that this implementation is also faster for the majority of operations than the C++ standard library. The only places where there is a consistent regression is in recursively creating directories, and this is directly connected to the approach of using `openat` as the basis. Even there, while the wall time regresses, the cycles and instructions are significantly improved. There are a number of operations not yet included here, I've focused on a core set of opening, closing, creating, and removing, and then adding those that I saw the current toolchain code using actively. I'll plan to expand the operations as needed going forward. A follow-up PR that I'll finish polishing and send next ports `//toolchain/install` to consistently use this library and `std::filesystem::path` to both exercise the library and showcase its use. I'll be working systematically across the toolchain to converge all the code, extending this library as needed. For reference, benchmark results on my macOS laptop: https://gist.github.com/chandlerc/29d1f4d465a835b8be5174a48dad2e8f Benchmark results on a Asahi Linux M1 Mac Mini: https://gist.github.com/chandlerc/c42d43dd6b9b91746ab314b2afa152f7 Benchmark results on a Linux server with weirdly slow FS operations: https://gist.github.com/chandlerc/48301a7383eb3972d53351b7e35e0561 --------- Co-authored-by: Dana Jansens <[email protected]>
1 parent b410ebd commit 42d2976

File tree

5 files changed

+2995
-0
lines changed

5 files changed

+2995
-0
lines changed

common/BUILD

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,56 @@ cc_test(
214214
],
215215
)
216216

217+
cc_library(
218+
name = "filesystem",
219+
srcs = ["filesystem.cpp"],
220+
hdrs = ["filesystem.h"],
221+
deps = [
222+
":build_data",
223+
":check",
224+
":error",
225+
":ostream",
226+
":raw_string_ostream",
227+
":template_string",
228+
"@llvm-project//llvm:Support",
229+
],
230+
)
231+
232+
cc_test(
233+
name = "filesystem_test",
234+
size = "small",
235+
srcs = ["filesystem_test.cpp"],
236+
deps = [
237+
":error_test_helpers",
238+
":filesystem",
239+
"//testing/base:gtest_main",
240+
"@googletest//:gtest",
241+
"@llvm-project//llvm:Support",
242+
],
243+
)
244+
245+
cc_binary(
246+
name = "filesystem_benchmark",
247+
testonly = 1,
248+
srcs = ["filesystem_benchmark.cpp"],
249+
deps = [
250+
":check",
251+
":filesystem",
252+
"//testing/base:benchmark_main",
253+
"@abseil-cpp//absl/hash",
254+
"@abseil-cpp//absl/random",
255+
"@google_benchmark//:benchmark",
256+
"@llvm-project//llvm:Support",
257+
],
258+
)
259+
260+
sh_test(
261+
name = "filesystem_benchmark_test",
262+
size = "small",
263+
srcs = [":filesystem_benchmark"],
264+
args = ["--benchmark_min_time=1x"],
265+
)
266+
217267
cc_library(
218268
name = "find",
219269
hdrs = ["find.h"],

0 commit comments

Comments
 (0)