-
Notifications
You must be signed in to change notification settings - Fork 14.7k
Work around documented Linux mmap bug. #152595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
On Linux, mmap doesn't always zero-fill slack bytes, despite being required to do so by POSIX. If the final page of a file is in the page cache and the bytes past the end of the file get overwritten by some process, those bytes then remain non-zero until the page falls out of the cache or another process overwrites them. Stop trusting that mmap behaves properly on Linux and instead check whether the buffer was indeed properly terminated. If not, fall back to using `read` to read the file contents. This fixes an obscure clang crash bug that can occur if another program (such as an editor) mmap's a source file and writes past the end of the mmap'd region shortly before clang or clangd attempts to parse the file.
@llvm/pr-subscribers-llvm-support Author: Richard Smith (zygoloid) ChangesOn Linux, mmap doesn't always zero-fill slack bytes, despite being required to do so by POSIX. If the final page of a file is in the page cache and the bytes past the end of the file get overwritten by some process, those bytes then remain non-zero until the page falls out of the cache or another process overwrites them. Stop trusting that mmap behaves properly on Linux and instead check whether the buffer was indeed properly terminated. If not, fall back to using This fixes an obscure clang crash bug that can occur if another program (such as an editor) mmap's a source file and writes past the end of the mmap'd region shortly before clang or clangd attempts to parse the file. Full diff: https://github.com/llvm/llvm-project/pull/152595.diff 1 Files Affected:
diff --git a/llvm/lib/Support/MemoryBuffer.cpp b/llvm/lib/Support/MemoryBuffer.cpp
index 601f11f6d23c8..ac686b5fb9099 100644
--- a/llvm/lib/Support/MemoryBuffer.cpp
+++ b/llvm/lib/Support/MemoryBuffer.cpp
@@ -501,8 +501,15 @@ getOpenFileImpl(sys::fs::file_t FD, const Twine &Filename, uint64_t FileSize,
std::unique_ptr<MB> Result(
new (NamedBufferAlloc(Filename)) MemoryBufferMMapFile<MB>(
RequiresNullTerminator, FD, MapSize, Offset, EC));
- if (!EC)
- return std::move(Result);
+ if (!EC) {
+#ifdef __linux__
+ // On Linux, mmap may return pages from the page cache that are not
+ // properly filled with trailing zeroes, if some prior user of the page
+ // wrote non-zero bytes. Detect this and don't use mmap in that case.
+ if (!RequiresNullTerminator || *Result->getBufferEnd() == '\0')
+#endif
+ return std::move(Result);
+ }
}
#ifdef __MVS__
|
llvm/lib/Support/MemoryBuffer.cpp
Outdated
if (!EC) | ||
return std::move(Result); | ||
if (!EC) { | ||
#ifdef __linux__ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How sure are we that Linux is the only OS which has this issue? Would it make sense to just remove the ifdef?
There's also a race condition here: if another processes modifies the file while it's mapped, you get undefined behavior. But we can't really do much about that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I expect that on any OS, if another process concurrently mmap
s the file with MAP_SHARED
and writes into the slack bytes, or truncates the file, we will probably crash, but Linux is the only OS that I've found that documents that this can happen even if the file is not concurrently modified. I wouldn't say I'm sure that other OSes don't have similar behavior; maybe they just don't document it?
I'm happy to generalize the check if you think it's worthwhile; as far as I can see, the only real consequence is that this would fault in the final page earlier, which is probably not an especially interesting difference given that we're presumably going to reach it pretty soon anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
On Linux, mmap doesn't always zero-fill slack bytes (man page), despite being required to do so by POSIX. If the final page of a file is in the page cache and the bytes past the end of the file get overwritten by some process, those bytes then remain non-zero until the page falls out of the cache or another process overwrites them.
Stop trusting that mmap behaves properly on Linux and instead check whether the buffer was indeed properly terminated. If not, fall back to using
read
to read the file contents.This fixes an obscure clang crash bug that can occur if another program (such as an editor) mmap's a source file and writes past the end of the mmap'd region shortly before clang or clangd attempts to parse the file.