Skip to content

Commit 4594b1f

Browse files
committed
Merge #30
30: Upgrade libfuzzer for LLVM 6.0 r=frewsxcv a=nagisa This PR pulls in the changes to libfuzzer, at commit llvm-mirror/compiler-rt@cc0ab3f. My observation is that this works just fine with current state of affairs. That is, given a: ``` $ rustc -Cllvm-args=-version LLVM (http://llvm.org/): LLVM version 6.0.0 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: znver1 ``` The following command, when run within the both of the `example` folders in this repository work "just" fine (should also be confirmed by the CI), which somewhat counteracts observations made in #29, that libfuzzer cannot work with sanitizers anymore (quite the contrary, it seems that sanitizers are still required). ``` cargo rustc --release -- -Cpasses=sancov -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cpanic=abort -Cllvm-args=-sanitizer-coverage-trace-divs -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Zsanitizer=address ``` If I remember correctly, cargo-fuzz generates fuzz targets with this git repository as a dependency. This means that all these fuzz targets will possibly break for people using older compilers once we upgrade, however not upgrading libfuzzer-sys will(?) break people using the new nightly. While our compatibility story is, I believe, that we support only the current nightly without any real back or forward compatibility, this might be a good time to think about how we want to go about our versioning and release flow. Perhaps we’ll be able to find some way that does not break everybody’s fuzz targets every time LLVMup happens. Fixes https://github.com/rust-fuzz/libfuzzer-sys/issues/29 r? @frewsxcv or @Manishearth cc @PaulGrandperrin
2 parents 737524f + 08b7053 commit 4594b1f

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

61 files changed

+4355
-1501
lines changed

.travis.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ notifications:
1212
email: false
1313
script:
1414
- cd example
15-
- cargo rustc --release -- -C passes='sancov' -C llvm-args='-sanitizer-coverage-level=4' -Z sanitizer=address
15+
- cargo rustc --release -- -Cpasses='sancov' -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cpanic=abort -Cllvm-args=-sanitizer-coverage-trace-divs -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Zsanitizer=address
1616
- (! ./target/release/example -runs=100000)
1717
- cd ../example_arbitrary
18-
- cargo rustc --release -- -C passes='sancov' -C llvm-args='-sanitizer-coverage-level=4' -Z sanitizer=address
18+
- cargo rustc --release -- -Cpasses='sancov' -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cpanic=abort -Cllvm-args=-sanitizer-coverage-trace-divs -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Zsanitizer=address
1919
- (! ./target/release/example -runs=10000000)

README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Barebones wrapper around libFuzzer runtime library.
22

3-
The CPP parts are extracted from llvm git repository with `git filter-branch`.
3+
The CPP parts are extracted from compiler-rt git repository with `git filter-branch`.
44

55
libFuzzer relies on LLVM sanitizer support. The Rust compiler has built-in support for LLVM sanitizer support, for now, it's limited to Linux. As a result, libfuzzer-sys only works on Linux.
66

@@ -50,4 +50,5 @@ $ ./target/debug/fuzzed # runs fuzzing
5050

5151
## License
5252

53-
All files in `llvm` are licensed NCSA. Everything else is dual-licensed Apache 2.0 and MIT.
53+
All files in `libfuzzer` directory are licensed NCSA.
54+
Everything else is dual-licensed Apache 2.0 and MIT.

build.rs

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ fn main() {
1414
println!("cargo:rustc-link-lib=stdc++");
1515
} else {
1616
let mut build = cc::Build::new();
17-
let sources = ::std::fs::read_dir("llvm/lib/Fuzzer")
17+
let sources = ::std::fs::read_dir("libfuzzer")
1818
.expect("listable source directory")
1919
.map(|de| de.expect("file in directory").path())
2020
.filter(|p| p.extension().map(|ext| ext == "cpp") == Some(true))
@@ -24,6 +24,7 @@ fn main() {
2424
}
2525
build.flag("-std=c++11");
2626
build.flag("-fno-omit-frame-pointer");
27+
build.flag("-w");
2728
build.cpp(true);
2829
build.compile("libfuzzer.a");
2930
}

libfuzzer/CMakeLists.txt

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
set(LIBFUZZER_SOURCES
2+
FuzzerClangCounters.cpp
3+
FuzzerCrossOver.cpp
4+
FuzzerDriver.cpp
5+
FuzzerExtFunctionsDlsym.cpp
6+
FuzzerExtFunctionsDlsymWin.cpp
7+
FuzzerExtFunctionsWeak.cpp
8+
FuzzerExtraCounters.cpp
9+
FuzzerIO.cpp
10+
FuzzerIOPosix.cpp
11+
FuzzerIOWindows.cpp
12+
FuzzerLoop.cpp
13+
FuzzerMerge.cpp
14+
FuzzerMutate.cpp
15+
FuzzerSHA1.cpp
16+
FuzzerShmemPosix.cpp
17+
FuzzerShmemWindows.cpp
18+
FuzzerTracePC.cpp
19+
FuzzerUtil.cpp
20+
FuzzerUtilDarwin.cpp
21+
FuzzerUtilFuchsia.cpp
22+
FuzzerUtilLinux.cpp
23+
FuzzerUtilPosix.cpp
24+
FuzzerUtilWindows.cpp
25+
)
26+
27+
CHECK_CXX_SOURCE_COMPILES("
28+
static thread_local int blah;
29+
int main() {
30+
return 0;
31+
}
32+
" HAS_THREAD_LOCAL)
33+
34+
set(LIBFUZZER_CFLAGS ${SANITIZER_COMMON_CFLAGS})
35+
36+
if("${CMAKE_SYSTEM_NAME}" STREQUAL "Linux" AND COMPILER_RT_LIBCXX_PATH)
37+
list(APPEND LIBFUZZER_CFLAGS -nostdinc++ -D_LIBCPP_ABI_VERSION=Fuzzer)
38+
# Remove -stdlib= which is unused when passing -nostdinc++.
39+
string(REGEX REPLACE "-stdlib=[a-zA-Z+]*" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
40+
endif()
41+
42+
append_list_if(COMPILER_RT_HAS_OMIT_FRAME_POINTER_FLAG -fno-omit-frame-pointer LIBFUZZER_CFLAGS)
43+
44+
if (CMAKE_CXX_FLAGS MATCHES "fsanitize-coverage")
45+
list(APPEND LIBFUZZER_CFLAGS -fno-sanitize-coverage=trace-pc-guard,edge,trace-cmp,indirect-calls,8bit-counters)
46+
endif()
47+
48+
if(NOT HAS_THREAD_LOCAL)
49+
list(APPEND LIBFUZZER_CFLAGS -Dthread_local=__thread)
50+
endif()
51+
52+
if(APPLE)
53+
set(FUZZER_SUPPORTED_OS osx)
54+
endif()
55+
56+
add_compiler_rt_object_libraries(RTfuzzer
57+
OS ${FUZZER_SUPPORTED_OS}
58+
ARCHS ${FUZZER_SUPPORTED_ARCH}
59+
SOURCES ${LIBFUZZER_SOURCES}
60+
CFLAGS ${LIBFUZZER_CFLAGS})
61+
62+
add_compiler_rt_object_libraries(RTfuzzer_main
63+
OS ${FUZZER_SUPPORTED_OS}
64+
ARCHS ${FUZZER_SUPPORTED_ARCH}
65+
SOURCES FuzzerMain.cpp
66+
CFLAGS ${LIBFUZZER_CFLAGS})
67+
68+
add_compiler_rt_runtime(clang_rt.fuzzer
69+
STATIC
70+
OS ${FUZZER_SUPPORTED_OS}
71+
ARCHS ${FUZZER_SUPPORTED_ARCH}
72+
OBJECT_LIBS RTfuzzer RTfuzzer_main
73+
CFLAGS ${LIBFUZZER_CFLAGS}
74+
PARENT_TARGET fuzzer)
75+
76+
add_compiler_rt_runtime(clang_rt.fuzzer_no_main
77+
STATIC
78+
OS ${FUZZER_SUPPORTED_OS}
79+
ARCHS ${FUZZER_SUPPORTED_ARCH}
80+
OBJECT_LIBS RTfuzzer
81+
CFLAGS ${LIBFUZZER_CFLAGS}
82+
PARENT_TARGET fuzzer)
83+
84+
if("${CMAKE_SYSTEM_NAME}" STREQUAL "Linux" AND COMPILER_RT_LIBCXX_PATH)
85+
macro(partially_link_libcxx name dir arch)
86+
set(cxx_${arch}_merge_dir "${CMAKE_CURRENT_BINARY_DIR}/cxx_${arch}_merge.dir")
87+
file(MAKE_DIRECTORY ${cxx_${arch}_merge_dir})
88+
add_custom_command(TARGET clang_rt.${name}-${arch} POST_BUILD
89+
COMMAND ${CMAKE_LINKER} --whole-archive "$<TARGET_LINKER_FILE:clang_rt.${name}-${arch}>" --no-whole-archive ${dir}/src/libcxx_fuzzer_${arch}-build/lib/libc++.a -r -o ${name}.o
90+
COMMAND ${CMAKE_OBJCOPY} --localize-hidden ${name}.o
91+
COMMAND ${CMAKE_COMMAND} -E remove "$<TARGET_LINKER_FILE:clang_rt.${name}-${arch}>"
92+
COMMAND ${CMAKE_AR} qcs "$<TARGET_LINKER_FILE:clang_rt.${name}-${arch}>" ${name}.o
93+
WORKING_DIRECTORY ${cxx_${arch}_merge_dir}
94+
)
95+
endmacro()
96+
97+
foreach(arch ${FUZZER_SUPPORTED_ARCH})
98+
get_target_flags_for_arch(${arch} TARGET_CFLAGS)
99+
set(LIBCXX_${arch}_PREFIX ${CMAKE_CURRENT_BINARY_DIR}/libcxx_fuzzer_${arch})
100+
add_custom_libcxx(libcxx_fuzzer_${arch} ${LIBCXX_${arch}_PREFIX}
101+
CFLAGS ${TARGET_CFLAGS}
102+
-D_LIBCPP_ABI_VERSION=Fuzzer
103+
-D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS=1
104+
-fvisibility=hidden
105+
CMAKE_ARGS -DLIBCXX_ENABLE_EXCEPTIONS=OFF
106+
-DLIBCXX_CXX_ABI=none)
107+
target_compile_options(RTfuzzer.${arch} PRIVATE -isystem ${COMPILER_RT_LIBCXX_PATH}/include)
108+
add_dependencies(RTfuzzer.${arch} libcxx_fuzzer_${arch}-build)
109+
target_compile_options(RTfuzzer_main.${arch} PRIVATE -isystem ${COMPILER_RT_LIBCXX_PATH}/include)
110+
add_dependencies(RTfuzzer_main.${arch} libcxx_fuzzer_${arch}-build)
111+
partially_link_libcxx(fuzzer_no_main ${LIBCXX_${arch}_PREFIX} ${arch})
112+
partially_link_libcxx(fuzzer ${LIBCXX_${arch}_PREFIX} ${arch})
113+
endforeach()
114+
endif()
115+
116+
if(COMPILER_RT_INCLUDE_TESTS)
117+
add_subdirectory(tests)
118+
endif()

libfuzzer/FuzzerClangCounters.cpp

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
//===- FuzzerExtraCounters.cpp - Extra coverage counters ------------------===//
2+
//
3+
// The LLVM Compiler Infrastructure
4+
//
5+
// This file is distributed under the University of Illinois Open Source
6+
// License. See LICENSE.TXT for details.
7+
//
8+
//===----------------------------------------------------------------------===//
9+
// Coverage counters from Clang's SourceBasedCodeCoverage.
10+
//===----------------------------------------------------------------------===//
11+
12+
// Support for SourceBasedCodeCoverage is experimental:
13+
// * Works only for the main binary, not DSOs yet.
14+
// * Works only on Linux.
15+
// * Does not implement print_pcs/print_coverage yet.
16+
// * Is not fully evaluated for performance and sensitivity.
17+
// We expect large performance drop due to 64-bit counters,
18+
// and *maybe* better sensitivity due to more fine-grained counters.
19+
// Preliminary comparison on a single benchmark (RE2) shows
20+
// a bit worse sensitivity though.
21+
22+
#include "FuzzerDefs.h"
23+
24+
#if LIBFUZZER_LINUX
25+
__attribute__((weak)) extern uint64_t __start___llvm_prf_cnts;
26+
__attribute__((weak)) extern uint64_t __stop___llvm_prf_cnts;
27+
namespace fuzzer {
28+
uint64_t *ClangCountersBegin() { return &__start___llvm_prf_cnts; }
29+
uint64_t *ClangCountersEnd() { return &__stop___llvm_prf_cnts; }
30+
} // namespace fuzzer
31+
#else
32+
// TODO: Implement on Mac (if the data shows it's worth it).
33+
//__attribute__((visibility("hidden")))
34+
//extern uint64_t CountersStart __asm("section$start$__DATA$__llvm_prf_cnts");
35+
//__attribute__((visibility("hidden")))
36+
//extern uint64_t CountersEnd __asm("section$end$__DATA$__llvm_prf_cnts");
37+
namespace fuzzer {
38+
uint64_t *ClangCountersBegin() { return nullptr; }
39+
uint64_t *ClangCountersEnd() { return nullptr; }
40+
} // namespace fuzzer
41+
#endif
42+
43+
namespace fuzzer {
44+
ATTRIBUTE_NO_SANITIZE_ALL
45+
void ClearClangCounters() { // hand-written memset, don't asan-ify.
46+
for (auto P = ClangCountersBegin(); P < ClangCountersEnd(); P++)
47+
*P = 0;
48+
}
49+
}

libfuzzer/FuzzerCommand.h

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
//===- FuzzerCommand.h - Interface representing a process -------*- C++ -* ===//
2+
//
3+
// The LLVM Compiler Infrastructure
4+
//
5+
// This file is distributed under the University of Illinois Open Source
6+
// License. See LICENSE.TXT for details.
7+
//
8+
//===----------------------------------------------------------------------===//
9+
// FuzzerCommand represents a command to run in a subprocess. It allows callers
10+
// to manage command line arguments and output and error streams.
11+
//===----------------------------------------------------------------------===//
12+
13+
#ifndef LLVM_FUZZER_COMMAND_H
14+
#define LLVM_FUZZER_COMMAND_H
15+
16+
#include "FuzzerDefs.h"
17+
#include "FuzzerIO.h"
18+
19+
#include <algorithm>
20+
#include <sstream>
21+
#include <string>
22+
#include <vector>
23+
24+
namespace fuzzer {
25+
26+
class Command final {
27+
public:
28+
// This command line flag is used to indicate that the remaining command line
29+
// is immutable, meaning this flag effectively marks the end of the mutable
30+
// argument list.
31+
static inline const char *ignoreRemainingArgs() {
32+
static const char *kIgnoreRemaining = "-ignore_remaining_args=1";
33+
return kIgnoreRemaining;
34+
}
35+
36+
Command() : CombinedOutAndErr(false) {}
37+
38+
explicit Command(const Vector<std::string> &ArgsToAdd)
39+
: Args(ArgsToAdd), CombinedOutAndErr(false) {}
40+
41+
explicit Command(const Command &Other)
42+
: Args(Other.Args), CombinedOutAndErr(Other.CombinedOutAndErr),
43+
OutputFile(Other.OutputFile) {}
44+
45+
Command &operator=(const Command &Other) {
46+
Args = Other.Args;
47+
CombinedOutAndErr = Other.CombinedOutAndErr;
48+
OutputFile = Other.OutputFile;
49+
return *this;
50+
}
51+
52+
~Command() {}
53+
54+
// Returns true if the given Arg is present in Args. Only checks up to
55+
// "-ignore_remaining_args=1".
56+
bool hasArgument(const std::string &Arg) const {
57+
auto i = endMutableArgs();
58+
return std::find(Args.begin(), i, Arg) != i;
59+
}
60+
61+
// Gets all of the current command line arguments, **including** those after
62+
// "-ignore-remaining-args=1".
63+
const Vector<std::string> &getArguments() const { return Args; }
64+
65+
// Adds the given argument before "-ignore_remaining_args=1", or at the end
66+
// if that flag isn't present.
67+
void addArgument(const std::string &Arg) {
68+
Args.insert(endMutableArgs(), Arg);
69+
}
70+
71+
// Adds all given arguments before "-ignore_remaining_args=1", or at the end
72+
// if that flag isn't present.
73+
void addArguments(const Vector<std::string> &ArgsToAdd) {
74+
Args.insert(endMutableArgs(), ArgsToAdd.begin(), ArgsToAdd.end());
75+
}
76+
77+
// Removes the given argument from the command argument list. Ignores any
78+
// occurrences after "-ignore_remaining_args=1", if present.
79+
void removeArgument(const std::string &Arg) {
80+
auto i = endMutableArgs();
81+
Args.erase(std::remove(Args.begin(), i, Arg), i);
82+
}
83+
84+
// Like hasArgument, but checks for "-[Flag]=...".
85+
bool hasFlag(const std::string &Flag) {
86+
std::string Arg("-" + Flag + "=");
87+
auto IsMatch = [&](const std::string &Other) {
88+
return Arg.compare(0, std::string::npos, Other, 0, Arg.length()) == 0;
89+
};
90+
return std::any_of(Args.begin(), endMutableArgs(), IsMatch);
91+
}
92+
93+
// Returns the value of the first instance of a given flag, or an empty string
94+
// if the flag isn't present. Ignores any occurrences after
95+
// "-ignore_remaining_args=1", if present.
96+
std::string getFlagValue(const std::string &Flag) {
97+
std::string Arg("-" + Flag + "=");
98+
auto IsMatch = [&](const std::string &Other) {
99+
return Arg.compare(0, std::string::npos, Other, 0, Arg.length()) == 0;
100+
};
101+
auto i = endMutableArgs();
102+
auto j = std::find_if(Args.begin(), i, IsMatch);
103+
std::string result;
104+
if (j != i) {
105+
result = j->substr(Arg.length());
106+
}
107+
return result;
108+
}
109+
110+
// Like AddArgument, but adds "-[Flag]=[Value]".
111+
void addFlag(const std::string &Flag, const std::string &Value) {
112+
addArgument("-" + Flag + "=" + Value);
113+
}
114+
115+
// Like RemoveArgument, but removes "-[Flag]=...".
116+
void removeFlag(const std::string &Flag) {
117+
std::string Arg("-" + Flag + "=");
118+
auto IsMatch = [&](const std::string &Other) {
119+
return Arg.compare(0, std::string::npos, Other, 0, Arg.length()) == 0;
120+
};
121+
auto i = endMutableArgs();
122+
Args.erase(std::remove_if(Args.begin(), i, IsMatch), i);
123+
}
124+
125+
// Returns whether the command's stdout is being written to an output file.
126+
bool hasOutputFile() const { return !OutputFile.empty(); }
127+
128+
// Returns the currently set output file.
129+
const std::string &getOutputFile() const { return OutputFile; }
130+
131+
// Configures the command to redirect its output to the name file.
132+
void setOutputFile(const std::string &FileName) { OutputFile = FileName; }
133+
134+
// Returns whether the command's stderr is redirected to stdout.
135+
bool isOutAndErrCombined() const { return CombinedOutAndErr; }
136+
137+
// Sets whether to redirect the command's stderr to its stdout.
138+
void combineOutAndErr(bool combine = true) { CombinedOutAndErr = combine; }
139+
140+
// Returns a string representation of the command. On many systems this will
141+
// be the equivalent command line.
142+
std::string toString() const {
143+
std::stringstream SS;
144+
for (auto arg : getArguments())
145+
SS << arg << " ";
146+
if (hasOutputFile())
147+
SS << ">" << getOutputFile() << " ";
148+
if (isOutAndErrCombined())
149+
SS << "2>&1 ";
150+
std::string result = SS.str();
151+
if (!result.empty())
152+
result = result.substr(0, result.length() - 1);
153+
return result;
154+
}
155+
156+
private:
157+
Command(Command &&Other) = delete;
158+
Command &operator=(Command &&Other) = delete;
159+
160+
Vector<std::string>::iterator endMutableArgs() {
161+
return std::find(Args.begin(), Args.end(), ignoreRemainingArgs());
162+
}
163+
164+
Vector<std::string>::const_iterator endMutableArgs() const {
165+
return std::find(Args.begin(), Args.end(), ignoreRemainingArgs());
166+
}
167+
168+
// The command arguments. Args[0] is the command name.
169+
Vector<std::string> Args;
170+
171+
// True indicates stderr is redirected to stdout.
172+
bool CombinedOutAndErr;
173+
174+
// If not empty, stdout is redirected to the named file.
175+
std::string OutputFile;
176+
};
177+
178+
} // namespace fuzzer
179+
180+
#endif // LLVM_FUZZER_COMMAND_H

0 commit comments

Comments
 (0)