Use Google FuzzTest as a fuzzing infrastructure

[gchatelet]

Google FuzzTest allows easy fuzzing of C++ code.
It is supported by CMake and Bazel, I think it could have detected #77080 quite easily.

@nickdesaulniers @michaelrj-google @lntue WDYT?

[llvmbot]

@llvm/issue-subscribers-libc

Author: Guillaume Chatelet (gchatelet)

[Google FuzzTest](https://github.com/google/fuzztest/blob/main/README.md) allows easy fuzzing of C++ code. It is supported by CMake and Bazel, I think it could have detected https://github.com//issues/77080 quite easily.

@nickdesaulniers @michaelrj-google @lntue WDYT?

[gchatelet]

So apparently -as of today- FuzzTest is not usable with Bazel
google/fuzztest#91 (comment)

[gchatelet]

Thinking this through again, it's not that hard to use the libfuzzer version directly as we already do. Fuzz tests are a bit harder to write but it shouldn't be too bad.

[michaelrj-google]

Given that we don't have a ton of fuzz tests, switching frameworks wouldn't be too difficult. What would be the benefit?

[nickdesaulniers]

Is Google FuzzTest distinct from https://google.github.io/oss-fuzz/?

[gchatelet]

FuzzTest is a layer on top of libFuzzer that we already use, it makes writing fuzzer trivial (they are coded as gtest unit tests)

void CheckMemcmp(const std::string &a, const std::string &b) {
  const auto sign = [](int value) -> int {
    if (value < 0)
      return -1;
    if (value > 0)
      return 1;
    return 0;
  };
  const int actual = LIBC_NAMESPACE::memcmp(a.data(), b.data(), a.size());
  const int reference = ::memcmp(a.data(), b.data(), a.size());
  EXPECT_EQ(sign(actual), sign(reference));
}

FUZZ_TEST(Memcmp, CheckMemcmp);

And it also provides great output

=================================================================
=== BUG FOUND!

llvm-project/libc/fuzzing/string/memcmp_test.cpp:20: Counterexample found for Memcmp.CheckMemcmp.
The test fails with input:
argument 0: "44\303\363\206\305\305\000\000\275\305\200\302\303\207x\306\271\300xX\201\303\370\305\305\305\305m\3352\335\335\335\335\335\335\335]\337\255\335\335\335\335\335\335\335\335\335\335\335\335\335\305\361"
argument 1: "\361\375\364\306"\262\024\334"

=================================================================
=== Reproducer test

TEST(Memcmp, CheckMemcmpRegression) {
  CheckMemcmp(
    std::string("44\303\363\206\305\305\000\000\275\305\200\302\303\207x\306\271\300xX\201\303\370\305\305\305\305m\3352\335\335\335\335\335\335\335]\337\255\335\335\335\335\335\335\335\335\335\335\335\335\335\305\361", 56),
    "\361\375\364\306\"\262\024\334"
  );
}

With libFuzzer only, we have to adapt data and size to our problem and provide additional code to generate a reproducer.

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
  const auto sign = [](int value) -> int {
    if (value < 0)
      return -1;
    if (value > 0)
      return 1;
    return 0;
  };
  const auto count = size / 2;
  const char *a = reinterpret_cast<const char *>(data);
  const char *b = reinterpret_cast<const char *>(data) + count;
  const int actual = LIBC_NAMESPACE::memcmp(a, b, count);
  const int reference = ::memcmp(a, b, count);
  if (sign(actual) == sign(reference))
    return 0;
  const auto print = [](const char *msg, const char *buffer, size_t size) {
    printf("%s\"", msg);
    for (size_t i = 0; i < size; ++i)
      printf("\\x%02x", (uint8_t)buffer[i]);
    printf("\"\n");
  };
  print("a     : ", a, count);
  print("b     : ", b, count);
  printf("count : %zu\n", count);
  printf("result: %d\n", reference);
  __builtin_trap();
}

@nickdesaulniers I believe oss-fuzz is an infrastructure to execute libFuzzer targets as a service. So yes FuzzTest is different from oss-fuzz.

[nickdesaulniers]

We've started an internal thread about the use of FuzzTest.

[vonosmas]

@nickdesaulniers Could you share the outcome of this discussion?

[nickdesaulniers]

I've forwarded the internal email thread to you.

[vonosmas]

I can see the benefits of https://github.com/google/fuzztest/blob/main/doc/overview.md, but looks like nobody in LLVM project is using it yet (so I wonder if we'd need a community buy-in), and it requires pulling in external dependency (on both gtest and fuzztest). I would defer to @michaelrj-google again to estimate the cost/benefit here.