Benchmarking the STL

Building and Running Benchmarks

Read our README's Benchmarking section.

Writing a New Benchmark

• Read Google Benchmark's User Guide.
• Review the existing benchmarks and follow our conventions.
  • Please avoid shadowing Standard Library names. For example, the benchmark for std::sort() shouldn't be called sort().
• Decide whether you should create a new file or extend an existing file.
  • If you create a new file, you'll need to add it to the build. At the bottom of benchmarks/CMakeLists.txt, insert add_benchmark(MEOW src/MEOW.cpp), where MEOW is the name of your file. Keep the list of benchmarks in sorted order.
• Your benchmark function should take (benchmark::State& state) and return void. A basic benchmark example:

  void meow(benchmark::State& state) {
      // setup code
      for (auto _ : state) {
          benchmark::DoNotOptimize(/* input */);
          // code that you want to benchmark
          benchmark::DoNotOptimize(/* output */);
      }
  }

• Use benchmark::DoNotOptimize(...) to prevent the compiler from optimizing away what you're trying to measure.
• Call your benchmark with BENCHMARK(meow);.
  • If you're creating a new file, you'll also need BENCHMARK_MAIN(); at the end.
• More advanced usage can call your benchmark function with varying arguments.
  • See examples where BENCHMARK(meow) is followed by ->Apply, ->Range, and so forth, where state.range(0) gets the varying arguments (and state.range(1) can get a second varying argument, etc.).