[libc++] Unify the benchmarks with the test suite

Instead of building the benchmarks separately via CMake and running them
separately from the test suite, this patch merges the benchmarks into
the test suite and handles both uniformly.

As a result:
- It is now possible to run individual benchmarks like we run tests
  (e.g. using libcxx-lit), which is a huge quality-of-life improvement.

- The benchmarks will be run under exactly the same configuration as
  the rest of the tests, which is a nice simplification. This does
  mean that one has to be careful to enable the desired optimization
  flags when running benchmarks, but that is easy with e.g.
  `libcxx-lit <...> --param optimization=speed`.

- Benchmarks can use the same annotations as the rest of the test
  suite, such as `// UNSUPPORTED` & friends.

When running the tests via `check-cxx`, we only compile the benchmarks
because running them would be too time consuming. This introduces a bit
of complexity in the testing setup, and instead it would be better to
allow passing a --dry-run flag to GoogleBenchmark executables, which is
the topic of google/benchmark#1827.

I am not really satisfied with the layering violation of adding the
%{benchmark_flags} substitution to cmake-bridge, however I believe
this can be improved in the future.

Author: ldionne

.github/workflows/libcxx-build-and-test.yaml

@@ -155,9 +155,7 @@ jobs:
           'generic-no-rtti',
           'generic-optimized-speed',
           'generic-static',
-          # TODO Find a better place for the benchmark and bootstrapping builds to live. They're either very expensive
-          # or don't provide much value since the benchmark run results are too noise on the bots.
-          'benchmarks',
+          # TODO Find a better place for the bootstrapping build to live, since it's very expensive.
           'bootstrapping-build'
         ]
         machine: [ 'libcxx-runners-8-set' ]

libcxx/CMakeLists.txt

@@ -154,13 +154,8 @@ message(STATUS "Using libc++ testing configuration: ${LIBCXX_TEST_CONFIG}")
 set(LIBCXX_TEST_PARAMS "" CACHE STRING
     "A list of parameters to run the Lit test suite with.")
 
-# Benchmark options -----------------------------------------------------------
 option(LIBCXX_INCLUDE_BENCHMARKS "Build the libc++ benchmarks and their dependencies" ON)
 
-set(LIBCXX_BENCHMARK_TEST_ARGS_DEFAULT --benchmark_min_time=0.01)
-set(LIBCXX_BENCHMARK_TEST_ARGS "${LIBCXX_BENCHMARK_TEST_ARGS_DEFAULT}" CACHE STRING
-    "Arguments to pass when running the benchmarks using check-cxx-benchmarks")
-
 option(LIBCXX_INCLUDE_DOCS "Build the libc++ documentation." ${LLVM_INCLUDE_DOCS})
 set(LIBCXX_LIBDIR_SUFFIX "${LLVM_LIBDIR_SUFFIX}" CACHE STRING
     "Define suffix of library directory name (32/64)")

libcxx/docs/TestingLibcxx.rst

@@ -392,6 +392,10 @@ Test Filenames`_ when determining the names for new test files.
        of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
        in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
 
+   * - ``FOO.bench.cpp``
+     - A benchmark test. These tests are linked against the GoogleBenchmark library and generally contain micro-benchmarks of individual
+       components of the library.
+
 
 libc++-Specific Lit Features
 ----------------------------
@@ -438,43 +442,29 @@ Libc++ contains benchmark tests separately from the test of the test suite.
 The benchmarks are written using the `Google Benchmark`_ library, a copy of which
 is stored in the libc++ repository.
 
-For more information about using the Google Benchmark library see the
+For more information about using the Google Benchmark library, see the
 `official documentation <https://github.com/google/benchmark>`_.
 
-.. _`Google Benchmark`: https://github.com/google/benchmark
-
-Building Benchmarks
--------------------
-
-The benchmark tests are not built by default. The benchmarks can be built using
-the ``cxx-benchmarks`` target.
-
-An example build would look like:
+The benchmarks are located under ``libcxx/test/benchmarks``. Running a benchmark
+works in the same way as running a test. Both the benchmarks and the tests share
+the same configuration, so make sure to enable the relevant optimization level
+when running the benchmarks. For example,
 
 .. code-block:: bash
 
-  $ ninja -C build cxx-benchmarks
-
-This will build all of the benchmarks under ``<libcxx>/test/benchmarks`` to be
-built against the just-built libc++. The compiled tests are output into
-``build/libcxx/test/benchmarks``.
+  $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/benchmarks/string.bench.cpp --param optimization=speed
 
-Running Benchmarks
-------------------
+If you want to see where a benchmark is located (e.g. you want to store the executable
+for subsequent analysis), you can print that information by passing ``--show-all`` to
+``lit``. That will print the command-lines being executed, which includes the location
+of the executable created for that benchmark.
 
-The benchmarks must be run manually by the user. Currently there is no way
-to run them as part of the build.
-
-For example:
-
-.. code-block:: bash
-
-  $ cd build/libcxx/test/benchmarks
-  $ ./find.bench.out # Runs all the benchmarks
-  $ ./find.bench.out --benchmark_filter="bm_ranges_find<std::vector<char>>" # Only runs that specific benchmark
-
-For more information about running benchmarks see `Google Benchmark`_.
+Note that benchmarks are only dry-run when run via the ``check-cxx`` target since
+we only want to make sure they don't rot. Do not rely on the results of benchmarks
+run through ``check-cxx`` for anything, instead run the benchmarks manually using
+the instructions for running individual tests.
 
+.. _`Google Benchmark`: https://github.com/google/benchmark
 
 .. _testing-hardening-assertions:
 
@@ -518,4 +508,3 @@ A toy example:
 
 Note that error messages are only tested (matched) if the ``debug``
 hardening mode is used.
-

libcxx/docs/VendorDocumentation.rst

@@ -244,7 +244,8 @@ General purpose options
 
   **Default**: ``ON`` (or value of ``LLVM_INCLUDE_TESTS``)
 
-  Build the libc++ tests.
+  Build the libc++ test suite, which includes various types of tests like conformance
+  tests, vendor-specific tests and benchmarks.
 
 .. option:: LIBCXX_INCLUDE_BENCHMARKS:BOOL
 
@@ -253,15 +254,6 @@ General purpose options
   Build the libc++ benchmark tests and the Google Benchmark library needed
   to support them.
 
-.. option:: LIBCXX_BENCHMARK_TEST_ARGS:STRING
-
-  **Default**: ``--benchmark_min_time=0.01``
-
-  A semicolon list of arguments to pass when running the libc++ benchmarks using the
-  ``check-cxx-benchmarks`` rule. By default we run the benchmarks for a very short amount of time,
-  since the primary use of ``check-cxx-benchmarks`` is to get test and sanitizer coverage, not to
-  get accurate measurements.
-
 .. option:: LIBCXX_ASSERTION_HANDLER_FILE:PATH
 
   **Default**:: ``"${CMAKE_CURRENT_SOURCE_DIR}/vendor/llvm/default_assertion_handler.in"``

libcxx/test/CMakeLists.txt

@@ -1,10 +1,6 @@
 include(HandleLitArguments)
 add_subdirectory(tools)
 
-if (LIBCXX_INCLUDE_BENCHMARKS)
-  add_subdirectory(benchmarks)
-endif()
-
 # Install the library at a fake location so we can run the test suite against it.
 # This ensures that we run the test suite against a setup that matches what we ship
 # in production as closely as possible (in terms of file paths, rpaths, etc).
@@ -66,6 +62,16 @@ set(SERIALIZED_LIT_PARAMS "# Lit parameters serialized here for llvm-lit to pick
 
 serialize_lit_string_param(SERIALIZED_LIT_PARAMS compiler "${CMAKE_CXX_COMPILER}")
 
+if (LIBCXX_INCLUDE_BENCHMARKS
+    AND LIBCXX_ENABLE_LOCALIZATION AND LIBCXX_ENABLE_THREADS AND LIBCXX_ENABLE_FILESYSTEM AND LIBCXX_ENABLE_RANDOM_DEVICE
+    AND LIBCXX_ENABLE_EXCEPTIONS AND LIBCXX_ENABLE_RTTI) # TODO: The benchmarks should work with exceptions/RTTI disabled
+  add_subdirectory(benchmarks)
+  set(_libcxx_benchmark_mode "dry-run")
+else()
+  serialize_lit_string_param(SERIALIZED_LIT_PARAMS enable_benchmarks "no")
+  set(_libcxx_benchmark_mode "no")
+endif()
+
 if (NOT LIBCXX_ENABLE_EXCEPTIONS)
   serialize_lit_param(SERIALIZED_LIT_PARAMS enable_exceptions False)
 endif()
@@ -102,4 +108,5 @@ configure_lit_site_cfg(
 add_lit_testsuite(check-cxx
   "Running libcxx tests"
   ${CMAKE_CURRENT_BINARY_DIR}
+  PARAMS enable_benchmarks="${_libcxx_benchmark_mode}"
   DEPENDS cxx-test-depends)

libcxx/test/benchmarks/CMakeLists.txt

@@ -1,10 +1,8 @@
-include(ExternalProject)
-include(CheckCXXCompilerFlag)
-
 #==============================================================================
 # Build Google Benchmark
 #==============================================================================
 
+include(ExternalProject)
 set(BENCHMARK_COMPILE_FLAGS
     -Wno-unused-command-line-argument
     -nostdinc++
@@ -39,173 +37,4 @@ ExternalProject_Add(google-benchmark
           -DBENCHMARK_USE_LIBCXX:BOOL=ON
           -DBENCHMARK_ENABLE_TESTING:BOOL=OFF)
 
-#==============================================================================
-# Benchmark tests configuration
-#==============================================================================
-add_custom_target(cxx-benchmarks)
-set(BENCHMARK_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR})
-set(BENCHMARK_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/google-benchmark)
-
-add_library(               cxx-benchmarks-flags INTERFACE)
-
-# TODO(cmake): remove. This is a workaround to prevent older versions of GCC
-# from failing the configure step because they don't support C++23.
-if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS "13.0")
-  return()
-endif()
-#TODO(cmake): remove the `add_compile_options`. Currently we have to explicitly
-# pass the `std:c++latest` flag on Windows to work around an issue where
-# requesting `cxx_std_23` results in an error -- somehow CMake fails to
-# translate the `c++23` flag into `c++latest`, and the highest numbered C++
-# version that MSVC flags support is C++20.
-if (MSVC)
-  add_compile_options(/std:c++latest)
-# ibm-clang does not recognize the cxx_std_23 flag, so use this as a temporary
-# workaround on AIX as well.
-elseif (${CMAKE_SYSTEM_NAME} MATCHES "AIX")
-  add_compile_options(-std=c++23)
-else()
-  target_compile_features( cxx-benchmarks-flags INTERFACE cxx_std_23)
-endif()
-
-target_compile_options(cxx-benchmarks-flags INTERFACE -fsized-deallocation -nostdinc++
-                                                      ${SANITIZER_FLAGS} -Wno-user-defined-literals -Wno-suggest-override)
-target_include_directories(cxx-benchmarks-flags INTERFACE "${LIBCXX_GENERATED_INCLUDE_DIR}"
-                                                INTERFACE "${BENCHMARK_INSTALL_DIR}/include"
-                                                INTERFACE "${LIBCXX_SOURCE_DIR}/test/support")
-target_link_options(cxx-benchmarks-flags INTERFACE -lm -nostdlib++
-                                                   "-L${BENCHMARK_INSTALL_DIR}/lib" "-L${BENCHMARK_INSTALL_DIR}/lib64"
-                                                   ${SANITIZER_FLAGS})
-
-set(libcxx_benchmark_targets)
-
-function(add_benchmark_test name source_file)
-  set(libcxx_target ${name}_libcxx)
-  list(APPEND libcxx_benchmark_targets ${libcxx_target})
-  add_executable(${libcxx_target} EXCLUDE_FROM_ALL ${source_file})
-  target_link_libraries(${libcxx_target} PRIVATE cxx-benchmarks-flags)
-  add_dependencies(${libcxx_target} cxx google-benchmark)
-  add_dependencies(cxx-benchmarks ${libcxx_target})
-  if (LIBCXX_ENABLE_SHARED)
-    target_link_libraries(${libcxx_target} PRIVATE cxx_shared)
-  else()
-    target_link_libraries(${libcxx_target} PRIVATE cxx_static)
-  endif()
-  target_link_libraries(${libcxx_target} PRIVATE cxx_experimental benchmark)
-  if (LLVM_USE_SANITIZER)
-    target_link_libraries(${libcxx_target} PRIVATE -ldl)
-  endif()
-  set_target_properties(${libcxx_target}
-    PROPERTIES
-          OUTPUT_NAME "${name}.bench.out"
-          RUNTIME_OUTPUT_DIRECTORY "${BENCHMARK_OUTPUT_DIR}"
-          CXX_EXTENSIONS NO)
-  cxx_link_system_libraries(${libcxx_target})
-endfunction()
-
-
-#==============================================================================
-# Register Benchmark tests
-#==============================================================================
-set(BENCHMARK_TESTS
-    algorithms.partition_point.bench.cpp
-    algorithms/count.bench.cpp
-    algorithms/equal.bench.cpp
-    algorithms/find.bench.cpp
-    algorithms/fill.bench.cpp
-    algorithms/for_each.bench.cpp
-    algorithms/lexicographical_compare.bench.cpp
-    algorithms/lower_bound.bench.cpp
-    algorithms/make_heap.bench.cpp
-    algorithms/make_heap_then_sort_heap.bench.cpp
-    algorithms/min.bench.cpp
-    algorithms/minmax.bench.cpp
-    algorithms/min_max_element.bench.cpp
-    algorithms/mismatch.bench.cpp
-    algorithms/pop_heap.bench.cpp
-    algorithms/pstl.stable_sort.bench.cpp
-    algorithms/push_heap.bench.cpp
-    algorithms/ranges_contains.bench.cpp
-    algorithms/ranges_ends_with.bench.cpp
-    algorithms/ranges_make_heap.bench.cpp
-    algorithms/ranges_make_heap_then_sort_heap.bench.cpp
-    algorithms/ranges_pop_heap.bench.cpp
-    algorithms/ranges_push_heap.bench.cpp
-    algorithms/ranges_sort.bench.cpp
-    algorithms/ranges_sort_heap.bench.cpp
-    algorithms/ranges_stable_sort.bench.cpp
-    algorithms/set_intersection.bench.cpp
-    algorithms/sort.bench.cpp
-    algorithms/sort_heap.bench.cpp
-    algorithms/stable_sort.bench.cpp
-    atomic_wait.bench.cpp
-    atomic_wait_vs_mutex_lock.bench.cpp
-    libcxxabi/dynamic_cast.bench.cpp
-    libcxxabi/dynamic_cast_old_stress.bench.cpp
-    allocation.bench.cpp
-    deque.bench.cpp
-    deque_iterator.bench.cpp
-    exception_ptr.bench.cpp
-    filesystem.bench.cpp
-    format/write_double_comparison.bench.cpp
-    format/write_int_comparison.bench.cpp
-    format/write_string_comparison.bench.cpp
-    format_to_n.bench.cpp
-    format_to.bench.cpp
-    format.bench.cpp
-    formatted_size.bench.cpp
-    formatter_float.bench.cpp
-    formatter_int.bench.cpp
-    function.bench.cpp
-    join_view.bench.cpp
-    lexicographical_compare_three_way.bench.cpp
-    map.bench.cpp
-    monotonic_buffer.bench.cpp
-    numeric/gcd.bench.cpp
-    ordered_set.bench.cpp
-    shared_mutex_vs_mutex.bench.cpp
-    stop_token.bench.cpp
-    std_format_spec_string_unicode.bench.cpp
-    std_format_spec_string_unicode_escape.bench.cpp
-    string.bench.cpp
-    stringstream.bench.cpp
-    system_error.bench.cpp
-    to_chars.bench.cpp
-    unordered_set_operations.bench.cpp
-    util_smartptr.bench.cpp
-    variant_visit_1.bench.cpp
-    variant_visit_2.bench.cpp
-    variant_visit_3.bench.cpp
-    vector_operations.bench.cpp
-    )
-
-foreach(test_path ${BENCHMARK_TESTS})
-  get_filename_component(test_file "${test_path}" NAME)
-  string(REPLACE ".bench.cpp" "" test_name "${test_file}")
-  if (NOT DEFINED ${test_name}_REPORTED)
-    message(STATUS "Adding Benchmark: ${test_file}")
-    # Only report the adding of the benchmark once.
-    set(${test_name}_REPORTED ON CACHE INTERNAL "")
-  endif()
-  add_benchmark_test(${test_name} ${test_path})
-endforeach()
-
-if (LIBCXX_INCLUDE_TESTS)
-  include(AddLLVM)
-
-  configure_lit_site_cfg(
-          ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py.in
-          ${CMAKE_CURRENT_BINARY_DIR}/lit.cfg.py)
-
-  configure_lit_site_cfg(
-          ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
-          ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py)
-
-  set(BENCHMARK_LIT_ARGS "--show-all --show-xfail --show-unsupported ${LIT_ARGS_DEFAULT}")
-
-  add_lit_target(check-cxx-benchmarks
-          "Running libcxx benchmarks tests"
-          ${CMAKE_CURRENT_BINARY_DIR}
-          DEPENDS cxx-benchmarks cxx-test-depends
-          ARGS ${BENCHMARK_LIT_ARGS})
-endif()
+add_dependencies(cxx-test-depends google-benchmark)

libcxx/test/benchmarks/lit.cfg.py.in

@@ -30,3 +30,4 @@ config.substitutions.append(('%{target-include-dir}', '@LIBCXX_TESTING_INSTALL_P
 config.substitutions.append(('%{lib-dir}', '@LIBCXX_TESTING_INSTALL_PREFIX@/@LIBCXX_INSTALL_LIBRARY_DIR@'))
 config.substitutions.append(('%{module-dir}', '@LIBCXX_TESTING_INSTALL_PREFIX@/@LIBCXX_INSTALL_MODULES_DIR@'))
 config.substitutions.append(('%{test-tools-dir}', '@LIBCXX_TEST_TOOLS_PATH@'))
+config.substitutions.append(('%{benchmark_flags}', '-I @LIBCXX_BINARY_DIR@/test/benchmarks/google-benchmark/include -L @LIBCXX_BINARY_DIR@/test/benchmarks/google-benchmark/lib -l benchmark'))