[libc++] Make benchmarks dry-run by default on the release branch

[ldionne]

As reported in #125510, doing a full run of the benchmarks during release testing breaks for some of the testers, and it also takes a long time. The proper fix would be for the release testing process to call check-cxx instead of running lit directly inside libc++'s test directory: that will also have the benefit of actually running all of our tests, not only the Lit ones.

However, since that fix may take longer to happen, this patch tries to reduce the pain of release testers by dry-running benchmarks by default instead.

[llvmbot]

@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

Changes

As reported in #125510, doing a full run of the benchmarks during release testing breaks for some of the testers, and it also takes a long time. The proper fix would be for the release testing process to call check-cxx instead of running lit directly inside libc++'s test directory: that will also have the benefit of actually running all of our tests, not only the Lit ones.

However, since that fix may take longer to happen, this patch tries to reduce the pain of release testers by dry-running benchmarks by default instead.

---

Full diff: https://github.com/llvm/llvm-project/pull/126441.diff

1 Files Affected:

• (modified) libcxx/utils/libcxx/test/params.py (+1-1)

diff --git a/libcxx/utils/libcxx/test/params.py b/libcxx/utils/libcxx/test/params.py
index 8fd3872cd8cbb05..6e13ad75ea94971 100644
--- a/libcxx/utils/libcxx/test/params.py
+++ b/libcxx/utils/libcxx/test/params.py
@@ -371,7 +371,7 @@ def getSuitableClangTidy(cfg):
         name="enable_benchmarks",
         choices=["no", "run", "dry-run"],
         type=str,
-        default="run",
+        default="dry-run",
         help="Whether to run the benchmarks in the test suite, to only dry-run them or to disable them entirely.",
         actions=lambda mode: [AddFeature(f"enable-benchmarks={mode}")],
     ),

[mordante]

Thanks, LGTM!

[github-actions]

@ldionne (or anyone else). If you would like to add a note about this fix in the release notes (completely optional). Please reply to this comment with a one or two sentence description of the fix. When you are done, please add the release:note label to this PR.

[rorth]

I've just seen the same issue in a 21.1.0 rc build on Ubuntu 22.04 (229 failures in llvm-libc++-shared.cfg.in :: benchmarks). It seems the LLVM 21 release needs the same fix/workaround.