[llvm][lit] Add option to run only the failed tests #158043

Michael137 · 2025-09-11T11:26:17Z

This patch adds a new --filter-failed option to llvm-lit, which when set, will only run the tests that have previously failed.

Michael137 · 2025-09-11T11:26:54Z

Couldn't find an existing flag that does this. If there's already a way to do this, happy to drop the PR

llvmbot · 2025-09-11T11:26:59Z

@llvm/pr-subscribers-testing-tools

Author: Michael Buch (Michael137)

Changes

This patch adds a new --filter-failed option to llvm-lit, which when set, will only run the tests that have previously failed.

Full diff: https://github.com/llvm/llvm-project/pull/158043.diff

5 Files Affected:

(modified) llvm/utils/lit/lit/cl_arguments.py (+6)
(modified) llvm/utils/lit/lit/main.py (+3)
(added) llvm/utils/lit/tests/Inputs/ignore-fail/pass.txt (+1)
(added) llvm/utils/lit/tests/filter-failed.py (+18)
(modified) llvm/utils/lit/tests/ignore-fail.py (+2)

diff --git a/llvm/utils/lit/lit/cl_arguments.py b/llvm/utils/lit/lit/cl_arguments.py
index 8238bc42395af..a87715f16df2b 100644
--- a/llvm/utils/lit/lit/cl_arguments.py
+++ b/llvm/utils/lit/lit/cl_arguments.py
@@ -302,6 +302,12 @@ def parse_args():
         help="Filter out tests with paths matching the given regular expression",
         default=os.environ.get("LIT_FILTER_OUT", "^$"),
     )
+    selection_group.add_argument(
+        "--filter-failed",
+        dest="filterFailed",
+        help="Only run tests which failed in the previous run.",
+        action="store_true",
+    )
     selection_group.add_argument(
         "--xfail",
         metavar="LIST",
diff --git a/llvm/utils/lit/lit/main.py b/llvm/utils/lit/lit/main.py
index a585cc0abdd48..6c650724bb33d 100755
--- a/llvm/utils/lit/lit/main.py
+++ b/llvm/utils/lit/lit/main.py
@@ -90,6 +90,9 @@ def main(builtin_params={}):
         and not opts.filter_out.search(t.getFullName())
     ]
 
+    if opts.filterFailed:
+        selected_tests = [t for t in selected_tests if t.previous_failure]
+
     if not selected_tests:
         sys.stderr.write(
             "error: filter did not match any tests "
diff --git a/llvm/utils/lit/tests/Inputs/ignore-fail/pass.txt b/llvm/utils/lit/tests/Inputs/ignore-fail/pass.txt
new file mode 100644
index 0000000000000..18efe9e49e95b
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/ignore-fail/pass.txt
@@ -0,0 +1 @@
+RUN: true
diff --git a/llvm/utils/lit/tests/filter-failed.py b/llvm/utils/lit/tests/filter-failed.py
new file mode 100644
index 0000000000000..074f14cf7fc34
--- /dev/null
+++ b/llvm/utils/lit/tests/filter-failed.py
@@ -0,0 +1,18 @@
+# Checks that --filter-failed only runs tests that previously failed.
+
+# RUN: not %{lit} %{inputs}/ignore-fail
+# RUN: not %{lit} --filter-failed %{inputs}/ignore-fail | FileCheck %s
+
+# END.
+
+# CHECK: Testing: 3 of 5 tests
+# CHECK-DAG: FAIL: ignore-fail :: fail.txt
+# CHECK-DAG: UNRESOLVED: ignore-fail :: unresolved.txt
+# CHECK-DAG: XPASS: ignore-fail :: xpass.txt
+
+#      CHECK: Testing Time:
+# CHECK: Total Discovered Tests:
+# CHECK-NEXT:   Excluded : 2 {{\([0-9]*\.[0-9]*%\)}}
+# CHECK-NEXT:   Unresolved : 1 {{\([0-9]*\.[0-9]*%\)}}
+# CHECK-NEXT:   Failed : 1 {{\([0-9]*\.[0-9]*%\)}}
+# CHECK-NEXT:   Unexpectedly Passed: 1 {{\([0-9]*\.[0-9]*%\)}}
diff --git a/llvm/utils/lit/tests/ignore-fail.py b/llvm/utils/lit/tests/ignore-fail.py
index 494c6e092c906..51196fbae9e5e 100644
--- a/llvm/utils/lit/tests/ignore-fail.py
+++ b/llvm/utils/lit/tests/ignore-fail.py
@@ -10,9 +10,11 @@
 # CHECK-DAG: UNRESOLVED: ignore-fail :: unresolved.txt
 # CHECK-DAG: XFAIL: ignore-fail :: xfail.txt
 # CHECK-DAG: XPASS: ignore-fail :: xpass.txt
+# CHECK-DAG: PASS: ignore-fail :: pass.txt
 
 #      CHECK: Testing Time:
 # CHECK: Total Discovered Tests:
+# CHECK-NEXT:   Passed : 1 {{\([0-9]*\.[0-9]*%\)}}
 # CHECK-NEXT:   Expectedly Failed : 1 {{\([0-9]*\.[0-9]*%\)}}
 # CHECK-NEXT:   Unresolved : 1 {{\([0-9]*\.[0-9]*%\)}}
 # CHECK-NEXT:   Failed : 1 {{\([0-9]*\.[0-9]*%\)}}

adrian-prantl · 2025-09-11T15:27:03Z

I'm surprised we didn't already have that — I guess it's because there's always a risk of breaking other tests while making changes, but that doesn't diminish the utility of this flag!

adrian-prantl · 2025-09-11T15:27:43Z

Cann you add documentation to this? https://llvm.org/docs/CommandGuide/lit.html

Michael137 · 2025-09-11T16:17:15Z

Cann you add documentation to this? https://llvm.org/docs/CommandGuide/lit.html

done in latest commit!

jh7370 · 2025-09-12T07:39:14Z

llvm/docs/CommandGuide/lit.rst

@@ -314,6 +314,10 @@ The timing data is stored in the `test_exec_root` in a file named
  place of this option, which is especially useful in environments where the
  call to ``lit`` is issued indirectly.

+.. option:: --filter-failed
+
+  Run only those tests that previously failed.


It would be helpful noting what happens a) to newly added tests and b) if lit hasn't been run before. These cases deserve testing too.

Added a test for this in the latest commit. Since i'm now echoing into the inputs directory I decided to create a dedicated one for this test (so it doesn't affect the ignore-fail test in case things go wrong).

jh7370 · 2025-09-12T07:40:49Z

llvm/utils/lit/lit/cl_arguments.py

+    selection_group.add_argument(
+        "--filter-failed",
+        dest="filterFailed",
+        help="Only run tests which failed in the previous run.",


Nit: there's a bit of a max of help text ending with and without '.'. I think the majority omit it.

jh7370 · 2025-09-12T07:42:33Z

llvm/utils/lit/tests/filter-failed.py

+#      CHECK: Testing Time:
+# CHECK: Total Discovered Tests:


What's with the inconsistent spacing here?

jh7370

Thanks, mostly looks good to me. I've got a couple more test suggestions:

Show the behaviour when a test failed on the first run and was deleted before being run again with --filter-failed.
Show that a failed test that then subsequently passes (under a --filter-failed run) doesn't get run if the tests are executed with --filter-failed again. In other words, the "failed" state is cleared once a test has been rerun successfully.

Michael137 · 2025-09-15T17:07:32Z

Thanks, mostly looks good to me. I've got a couple more test suggestions:

Show the behaviour when a test failed on the first run and was deleted before being run again with --filter-failed.

Show that a failed test that then subsequently passes (under a --filter-failed run) doesn't get run if the tests are executed with --filter-failed again. In other words, the "failed" state is cleared once a test has been rerun successfully.

Yup those seem like good things to test, thanks! Added in latest commit

jh7370 · 2025-09-16T08:28:23Z

llvm/utils/lit/tests/filter-failed-delete.py

+# RUN: not %{lit} --filter-failed %{inputs}/filter-failed-delete > %s.rerun.log
+# RUN: mv %{inputs}/filter-failed-delete/fail.txt.bk %{inputs}/filter-failed-delete/fail.txt
+#
+# RUN: cat %s.rerun.log | FileCheck %s --check-prefix=CHECK-RERUN


Suggested change

# RUN: cat %s.rerun.log | FileCheck %s --check-prefix=CHECK-RERUN

# RUN: FileCheck %s --input-file=%s.rerun.log --check-prefix=CHECK-RERUN

jh7370 · 2025-09-16T08:30:20Z

llvm/utils/lit/tests/filter-failed-delete.py

+#
+# RUN: mv %{inputs}/filter-failed-delete/fail.txt %{inputs}/filter-failed-delete/fail.txt.bk
+# RUN: not %{lit} --filter-failed %{inputs}/filter-failed-delete > %s.rerun.log
+# RUN: mv %{inputs}/filter-failed-delete/fail.txt.bk %{inputs}/filter-failed-delete/fail.txt


I've seen changes trying to make tests work in a read-only context, which this certainly won't. Could this test all be run in a temporary directory, copied from some source?

Good point! In the latest commit i copy the inputs to %t and do all the manipulation there. That makes the tests much shorter and removes the need for three separate input directories. Is this what you meant? Not sure if the rm -rf %t is valid for read-only contexts though.

jh7370 · 2025-09-16T08:32:12Z

llvm/utils/lit/tests/filter-failed-rerun.py

+# RUN: cp %{inputs}/filter-failed-rerun/pass.txt %{inputs}/filter-failed-rerun/fail.txt
+# RUN: not %{lit} %{inputs}/filter-failed-rerun > %s.rerun-1.log
+# RUN: not %{lit} --filter-failed %{inputs}/filter-failed-rerun > %s.rerun-2.log
+# RUN: mv %{inputs}/filter-failed-rerun/fail.txt.bk %{inputs}/filter-failed-rerun/fail.txt


Same comments in this file as the other test.

This patch adds a new `--filter-failed` option to `llvm-lit`, which when set, will only run the tests that have previously failed.

jh7370

Basically looks good. Couple more minor points and then we're good.

On the note of %t, it should work in a read-only context, since regular tests generally write objects there without issue.

llvm/utils/lit/tests/filter-failed-delete.py

llvm/utils/lit/tests/filter-failed-rerun.py

jh7370

LGTM.

Michael137 · 2025-09-19T08:58:04Z

Looks like Windows CI isn't happy. Checking..

Michael137 · 2025-09-19T08:59:23Z

Oh probably because I'm echoing the file incorrectly on Windows:

echo "RUN: false" > C:\_work\llvm-project\llvm-project\build\utils\lit\tests\Output\filter-failed.py.tmp/new-fail.txt

Michael137 · 2025-09-20T18:51:26Z

Hmm a bit confused. It looks like UNRESOLVED test isn't being re-run on Windows. Is Windows not marking those as failures? It does work on Linux...

Michael137 · 2025-09-22T09:48:11Z

@jh7370 do you have any idea what could be going on here? Do you see anything wrong here that would cause the tests not to work on Windows. For some reason in filter-failed.py and filter-failed-rerun.py the UNRESOLVED tests don't seem to get marked as failures, so they don't get picked up by filter-failed. But it does work for filter-failed-delete.py. Which confuses me, since they use the same test inputs.

jh7370 · 2025-09-22T10:52:55Z

@jh7370 do you have any idea what could be going on here? Do you see anything wrong here that would cause the tests not to work on Windows. For some reason in filter-failed.py and filter-failed-rerun.py the UNRESOLVED tests don't seem to get marked as failures, so they don't get picked up by filter-failed. But it does work for filter-failed-delete.py. Which confuses me, since they use the same test inputs.

I don't know why CI isn't showing it up as failing, but filter-failed-delete.py fails for me, as well as the other two.

Michael137 requested review from JDevlieghere, adrian-prantl, jh7370 and pogo59 September 11, 2025 11:26

llvmbot added llvm-lit testing-tools labels Sep 11, 2025

jh7370 reviewed Sep 12, 2025

View reviewed changes

jh7370 reviewed Sep 15, 2025

View reviewed changes

jh7370 reviewed Sep 16, 2025

View reviewed changes

Michael137 added 9 commits September 16, 2025 11:47

[llvm][lit] Add option to run only the failed tests

fe20c20

This patch adds a new `--filter-failed` option to `llvm-lit`, which when set, will only run the tests that have previously failed.

fixup! docs

c4ceeac

fixup! omit terminating '.' in help text

9e36ec2

fixup! expand docs

5eabe3b

fixup! fix spacing in test

07e2f75

fixup! add 'newly added tests' test case

652a733

fixup! revert changes to ignore-fail

93ef008

fixup! add more tests

fc70e28

fixup! use --input-file

c3d1e18

Michael137 force-pushed the llvm/lit-filter-failed branch from cab413c to 4b39c15 Compare September 16, 2025 11:15

fixup! use temporary directory for input manipulation

4aa6397

Michael137 force-pushed the llvm/lit-filter-failed branch from 4b39c15 to 4aa6397 Compare September 16, 2025 11:17

jh7370 reviewed Sep 18, 2025

View reviewed changes

llvm/utils/lit/tests/filter-failed-delete.py Outdated Show resolved Hide resolved

llvm/utils/lit/tests/filter-failed-rerun.py Outdated Show resolved Hide resolved

Michael137 added 2 commits September 19, 2025 07:58

fixup! typo; check that test-suite ran in test

436f0b0

fixup! rename CHECK-RERUN directives

bcb3f32

jh7370 approved these changes Sep 19, 2025

View reviewed changes

Michael137 added 2 commits September 19, 2025 10:03

fixup! use %{fs-sep}

d17641e

fixup! more path separator fixes

24364e2

Michael137 enabled auto-merge (squash) September 19, 2025 15:17

Michael137 added 3 commits September 22, 2025 08:50

fixup! try modified unresolved.txt

f363066

fixup! try modified unresolved.txt

138cdbc

fixup! try modified unresolved.txt

39e562d

	# RUN: cat %s.rerun.log \| FileCheck %s --check-prefix=CHECK-RERUN
	# RUN: FileCheck %s --input-file=%s.rerun.log --check-prefix=CHECK-RERUN

[llvm][lit] Add option to run only the failed tests #158043

Are you sure you want to change the base?

[llvm][lit] Add option to run only the failed tests #158043

Conversation

Michael137 commented Sep 11, 2025

Uh oh!

Michael137 commented Sep 11, 2025

Uh oh!

llvmbot commented Sep 11, 2025

Uh oh!

adrian-prantl commented Sep 11, 2025

Uh oh!

adrian-prantl commented Sep 11, 2025

Uh oh!

Michael137 commented Sep 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jh7370 left a comment

Choose a reason for hiding this comment

Uh oh!

Michael137 commented Sep 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jh7370 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jh7370 left a comment

Choose a reason for hiding this comment

Uh oh!

Michael137 commented Sep 19, 2025

Uh oh!

Michael137 commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Michael137 commented Sep 20, 2025

Uh oh!

Michael137 commented Sep 22, 2025

Uh oh!

jh7370 commented Sep 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Michael137 commented Sep 19, 2025 •

edited

Loading