Skip to content

[lit] Add an option to read an xfail list from a file #163959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dcandler wants to merge 1 commit into from
Closed

[lit] Add an option to read an xfail list from a file #163959

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions llvm/docs/CommandGuide/lit.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,10 @@ The timing data is stored in the `test_exec_root` in a file named

LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"

.. option:: --xfail-from-file PATH

Read a line separated list of tests from a file to be used by :option:`--xfail`.
Copy link
Collaborator

@DavidSpickett DavidSpickett Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "line separated" is a standard term but I don't know if it means:

thing

thing

Or:

thing\nthing

I guess the latter?

Or in other words: one test name/pattern per line.

Copy link
Collaborator

@DavidSpickett DavidSpickett Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And yes I see below, the splitlines() call.

Copy link
Collaborator

@DavidSpickett DavidSpickett Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you've got a citation for the term, fine, but it's not familiar to me.


.. option:: --xfail-not LIST

Do not treat the specified tests as ``XFAIL``. The environment variable
Expand All @@ -356,6 +360,10 @@ The timing data is stored in the `test_exec_root` in a file named
primary purpose is to suppress an ``XPASS`` result without modifying a test
case that uses the ``XFAIL`` directive.

.. option:: --xfail-not-from-file PATH

Read a line separated list of tests from a file to be used by :option:`--xfail-not`.
Copy link
Collaborator

@DavidSpickett DavidSpickett Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Saying it's used by the option is detail that users don't really need. What they might want to know is that this option does add to that other one. So you could say:

Acts as --xfail-not, except that the paths are read from the specified file. If this option and --xfail-not are specified at the same time, both sets of paths are considered.

Then you're also going back to the "this option but from a file" idea that you stated in the PR description.


.. option:: --exclude-xfail

``XFAIL`` tests won't be run, unless they are listed in the ``--xfail-not``
Expand Down
19 changes: 19 additions & 0 deletions llvm/utils/lit/lit/cl_arguments.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,13 +309,25 @@ def parse_args():
help="XFAIL tests with paths in the semicolon separated list",
default=os.environ.get("LIT_XFAIL", ""),
)
selection_group.add_argument(
"--xfail-from-file",
metavar="PATH",
help="XFAIL tests with paths in the line separated list contained in "
"the specified file",
Copy link
Collaborator

@DavidSpickett DavidSpickett Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe reword to split purpose and format:
XFAIL tests whose paths are contained in the specified file. Each path should be newline separated in the file.

Copy link
Collaborator

@DavidSpickett DavidSpickett Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But also consider like above, saying it is just that other option but sourced from a file.

)
selection_group.add_argument(
"--xfail-not",
metavar="LIST",
type=_semicolon_list,
help="do not XFAIL tests with paths in the semicolon separated list",
default=os.environ.get("LIT_XFAIL_NOT", ""),
)
selection_group.add_argument(
"--xfail-not-from-file",
metavar="PATH",
help="do not XFAIL tests with paths in the line separated list "
"contained in the specified file",
)
selection_group.add_argument(
"--exclude-xfail",
help="exclude XFAIL tests (unless they are in the --xfail-not list). "
Expand Down Expand Up @@ -396,6 +408,13 @@ def parse_args():
for report in opts.reports:
report.use_unique_output_file_name = opts.use_unique_output_file_name

if opts.xfail_from_file:
with open(opts.xfail_from_file, "r", encoding="utf-8") as f:
opts.xfail.extend(f.read().splitlines())
if opts.xfail_not_from_file:
with open(opts.xfail_not_from_file, "r", encoding="utf-8") as f:
opts.xfail_not.extend(f.read().splitlines())

return opts


Expand Down
2 changes: 2 additions & 0 deletions llvm/utils/lit/tests/Inputs/xfail-cl/xfail-not.list
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
true-xfail.txt
top-level-suite :: a :: test-xfail.txt
3 changes: 3 additions & 0 deletions llvm/utils/lit/tests/Inputs/xfail-cl/xfail.list
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
false.txt
false2.txt
top-level-suite :: b :: test.txt
4 changes: 4 additions & 0 deletions llvm/utils/lit/tests/xfail-cl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
# RUN: %{inputs}/xfail-cl \
# RUN: | FileCheck --check-prefixes=CHECK-EXCLUDED,CHECK-EXCLUDED-OVERRIDE %s

# RUN: %{lit} --xfail-from-file %{inputs}/xfail-cl/xfail.list \
# RUN: --xfail-not-from-file %{inputs}/xfail-cl/xfail-not.list \
# RUN: %{inputs}/xfail-cl \
# RUN: | FileCheck --check-prefix=CHECK-FILTER %s

# RUN: env LIT_XFAIL='false.txt;false2.txt;top-level-suite :: b :: test.txt' \
# RUN: LIT_XFAIL_NOT='true-xfail.txt;top-level-suite :: a :: test-xfail.txt' \
Expand Down