Skip to content

[Backend Tester] Add test flow CLI arg #13360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 206 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
206 commits
Select commit Hold shift + click to select a range
f120e70
Update
GregoryComer Jul 18, 2025
0fb85e6
Update
GregoryComer Jul 18, 2025
4d8d844
Update
GregoryComer Jul 19, 2025
dc12b40
Update
GregoryComer Jul 21, 2025
ead0616
Update
GregoryComer Jul 22, 2025
0f13676
Update
GregoryComer Jul 22, 2025
b0b01f2
Update
GregoryComer Jul 22, 2025
8b9c9ef
Update
GregoryComer Jul 22, 2025
06bf03a
Update
GregoryComer Jul 22, 2025
2f8f49b
Update
GregoryComer Jul 22, 2025
8ca7766
Update
GregoryComer Jul 22, 2025
bffb95f
Update
GregoryComer Jul 22, 2025
d21492b
Update
GregoryComer Jul 22, 2025
e2c4ea5
Update
GregoryComer Jul 22, 2025
8230848
Update
GregoryComer Jul 22, 2025
2a1f564
Update
GregoryComer Jul 22, 2025
b35e7b1
Update
GregoryComer Jul 22, 2025
5c4c6ce
Update
GregoryComer Jul 22, 2025
9397803
Update
GregoryComer Jul 22, 2025
9dfeb5a
Update
GregoryComer Jul 22, 2025
ff5c4a5
Update
GregoryComer Jul 22, 2025
42a5de5
Update
GregoryComer Jul 22, 2025
402d8f5
Update
GregoryComer Jul 22, 2025
34d3ab3
Update
GregoryComer Jul 22, 2025
1105e04
Update
GregoryComer Jul 22, 2025
482bd21
Update
GregoryComer Jul 22, 2025
ea548b7
Update
GregoryComer Jul 23, 2025
4108f54
Update
GregoryComer Jul 23, 2025
7ef236b
Update
GregoryComer Jul 23, 2025
4a58c9d
Update
GregoryComer Jul 23, 2025
3b866b4
Update
GregoryComer Jul 23, 2025
5ba25cb
Update
GregoryComer Jul 23, 2025
19760fc
Update
GregoryComer Jul 23, 2025
81dfb07
Update
GregoryComer Jul 23, 2025
4d50265
Update
GregoryComer Jul 23, 2025
5f66043
Update
GregoryComer Jul 23, 2025
24e919d
Update
GregoryComer Jul 23, 2025
523cc20
Update
GregoryComer Jul 23, 2025
74c95fe
Update
GregoryComer Jul 23, 2025
5d437b1
Update
GregoryComer Jul 23, 2025
89757ce
Update
GregoryComer Jul 23, 2025
423f79a
Update
GregoryComer Jul 23, 2025
69f7f9c
Update
GregoryComer Jul 23, 2025
c0f6224
Update
GregoryComer Jul 23, 2025
e2ea2a3
Update
GregoryComer Jul 23, 2025
7a2fab5
Update
GregoryComer Jul 23, 2025
033c231
Update
GregoryComer Jul 23, 2025
a9ed762
Update
GregoryComer Jul 23, 2025
64b174a
Update
GregoryComer Jul 23, 2025
3976629
Update
GregoryComer Jul 23, 2025
27cd171
Update
GregoryComer Jul 23, 2025
7bdd3e5
Update
GregoryComer Jul 23, 2025
b1254cd
Update
GregoryComer Jul 23, 2025
f2e2289
Update
GregoryComer Jul 23, 2025
cdd15c1
Update
GregoryComer Jul 23, 2025
e2df06e
Update
GregoryComer Jul 23, 2025
4461bd8
Update
GregoryComer Jul 23, 2025
7e97fd0
Update
GregoryComer Jul 23, 2025
bcb697c
Update
GregoryComer Jul 23, 2025
11a5a02
Update
GregoryComer Jul 24, 2025
244b146
Update
GregoryComer Jul 24, 2025
de21ac2
Update
GregoryComer Jul 24, 2025
fd26fc7
Update
GregoryComer Jul 24, 2025
4ae840d
Update
GregoryComer Jul 24, 2025
710ea49
Update
GregoryComer Jul 24, 2025
32f54b0
Update
GregoryComer Jul 24, 2025
a27d18c
Update
GregoryComer Jul 24, 2025
2eb59fc
Update
GregoryComer Jul 24, 2025
5cc4941
Update
GregoryComer Jul 24, 2025
ef7af5c
Update
GregoryComer Jul 24, 2025
18e89c1
Update
GregoryComer Jul 24, 2025
4719c90
Update
GregoryComer Jul 25, 2025
dd09555
Update
GregoryComer Aug 8, 2025
f1db3a0
Update
GregoryComer Aug 8, 2025
e0700b2
Update
GregoryComer Aug 8, 2025
f260b50
Update
GregoryComer Aug 8, 2025
d62ee60
Update
GregoryComer Aug 8, 2025
b2ab3a5
Update
GregoryComer Aug 8, 2025
c23c3e9
Update
GregoryComer Aug 8, 2025
c99c41a
Update
GregoryComer Aug 9, 2025
bf57d6c
Update
GregoryComer Aug 9, 2025
f261355
Update
GregoryComer Aug 11, 2025
c3a24f9
Update
GregoryComer Aug 11, 2025
1697cbc
Update
GregoryComer Aug 11, 2025
b94b45e
Update
GregoryComer Aug 11, 2025
5740f0a
Update
GregoryComer Aug 11, 2025
ed6840d
Update
GregoryComer Aug 11, 2025
f2a7e1f
Update
GregoryComer Aug 11, 2025
0e162ab
Update
GregoryComer Aug 11, 2025
c6bd56b
Update
GregoryComer Aug 11, 2025
144a8ae
Update
GregoryComer Aug 12, 2025
6f85fc1
Update
GregoryComer Aug 12, 2025
2439022
Update
GregoryComer Aug 12, 2025
bd79ef2
Update
GregoryComer Aug 12, 2025
8932c29
Update
GregoryComer Aug 12, 2025
ea2549c
Update
GregoryComer Aug 12, 2025
ffaa1c3
Update
GregoryComer Aug 12, 2025
bba2fa9
Update
GregoryComer Aug 12, 2025
3a3e026
Update
GregoryComer Aug 12, 2025
2c4488f
Update
GregoryComer Aug 12, 2025
491ec2b
Update
GregoryComer Aug 12, 2025
375b076
Update
GregoryComer Aug 12, 2025
fc1c2a7
Update
GregoryComer Aug 12, 2025
78086b4
Update
GregoryComer Aug 12, 2025
285e058
Update
GregoryComer Aug 12, 2025
cfbc408
Update
GregoryComer Aug 12, 2025
f4b0dc2
Update
GregoryComer Aug 12, 2025
5e92884
Update
GregoryComer Aug 12, 2025
605a0ad
Update
GregoryComer Aug 12, 2025
aa27776
Update
GregoryComer Aug 12, 2025
90ce443
Update
GregoryComer Aug 12, 2025
67f60e4
Update
GregoryComer Aug 12, 2025
1b9315e
Update
GregoryComer Aug 12, 2025
54563ee
Update
GregoryComer Aug 12, 2025
94d89c4
Update
GregoryComer Aug 12, 2025
d79d1a8
Update
GregoryComer Aug 12, 2025
733d4f9
Update
GregoryComer Aug 12, 2025
1e3e79c
Update
GregoryComer Aug 12, 2025
6043952
Update
GregoryComer Aug 12, 2025
8f476ea
Update
GregoryComer Aug 12, 2025
1c136d7
Update
GregoryComer Aug 12, 2025
780519f
Update
GregoryComer Aug 12, 2025
a62c6d0
Update
GregoryComer Aug 12, 2025
7e1a002
Update
GregoryComer Aug 12, 2025
a628d29
Update
GregoryComer Aug 12, 2025
3615d89
Update
GregoryComer Aug 12, 2025
e994bc1
Update
GregoryComer Aug 12, 2025
0aba8e1
Update
GregoryComer Aug 12, 2025
4329bf6
Update
GregoryComer Aug 12, 2025
105aabc
Update
GregoryComer Aug 12, 2025
c1a51ee
Update
GregoryComer Aug 12, 2025
dd30127
Update
GregoryComer Aug 12, 2025
4216805
Update
GregoryComer Aug 12, 2025
092b8d1
Update
GregoryComer Aug 12, 2025
bd786cd
Update
GregoryComer Aug 12, 2025
1d34f49
Update
GregoryComer Aug 12, 2025
933fba2
Update
GregoryComer Aug 12, 2025
d468ae4
Update
GregoryComer Aug 12, 2025
acbd480
Update
GregoryComer Aug 12, 2025
e515bf1
Update
GregoryComer Aug 12, 2025
803db00
Update
GregoryComer Aug 12, 2025
ab18089
Update
GregoryComer Aug 12, 2025
49944fa
Update
GregoryComer Aug 12, 2025
60f3932
Update
GregoryComer Aug 12, 2025
fe201b8
Update
GregoryComer Aug 12, 2025
a807a90
Update
GregoryComer Aug 12, 2025
1897d4e
Update
GregoryComer Aug 12, 2025
f65d80f
Update
GregoryComer Aug 12, 2025
0d1f097
Update
GregoryComer Aug 12, 2025
f0c2490
Update
GregoryComer Aug 12, 2025
0046b02
Update
GregoryComer Aug 12, 2025
32e1029
Update
GregoryComer Aug 12, 2025
f4df9c9
Update
GregoryComer Aug 12, 2025
832cea5
Update
GregoryComer Aug 12, 2025
40819b9
Update
GregoryComer Aug 12, 2025
692f0fa
Update
GregoryComer Aug 12, 2025
871312a
Update
GregoryComer Aug 12, 2025
53990fe
Update
GregoryComer Aug 12, 2025
567d055
Update
GregoryComer Aug 12, 2025
cd998cf
Update
GregoryComer Aug 12, 2025
2a837ab
Update
GregoryComer Aug 12, 2025
51c392a
Update
GregoryComer Aug 12, 2025
f066d46
Update
GregoryComer Aug 12, 2025
ab5415f
Update
GregoryComer Aug 12, 2025
e96c2ef
Update
GregoryComer Aug 12, 2025
31bc137
Update
GregoryComer Aug 12, 2025
dae5d43
Update
GregoryComer Aug 12, 2025
06b5532
Update
GregoryComer Aug 12, 2025
a343abc
Update
GregoryComer Aug 12, 2025
778f7cd
Update
GregoryComer Aug 12, 2025
c19a8ea
Update
GregoryComer Aug 12, 2025
b4a8428
Update
GregoryComer Aug 12, 2025
8aa25c7
Update
GregoryComer Aug 12, 2025
637b8a2
Update
GregoryComer Aug 12, 2025
7141f6c
Update
GregoryComer Aug 12, 2025
4b43363
Update
GregoryComer Aug 12, 2025
88d1dd3
Update
GregoryComer Aug 12, 2025
c583fa1
Update
GregoryComer Aug 12, 2025
7ba2a7f
Update
GregoryComer Aug 12, 2025
e7b7975
Update
GregoryComer Aug 13, 2025
29c1061
Update
GregoryComer Aug 13, 2025
995c4b5
Update
GregoryComer Aug 13, 2025
0fc4475
Update
GregoryComer Aug 13, 2025
1fec71c
Update
GregoryComer Aug 13, 2025
ad8ab13
Update
GregoryComer Aug 13, 2025
f955b94
Update
GregoryComer Aug 13, 2025
765e960
Update
GregoryComer Aug 13, 2025
a3c2dbe
Update
GregoryComer Aug 13, 2025
1008ad4
Update
GregoryComer Aug 13, 2025
e94e9ae
Update
GregoryComer Aug 13, 2025
0225d15
Update
GregoryComer Aug 13, 2025
724e992
Update
GregoryComer Aug 13, 2025
98eee4d
Update
GregoryComer Aug 13, 2025
ad04155
Update
GregoryComer Aug 13, 2025
20574c3
Update
GregoryComer Aug 13, 2025
4fd031e
Update
GregoryComer Aug 13, 2025
a5f702f
Update
GregoryComer Aug 13, 2025
569916c
Update
GregoryComer Aug 13, 2025
8303207
Update
GregoryComer Aug 13, 2025
0a017d0
Update
GregoryComer Aug 13, 2025
b299659
Update
GregoryComer Aug 13, 2025
b7d4b4e
Update
GregoryComer Aug 13, 2025
5a717d5
Update
GregoryComer Aug 13, 2025
5fc71bd
Update
GregoryComer Aug 13, 2025
d19e0db
Update
GregoryComer Aug 13, 2025
ceb875b
Update
GregoryComer Aug 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions backends/test/suite/discovery.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ class TestFilter:
backends: set[str] | None
""" The set of backends to include. If None, all backends are included. """

flows: set[str] | None
""" The set of test flows to include. If None, all backends are included. """

name_regex: Pattern[str] | None
""" A regular expression to filter test names. If None, all tests are included. """

Expand Down Expand Up @@ -86,6 +89,9 @@ def _is_test_enabled(test_case: unittest.TestCase, test_filter: TestFilter) -> b
if test_filter.backends is not None and flow.backend not in test_filter.backends:
return False

if test_filter.flows is not None and flow.name not in test_filter.flows:
return False

if test_filter.name_regex is not None and not test_filter.name_regex.search(
test_case.id()
):
Expand Down
2 changes: 2 additions & 0 deletions backends/test/suite/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ def parse_args():
parser.add_argument(
"-b", "--backend", nargs="*", help="The backend or backends to test."
)
parser.add_argument("-l", "--flow", nargs="*", help="The flow or flows to test.")
parser.add_argument(
"-f", "--filter", nargs="?", help="A regular expression filter for test names."
)
Expand All @@ -273,6 +274,7 @@ def parse_args():
def build_test_filter(args: argparse.Namespace) -> TestFilter:
return TestFilter(
backends=set(args.backend) if args.backend is not None else None,
flows=set(args.flow) if args.flow is not None else None,
name_regex=re.compile(args.filter) if args.filter is not None else None,
)

Expand Down
Loading