How to run a selection of cases in a parameterized test?

[alanmcruickshank]

I'm one of the maintainers of sqlfluff: https://github.com/sqlfluff/sqlfluff

We have many parameterized tests, with functions to generate those cases from yaml files. This is great for reusable code, but it means some test functions have 100+ different combinations of situations being passed to them. We make use of the id function of parameterize to refer to individual tests and give them specific names.

Because of the number of tests, it's sometimes inconvenient to run all of the cases, but also slow to call individual cases one by one. Is there a way to use wildcards, or smart naming of cases, to be able to run a set of parameter cases matching some pattern or with some prefix?

As an example, if we had the following case:

@pytest.mark.parametrize(
    "a,b,expected",
    [
        pytest.param(1, 2, id="foo-abc"),
        pytest.param(2, 3, id="foo-abcd"),
        pytest.param(3, 4, id="foo-abcde"),
        pytest.param(8, 9, id="bar-def"),
        pytest.param(7, 8, id="bar-defg"),
        pytest.param(6, 7, id="bar-defgh"),
    ],
)
def test_example(a, b):
    do_something_with(a, b)

I know we can run pytest test.py::test_example and run all of them.

We can also call pytest test.py::test_example[foo-abcde] and run a specific one.

Is it possible to somehow effectively run all of the tests which start with foo but not bar, either by using a wildcard [foo-*] or specifying only part of the name?

I'm open to solutions which involve changing the setup to make this easier.

[kalekundert]

The -k option is pretty close to what you want. It just does a substring search, so it's not quite as powerful as a wildcard, but it can easily distinguish between foo and bar in the example you gave:

$ pytest test.py -k foo
...
======================= 3 passed, 3 deselected in 0.03s ========================

[alanmcruickshank]

This is exactly what I was looking for! Thanks.

To give the fully fledged example for anyone finding this (and because you can try it on the sqlfluff repo which is public if people want a real example).

The eventual command I'm running is pytest .\test\rules\yaml_test_cases_test.py::test__rule_test_case -k L052.

This runs exactly the test that I want but only the cases with reference L052 (one of our linting rules). 🎉