Show test duration subtotals grouped by file or class

[sparrowt]

--durations is useful but in a suite with thousands of tests it is just showing me some tests which I know are slow and there's not much I can do about them.

What I would like to know is which "groupings" of tests are together contributing the most to our total duration:

• by file name
• possibly by class name (less important, but useful for older codebases with lots of unittest style TestCase classes)

For example currently I see this

============================= slowest 20 durations =============================
 9.82s call     tests/foo/test_foo.py::test_abc
 5.87s call     tests/baz/test_bazarre.py::test_mno
 5.47s call     tests/bar/test_barrito.py::test_xyz
...
 4.41s call     tests/qux/test_qux_quizzles.py::test_quotas
 3.90s call     tests/qux/test_qux_quizzles.py::test_quilted
 1.93s call     tests/thing/test_thing_full.py::ThingTester::test_thing_does_blah
 1.98s call     tests/qux/test_qux_quizzles.py::test_queen
 1.23s call     tests/thing/test_thing_full.py::ThingTester::test_other_thing
...

but what would be more useful is something like this:

========================= slowest 20 durations grouped =========================
59.21s call     tests/qux/test_qux_quizzles.py
48.93s call     tests/thing/test_thing_full.py::ThingTester
 9.82s call     tests/foo/test_foo.py
...

which tells me that, together, all the tests in test_qux_quizzles.py are actually the worst offender for slow times (even though many of them may not even make it into the top N slowest) - so if I can speed up something which all those tests do then this would be a better use of my time than trying to shave little bits off the slowest individual test test_foo.py:test_abc.

I could probably get the data for this from --durations 0 and then parsing the full set of results in some other script but that would be yuck.

Is there already a way of doing so? If not then this is a 'feature suggestion' I guess!

[nicoddemus]

@sparrowt this should be easy enough to accomplish yourself.

1. Implement pytest_report_teststatus in your root conftest.py file, and save the elapsed time of each test in a global dict, along with any information you find useful.
2. Implement pytest_terminal_summary, printing the data in the way you want.

[sparrowt]

Thanks for the hints 🙏 after some digging this is what I've got so far, in case it's useful to others:

@pytest.hookimpl()
def pytest_report_teststatus(report, config):
    # Store durations by path (to the test .py file) and phase (setup/call/teardown) for aggregation in pytest_terminal_summary
    config.stash.setdefault('test_durations_grouped', {}).setdefault((report.fspath, report.when), []).append(report.duration)


@pytest.hookimpl(trylast=True)
def pytest_terminal_summary(terminalreporter, exitstatus, config):
    # Piggy-back off the existing --durations option
    top_n = terminalreporter.config.option.durations
    grouped_durations: dict[tuple[str, str], list[float]] = config.stash.get('test_durations_grouped', {})
    if top_n is None or not grouped_durations:
        return

    # Sort by total duration descending
    sorted_items = sorted(grouped_durations.items(), key=lambda item: sum(item[1]), reverse=True)
    terminalreporter.write_sep("=", f"slowest {top_n} durations grouped by file")
    for (path, phase), durations in sorted_items[:top_n]:
        phase_total = sum(durations)
        terminalreporter.write_line(f"{phase_total:5.2f}s {phase:>8} x {len(durations):<3} {path}")

which adds something like this if you say --durations 10 (after the per-test-method top 10)

================= slowest 10 durations grouped by file ==================
78.10s     call x 193 tests/baz/qux.py
11.00s     call x 7   tests/foo.py
 1.34s    setup x 2   tests/bar.py
 1.21s teardown x 193 tests/baz/qux.py
 0.87s    setup x 193 tests/baz/qux.py
 0.04s teardown x 7   tests/foo.py
 0.03s    setup x 7   tests/foo.py
 0.01s teardown x 8   tests/other.py
 0.01s    setup x 8   tests/other.py
 0.01s teardown x 2   tests/bar.py

which is just what I want for finding which groupings of tests contribute most to the overall runtime 🎉

Note: I found that this didn't work if pytest-sugar was installed because pytest_report_teststatus wasn't called which seems like a bug, but -p no:sugar avoids that temporarily.

[sparrowt]

Note: despite my searching before I had failed to uncover these 2 options until after doing the above (!)

• https://pypi.org/project/pytest-duration-insights/ which looks good for a manual deep-dive
• https://github.com/gabrieldemarmiesse/pytest-extra-durations/blob/master/pytest_extra_durations.py

Nevertheless the above solution gives a nice overview in CI without needing to install any extra packages & is easily customisable.

[sparrowt]

P.S. after suggesting the idea over here blake-r/pytest-durations#26 the pytest-durations plugin has now (amazingly quickly!) implemented it via --pytest-durations-group-by=module wow! 👏