Flaking coverage for templates

[sobolevn]

Hi!

What's wrong

From time to time I get random CI failures, because some .html template files are reported to be not convered. However, this is not the case. We have a full 100% coverage all the time.

Report example:

================================ tests coverage ================================
_______________ coverage: platform linux, python 3.12.10-final-0 _______________
Name                                                                      Stmts   Miss Branch BrPart  Cover   Missing
---------------------------------------------------------------------------------------------------------------------
server/apps/identity/templates/identity/emails/verification_email.html       10      9      0      0    10%   4-12
server/apps/identity/templates/identity/pages/verification_pending.html       6      5      0      0    17%   5-10
server/apps/identity/templates/password_reset/recovery_done.html              7      5      0      0    29%   3-7, 9
server/apps/identity/templates/password_reset/recovery_form.html             15     12      0      0    20%   3-12, 15-17
server/apps/identity/templates/password_reset/reset.html                     23     16      0      0    30%   3-12, 14-15, 17-18, 23-25
server/apps/identity/templates/password_reset/reset_sent.html                 9      8      0      0    11%   3-11
---------------------------------------------------------------------------------------------------------------------
TOTAL                                                                      3322     55    100      0    98%
161 files skipped due to complete coverage.
Coverage HTML written to dir htmlcov
Coverage XML written to file coverage.xml
FAIL Required test coverage of 100% not reached. Total coverage: 98.39%
============================= 150 passed in 50.03s =============================

On other runs - other .html files might be reported.

It started recently. Not sure why. The setup is rather complex, I am not able to reproduce it locally on macos, only in CI.

Repro

I have a rather complex setup basad on https://github.com/wemake-services/wemake-django-template/tree/master/%7B%7Bcookiecutter.project_name%7D%7D

Important parts:

• coverage plugins: https://github.com/wemake-services/wemake-django-template/blob/56a8f10f1d1d847f6f9cfd64bd936509dc6b6686/%7B%7Bcookiecutter.project_name%7D%7D/setup.cfg#L79-L86
• pytest config: https://github.com/wemake-services/wemake-django-template/blob/56a8f10f1d1d847f6f9cfd64bd936509dc6b6686/%7B%7Bcookiecutter.project_name%7D%7D/setup.cfg#L59-L73
• TEMPLATES config: https://github.com/wemake-services/wemake-django-template/blob/56a8f10f1d1d847f6f9cfd64bd936509dc6b6686/%7B%7Bcookiecutter.project_name%7D%7D/tests/plugins/django_settings.py#L30-L35 debug=True is set
• All testing related deps: https://github.com/wemake-services/wemake-django-template/blob/56a8f10f1d1d847f6f9cfd64bd936509dc6b6686/%7B%7Bcookiecutter.project_name%7D%7D/pyproject.toml#L51-L59

Versions:

• pytest: 8.3.5
• pytest-cov: 6.1.1
• django_coverage_plugin: 3.1.0
• coverage: 7.8.0
• covdefaults: 2.3.0
• django: 5.1.8
• python: 3.12.10

Things I've tried

• Changing coverage plugins order
• Setting debug=True for templates in settings.py
• Setting setting.DEBUG=True in https://github.com/wemake-services/wemake-django-template/blob/56a8f10f1d1d847f6f9cfd64bd936509dc6b6686/%7B%7Bcookiecutter.project_name%7D%7D/tests/plugins/django_settings.py#L30-L35
• Removing --compile option from poetry, so it won't create bytecode before the tests start
• Removing covdefaults plugin
• De-randomize the tests order, I am using pytest-randomly by default

Nothing really helped, except commenting out django_coverage_plugin, so .html coverage is not even calculated. It is not a good solution, because we use templates in our app.

I understand that it is rather hard to make anything without a repro, but I cannot make one just yet :(
Any help is appreciated.