Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,10 +329,10 @@ for family, grp in itertools.groupby(collected.checks.items(), key=lambda x: x[1
- [`PP005`](https://learn.scientific-python.org/development/guides/packaging-simple#PP005): Using SPDX project.license should not use deprecated trove classifiers
- [`PP006`](https://learn.scientific-python.org/development/guides/packaging-simple#PP006): The dev dependency group should be defined
- [`PP301`](https://learn.scientific-python.org/development/guides/pytest#PP301): Has pytest in pyproject
- [`PP302`](https://learn.scientific-python.org/development/guides/pytest#PP302): Sets a minimum pytest to at least 6
- [`PP302`](https://learn.scientific-python.org/development/guides/pytest#PP302): Sets a minimum pytest to at least 6 or 9
- [`PP303`](https://learn.scientific-python.org/development/guides/pytest#PP303): Sets the test paths
- [`PP304`](https://learn.scientific-python.org/development/guides/pytest#PP304): Sets the log level in pytest
- [`PP305`](https://learn.scientific-python.org/development/guides/pytest#PP305): Specifies xfail_strict
- [`PP305`](https://learn.scientific-python.org/development/guides/pytest#PP305): Specifies strict xfail
- [`PP306`](https://learn.scientific-python.org/development/guides/pytest#PP306): Specifies strict config
- [`PP307`](https://learn.scientific-python.org/development/guides/pytest#PP307): Specifies strict markers
- [`PP308`](https://learn.scientific-python.org/development/guides/pytest#PP308): Specifies useful pytest summary
Expand Down
56 changes: 38 additions & 18 deletions docs/pages/guides/pytest.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,12 @@ This looks simple, but it is doing several things:
### Configuring pytest

pytest supports configuration in `pytest.ini`, `setup.cfg`, or, since version 6,
`pyproject.toml` {% rr PP301 %}. Remember, pytest is a developer requirement,
not a user one, so always require 6+ (or 7+) and use `pyproject.toml`. This is
an example configuration:
`pyproject.toml` {% rr PP301 %}, or, since version 9, `pytest.toml` or
`.pytest.toml`. Remember, pytest is a developer requirement, not a user one, so
always require 6+ (or 9+) and use `pyproject.toml` or the pytest TOML ones. This
is an example configuration:

{% tabs %} {% tab classic "Pytest 6+" %}

```toml
[tool.pytest.ini_options]
Expand All @@ -82,23 +85,40 @@ testpaths = [
]
```

{% endtab %} {% tab modern "Pytest 9+" %}

```toml
[tool.pytest]
minversion = "9.0"
addopts = ["-ra", "--showlocals"]
strict = true
filterwarnings = ["error"]
log_level = "INFO"
testpaths = [
"tests",
]
```

{% endtab %} {% endtabs %}

{% rr PP302 %} The `minversion` will print a nicer error if your `pytest` is too
old (though, ironically, it won't read this if the version is too old, so
setting "6" or less in `pyproject.toml` is rather pointless). The `addopts`
setting will add whatever you put there to the command line when you run;
{% rr PP308 %} `-ra` will print a summary "r"eport of "a"ll results, which gives
you a quick way to review what tests failed and were skipped, and why.
`--showlocals` will print locals in tracebacks. {% rr PP307 %}
`--strict-markers` will make sure you don't try to use an unspecified fixture.
{% rr PP306 %} And `--strict-config` will error if you make a mistake in your
config. {% rr PP305 %} `xfail_strict` will change the default for `xfail` to
fail the tests if it doesn't fail - you can still override locally in a specific
xfail for a flaky failure. {% rr PP309 %} `filter_warnings` will cause all
warnings to be errors (you can add allowed warnings here too, see below).
{% rr PP304 %} `log_level` will report `INFO` and above log messages on a
failure. {% rr PP303 %} Finally, `testpaths` will limit `pytest` to just looking
in the folders given - useful if it tries to pick up things that are not tests
from other directories.
setting "6" or less in `pyproject.toml` is rather pointless, similarly for 9 if
using the new config location). The `addopts` setting will add whatever you put
there to the command line when you run; {% rr PP308 %} `-ra` will print a
summary "r"eport of "a"ll results, which gives you a quick way to review what
tests failed and were skipped, and why. `--showlocals` will print locals in
tracebacks - depending on your tests, you might or might not like this one.
{% rr PP307 %} `--strict-markers` will make sure you don't try to use an
unspecified fixture. {% rr PP306 %} And `--strict-config` will error if you make
a mistake in your config. {% rr PP305 %} `xfail_strict` will change the default
for `xfail` to fail the tests if it doesn't fail - you can still override
locally in a specific xfail for a flaky failure. {% rr PP309 %}
`filter_warnings` will cause all warnings to be errors (you can add allowed
warnings here too, see below). {% rr PP304 %} `log_level` will report `INFO` and
above log messages on a failure. {% rr PP303 %} Finally, `testpaths` will limit
`pytest` to just looking in the folders given - useful if it tries to pick up
things that are not tests from other directories.
[See the docs](https://docs.pytest.org/en/stable/customize.html) for more
options.

Expand Down
5 changes: 3 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,14 @@ setupcfg = "sp_repo_review.checks.setupcfg:repo_review_checks"
noxfile = "sp_repo_review.checks.noxfile:repo_review_checks"

[project.entry-points."repo_review.fixtures"]
workflows = "sp_repo_review.checks.github:workflows"
dependabot = "sp_repo_review.checks.github:dependabot"
noxfile = "sp_repo_review.checks.noxfile:noxfile"
precommit = "sp_repo_review.checks.precommit:precommit"
pytest = "sp_repo_review.checks.pyproject:pytest"
readthedocs = "sp_repo_review.checks.readthedocs:readthedocs"
ruff = "sp_repo_review.checks.ruff:ruff"
setupcfg = "sp_repo_review.checks.setupcfg:setupcfg"
noxfile = "sp_repo_review.checks.noxfile:noxfile"
workflows = "sp_repo_review.checks.github:workflows"

[project.entry-points."repo_review.families"]
scikit-hep = "sp_repo_review.families:get_families"
Expand Down
131 changes: 90 additions & 41 deletions src/sp_repo_review/checks/pyproject.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,46 @@
from __future__ import annotations

import enum
from typing import TYPE_CHECKING, Any

from .._compat import tomllib
from . import mk_url

if TYPE_CHECKING:
from configparser import ConfigParser

from .._compat.importlib.resources.abc import Traversable


class PytestFile(enum.Enum):
PYTEST_TOML = enum.auto()
MODERN_PYPROJECT = enum.auto()
LEGACY_PYPROJECT = enum.auto()
NONE = enum.auto()


def pytest(
pyproject: dict[str, Any], root: Traversable
) -> tuple[PytestFile, dict[str, Any]]:
"""
Returns the pytest configuration, or None if the configuration doesn't exist.
Respects toml configurations only.
"""
paths = [root.joinpath("pytest.toml"), root.joinpath(".pytest.toml")]
for path in paths:
if path.is_file():
with path.open("rb") as f:
contents = tomllib.load(f)
return (PytestFile.PYTEST_TOML, contents.get("pytest", {}))

match pyproject:
case {"tool": {"pytest": {"ini_options": config}}}:
return (PytestFile.LEGACY_PYPROJECT, config)
case {"tool": {"pytest": config}}:
return (PytestFile.MODERN_PYPROJECT, config)
case _:
return (PytestFile.NONE, {})


def get_requires_python(
pyproject: dict[str, Any], setupcfg: ConfigParser | None
Expand Down Expand Up @@ -149,45 +183,45 @@ def check(pyproject: dict[str, Any]) -> bool:
class PP301(PyProject):
"Has pytest in pyproject"

requires = {"PY001"}
requires = set[str]()
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
Must have a `[tool.pytest.ini_options]` configuration section in
pyproject.toml. If you must have it somewhere else (such as to support
`pytest<6`), ignore this check.
Must have a `[tool.pytest]` (pytest 9+) or `[tool.pytest.ini_options]`
(pytest 6+) configuration section in pyproject.toml. If you must have it
in ini format, ignore this check. pytest.toml and .pytest.toml files
(pytest 9+) are also supported.
"""

match pyproject:
case {"tool": {"pytest": {"ini_options": {}}}}:
return True
case _:
return False
loc, _ = pytest
return loc is not PytestFile.NONE


class PP302(PyProject):
"Sets a minimum pytest to at least 6"
"Sets a minimum pytest to at least 6 or 9"

requires = {"PP301"}
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
Must have a `minversion=`, and must be at least 6 (first version to
support `pyproject.toml` configuration).
support `pyproject.toml` ini configuration) or 9 (first version to
support native configuration and toml config files).

```toml
[tool.pytest.ini_options]
minversion = "7"
minversion = "9"
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
loc, options = pytest
minversion = 6 if loc is PytestFile.LEGACY_PYPROJECT else 9
return (
"minversion" in options
and int(str(options["minversion"]).split(".", maxsplit=1)[0]) >= 6
and int(str(options["minversion"]).split(".", maxsplit=1)[0]) >= minversion
)


Expand All @@ -198,7 +232,7 @@ class PP303(PyProject):
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
The `testpaths` setting should be set to a reasonable default.

Expand All @@ -207,7 +241,7 @@ def check(pyproject: dict[str, Any]) -> bool:
testpaths = ["tests"]
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
_, options = pytest
return "testpaths" in options


Expand All @@ -218,7 +252,7 @@ class PP304(PyProject):
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
`log_level` should be set. This will allow logs to be displayed on
failures.
Expand All @@ -228,29 +262,34 @@ def check(pyproject: dict[str, Any]) -> bool:
log_level = "INFO"
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
_, options = pytest
return "log_cli_level" in options or "log_level" in options


class PP305(PyProject):
"Specifies xfail_strict"
"Specifies strict xfail"

requires = {"PP301"}
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
`xfail_strict` should be set. You can manually specify if a check
should be strict when setting each xfail.
`xfail_strict`, or if using pytest 9+, `strict_xfail` or `strict` should
be set. You can manually specify if a check should be strict when
setting each xfail.

```toml
[tool.pytest.ini_options]
xfail_strict = true
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
return "xfail_strict" in options
_, options = pytest
return (
"xfail_strict" in options
or "strict_xfail" in options
or "strict" in options
)


class PP306(PyProject):
Expand All @@ -260,18 +299,23 @@ class PP306(PyProject):
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
`--strict-config` should be in `addopts = [...]`. This forces an error
if a config setting is misspelled.
`--strict-config` should be in `addopts = [...]` or (pytest 9+)
`strict_config` or `strict` should be set. This forces an error if a
config setting is misspelled.

```toml
[tool.pytest.ini_options]
addopts = ["-ra", "--strict-config", "--strict-markers"]
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
return "--strict-config" in options.get("addopts", [])
_, options = pytest
return (
"strict" in options
or "strict_config" in options
or "--strict-config" in options.get("addopts", [])
)


class PP307(PyProject):
Expand All @@ -281,18 +325,23 @@ class PP307(PyProject):
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
`--strict-markers` should be in `addopts = [...]`. This forces all
markers to be specified in config, avoiding misspellings.
`--strict-markers` should be in `addopts = [...]` or (pytest 9+)
`strict_markers` or `strict` should be set. This forces test markers to
be specified in config, avoiding misspellings.

```toml
[tool.pytest.ini_options]
addopts = ["-ra", "--strict-config", "--strict-markers"]
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
return "--strict-markers" in options.get("addopts", [])
_, options = pytest
return (
"strict" in options
or "strict_markers" in options
or "--strict-markers" in options.get("addopts", [])
)


class PP308(PyProject):
Expand All @@ -302,7 +351,7 @@ class PP308(PyProject):
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
An explicit summary flag like `-ra` should be in `addopts = [...]`
(print summary of all fails/errors).
Expand All @@ -312,9 +361,9 @@ def check(pyproject: dict[str, Any]) -> bool:
addopts = ["-ra", "--strict-config", "--strict-markers"]
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
loc, options = pytest
addopts = options.get("addopts", [])
if isinstance(addopts, str):
if loc is PytestFile.LEGACY_PYPROJECT and isinstance(addopts, str):
addopts = addopts.split()
return any(opt.startswith("-r") for opt in addopts)

Expand All @@ -326,7 +375,7 @@ class PP309(PyProject):
url = mk_url("pytest")

@staticmethod
def check(pyproject: dict[str, Any]) -> bool:
def check(pytest: tuple[PytestFile, dict[str, Any]]) -> bool:
"""
`filterwarnings` must be set (probably to at least `["error"]`). Python
will hide important warnings otherwise, like deprecations.
Expand All @@ -336,7 +385,7 @@ def check(pyproject: dict[str, Any]) -> bool:
filterwarnings = ["error"]
```
"""
options = pyproject["tool"]["pytest"]["ini_options"]
_, options = pytest
return "filterwarnings" in options


Expand Down
Loading
Loading