Skip to content

chore: Clean up toxgen #4855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 24, 2025
Merged

chore: Clean up toxgen #4855

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8d1daa6
chore: Clean up toxgen
sentrivana Sep 24, 2025
cdf8f26
Merge branch 'master' into ivana/toxgen/clean-more-stuff-up
sentrivana Sep 24, 2025
b2bb64c
.
sentrivana Sep 24, 2025
ce01379
Merge branch 'master' into ivana/toxgen/clean-more-stuff-up
sentrivana Sep 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 24 additions & 40 deletions scripts/populate_tox/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ The `populate_tox.py` script fills out the auto-generated part of that template.
It does this by querying PyPI for each framework's package and its metadata and
then determining which versions make sense to test to get good coverage.

The lowest supported and latest version of a framework are always tested, with
a number of releases in between:
By default, the lowest supported and latest version of a framework are always
tested, with a number of releases in between:
- If the package has majors, we pick the highest version of each major.
- If the package doesn't have multiple majors, we pick two versions in between
lowest and highest.
Expand All @@ -34,7 +34,8 @@ the main package (framework, library) to test with; any additional test
dependencies, optionally gated behind specific conditions; and optionally
the Python versions to test on.

Constraints are defined using the format specified below. The following sections describe each key.
Constraints are defined using the format specified below. The following sections
describe each key.

```
integration_name: {
Expand All @@ -43,7 +44,7 @@ integration_name: {
rule1: [package1, package2, ...],
rule2: [package3, package4, ...],
},
"python": python_version_specifier,
"python": python_version_specifier | dict[package_version_specifier, python_version_specifier],
"include": package_version_specifier,
"integration_name": integration_name,
"num_versions": int,
Expand Down Expand Up @@ -102,15 +103,14 @@ Python versions, you can say:
...
}
```

This key is optional.

### `python`

Sometimes, the whole test suite should only run on specific Python versions.
This can be achieved via the `python` key.

There are two variants how to define the Python versions to run the test suite
on.
This can be achieved via the `python` key. There are two variants how to define
the Python versions to run the test suite on.

If you want the test suite to only be run on specific Python versions, you can
set `python` to a version specifier. For example, if you want AIOHTTP tests to
Expand Down Expand Up @@ -142,18 +142,17 @@ say:
The `python` key is optional, and when possible, it should be omitted. The script
should automatically detect which Python versions the package supports. However,
if a package has broken metadata or the SDK is explicitly not supporting some
packages on specific Python versions (because of, for example, broken context
vars), the `python` key can be used.
packages on specific Python versions, the `python` key can be used.

### `include`

Sometimes we only want to consider testing some specific versions of packages.
For example, the Starlite package has two alpha prereleases of version 2.0.0, but
we do not want to test these, since Starlite 2.0 was renamed to Litestar.
Sometimes we only want to test specific versions of packages. For example, the
Starlite package has two alpha prereleases of version 2.0.0, but we do not want
to test these, since Starlite 2.0 was renamed to Litestar.

The value of the `include` key expects a version specifier defining which
versions should be considered for testing. For example, since we only want to test
versions below 2.x in Starlite, we can use
versions below 2.x in Starlite, we can use:

```python
"starlite": {
Expand All @@ -178,13 +177,21 @@ be expressed like so:

Sometimes, the name of the test suite doesn't match the name of the integration.
For example, we have the `openai_base` and `openai_notiktoken` test suites, both
of which are actually testing the `openai` integration. If this is the case, you can use the `integration_name` key to define the name of the integration. If not provided, it will default to the name of the test suite.
of which are actually testing the `openai` integration. If this is the case, you
can use the `integration_name` key to define the name of the integration. If not
provided, it will default to the name of the test suite.

Linking an integration to a test suite allows the script to access integration configuration like for example the minimum version defined in `sentry_sdk/integrations/__init__.py`.
Linking an integration to a test suite allows the script to access integration
configuration like, for example, the minimum supported version defined in
`sentry_sdk/integrations/__init__.py`.

### `num_versions`

With this option you can tweak the default version picking behavior by specifying how many package versions should be tested. It accepts an integer equal to or greater than 2, as the oldest and latest supported versions will always be picked. Additionally, if there is a recent prerelease, it'll also always be picked (this doesn't count towards `num_versions`).
With this option you can tweak the default version picking behavior by specifying
how many package versions should be tested. It accepts an integer equal to or
greater than 2, as the oldest and latest supported versions will always be
picked. Additionally, if there is a recent prerelease, it'll also always be
picked (this doesn't count towards `num_versions`).


## How-Tos
Expand All @@ -202,26 +209,3 @@ With this option you can tweak the default version picking behavior by specifyin
`scripts/split_tox_gh_actions/split_tox_gh_actions.py`.
4. Add the `TESTPATH` for the test suite in `tox.jinja`'s `setenv` section.
5. Run `scripts/generate-test-files.sh` and commit the changes.

### Migrate a test suite to populate_tox.py

A handful of integration test suites are still hardcoded. The goal is to migrate
them all to `populate_tox.py` over time.

1. Remove the integration from the `IGNORE` list in `populate_tox.py`.
2. Remove the hardcoded entries for the integration from the `envlist` and `deps` sections of `tox.jinja`.
3. Run `scripts/generate-test-files.sh`.
4. Run the test suite, either locally or by creating a PR.
5. Address any test failures that happen.

You might have to introduce additional version bounds on the dependencies of the
package. Try to determine the source of the failure and address it.

Common scenarios:
- An old version of the tested package installs a dependency without defining
an upper version bound on it. A new version of the dependency is installed that
is incompatible with the package. In this case you need to determine which
versions of the dependency don't contain the breaking change and restrict this
in `TEST_SUITE_CONFIG`.
- Tests are failing on an old Python version. In this case first double-check
whether we were even testing them on that version in the original `tox.ini`.
49 changes: 30 additions & 19 deletions scripts/populate_tox/populate_tox.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
"""
This script populates tox.ini automatically using release data from PyPI.

See scripts/populate_tox/README.md for more info.
"""

import functools
Expand Down Expand Up @@ -123,6 +125,7 @@ def _save_to_cache(package: str, version: Version, release: Optional[dict]) -> N
releases_cache.write(json.dumps(_normalize_release(release)) + "\n")

CACHE[_normalize_name(package)][str(version)] = release
CACHE[_normalize_name(package)][str(version)]["_accessed"] = True


def _prefilter_releases(
Expand Down Expand Up @@ -150,7 +153,8 @@ def _prefilter_releases(
min_supported = Version(".".join(map(str, min_supported)))
else:
print(
f" {integration} doesn't have a minimum version defined in sentry_sdk/integrations/__init__.py. Consider defining one"
f" {integration} doesn't have a minimum version defined in "
f"sentry_sdk/integrations/__init__.py. Consider defining one"
)

include_versions = None
Expand Down Expand Up @@ -225,7 +229,8 @@ def get_supported_releases(
Get a list of releases that are currently supported by the SDK.

This takes into account a handful of parameters (Python support, the lowest
version we've defined for the framework, the date of the release).
supported version we've defined for the framework, optionally the date
of the release).

We return the list of supported releases and optionally also the newest
prerelease, if it should be tested (meaning it's for a version higher than
Expand Down Expand Up @@ -272,9 +277,9 @@ def pick_releases_to_test(
) -> list[Version]:
"""Pick a handful of releases to test from a sorted list of supported releases."""
# If the package has majors (or major-like releases, even if they don't do
# semver), we want to make sure we're testing them all (unless there's too
# many). If not, we just pick the oldest, the newest, and a couple
# in between.
# semver), we want to make sure we're testing them all. If it doesn't have
# multiple majors, we just pick the oldest, the newest, and a couple of
# releases in between.
#
# If there is a relevant prerelease, also test that in addition to the above.
num_versions = TEST_SUITE_CONFIG[integration].get("num_versions")
Expand Down Expand Up @@ -342,7 +347,7 @@ def supported_python_versions(
custom_supported_versions: Optional[
Union[SpecifierSet, dict[SpecifierSet, SpecifierSet]]
] = None,
version: Optional[Version] = None,
release_version: Optional[Version] = None,
) -> list[Version]:
"""
Get the intersection of Python versions supported by the package and the SDK.
Expand All @@ -362,6 +367,12 @@ def supported_python_versions(
on Python 3.7, so we can provide this as `custom_supported_versions`. The
result of this function will then by the intersection of all three, i.e.,
[3.7].
- The Python SDK supports Python 3.6-3.13. The package supports 3.5-3.8.
Additionally, we have a limitation in place to only test this framework on
Python 3.5 if the framework version is <2.0. `custom_supported_versions`
will contain this restriction, and `release_version` will contain the
version of the package we're currently looking at, to determine whether the
<2.0 restriction applies in this case.
"""
supported = []

Expand All @@ -377,11 +388,11 @@ def supported_python_versions(
if curr in custom_supported_versions:
supported.append(curr)

elif version is not None and isinstance(
elif release_version is not None and isinstance(
custom_supported_versions, dict
):
for v, py in custom_supported_versions.items():
if version in v:
if release_version in v:
if curr in py:
supported.append(curr)
break
Expand Down Expand Up @@ -435,8 +446,10 @@ def _parse_python_versions_from_classifiers(classifiers: list[str]) -> list[Vers

def determine_python_versions(pypi_data: dict) -> Union[SpecifierSet, list[Version]]:
"""
Given data from PyPI's release endpoint, determine the Python versions supported by the package
from the Python version classifiers, when present, or from `requires_python` if there are no classifiers.
Determine the Python versions supported by the package from PyPI data.

We're looking at Python version classifiers, if present, and
`requires_python` if there are no classifiers.
"""
try:
classifiers = pypi_data["info"]["classifiers"]
Expand All @@ -453,7 +466,7 @@ def determine_python_versions(pypi_data: dict) -> Union[SpecifierSet, list[Versi

# We only use `requires_python` if there are no classifiers. This is because
# `requires_python` doesn't tell us anything about the upper bound, which
# depends on when the release first came out
# implicitly depends on when the release first came out.
try:
requires_python = pypi_data["info"]["requires_python"]
except (AttributeError, KeyError):
Expand Down Expand Up @@ -666,7 +679,8 @@ def main(fail_on_changes: bool = False) -> None:
# timestamp so that we don't fail CI on a PR just because a new package
# version was released, leading to unrelated changes in tox.ini.
print(
f"Since we're in fail_on_changes mode, we're only considering releases before the last tox.ini update at {last_updated.isoformat()}."
f"Since we're in fail_on_changes mode, we're only considering "
f"releases before the last tox.ini update at {last_updated.isoformat()}."
)

global MIN_PYTHON_VERSION, MAX_PYTHON_VERSION
Expand Down Expand Up @@ -789,19 +803,16 @@ def main(fail_on_changes: bool = False) -> None:

Please don't make manual changes to `tox.ini`. Instead, make the
changes to the `tox.jinja` template and/or the `populate_tox.py`
script (as applicable) and regenerate the `tox.ini` file with:

python -m venv toxgen.env
. toxgen.env/bin/activate
pip install -r scripts/populate_tox/requirements.txt
python scripts/populate_tox/populate_tox.py
script (as applicable) and regenerate the `tox.ini` file by
running scripts/generate-test-files.sh
"""
)
)
print("Done checking tox.ini. Looking good!")
else:
print(
"Done generating tox.ini. Make sure to also update the CI YAML files to reflect the new test targets."
"Done generating tox.ini. Make sure to also update the CI YAML "
"files to reflect the new test targets."
)


Expand Down
Loading