|
| 1 | +# Populate Tox |
| 2 | + |
| 3 | +We integrate with a number of frameworks and libraries and have a test suite for |
| 4 | +each. The tests run against different versions of the framework/library to make |
| 5 | +sure we support everything we claim to. |
| 6 | + |
| 7 | +This `populate_tox.py` script is responsible for picking reasonable versions to |
| 8 | +test automatically and generating parts of `tox.ini` to capture this. |
| 9 | + |
| 10 | +## How it works |
| 11 | + |
| 12 | +There is a template in this directory called `tox.jinja` which contains a |
| 13 | +combination of hardcoded and generated entries. |
| 14 | + |
| 15 | +The `populate_tox.py` script fills out the auto-generated part of that template. |
| 16 | +It does this by querying PyPI for each framework's package and its metadata and |
| 17 | +then determining which versions make sense to test to get good coverage. |
| 18 | + |
| 19 | +The lowest supported and latest version of a framework are always tested, with |
| 20 | +a number of releases in between: |
| 21 | +- If the package has majors, we pick the highest version of each major. For the |
| 22 | + latest major, we also pick the lowest version in that major. |
| 23 | +- If the package doesn't have multiple majors, we pick two versions in between |
| 24 | + lowest and highest. |
| 25 | + |
| 26 | +#### Caveats |
| 27 | + |
| 28 | +- Make sure the integration name is the same everywhere. If it consists of |
| 29 | + multiple words, use an underscore instead of a hyphen. |
| 30 | + |
| 31 | +## Defining constraints |
| 32 | + |
| 33 | +The `TEST_SUITE_CONFIG` dictionary defines, for each integration test suite, |
| 34 | +the main package (framework, library) to test with; any additional test |
| 35 | +dependencies, optionally gated behind specific conditions; and optionally |
| 36 | +the Python versions to test on. |
| 37 | + |
| 38 | +Constraints are defined using the format specified below. The following sections describe each key. |
| 39 | + |
| 40 | +``` |
| 41 | +integration_name: { |
| 42 | + "package": name_of_main_package_on_pypi, |
| 43 | + "deps": { |
| 44 | + rule1: [package1, package2, ...], |
| 45 | + rule2: [package3, package4, ...], |
| 46 | + }, |
| 47 | + "python": python_version_specifier, |
| 48 | +} |
| 49 | +``` |
| 50 | + |
| 51 | +### `package` |
| 52 | + |
| 53 | +The name of the third party package as it's listed on PyPI. The script will |
| 54 | +be picking different versions of this package to test. |
| 55 | + |
| 56 | +This key is mandatory. |
| 57 | + |
| 58 | +### `deps` |
| 59 | + |
| 60 | +The test dependencies of the test suite. They're defined as a dictionary of |
| 61 | +`rule: [package1, package2, ...]` key-value pairs. All packages |
| 62 | +in the package list of a rule will be installed as long as the rule applies. |
| 63 | + |
| 64 | +`rule`s are predefined. Each `rule` must be one of the following: |
| 65 | + - `*`: packages will be always installed |
| 66 | + - a version specifier on the main package (e.g. `<=0.32`): packages will only |
| 67 | + be installed if the main package falls into the version bounds specified |
| 68 | + - specific Python version(s) in the form `py3.8,py3.9`: packages will only be |
| 69 | + installed if the Python version matches one from the list |
| 70 | + |
| 71 | +Rules can be used to specify version bounds on older versions of the main |
| 72 | +package's dependencies, for example. If e.g. Flask tests generally need |
| 73 | +Werkzeug and don't care about its version, but Flask older than 3.0 needs |
| 74 | +a specific Werkzeug version to work, you can say: |
| 75 | + |
| 76 | +```python |
| 77 | +"flask": { |
| 78 | + "deps": { |
| 79 | + "*": ["Werkzeug"], |
| 80 | + "<3.0": ["Werkzeug<2.1.0"], |
| 81 | + }, |
| 82 | + ... |
| 83 | +} |
| 84 | +``` |
| 85 | + |
| 86 | +If you need to install a specific version of a secondary dependency on specific |
| 87 | +Python versions, you can say: |
| 88 | + |
| 89 | +```python |
| 90 | +"celery": { |
| 91 | + "deps": { |
| 92 | + "*": ["newrelic", "redis"], |
| 93 | + "py3.7": ["importlib-metadata<5.0"], |
| 94 | + }, |
| 95 | + ... |
| 96 | +} |
| 97 | +``` |
| 98 | +This key is optional. |
| 99 | + |
| 100 | +### `python` |
| 101 | + |
| 102 | +Sometimes, the whole test suite should only run on specific Python versions. |
| 103 | +This can be achieved via the `python` key, which expects a version specifier. |
| 104 | + |
| 105 | +For example, if you want AIOHTTP tests to only run on Python 3.7+, you can say: |
| 106 | + |
| 107 | +```python |
| 108 | +"aiohttp": { |
| 109 | + "python": ">=3.7", |
| 110 | + ... |
| 111 | +} |
| 112 | +``` |
| 113 | + |
| 114 | +The `python` key is optional, and when possible, it should be omitted. The script |
| 115 | +should automatically detect which Python versions the package supports. |
| 116 | +However, if a package has broken |
| 117 | +metadata or the SDK is explicitly not supporting some packages on specific |
| 118 | +Python versions (because of, for example, broken context vars), the `python` |
| 119 | +key can be used. |
| 120 | + |
| 121 | + |
| 122 | +## How-Tos |
| 123 | + |
| 124 | +### Add a new test suite |
| 125 | + |
| 126 | +1. Add the minimum supported version of the framework/library to `_MIN_VERSIONS` |
| 127 | + in `integrations/__init__.py`. This should be the lowest version of the |
| 128 | + framework that we can guarantee works with the SDK. If you've just added the |
| 129 | + integration, you should generally set this to the latest version of the framework |
| 130 | + at the time. |
| 131 | +2. Add the integration and any constraints to `TEST_SUITE_CONFIG`. See the |
| 132 | + "Defining constraints" section for the format. |
| 133 | +3. Add the integration to one of the groups in the `GROUPS` dictionary in |
| 134 | + `scripts/split_tox_gh_actions/split_tox_gh_actions.py`. |
| 135 | +4. Add the `TESTPATH` for the test suite in `tox.jinja`'s `setenv` section. |
| 136 | +5. Run `scripts/generate-test-files.sh` and commit the changes. |
| 137 | + |
| 138 | +### Migrate a test suite to populate_tox.py |
| 139 | + |
| 140 | +A handful of integration test suites are still hardcoded. The goal is to migrate |
| 141 | +them all to `populate_tox.py` over time. |
| 142 | + |
| 143 | +1. Remove the integration from the `IGNORE` list in `populate_tox.py`. |
| 144 | +2. Remove the hardcoded entries for the integration from the `envlist` and `deps` sections of `tox.jinja`. |
| 145 | +3. Run `scripts/generate-test-files.sh`. |
| 146 | +4. Run the test suite, either locally or by creating a PR. |
| 147 | +5. Address any test failures that happen. |
| 148 | + |
| 149 | +You might have to introduce additional version bounds on the dependencies of the |
| 150 | +package. Try to determine the source of the failure and address it. |
| 151 | + |
| 152 | +Common scenarios: |
| 153 | +- An old version of the tested package installs a dependency without defining |
| 154 | + an upper version bound on it. A new version of the dependency is installed that |
| 155 | + is incompatible with the package. In this case you need to determine which |
| 156 | + versions of the dependency don't contain the breaking change and restrict this |
| 157 | + in `TEST_SUITE_CONFIG`. |
| 158 | +- Tests are failing on an old Python version. In this case first double-check |
| 159 | + whether we were even testing them on that version in the original `tox.ini`. |
0 commit comments