Skip to content

Commit b0a0772

Browse files
sentrivanasentrivana
committed
docs
1 parent 0adedc7 commit b0a0772

File tree

6 files changed

+137
-37
lines changed

6 files changed

+137
-37
lines changed

‎scripts/generate-test-files.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
3+
# This script generates tox.ini and CI YAML files in one go.
4+
5+
set -xe
6+
7+
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
8+
9+
python -m venv .venv
10+
source .venv/bin/activate
11+
12+
pip install -r "$SCRIPT_DIR/populate_tox/requirements.txt"
13+
pip install -r "$SCRIPT_DIR/split_tox_gh_actions/requirements.txt"
14+
15+
python scripts/populate_tox/populate_tox.py
16+
python scripts/split_tox_gh_actions/split_tox_gh_actions.py

‎scripts/populate_tox/README.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Populate Tox
2+
3+
We integrate with a number of frameworks and libraries and have a test suite for
4+
each. The tests run against different versions of the framework/library to make
5+
sure we support everything we claim to.
6+
7+
This `populate_tox.py` script is responsible for picking reasonable versions to
8+
test automatically and generating parts of `tox.ini` to capture this.
9+
10+
## How it works
11+
12+
There is a template in this directory called `tox.jinja` which contains a
13+
combination of hardcoded and generated entries.
14+
15+
The `populate_tox.py` script fills out the auto-generated part of that template.
16+
It does this by querying PYPI for each framework's package and its metadata and
17+
then determining which versions make sense to test to get good coverage.
18+
19+
The lowest supported and latest version of a framework are always tested, with
20+
a number of releases in between:
21+
- If the package has majors, we pick the highest version of each major. For the
22+
latest major, we also pick the lowest version in that major.
23+
- If the package doesn't have multiple majors, we pick two versions in between
24+
lowest and highest.
25+
26+
27+
### How to add a test suite
28+
29+
1. Add the minimum supported version of the framework/library to `_MIN_VERSIONS`
30+
in `integrations/__init__.py`. This should be the lowest version of the
31+
framework that we can guarantee works with the SDK. If you've just added the
32+
integration, it's fine to set this to the latest version of the framework
33+
at the time.
34+
2. Add the integration and any constraints to `TEST_SUITE_CONFIG`. See below
35+
for the format (or copy-paste one of the existing entries).
36+
3. Add the integration to one of the groups in the `GROUPS` dictionary in
37+
`scripts/split_tox_gh_actions/split_tox_gh_actions.py`.
38+
4. Add the `TESTPATH` for the test suite in `tox.jinja`'s `setenv` section.
39+
5. Run `scripts/generate-test-files.sh` and commit the changes.
40+
41+
#### Caveats
42+
43+
- Make sure the integration name is the same everywhere. If it consists of
44+
multiple words, use an underscore instead of a hyphen.
45+
46+
## Defining constraints
47+
48+
The `TEST_SUITE_CONFIG` dictionary defines, for each integration test suite,
49+
the main package (framework, library) to test with; any additional test
50+
dependencies, optionally gated behind specific conditions; and optionally
51+
the Python versions to test on.
52+
53+
The format is:
54+
55+
```
56+
integration_name: {
57+
"package": name_of_main_package_on_pypi,
58+
"deps": {
59+
rule1: [package1, package2, ...],
60+
rule2: [package3, package4, ...],
61+
},
62+
"python": python_version_specifier,
63+
}
64+
```
65+
66+
The following can be set as a rule:
67+
- `*`: packages will be always installed
68+
- a version specifier on the main package (e.g. `<=0.32`): packages will only
69+
be installed if the main package falls into the version bounds specified
70+
- specific Python version(s) in the form `py3.8,py3.9`: packages will only be
71+
installed if the Python version matches one from the list
72+
73+
Rules can be used to specify version bounds on older versions of the main
74+
package's dependencies, for example. If e.g. Flask tests generally need
75+
Werkzeug and don't care about its version, but Flask older than 3.0 needs
76+
a specific Werkzeug version to work, you can say:
77+
78+
```
79+
"flask": {
80+
"deps": {
81+
"*": ["Werkzeug"],
82+
"<3.0": ["Werkzeug<2.1.0"],
83+
}
84+
}
85+
````
86+
87+
Sometimes, things depend on the Python version installed. If the integration
88+
test should only run on specific Python version, e.g. if you want AIOHTTP
89+
tests to only run on Python 3.7+, you can say:
90+
91+
```
92+
"aiohttp": {
93+
...
94+
"python": ">=3.7",
95+
}
96+
```
97+
98+
If, on the other hand, you need to install a specific version of a secondary
99+
dependency on specific Python versions (so the test suite should still run on
100+
said Python versions, just with different dependency-of-a-dependency bounds),
101+
you can say:
102+
103+
```
104+
"celery": {
105+
...
106+
"deps": {
107+
"*": ["newrelic", "redis"],
108+
"py3.7": ["importlib-metadata<5.0"],
109+
},
110+
},
111+
```

‎scripts/populate_tox/config.py‎

Lines changed: 1 addition & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -3,39 +3,7 @@
33
# dependencies, optionally gated behind specific conditions; and optionally
44
# the Python versions to test on.
55
#
6-
# The format is:
7-
# ```
8-
# integration_name: {
9-
# "package": name_of_main_package_on_pypi,
10-
# "deps": {
11-
# rule1: [package1, package2, ...],
12-
# rule2: [package3, package4, ...],
13-
# },
14-
# "python": python_version_specifier,
15-
# }
16-
# ```
17-
#
18-
# The following can be set as a rule:
19-
# - `*`: packages will be always installed
20-
# - a version specifier on the main package (e.g. `<=0.32`): packages will only
21-
# be installed if the main package falls into the version bounds specified
22-
# - specific Python version(s) in the form `py3.8,py3.9`: packages will only be
23-
# installed if the Python version matches one from the list
24-
#
25-
# Rules can be used to specify version bounds on older versions of the main
26-
# package's dependencies, for example. If e.g. Flask tests generally need
27-
# Werkzeug and don't care about its version, but Flask older than 3.0 needs
28-
# a specific Werkzeug version to work, you can say:
29-
#
30-
# ```
31-
# "flask": {
32-
# "deps": {
33-
# "*": ["Werkzeug"],
34-
# "<3.0": ["Werkzeug<2.1.0"],
35-
# }
36-
# }
37-
# ````
38-
6+
# See scripts/populate_tox/README.md for more info on the format and examples.
397

408
TEST_SUITE_CONFIG = {
419
"aiohttp": {

‎scripts/populate_tox/populate_tox.py‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -424,8 +424,7 @@ def write_tox_file(packages: dict) -> None:
424424
release.python_versions
425425
)
426426

427-
# Give PYPI some breathing room
428-
time.sleep(0.25)
427+
time.sleep(0.1) # give PYPI some breathing room
429428

430429
test_releases = [
431430
release for release in test_releases if release.python_versions

‎scripts/populate_tox/requirements.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
jinja2
2+
packaging
3+
requests

‎scripts/populate_tox/tox.jinja‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,11 @@
44
# and then run "tox" from this directory.
55
#
66
# This file has been generated from a template
7-
# by scripts/populate_tox/populate_tox.py. Any changes to the file should be made
8-
# in the script and the file regenerated.
7+
# by "scripts/populate_tox/populate_tox.py". Any changes to the file should
8+
# be made in the template (if you want to change a hardcoded part of the file)
9+
# or in the script (if you want to change the auto-generated part).
10+
# The file (and all resulting CI YAMLs) then need to be regenerated via
11+
# "scripts/generate-test-files.sh".
912

1013
[tox]
1114
requires =

0 commit comments

Comments
 (0)