Merge branch 'master' into antonpirker/openai-agents-guardrail-span

Author: antonpirker

CONTRIBUTING.md

@@ -74,19 +74,37 @@ That's it. You should be ready to make changes, run tests, and make commits! If
 
 ## Running Tests
 
-You can run all tests with the following command:
+We test against a number of Python language and library versions, which are automatically generated and stored in the [tox.ini](tox.ini) file. The `envlist` defines the environments you can choose from when running tests, and correspond to package versions and environment variables. The `TESTPATH` environment variable, in turn, determines which tests are run.
+
+The tox CLI tool is required to run the tests locally. Follow [the installation instructions](https://tox.wiki/en/latest/installation.html) for tox. Dependencies are installed for you when you run the command below, but _you_ need to bring an appropriate Python interpreter.
+
+[Pyenv](https://github.com/pyenv/pyenv) is a cross-platform utility for managing Python versions. You can also use a conventional package manager, but not all versions may be distributed in the package manager of your choice. For macOS, Versions 3.8 and up can be installed with Homebrew.
+
+An environment consists of the Python major and minor version and the library name and version. The exception to the rule is that you can provide `common` instead of the library information. The environments tied to a specific library usually run the corresponding test suite, while `common` targets all tests but skips those that require uninstalled dependencies.
+
+To run Celery tests for version v5.5.3 of its Python library using a 3.12 interpreter, use
 
 ```bash
-pytest tests/
+tox -p auto -o -e py3.12-celery-v5.5.3
 ```
 
-If you would like to run the tests for a specific integration, use a command similar to the one below:
+or to use the `common` environment, run
 
 ```bash
-pytest -rs tests/integrations/flask/  # Replace "flask" with the specific integration you wish to test
+tox -p auto -o -e py3.12-common
 ```
 
-**Hint:** Tests of integrations need additional dependencies. The switch `-rs` will show you why tests were skipped and what dependencies you need to install for the tests to run. (You can also consult the [tox.ini](tox.ini) file to see what dependencies are installed for each integration)
+To select specific tests, you can forward arguments to `pytest` like so
+
+```bash
+tox -p auto -o -e py3.12-celery-v5.5.3 -- -k test_transaction_events
+```
+
+In general, you use
+
+```bash
+tox -p auto -o -e <tox_env> -- <pytest_args>
+```
 
 ## Adding a New Integration
 

scripts/populate_tox/README.md

@@ -16,8 +16,8 @@ The `populate_tox.py` script fills out the auto-generated part of that template.
 It does this by querying PyPI for each framework's package and its metadata and
 then determining which versions make sense to test to get good coverage.
 
-The lowest supported and latest version of a framework are always tested, with
-a number of releases in between:
+By default, the lowest supported and latest version of a framework are always
+tested, with a number of releases in between:
 - If the package has majors, we pick the highest version of each major.
 - If the package doesn't have multiple majors, we pick two versions in between
   lowest and highest.
@@ -34,7 +34,8 @@ the main package (framework, library) to test with; any additional test
 dependencies, optionally gated behind specific conditions; and optionally
 the Python versions to test on.
 
-Constraints are defined using the format specified below. The following sections describe each key.
+Constraints are defined using the format specified below. The following sections
+describe each key.
 
 ```
 integration_name: {
@@ -43,7 +44,7 @@ integration_name: {
          rule1: [package1, package2, ...],
          rule2: [package3, package4, ...],
      },
-     "python": python_version_specifier,
+     "python": python_version_specifier | dict[package_version_specifier, python_version_specifier],
      "include": package_version_specifier,
      "integration_name": integration_name,
      "num_versions": int,
@@ -102,15 +103,14 @@ Python versions, you can say:
     ...
 }
 ```
+
 This key is optional.
 
 ### `python`
 
 Sometimes, the whole test suite should only run on specific Python versions.
-This can be achieved via the `python` key.
-
-There are two variants how to define the Python versions to run the test suite
-on.
+This can be achieved via the `python` key. There are two variants how to define
+the Python versions to run the test suite on.
 
 If you want the test suite to only be run on specific Python versions, you can
 set `python` to a version specifier. For example, if you want AIOHTTP tests to
@@ -142,18 +142,17 @@ say:
 The `python` key is optional, and when possible, it should be omitted. The script
 should automatically detect which Python versions the package supports. However,
 if a package has broken metadata or the SDK is explicitly not supporting some
-packages on specific Python versions (because of, for example, broken context
-vars), the `python` key can be used.
+packages on specific Python versions, the `python` key can be used.
 
 ### `include`
 
-Sometimes we only want to consider testing some specific versions of packages.
-For example, the Starlite package has two alpha prereleases of version 2.0.0, but
-we do not want to test these, since Starlite 2.0 was renamed to Litestar.
+Sometimes we only want to test specific versions of packages. For example, the
+Starlite package has two alpha prereleases of version 2.0.0, but we do not want
+to test these, since Starlite 2.0 was renamed to Litestar.
 
 The value of the `include` key expects a version specifier defining which
 versions should be considered for testing. For example, since we only want to test
-versions below 2.x in Starlite, we can use
+versions below 2.x in Starlite, we can use:
 
 ```python
 "starlite": {
@@ -178,13 +177,21 @@ be expressed like so:
 
 Sometimes, the name of the test suite doesn't match the name of the integration.
 For example, we have the `openai_base` and `openai_notiktoken` test suites, both
-of which are actually testing the `openai` integration. If this is the case, you can use the `integration_name` key to define the name of the integration. If not provided, it will default to the name of the test suite.
+of which are actually testing the `openai` integration. If this is the case, you
+can use the `integration_name` key to define the name of the integration. If not
+provided, it will default to the name of the test suite.
 
-Linking an integration to a test suite allows the script to access integration configuration like for example the minimum version defined in `sentry_sdk/integrations/__init__.py`.
+Linking an integration to a test suite allows the script to access integration
+configuration like, for example, the minimum supported version defined in
+`sentry_sdk/integrations/__init__.py`.
 
 ### `num_versions`
 
-With this option you can tweak the default version picking behavior by specifying how many package versions should be tested. It accepts an integer equal to or greater than 2, as the oldest and latest supported versions will always be picked. Additionally, if there is a recent prerelease, it'll also always be picked (this doesn't count towards `num_versions`).
+With this option you can tweak the default version picking behavior by specifying
+how many package versions should be tested. It accepts an integer equal to or
+greater than 2, as the oldest and latest supported versions will always be
+picked. Additionally, if there is a recent prerelease, it'll also always be
+picked (this doesn't count towards `num_versions`).
 
 
 ## How-Tos
@@ -202,26 +209,3 @@ With this option you can tweak the default version picking behavior by specifyin
    `scripts/split_tox_gh_actions/split_tox_gh_actions.py`.
 4. Add the `TESTPATH` for the test suite in `tox.jinja`'s `setenv` section.
 5. Run `scripts/generate-test-files.sh` and commit the changes.
-
-### Migrate a test suite to populate_tox.py
-
-A handful of integration test suites are still hardcoded. The goal is to migrate
-them all to `populate_tox.py` over time.
-
-1. Remove the integration from the `IGNORE` list in `populate_tox.py`.
-2. Remove the hardcoded entries for the integration from the `envlist` and `deps` sections of `tox.jinja`.
-3. Run `scripts/generate-test-files.sh`.
-4. Run the test suite, either locally or by creating a PR.
-5. Address any test failures that happen.
-
-You might have to introduce additional version bounds on the dependencies of the
-package. Try to determine the source of the failure and address it.
-
-Common scenarios:
-- An old version of the tested package installs a dependency without defining
-  an upper version bound on it. A new version of the dependency is installed that
-  is incompatible with the package. In this case you need to determine which
-  versions of the dependency don't contain the breaking change and restrict this
-  in `TEST_SUITE_CONFIG`.
-- Tests are failing on an old Python version. In this case first double-check
-  whether we were even testing them on that version in the original `tox.ini`.