fix(tests): use riot lockfiles [backport #5246 to 1.9] (#5624)

Backport of #5246 to 1.9

This change makes the Riot test suite build its virtual environments
using fully-specified dependency tree files that live in source control.

## How to Review

1. Review #5227 and leave
your comments on this change, or use [this filtered diff
view](https://github.com/DataDog/dd-trace-py/pull/5246/files?file-filters%5B%5D=.py&file-filters%5B%5D=.rst&file-filters%5B%5D=.yml&file-filters%5B%5D=No+extension&file-filters%5B%5D=dotfile&show-deleted-files=true&show-viewed-files=true)

## Why?

The computation involved in building the tree of dependencies for a
given test virtual environment is not small. By performing that work
only when the tree changes rather than on every test run, we make our CI
build time shorter.

Having the entire dependency tree in source control allows developers to
unambiguously understand which versions of nth-order dependencies are in
use in a given test environment without needing to run the tests
locally, which can make debugging tests easier.

Using `riot run -c` to automatically rebuild these requirements files
when necessary facilitates future changes like [having a bot rebuild and
PR them
periodically](https://github.com/DataDog/dd-trace-py/tree/emmett.butler/latest-automation).

## How?

By depending on riot==0.17.1, Riot runs on this branch look for
requirements lockfiles and create them if they don't exist. That
combined with the fact that the repo now contains a lockfile for each
venv means that there are no changes necessary to the testing logic
itself.

This change adds `scripts/compile-and-prune-test-requirements`. That
script builds requirements.txt lockfiles for all Riot hashes in the
current working tree that don't already have one in
`.riot/requirements`. It also deletes such files for Riot hashes that
don't exist. In a future change, this script could run under automation,
commit its changes, and open pull requests.

## Notes

**If you're running tests locally and making changes to the riotfile,
you'll need to run `riot` with the `-c` flag to ensure that those
changes are reflected in the new dependency trees. You should commit
these changes to dependency tree lockfiles and include them in pull
requests that change the riotfile.**

This change removes the list of latest tested versions in
`dependencies.py` as well as the logic that parses that list and uses it
in tests. The finding of the latest version is now handled by Riot,
which runs `pip-compile` internally.

Most of the changes to the riotfile in this diff are either:

* workarounds for the fact that versions of `pip-compile` prior to 3.7
don't do
"[backtracking](https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking)"
during dependency resolution
* pins to the same versions as in the deleted `dependencies.py` file due
to compile or test failures with newer versions

## Risk

The main risk I can think of is that this adds some confusion for
contributors about how to get riot to notice their venv changes. I've
updated the contributor docs accordingly.

## Checklist

- [x] Finish riot pull request DataDog/riot#196
- [x] Finish the next riot pull request
DataDog/riot#202
- [x] Generate all requirements lockfiles
- [x] Fix riotfile to make tests pass on this PR
- [x] Move automation changes to separate PR
- [x] Change(s) are motivated and described in the PR description.
- [x] Testing strategy is described if automated tests are not included
in the PR.
- [x] Risk is outlined (performance impact, potential for breakage,
maintainability, etc).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines)
are followed.
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/)).
- [x] Author is aware of the performance implications of this PR as
reported in the benchmarks PR comment.

## Reviewer Checklist

- [x] Title is accurate.
- [x] No unnecessary changes are introduced.
- [x] Description motivates each change.
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes unless absolutely necessary.
- [x] Testing strategy adequately addresses listed risk(s).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] Release note makes sense to a user of the library.
- [x] Reviewer is aware of, and discussed the performance implications
of this PR as reported in the benchmarks PR comment.

---------

Co-authored-by: Emmett Butler <[email protected]>
Co-authored-by: Emmett Butler <[email protected]>

Author: brettlangdon

.circleci/dependencies.py

@@ -123,7 +123,8 @@ ENV/
 .vscode/
 
 # Riot
-.riot/
+.riot/venv*
+.riot/requirements/*.in
 
 # Auto-generated version file
 ddtrace/_version.py

.gitignore

@@ -0,0 +1,25 @@
+#
+# This file is autogenerated by pip-compile with Python 3.10
+# by the following command:
+#
+#    pip-compile --no-annotate --resolver=backtracking .riot/requirements/1011db1.in
+#
+aniso8601==9.0.1
+attrs==22.2.0
+coverage[toml]==7.2.2
+exceptiongroup==1.1.1
+graphene==3.2.2
+graphql-core==3.1.7
+graphql-relay==3.1.5
+hypothesis==6.45.0
+iniconfig==2.0.0
+mock==5.0.1
+opentracing==2.4.0
+packaging==23.0
+pluggy==1.0.0
+pytest==7.2.2
+pytest-asyncio==0.20.3
+pytest-cov==4.0.0
+pytest-mock==3.10.0
+sortedcontainers==2.4.0
+tomli==2.0.1

.riot/requirements/1011db1.txt

@@ -0,0 +1,27 @@
+#
+# This file is autogenerated by pip-compile with python 3.6
+# To update, run:
+#
+#    pip-compile --no-annotate .riot/requirements/1017b7b.in
+#
+amqp==2.6.1
+attrs==22.2.0
+coverage[toml]==6.2
+hypothesis==6.31.6
+importlib-metadata==4.8.3
+iniconfig==1.1.1
+kombu==4.0.2
+mock==5.0.1
+opentracing==2.4.0
+packaging==21.3
+pluggy==1.0.0
+py==1.11.0
+pyparsing==3.0.9
+pytest==7.0.1
+pytest-cov==4.0.0
+pytest-mock==3.6.1
+sortedcontainers==2.4.0
+tomli==1.2.3
+typing-extensions==4.1.1
+vine==1.3.0
+zipp==3.6.0

.riot/requirements/1017b7b.txt

@@ -0,0 +1,22 @@
+#
+# This file is autogenerated by pip-compile with Python 3.9
+# by the following command:
+#
+#    pip-compile --no-annotate --resolver=backtracking .riot/requirements/1018a90.in
+#
+attrs==22.2.0
+coverage[toml]==7.2.2
+exceptiongroup==1.1.1
+hypothesis==6.45.0
+iniconfig==2.0.0
+mock==5.0.1
+opentracing==2.4.0
+packaging==23.0
+pluggy==1.0.0
+pytest==7.2.2
+pytest-cov==4.0.0
+pytest-mock==3.10.0
+redis==3.0.1
+redis-py-cluster==2.0.0
+sortedcontainers==2.4.0
+tomli==2.0.1

.riot/requirements/1018a90.txt

@@ -0,0 +1,26 @@
+#
+# This file is autogenerated by pip-compile with python 3.6
+# To update, run:
+#
+#    pip-compile --no-annotate .riot/requirements/10192c8.in
+#
+attrs==22.2.0
+coverage[toml]==6.2
+hypothesis==6.31.6
+importlib-metadata==4.8.3
+iniconfig==1.1.1
+mako==1.1.6
+markupsafe==2.0.1
+mock==5.0.1
+opentracing==2.4.0
+packaging==21.3
+pluggy==1.0.0
+py==1.11.0
+pyparsing==3.0.9
+pytest==7.0.1
+pytest-cov==4.0.0
+pytest-mock==3.6.1
+sortedcontainers==2.4.0
+tomli==1.2.3
+typing-extensions==4.1.1
+zipp==3.6.0

.riot/requirements/10192c8.txt

@@ -0,0 +1,27 @@
+#
+# This file is autogenerated by pip-compile with Python 3.7
+# by the following command:
+#
+#    pip-compile --no-annotate --resolver=backtracking .riot/requirements/10212ca.in
+#
+attrs==22.2.0
+coverage[toml]==7.2.2
+exceptiongroup==1.1.1
+googleapis-common-protos==1.58.0
+grpcio==1.51.3
+hypothesis==6.45.0
+importlib-metadata==6.0.0
+iniconfig==2.0.0
+mock==5.0.1
+opentracing==2.4.0
+packaging==23.0
+pluggy==1.0.0
+protobuf==4.22.1
+pytest==7.2.2
+pytest-asyncio==0.20.3
+pytest-cov==4.0.0
+pytest-mock==3.10.0
+sortedcontainers==2.4.0
+tomli==2.0.1
+typing-extensions==4.5.0
+zipp==3.15.0

.riot/requirements/10212ca.txt

@@ -0,0 +1,70 @@
+#
+# This file is autogenerated by pip-compile with Python 3.8
+# by the following command:
+#
+#    pip-compile --no-annotate --resolver=backtracking .riot/requirements/102421e.in
+#
+attrs==22.2.0
+aws-sam-translator==1.62.0
+aws-xray-sdk==2.11.0
+boto==2.49.0
+boto3==1.26.92
+botocore==1.29.92
+certifi==2022.12.7
+cffi==1.15.1
+cfn-lint==0.53.1
+charset-normalizer==3.1.0
+coverage[toml]==7.2.2
+cryptography==39.0.2
+docker==6.0.1
+ecdsa==0.14.1
+exceptiongroup==1.1.1
+hypothesis==6.45.0
+idna==2.10
+iniconfig==2.0.0
+jinja2==2.11.3
+jmespath==1.0.1
+jsondiff==2.0.0
+jsonpatch==1.32
+jsonpointer==2.3
+jsonschema==3.2.0
+junit-xml==1.9
+markupsafe==1.1.1
+mock==5.0.1
+more-itertools==9.1.0
+moto==1.3.16
+networkx==2.8.8
+opentracing==2.4.0
+packaging==23.0
+pluggy==1.0.0
+pyasn1==0.4.8
+pycparser==2.21
+pydantic==1.10.6
+pynamodb==5.4.1
+pyrsistent==0.19.3
+pytest==7.2.2
+pytest-cov==4.0.0
+pytest-mock==3.10.0
+python-dateutil==2.8.2
+python-jose[cryptography]==3.3.0
+pytz==2022.7.1
+pyyaml==6.0
+requests==2.28.2
+responses==0.23.1
+rsa==4.9
+s3transfer==0.6.0
+six==1.16.0
+sortedcontainers==2.4.0
+sshpubkeys==3.3.1
+tomli==2.0.1
+types-pyyaml==6.0.12.8
+typing-extensions==4.5.0
+urllib3==1.26.15
+websocket-client==1.5.1
+werkzeug==2.1.2
+wrapt==1.15.0
+xmltodict==0.13.0
+zipp==3.15.0
+
+# The following packages are considered to be unsafe in a requirements file:
+# setuptools