Skip to content

toxgen: Add variants & move OpenAI under toxgen #4730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Sep 2, 2025
+77 −53
Merged

toxgen: Add variants & move OpenAI under toxgen #4730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7dcb27a
toxgen: Add variants
sentrivana Sep 1, 2025
aedabf8
Merge branch 'master' into ivana/toxgen/variants
sentrivana Sep 1, 2025
8cfb058
.
sentrivana Sep 1, 2025
ef71334
Merge branch 'master' into ivana/toxgen/variants
sentrivana Sep 1, 2025
cd467bf
.
sentrivana Sep 1, 2025
67f2678
.
sentrivana Sep 1, 2025
6d47378
.
sentrivana Sep 1, 2025
7192946
readme
sentrivana Sep 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions .github/workflows/test-integrations-ai.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,14 @@ jobs:
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-langchain-latest"
- name: Test openai latest
- name: Test openai_base latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-openai-latest"
./scripts/runtox.sh "py${{ matrix.python-version }}-openai_base-latest"
- name: Test openai_notiktoken latest
run: |
set -x # print commands that are executed
./scripts/runtox.sh "py${{ matrix.python-version }}-openai_notiktoken-latest"
- name: Test openai_agents latest
run: |
set -x # print commands that are executed
Expand Down Expand Up @@ -141,10 +145,14 @@ jobs:
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-langchain"
- name: Test openai pinned
- name: Test openai_base pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-openai_base"
- name: Test openai_notiktoken pinned
run: |
set -x # print commands that are executed
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-openai"
./scripts/runtox.sh --exclude-latest "py${{ matrix.python-version }}-openai_notiktoken"
- name: Test openai_agents pinned
run: |
set -x # print commands that are executed
Expand Down
8 changes: 8 additions & 0 deletions scripts/populate_tox/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,14 @@ be expressed like so:
}
```

### `integration_name`

Sometimes, the name of the test suite doesn't match the name of the integration.
For example, we have the `openai_base` and `openai_notiktoken` test suites, both
of which are actually testing the `openai` integration. If this is the case, you can use the `integration_name` key to define the name of the integration. If not provided, it will default to the name of the test suite.

Linking an integration to a test suite allows the script to access integration configuration like for example the minimum version defined in `sentry_sdk/integrations/__init__.py`.


## How-Tos

Expand Down
18 changes: 18 additions & 0 deletions scripts/populate_tox/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,24 @@
"loguru": {
"package": "loguru",
},
"openai_base": {
"package": "openai",
"integration_name": "openai",
"deps": {
"*": ["pytest-asyncio", "tiktoken"],
"<1.55": ["httpx<0.28"],
},
"python": ">=3.8",
},
"openai_notiktoken": {
"package": "openai",
"integration_name": "openai",
"deps": {
"*": ["pytest-asyncio"],
"<1.55": ["httpx<0.28"],
},
"python": ">=3.8",
},
"openai_agents": {
"package": "openai-agents",
"deps": {
Expand Down
8 changes: 5 additions & 3 deletions scripts/populate_tox/populate_tox.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,6 @@
"httpx",
"langchain",
"langchain_notiktoken",
"openai",
"openai_notiktoken",
"pure_eval",
"quart",
"ray",
Expand Down Expand Up @@ -141,7 +139,11 @@ def _prefilter_releases(
- the list of prefiltered releases
- an optional prerelease if there is one that should be tested
"""
min_supported = _MIN_VERSIONS.get(integration)
integration_name = (
TEST_SUITE_CONFIG[integration].get("integration_name") or integration
)

min_supported = _MIN_VERSIONS.get(integration_name)
if min_supported is not None:
min_supported = Version(".".join(map(str, min_supported)))
else:
Expand Down
24 changes: 2 additions & 22 deletions scripts/populate_tox/tox.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,13 +83,6 @@ envlist =
{py3.9,py3.11,py3.12}-langchain-latest
{py3.9,py3.11,py3.12}-langchain-notiktoken

# OpenAI
{py3.9,py3.11,py3.12}-openai-v1.0
{py3.9,py3.11,py3.12}-openai-v1.22
{py3.9,py3.11,py3.12}-openai-v1.55
{py3.9,py3.11,py3.12}-openai-latest
{py3.9,py3.11,py3.12}-openai-notiktoken

# OpenTelemetry (OTel)
{py3.7,py3.9,py3.12,py3.13}-opentelemetry

Expand Down Expand Up @@ -252,20 +245,6 @@ deps =
langchain-{latest,notiktoken}: openai>=1.6.1
langchain-latest: tiktoken~=0.6.0

# OpenAI
openai: pytest-asyncio
openai-v1.0: openai~=1.0.0
openai-v1.0: tiktoken
openai-v1.0: httpx<0.28.0
openai-v1.22: openai~=1.22.0
openai-v1.22: tiktoken
openai-v1.22: httpx<0.28.0
openai-v1.55: openai~=1.55.0
openai-v1.55: tiktoken
openai-latest: openai
openai-latest: tiktoken~=0.6.0
openai-notiktoken: openai

# OpenTelemetry (OTel)
opentelemetry: opentelemetry-distro

Expand Down Expand Up @@ -401,7 +380,8 @@ setenv =
launchdarkly: TESTPATH=tests/integrations/launchdarkly
litestar: TESTPATH=tests/integrations/litestar
loguru: TESTPATH=tests/integrations/loguru
openai: TESTPATH=tests/integrations/openai
openai_base: TESTPATH=tests/integrations/openai
openai_notiktoken: TESTPATH=tests/integrations/openai
openai_agents: TESTPATH=tests/integrations/openai_agents
openfeature: TESTPATH=tests/integrations/openfeature
opentelemetry: TESTPATH=tests/integrations/opentelemetry
Expand Down
3 changes: 2 additions & 1 deletion scripts/split_tox_gh_actions/split_tox_gh_actions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,8 @@
"anthropic",
"cohere",
"langchain",
"openai",
"openai_base",
"openai_notiktoken",
"openai_agents",
"huggingface_hub",
],
Expand Down
53 changes: 30 additions & 23 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
# The file (and all resulting CI YAMLs) then need to be regenerated via
# "scripts/generate-test-files.sh".
#
# Last generated: 2025-09-01T12:08:33.833560+00:00
# Last generated: 2025-09-01T14:09:50.564158+00:00

[tox]
requires =
Expand Down Expand Up @@ -83,13 +83,6 @@ envlist =
{py3.9,py3.11,py3.12}-langchain-latest
{py3.9,py3.11,py3.12}-langchain-notiktoken

# OpenAI
{py3.9,py3.11,py3.12}-openai-v1.0
{py3.9,py3.11,py3.12}-openai-v1.22
{py3.9,py3.11,py3.12}-openai-v1.55
{py3.9,py3.11,py3.12}-openai-latest
{py3.9,py3.11,py3.12}-openai-notiktoken

# OpenTelemetry (OTel)
{py3.7,py3.9,py3.12,py3.13}-opentelemetry

Expand Down Expand Up @@ -145,6 +138,16 @@ envlist =
{py3.9,py3.11,py3.12}-cohere-v5.13.12
{py3.9,py3.11,py3.12}-cohere-v5.17.0

{py3.8,py3.11,py3.12}-openai_base-v1.0.1
{py3.8,py3.11,py3.12}-openai_base-v1.35.15
{py3.8,py3.11,py3.12}-openai_base-v1.69.0
{py3.8,py3.12,py3.13}-openai_base-v1.102.0

{py3.8,py3.11,py3.12}-openai_notiktoken-v1.0.1
{py3.8,py3.11,py3.12}-openai_notiktoken-v1.35.15
{py3.8,py3.11,py3.12}-openai_notiktoken-v1.69.0
{py3.8,py3.12,py3.13}-openai_notiktoken-v1.102.0

{py3.10,py3.11,py3.12}-openai_agents-v0.0.19
{py3.10,py3.12,py3.13}-openai_agents-v0.1.0
{py3.10,py3.12,py3.13}-openai_agents-v0.2.10
Expand Down Expand Up @@ -423,20 +426,6 @@ deps =
langchain-{latest,notiktoken}: openai>=1.6.1
langchain-latest: tiktoken~=0.6.0

# OpenAI
openai: pytest-asyncio
openai-v1.0: openai~=1.0.0
openai-v1.0: tiktoken
openai-v1.0: httpx<0.28.0
openai-v1.22: openai~=1.22.0
openai-v1.22: tiktoken
openai-v1.22: httpx<0.28.0
openai-v1.55: openai~=1.55.0
openai-v1.55: tiktoken
openai-latest: openai
openai-latest: tiktoken~=0.6.0
openai-notiktoken: openai

# OpenTelemetry (OTel)
opentelemetry: opentelemetry-distro

Expand Down Expand Up @@ -527,6 +516,23 @@ deps =
cohere-v5.13.12: cohere==5.13.12
cohere-v5.17.0: cohere==5.17.0

openai_base-v1.0.1: openai==1.0.1
openai_base-v1.35.15: openai==1.35.15
openai_base-v1.69.0: openai==1.69.0
openai_base-v1.102.0: openai==1.102.0
openai_base: pytest-asyncio
openai_base: tiktoken
openai_base-v1.0.1: httpx<0.28
openai_base-v1.35.15: httpx<0.28

openai_notiktoken-v1.0.1: openai==1.0.1
openai_notiktoken-v1.35.15: openai==1.35.15
openai_notiktoken-v1.69.0: openai==1.69.0
openai_notiktoken-v1.102.0: openai==1.102.0
openai_notiktoken: pytest-asyncio
openai_notiktoken-v1.0.1: httpx<0.28
openai_notiktoken-v1.35.15: httpx<0.28

openai_agents-v0.0.19: openai-agents==0.0.19
openai_agents-v0.1.0: openai-agents==0.1.0
openai_agents-v0.2.10: openai-agents==0.2.10
Expand Down Expand Up @@ -831,7 +837,8 @@ setenv =
launchdarkly: TESTPATH=tests/integrations/launchdarkly
litestar: TESTPATH=tests/integrations/litestar
loguru: TESTPATH=tests/integrations/loguru
openai: TESTPATH=tests/integrations/openai
openai_base: TESTPATH=tests/integrations/openai
openai_notiktoken: TESTPATH=tests/integrations/openai
openai_agents: TESTPATH=tests/integrations/openai_agents
openfeature: TESTPATH=tests/integrations/openfeature
opentelemetry: TESTPATH=tests/integrations/opentelemetry
Expand Down