fix: use openai client 1.76.0 in vllm_inference.py #1143
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The suggested 1.13.3 is incompatible with httpx 0.28.0 and higher. With Python 3.13.3, my `pip install openai==1.13.3` installed httpx 0.28.1. Then `python openai_compatible/client.py` failed with `TypeError: Client.__init__() got an unexpected keyword argument 'proxies'`. This is a bug in older versions of openai. It was first reported in openai/openai-python#1902 and fixed in openai/openai-python#1905 `pip install openai==1.76.0 && python openai_compatible/client.py` worked. Signed-off-by: David Xia <[email protected]>
before$ pip install openai==1.13.3
Collecting openai==1.13.3
Downloading openai-1.13.3-py3-none-any.whl.metadata (18 kB)
Requirement already satisfied: anyio<5,>=3.5.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.13.3) (4.9.0)
Collecting distro<2,>=1.7.0 (from openai==1.13.3)
Using cached distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)
Collecting httpx<1,>=0.23.0 (from openai==1.13.3)
Using cached httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB)
Requirement already satisfied: pydantic<3,>=1.9.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.13.3) (2.11.3)
Requirement already satisfied: sniffio in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.13.3) (1.3.1)
Collecting tqdm>4 (from openai==1.13.3)
Using cached tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
Requirement already satisfied: typing-extensions<5,>=4.7 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.13.3) (4.13.2)
Requirement already satisfied: idna>=2.8 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from anyio<5,>=3.5.0->openai==1.13.3) (3.10)
Requirement already satisfied: certifi in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from httpx<1,>=0.23.0->openai==1.13.3) (2025.4.26)
Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai==1.13.3)
Downloading httpcore-1.0.9-py3-none-any.whl.metadata (21 kB)
Collecting h11>=0.16 (from httpcore==1.*->httpx<1,>=0.23.0->openai==1.13.3)
Downloading h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)
Requirement already satisfied: annotated-types>=0.6.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from pydantic<3,>=1.9.0->openai==1.13.3) (0.7.0)
Requirement already satisfied: pydantic-core==2.33.1 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from pydantic<3,>=1.9.0->openai==1.13.3) (2.33.1)
Requirement already satisfied: typing-inspection>=0.4.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from pydantic<3,>=1.9.0->openai==1.13.3) (0.4.0)
Downloading openai-1.13.3-py3-none-any.whl (227 kB)
Using cached distro-1.9.0-py3-none-any.whl (20 kB)
Using cached httpx-0.28.1-py3-none-any.whl (73 kB)
Downloading httpcore-1.0.9-py3-none-any.whl (78 kB)
Using cached tqdm-4.67.1-py3-none-any.whl (78 kB)
Downloading h11-0.16.0-py3-none-any.whl (37 kB)
Installing collected packages: tqdm, h11, distro, httpcore, httpx, openai
Successfully installed distro-1.9.0 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 openai-1.13.3 tqdm-4.67.1
$ python openai_compatible/client.py
Traceback (most recent call last):
File "/Users/dxia/src/github.com/modal-labs/modal-examples/06_gpu_and_ml/llm-serving/openai_compatible/client.py", line 235, in <module>
main()
~~~~^^
File "/Users/dxia/src/github.com/modal-labs/modal-examples/06_gpu_and_ml/llm-serving/openai_compatible/client.py", line 128, in main
client = OpenAI(api_key=args.api_key)
File "/Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages/openai/_client.py", line 112, in __init__
super().__init__(
~~~~~~~~~~~~~~~~^
version=__version__,
^^^^^^^^^^^^^^^^^^^^
...<6 lines>...
_strict_response_validation=_strict_response_validation,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages/openai/_base_client.py", line 793, in __init__
self._client = http_client or SyncHttpxClientWrapper(
~~~~~~~~~~~~~~~~~~~~~~^
base_url=base_url,
^^^^^^^^^^^^^^^^^^
...<5 lines>...
follow_redirects=True,
^^^^^^^^^^^^^^^^^^^^^^
)
^
TypeError: Client.__init__() got an unexpected keyword argument 'proxies'after$ pip install openai==1.76.0
Collecting openai==1.76.0
Downloading openai-1.76.0-py3-none-any.whl.metadata (25 kB)
Requirement already satisfied: anyio<5,>=3.5.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.76.0) (4.9.0)
Requirement already satisfied: distro<2,>=1.7.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.76.0) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.76.0) (0.28.1)
Collecting jiter<1,>=0.4.0 (from openai==1.76.0)
Downloading jiter-0.9.0-cp313-cp313-macosx_11_0_arm64.whl.metadata (5.2 kB)
Requirement already satisfied: pydantic<3,>=1.9.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.76.0) (2.11.3)
Requirement already satisfied: sniffio in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.76.0) (1.3.1)
Requirement already satisfied: tqdm>4 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.76.0) (4.67.1)
Requirement already satisfied: typing-extensions<5,>=4.11 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from openai==1.76.0) (4.13.2)
Requirement already satisfied: idna>=2.8 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from anyio<5,>=3.5.0->openai==1.76.0) (3.10)
Requirement already satisfied: certifi in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from httpx<1,>=0.23.0->openai==1.76.0) (2025.4.26)
Requirement already satisfied: httpcore==1.* in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from httpx<1,>=0.23.0->openai==1.76.0) (1.0.9)
Requirement already satisfied: h11>=0.16 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai==1.76.0) (0.16.0)
Requirement already satisfied: annotated-types>=0.6.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from pydantic<3,>=1.9.0->openai==1.76.0) (0.7.0)
Requirement already satisfied: pydantic-core==2.33.1 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from pydantic<3,>=1.9.0->openai==1.76.0) (2.33.1)
Requirement already satisfied: typing-inspection>=0.4.0 in /Users/dxia/.pyenv/versions/3.13.3/envs/modal-examples/lib/python3.13/site-packages (from pydantic<3,>=1.9.0->openai==1.76.0) (0.4.0)
Downloading openai-1.76.0-py3-none-any.whl (661 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 661.2/661.2 kB 8.8 MB/s eta 0:00:00
Downloading jiter-0.9.0-cp313-cp313-macosx_11_0_arm64.whl (318 kB)
Installing collected packages: jiter, openai
Attempting uninstall: openai
Found existing installation: openai 1.13.3
Uninstalling openai-1.13.3:
Successfully uninstalled openai-1.13.3
Successfully installed jiter-0.9.0 openai-1.76.0
[notice] A new release of pip is available: 25.0.1 -> 25.1
[notice] To update, run: pip install --upgrade pip
$ python openai_compatible/client.py
🔎: Looking up available models on server at https://davidxia--example-vllm-openai-compatible-serve.modal.run/v1/. This may trigger a model load!
🧠: Using neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16
🧠: Using system prompt: You are a poetic assistant, skilled in writing satirical doggerel with creative flair.
You: Compose a limerick about baboons and racoons.
🤖:There once was a baboon named Pete,
Whose raccoon friend, Rocky, would meet,
They'd raid with a grin,
And steal from within,
Their mischief was quite hard to beat. |
|
Are there tests for these small examples? |
|
@davidxia thanks for the PR! And the others as well. There are tests for these examples -- details in The most important tests run the examples on Modal, aiming to mimic the way a user interacts with them. They are controlled by the YAML frontmatter that you can see in some of the example files. You should be able to execute those tests by running These tests aren't sophisticated enough to catch things like this regression in a transitive dependency from a package mentioned in a Markdown code snippet though. |
|
Thanks for reviewing and merging! That's helpful info. 🙏 |
ben-modal
pushed a commit
that referenced
this pull request
May 5, 2025
The suggested 1.13.3 is incompatible with httpx 0.28.0 and higher. With Python 3.13.3, my `pip install openai==1.13.3` installed httpx 0.28.1. Then `python openai_compatible/client.py` failed with `TypeError: Client.__init__() got an unexpected keyword argument 'proxies'`. This is a bug in older versions of openai. It was first reported in openai/openai-python#1902 and fixed in openai/openai-python#1905 `pip install openai==1.76.0 && python openai_compatible/client.py` worked. Signed-off-by: David Xia <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The suggested 1.13.3 is incompatible with httpx 0.28.0 and higher. With Python 3.13.3, my
pip install openai==1.13.3installed httpx 0.28.1. Thenpython openai_compatible/client.pyfailed withTypeError: Client.__init__() got an unexpected keyword argument 'proxies'.This is a bug in older versions of openai. It was first reported in openai/openai-python#1902 and fixed in openai/openai-python#1905
pip install openai==1.76.0 && python openai_compatible/client.pyworked.Type of Change
Checklist
lambda-test: falseis added to example frontmatter (---)modal runor an alternativecmdis provided in the example frontmatter (e.g.cmd: ["modal", "deploy"])argsare provided in the example frontmatter (e.g.args: ["--prompt", "Formula for room temperature superconductor:"]latestpython_versionfor the base image, if it is used~=x.y.zor==x.yversion < 1are pinned to patch version,==0.y.z