Running Ollama Cloud with Pydantic AI

[knana1662]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

I have this code where i am using ollama cloud with pydantic ai. Please kindly assist as i am currently getting the error below:

C:\Users\NanaKwameAsanteDanso>python -u "c:\Users\NanaKwameAsanteDanso\test_ollama.py"
Traceback (most recent call last):
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\models\openai.py", line 523, in _process_response
    response = chat.ChatCompletion.model_validate(response.model_dump())
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic\main.py", line 705, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        obj, strict=strict, from_attributes=from_attributes, context=context, by_alias=by_alias, by_name=by_name
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ChatCompletion
id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/string_type
choices.0.index
  Input should be a valid integer [type=int_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/int_type
object
  Input should be 'chat.completion' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/literal_error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\NanaKwameAsanteDanso\test_ollama.py", line 17, in <module>
    response = agent.run_sync("Why is the sky blue?")  # Use run_sync instead of run
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\agent\abstract.py", line 317, in run_sync
    return get_event_loop().run_until_complete(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.run(
        ^^^^^^^^^
    ...<12 lines>...
        )
        ^
    )
    ^
  File "C:\Users\NanaKwameAsanteDanso\AppData\Local\Programs\Python\Python313\Lib\asyncio\base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\agent\abstract.py", line 218, in run
    async for node in agent_run:
    ...<4 lines>...
                await event_stream_handler(_agent_graph.build_run_context(agent_run.ctx), stream)
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\run.py", line 149, in __anext__
    next_node = await self._graph_run.__anext__()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_graph\graph.py", line 758, in __anext__
    return await self.next(self._next_node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_graph\graph.py", line 731, in next
    self._next_node = await node.run(ctx)
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\_agent_graph.py", line 399, in run
    return await self._make_request(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\_agent_graph.py", line 441, in _make_request
    model_response = await ctx.deps.model.request(message_history, model_settings, model_request_parameters)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\models\openai.py", line 399, in request
    model_response = self._process_response(response)
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\models\openai.py", line 525, in _process_response
    raise UnexpectedModelBehavior(f'Invalid response from OpenAI chat completions endpoint: {e}') from e
pydantic_ai.exceptions.UnexpectedModelBehavior: Invalid response from OpenAI chat completions endpoint: 3 validation errors for ChatCompletion
id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/string_type
choices.0.index
  Input should be a valid integer [type=int_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/int_type
object
  Input should be 'chat.completion' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/literal_error

C:\Users\NanaKwameAsanteDanso>python -u "c:\Users\NanaKwameAsanteDanso\test_ollama.py"
Traceback (most recent call last):
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\models\openai.py", line 523, in _process_response
    response = chat.ChatCompletion.model_validate(response.model_dump())
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic\main.py", line 705, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        obj, strict=strict, from_attributes=from_attributes, context=context, by_alias=by_alias, by_name=by_name
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ChatCompletion
id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/string_type
choices.0.index
  Input should be a valid integer [type=int_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/int_type
object
  Input should be 'chat.completion' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/literal_error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\NanaKwameAsanteDanso\test_ollama.py", line 17, in <module>
    response = agent.run_sync("Why is the sky blue?")  # Use run_sync instead of run
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\agent\abstract.py", line 317, in run_sync
    return get_event_loop().run_until_complete(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.run(
        ^^^^^^^^^
    ...<12 lines>...
        )
        ^
    )
    ^
  File "C:\Users\NanaKwameAsanteDanso\AppData\Local\Programs\Python\Python313\Lib\asyncio\base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\agent\abstract.py", line 218, in run
    async for node in agent_run:
    ...<4 lines>...
                await event_stream_handler(_agent_graph.build_run_context(agent_run.ctx), stream)
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\run.py", line 149, in __anext__
    next_node = await self._graph_run.__anext__()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_graph\graph.py", line 758, in __anext__
    return await self.next(self._next_node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_graph\graph.py", line 731, in next
    self._next_node = await node.run(ctx)
                      ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\_agent_graph.py", line 399, in run
    return await self._make_request(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\_agent_graph.py", line 441, in _make_request
    model_response = await ctx.deps.model.request(message_history, model_settings, model_request_parameters)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\models\openai.py", line 399, in request
    model_response = self._process_response(response)
  File "C:\Users\NanaKwameAsanteDanso\OneDrive - Dynamic Data Solutions Ltd\Desktop\core\Lib\site-packages\pydantic_ai\models\openai.py", line 525, in _process_response
    raise UnexpectedModelBehavior(f'Invalid response from OpenAI chat completions endpoint: {e}') from e
pydantic_ai.exceptions.UnexpectedModelBehavior: Invalid response from OpenAI chat completions endpoint: 3 validation errors for ChatCompletion
id
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/string_type
choices.0.index
  Input should be a valid integer [type=int_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/int_type
object
  Input should be 'chat.completion' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/literal_error

Example Code

from pydantic_ai.models.openai import OpenAIChatModel  # Updated class name
from pydantic_ai.providers.openai import OpenAIProvider  # Correct import
from pydantic_ai import Agent

# Ollama running with OpenAI-compatible endpoint (Ollama Cloud)
ollama_cloud_gpt_oss_120b = OpenAIChatModel(
    model_name="gpt-oss:120b",
    provider=OpenAIProvider(
        base_url="https://ollama.com/v1",   # OpenAI-compatible endpoint
        api_key="<api_key>"                       # API key if required
    )
)

# --- Usage Example ---
agent = Agent(ollama_cloud_gpt_oss_120b)

response = agent.run_sync("Why is the sky blue?")  # Use run_sync instead of run
print(response.output)  # Use response.output instead of response.data

Python, Pydantic AI & LLM client version

Python 3.13.2
pydantic-ai==1.0.10

[DouweM]

@knana1662 Thanks for the report. Can you try to capture the HTTP response using https://ai.pydantic.dev/logfire/ (logfire.instrument_httpx(capture_all=True)) and share it?

[gonzoramos]

Adding this comment at the request ot @DouweM

Calling Ollama cloud with the following code (to a non-streaming response)

import asyncio
import os
import random

import logfire
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.models.groq import GroqModel, GroqModelSettings
from pydantic_ai.providers.openai import OpenAIProvider


from pydantic import BaseModel
from dotenv import load_dotenv

# Load environment variables from a .env file if it exists
env_ok = load_dotenv(os.path.join('..', ".env"))

if not env_ok:
    print("Warning: .env file not found. Environment variables may not be set correctly.")

logfire.configure()
logfire.instrument_httpx(capture_all=True)
logfire.instrument_pydantic_ai()


def roll_dice() -> str:
    """Roll a six-sided die and return the result."""
    return str(random.randint(1, 6))

# --- Agent and Model Configuration ---


ollama_model = OpenAIModel(
    model_name='gpt-oss:120b-cloud', provider=OpenAIProvider(base_url='http://localhost:11434/v1'), settings={"temperature": 0.2}
)

groq_model = GroqModel(
    model_name='qwen/qwen3-32b', settings={"temperature": 0.2},
)

default_system_prompt = """
You are a helpful assistant. 
Do not ask any follow-up questions.
"""

model_settings = {
    "temperature": 0.2,
}

tools = [roll_dice]

agent = Agent(model=ollama_model, tools=tools,
              system_prompt=default_system_prompt, output_type=str, model_settings={})


async def main():
    prompt = 'roll the dice'

    result = await agent.run(prompt)
    print(f"Result: {result}")

# ---

if __name__ == '__main__':
    asyncio.run(main())

and then looking at my logfire dashboard:

I see the followng response:

{
  "start_timestamp": "2025-09-26T18:33:51.286122Z",
  "trace_id": "0199874d877aad385102e59237b65da1",
  "span_id": "47295058fd1b9dc7",
  "parent_span_id": "8fa34378e9632003",
  "span_name": "Reading response body",
  "level": 9,
  "service_name": "unknown_service",
  "otel_scope_name": "logfire.httpx",
  "tags": [],
  "created_at": 1758911632763.902,
  "end_timestamp": "2025-09-26T18:33:51.288111Z",
  "kind": "span",
  "message": "Reading response body",
  "is_exception": false,
  "exception_type": null,
  "otel_scope_version": "4.10.0",
  "service_version": "3e2b97c25251b664c31f7dc105926dba5d5f577a",
  "http_response_status_code": null,
  "gen_ai_operation_name": null,
  "gen_ai_request_model": null,
  "gen_ai_response_model": null,
  "gen_ai_system": null,
  "gen_ai_usage_input_tokens": 0,
  "gen_ai_usage_output_tokens": 0,
  "gen_ai_response_id": null,
  "matched_filter": true,
  "user_columns": {},
  "is_extra_span": false,
  "day": "2025-09-26",
  "duration": 0.001989,
  "otel_status_code": "UNSET",
  "otel_status_message": null,
  "otel_links": [],
  "otel_events": [],
  "url_path": null,
  "url_query": null,
  "url_full": null,
  "http_route": null,
  "http_method": null,
  "attributes": {
    "code.filepath": "test.py",
    "code.function": "main",
    "code.lineno": 59,
    "http.response.body.text": {
      "id": "chatcmpl-544",
      "object": "chat.completion",
      "created": 1758911631,
      "model": "gpt-oss:120b-cloud",
      "system_fingerprint": "fp_ollama",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "The die rolled a **6**.",
            "reasoning": "The user asked \"roll the dice\". We rolled and got 6. Need to respond with result."
          },
          "finish_reason": null
        }
      ],
      "usage": {
        "prompt_tokens": 159,
        "completion_tokens": 39,
        "total_tokens": 198
      }
    },
    "logfire.msg_template": "Reading response body"
  },
  "attributes_json_schema": "{\"type\":\"object\",\"properties\":{\"http.response.body.text\":{\"type\":\"object\"}}}",
  "otel_scope_attributes": {},
  "service_namespace": "",
  "service_instance_id": "5de721cc0c75403d81463054b1410109",
  "process_pid": 33638,
  "otel_resource_attributes": {
    "process.pid": 33638,
    "process.runtime.description": "3.11.13 (main, Jun  7 2025, 16:56:45) [Clang 16.0.0 (clang-1600.0.26.6)]",
    "process.runtime.name": "cpython",
    "process.runtime.version": "3.11.13",
    "service.instance.id": "5de721cc0c75403d81463054b1410109",
    "service.name": "unknown_service",
    "service.version": "3e2b97c25251b664c31f7dc105926dba5d5f577a",
    "telemetry.sdk.language": "python",
    "telemetry.sdk.name": "opentelemetry",
    "telemetry.sdk.version": "1.35.0"
  },
  "telemetry_sdk_name": "opentelemetry",
  "telemetry_sdk_language": "python",
  "telemetry_sdk_version": "1.35.0",
  "deployment_environment": null
}

From what I see, there is a "finish_reason": null , which seems to be a problem, since I get this error:

Traceback (most recent call last):
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/pydantic_ai/models/openai.py", line 380, in _process_response
    response = chat.ChatCompletion.model_validate(response.model_dump())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/pydantic/main.py", line 705, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatCompletion
choices.0.finish_reason
  Input should be 'stop', 'length', 'tool_calls', 'content_filter' or 'function_call' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/literal_error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
    yield span
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/pydantic_ai/models/instrumented.py", line 328, in _instrument
    yield finish
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/pydantic_ai/models/instrumented.py", line 215, in request
    response = await super().request(messages, model_settings, model_request_parameters)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/pydantic_ai/models/wrapper.py", line 30, in request
    return await self.wrapped.request(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/pydantic_ai/models/openai.py", line 247, in request
    model_response = self._process_response(response)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gonzaloramos/Git/marginalia/backend/.venv/lib/python3.11/site-packages/pydantic_ai/models/openai.py", line 382, in _process_response
    raise UnexpectedModelBehavior(f'Invalid response from OpenAI chat completions endpoint: {e}') from e
pydantic_ai.exceptions.UnexpectedModelBehavior: Invalid response from OpenAI chat completions endpoint: 1 validation error for ChatCompletion
choices.0.finish_reason
  Input should be 'stop', 'length', 'tool_calls', 'content_filter' or 'function_call' [type=literal_error, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/literal_error

[DouweM]

@knana1662 Did you happen to have a chance to capture the failing model response?

[jefftriplett]

I'm seeing this too for any Ollama Cloud (the hosted) models that use tools in case that context is helpful.

[DouweM]

@jefftriplett Any chance you can capture the failing HTTP response body using ai.pydantic.dev/logfire (logfire.instrument_httpx(capture_all=True)) and share it? Specifically the http.response.body.text that you can also see in @gonzoramos's post above.

[DouweM]

@knana1662 The example you've shared works for me. It also works when using OllamaProvider and setting the OLLAMA_BASE_URL and OLLAMA_API_KEY environment variables.

I can reproduce the issue @gonzoramos is seeing when accessing Ollama Cloud through the locally running server, but it works when connecting with Ollama Cloud directly using the OllamaProvider base_url. I've fixed that in #3045.

@jefftriplett I cannot reproduce the issue with Cloud and tools. This example runs fine for me (assuming OLLAMA_API_KEY is set):

import asyncio
import random

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.ollama import OllamaProvider

ollama_model = OpenAIChatModel(
    model_name='gpt-oss:120b',
    provider=OllamaProvider(base_url='https://ollama.com/v1'),
)


agent = Agent(model=ollama_model)


@agent.tool_plain
def roll_dice() -> str:
    """Roll a six-sided die and return the result."""
    return str(random.randint(1, 6))


async def main():
    result = await agent.run('roll the dice')
    print(result)


if __name__ == '__main__':
    asyncio.run(main())

[jefftriplett]

@DouweM Thank you, that's super helpful. It's working now with the latest Pydantic AI version.

From another issue, I saw a recommendation to use: http://localhost:11434/v1 and gpt-oss:120b-cloud which worked for all of my non-tool invocations. (Not sure why, but I had to use -cloud to get Ollam to proxy out to the correct model.)

Your suggestion of using https://ollama.com/v1 and dropping the -cloud from the model name works. The latest version is passing through an error that the model name is wrong, which is also helpful. I was getting the same type=literal_error previously.

Thank you again!

[DouweM]

@jefftriplett Glad to hear that resolved it. I'll close this issue as I couldn't reproduce the problem in the description, and the one problem I could reproduce has been fixed in #3045.

If anyone runs into more issues with Ollama Cloud, please post them here with reproducible code and I'll reopen.