Support gpt-oss directly via Harmony format

[eksno]

Description

Request to support for openai's harmony response format

References

similar issue on langchain: langchain-ai/langchain#32425

[DouweM]

Note that gpt-oss is already supported through cloud and local inference providers:

• Groq
  • Agent('groq:openai/gpt-oss-120b')
  • model library
  • Pydantic AI docs)
• HuggingFace
  • Agent('huggingface:openai/gpt-oss-120b')
  • model library
  • Pydantic AI docs
• Ollama
  • ollama run gpt-oss:20b + OpenAIModel('gpt-oss:20b', provider=OpenAIProvider(base_url='...')
  • model library
  • Pydantic AI docs

If you run into any issues using gpt-oss with Pydantic AI through their APIs, please file an issue. Be sure to check first if it's not an issue with their implementation of gpt-oss/Harmony.

---

That said, I agree it would also be cool to have Pydantic AI directly speak the Harmony format and send/receive tokens directly to/from gpt-oss using something like transformers to run the model (see OpenAI's docs on transformers + gpt-oss). The concrete advantage of this over one of the previously mentioned APIs (including the local API provided by Ollama) isn't clear to me however.

If you have a specific use case in mind where bringing this into Pydantic AI would work better than using a dedicated API-to-tokens layer, please share!

[eksno]

@DouweM Thanks for the reply, I've been looking a lot into gpt-oss and the harmony format and can see you're right on the providers being the issue.

We've been using openrouter which have had big issues with the output (see picture below, it's actually quite insightful into how the raw reply of gpt-oss using harmony looks). I will try out groq provider directly and see if we have the same issue, if we do I'll chuck it here with my findings to see if it has anything to do with pydantic-ai.

Otherwise as a note, have loved the framework ever since we moved from langchain!

[jgslazzaro]

Note that gpt-oss is already supported through cloud and local inference providers:

• Groq

  • Agent('groq:openai/gpt-oss-120b')
  • model library
  • Pydantic AI docs)

• HuggingFace

  • Agent('huggingface:openai/gpt-oss-120b')
  • model library
  • Pydantic AI docs

• Ollama

  • ollama run gpt-oss:20b + OpenAIModel('gpt-oss:20b', provider=OpenAIProvider(base_url='...')
  • model library
  • Pydantic AI docs

If you run into any issues using gpt-oss with Pydantic AI through their APIs, please file an issue. Be sure to check first if it's not an issue with their implementation of gpt-oss/Harmony.

That said, I agree it would also be cool to have Pydantic AI directly speak the Harmony format and send/receive tokens directly to/from gpt-oss using something like transformers to run the model (see OpenAI's docs on transformers + gpt-oss). The concrete advantage of this over one of the previously mentioned APIs (including the local API provided by Ollama) isn't clear to me however.

If you have a specific use case in mind where bringing this into Pydantic AI would work better than using a dedicated API-to-tokens layer, please share!

I want to run with transformers so I can finetune the model as I wish and use it with LangChain. The providers make it harder to do it.

[DouweM]

@jgslazzaro Note that you can use transformers serve to spin up a Chat Completions-compatible API (see gpt-oss + transformers docs) that you can then use with OpenAIChatModel + OpenAIProvider.

[eksno]

Closing since it's stale and some providers have moreso gotten their act together. I can vouch for groq as a decent provider.