Feature Request: Support for custom OpenAI-compatible endpoints (vLLM)

[sfc-gh-valamuri]

Problem Description

Currently, streamlit-openai appears to be tightly coupled to OpenAI's full platform ecosystem, including containers, assistants, and other platform-specific APIs. This prevents usage with OpenAI-compatible inference servers like vLLM that only implement the chat completions API.

Use Case

Many organizations run their own LLM inference servers (vLLM, TGI, etc.) that provide OpenAI-compatible APIs for cost, privacy, or custom model requirements. It would be valuable to use streamlit-openai's excellent UI/UX with these deployments.

Current Behavior

When attempting to use a custom endpoint, the library fails during initialization:

import streamlit_openai
import os

os.environ["OPENAI_API_KEY"] = "dummy-key"
os.environ["OPENAI_BASE_URL"] = "https://my-vllm-server.com/v1"

# This fails with 406 error because vLLM does not support the other endpoints (like Container)
chat = streamlit_openai.Chat(model="meta-llama/Llama-3.1-8B-Instruct")

Proposed Solutions

1. Configuration Option: Add a parameter to disable OpenAI-specific features:

   chat = streamlit_openai.Chat(
       model="custom-model",
       openai_compatible_mode=True  # Disables containers, assistants, etc.
   )

2. Client Injection: Allow passing a pre-configured OpenAI client:

   custom_client = OpenAI(base_url="...", default_headers={"Auth": "..."})
   chat = streamlit_openai.Chat(client=custom_client)

Benefits

• Broader compatibility with the growing ecosystem of LLM deployment tools
• Maintains the excellent UX of streamlit-openai for custom deployments
• Could significantly expand the user base

Environment

• streamlit-openai version: 0.1.3
• Python version: 3.11
• Testing with: vLLM server with OpenAI-compatible API

TLDR

It would be amazing if this library was less tightly coupled with the OpenAI backend and had flexibility to plug in with other OpenAI compatible backends like vLLM!

[sbslee]

@sfc-gh-valamuri,

Thank you so much for the detailed post.

Quick clarification: I'm not familiar with vLLM or other custom endpoints. Do they follow OpenAI's Responses API (e.g., client.responses.create()), which is the API currently used by streamlit-openai? Your second proposed solution, client injection, is probably the simplest solution, but I'm concerned it might not work because custom endpoints likely follow OpenAI's Chat Completions API instead of the Responses API.

[sfc-gh-valamuri]

Hey @sbslee!

Yeah they use the Completions and Chat Completions API. It doesn't look like they follow the Responses API (at least yet, maybe in a future release).

FYI here's the full list: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#supported-apis.

Having said that, is it possible to make the existing streamlit-openai codebase compatible with the Chat Completions API? The HuggingFace LLM models we use have a chat template.