Skip to content

Feature request: Add CometAPI as an LLM provider #2269

@TensorNull

Description

@TensorNull

Is your feature request related to a problem? Please describe.
We need more robust, region-diverse, and cost-effective LLM options in Haystack pipelines. Relying on a small set of providers can cause issues during rate limiting, regional outages, or price-sensitive workloads. Adding CometAPI as an OpenAI-compatible provider would improve availability, portability, and resilience without disrupting existing workflows.

Describe the solution you'd like
Add CometAPI as a first-class LLM provider in Haystack, or officially support configuring the existing OpenAI components via base_url to point to CometAPI. Desired capabilities:

  • Chat and text generation via OpenAI-compatible endpoints
  • Model listing and selection via CometAPI model APIs
  • API Key configuration and credential management
  • Streaming and non-streaming support (leveraging Haystack abstractions if available)
  • Minimal example + docs showing how to use CometAPI in a Haystack pipeline

Describe alternatives you've considered

  • Use current OpenAI adapters with a custom base_url (if already supported). This works but lacks discoverability and may have subtle compatibility gaps.
  • Go through third-party proxies/aggregators. This adds an extra dependency and less control.
  • Stay with current providers only. This limits resilience, vendor flexibility, and cost optimization.

Additional context
CometAPI Resources:

Compatibility:

  • CometAPI exposes OpenAI-compatible endpoints, enabling low-friction integration by reusing Haystack's existing OpenAI components where possible.

Implementation Offer:

  • We’re happy to implement this integration and open a PR following your contribution guidelines (including examples and docs). We can also provide a temporary test API key and a small demo notebook/script if helpful. Please advise on the preferred integration point (new provider vs. extending OpenAI components) and acceptance criteria.

Metadata

Metadata

Assignees

Labels

P3new integrationDiscuss the creation of a new integration in Core

Type

No type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions