[Feature Request]: Built-in LLM Failover for Reliability

[YassinNouh21]

Feature Description

Introduce a first-class failover mechanism so that when a primary LLM request fails (timeouts, 429s, 5xx, network), LlamaIndex automatically routes the same request to the next configured provider—without surfacing errors to end users and without code duplication across apps.

Reason

• Today, apps built on LlamaIndex are tightly bound to a single LLM per call. Any transient provider issue becomes a user-visible failure or requires every app to re-implement retries and fallbacks in inconsistent ways.
• This gap is more acute during peak periods (rate limits), provider incidents, or regional network degradation.
• Teams need dependable behavior across standard and streaming responses; manual fallbacks cause UX glitches and slowdown.
• There is clear market precedent to aggregate across multiple models/providers under one surface (e.g., OpenRouter) but users still need a unified, provider-agnostic failover within LlamaIndex itself for end-to-end consistency. See: OpenRouter example docs.

Value of Feature

• Minimizes business impact during provider outages or throttling; requests continue seamlessly with the next available model.
• Clear pathway for enterprises to meet reliability/compliance targets without bespoke plumbing per product team.

[debu-sinha]

I'd like to take this on. I've been working on LlamaIndex's resilience story recently with the token-bucket rate limiter (#20712, merged) and Retry-After header support (#20813, in review), and this is the natural next piece.

I put together a design doc covering the proposed architecture: LLM Failover and Circuit Breaker

The short version:

FallbackLLM(LLM) -- wraps a primary LLM and an ordered list of fallbacks. Follows the same composition pattern as StructuredLLM: subclasses LLM, delegates metadata to the primary, works transparently with Settings.llm and everything downstream.

from llama_index.core.llms import FallbackLLM
from llama_index.llms.openai import OpenAI
from llama_index.llms.anthropic import Anthropic

Settings.llm = FallbackLLM(
    llm=OpenAI(model="gpt-4o"),
    fallbacks=[Anthropic(model="claude-sonnet-4-20250514")],
)
# Everything downstream works unchanged

When the primary raises a transient error (connection failure, timeout, rate limit), the request routes to the next provider. Non-transient errors (auth, bad request) raise immediately. The exception filter is user-configurable.

CircuitBreaker (optional) -- tracks consecutive failures per provider and short-circuits requests to unhealthy ones. Standard closed/open/half-open state machine. Prevents wasted latency when a provider is clearly down.

Settings.llm = FallbackLLM(
    llm=OpenAI(model="gpt-4o"),
    fallbacks=[Anthropic(model="claude-sonnet-4-20250514")],
    circuit_breaker=CircuitBreaker(failure_threshold=5, recovery_timeout=60.0),
)

Key design decisions:

• Failover triggers after provider-level retries are exhausted (OpenAI's tenacity retries, Anthropic's SDK retries all run first)
• Default exception filter is (ConnectionError, TimeoutError) which covers openai.APIConnectionError and openai.APITimeoutError via inheritance. Users add provider-specific exceptions like openai.RateLimitError as needed
• Streaming: failover on generator creation error only (same as LangChain's approach). Mid-stream retry is future work
• Each provider keeps its own rate_limiter, so per-provider throttling composes naturally
• Circuit breaker is a separate class, not baked into FallbackLLM

Together with the rate limiter and Retry-After support, this would give LlamaIndex a complete resilience stack comparable to LangChain's RunnableWithFallbacks and Semantic Kernel's retry + circuit breaker system.

Happy to hear feedback on the approach before I start coding. @AstraBert @logan-markewich