Skip to content

Commit cc4f024

Browse files
Sync Haystack API reference on Docusaurus (#9892)
Co-authored-by: vblagoje <[email protected]>
1 parent 90edcda commit cc4f024

File tree

2 files changed

+607
-469
lines changed

2 files changed

+607
-469
lines changed

docs-website/reference/haystack-api/generators_api.md

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1752,3 +1752,141 @@ If set, it will override the `tools_strict` parameter set during component initi
17521752
A dictionary with the following key:
17531753
- `replies`: A list containing the generated responses as ChatMessage instances.
17541754

1755+
<a id="chat/fallback"></a>
1756+
1757+
# Module chat/fallback
1758+
1759+
<a id="chat/fallback.FallbackChatGenerator"></a>
1760+
1761+
## FallbackChatGenerator
1762+
1763+
A chat generator wrapper that tries multiple chat generators sequentially.
1764+
1765+
It forwards all parameters transparently to the underlying chat generators and returns the first successful result.
1766+
Calls chat generators sequentially until one succeeds. Falls back on any exception raised by a generator.
1767+
If all chat generators fail, it raises a RuntimeError with details.
1768+
1769+
Timeout enforcement is fully delegated to the underlying chat generators. The fallback mechanism will only
1770+
work correctly if the underlying chat generators implement proper timeout handling and raise exceptions
1771+
when timeouts occur. For predictable latency guarantees, ensure your chat generators:
1772+
- Support a `timeout` parameter in their initialization
1773+
- Implement timeout as total wall-clock time (shared deadline for both streaming and non-streaming)
1774+
- Raise timeout exceptions (e.g., TimeoutError, asyncio.TimeoutError, httpx.TimeoutException) when exceeded
1775+
1776+
Note: Most well-implemented chat generators (OpenAI, Anthropic, Cohere, etc.) support timeout parameters
1777+
with consistent semantics. For HTTP-based LLM providers, a single timeout value (e.g., `timeout=30`)
1778+
typically applies to all connection phases: connection setup, read, write, and pool. For streaming
1779+
responses, read timeout is the maximum gap between chunks. For non-streaming, it's the time limit for
1780+
receiving the complete response.
1781+
1782+
Failover is automatically triggered when a generator raises any exception, including:
1783+
- Timeout errors (if the generator implements and raises them)
1784+
- Rate limit errors (429)
1785+
- Authentication errors (401)
1786+
- Context length errors (400)
1787+
- Server errors (500+)
1788+
- Any other exception
1789+
1790+
<a id="chat/fallback.FallbackChatGenerator.__init__"></a>
1791+
1792+
#### FallbackChatGenerator.\_\_init\_\_
1793+
1794+
```python
1795+
def __init__(chat_generators: list[ChatGenerator])
1796+
```
1797+
1798+
Creates an instance of FallbackChatGenerator.
1799+
1800+
**Arguments**:
1801+
1802+
- `chat_generators`: A non-empty list of chat generator components to try in order.
1803+
1804+
<a id="chat/fallback.FallbackChatGenerator.to_dict"></a>
1805+
1806+
#### FallbackChatGenerator.to\_dict
1807+
1808+
```python
1809+
def to_dict() -> dict[str, Any]
1810+
```
1811+
1812+
Serialize the component, including nested chat generators when they support serialization.
1813+
1814+
<a id="chat/fallback.FallbackChatGenerator.from_dict"></a>
1815+
1816+
#### FallbackChatGenerator.from\_dict
1817+
1818+
```python
1819+
@classmethod
1820+
def from_dict(cls, data: dict[str, Any]) -> FallbackChatGenerator
1821+
```
1822+
1823+
Rebuild the component from a serialized representation, restoring nested chat generators.
1824+
1825+
<a id="chat/fallback.FallbackChatGenerator.run"></a>
1826+
1827+
#### FallbackChatGenerator.run
1828+
1829+
```python
1830+
@component.output_types(replies=list[ChatMessage], meta=dict[str, Any])
1831+
def run(
1832+
messages: list[ChatMessage],
1833+
generation_kwargs: Union[dict[str, Any], None] = None,
1834+
tools: Union[list[Tool], Toolset, None] = None,
1835+
streaming_callback: Union[StreamingCallbackT,
1836+
None] = None) -> dict[str, Any]
1837+
```
1838+
1839+
Execute chat generators sequentially until one succeeds.
1840+
1841+
**Arguments**:
1842+
1843+
- `messages`: The conversation history as a list of ChatMessage instances.
1844+
- `generation_kwargs`: Optional parameters for the chat generator (e.g., temperature, max_tokens).
1845+
- `tools`: Optional Tool instances or Toolset for function calling capabilities.
1846+
- `streaming_callback`: Optional callable for handling streaming responses.
1847+
1848+
**Raises**:
1849+
1850+
- `RuntimeError`: If all chat generators fail.
1851+
1852+
**Returns**:
1853+
1854+
A dictionary with:
1855+
- "replies": Generated ChatMessage instances from the first successful generator.
1856+
- "meta": Execution metadata including successful_chat_generator_index, successful_chat_generator_class,
1857+
total_attempts, failed_chat_generators, plus any metadata from the successful generator.
1858+
1859+
<a id="chat/fallback.FallbackChatGenerator.run_async"></a>
1860+
1861+
#### FallbackChatGenerator.run\_async
1862+
1863+
```python
1864+
@component.output_types(replies=list[ChatMessage], meta=dict[str, Any])
1865+
async def run_async(
1866+
messages: list[ChatMessage],
1867+
generation_kwargs: Union[dict[str, Any], None] = None,
1868+
tools: Union[list[Tool], Toolset, None] = None,
1869+
streaming_callback: Union[StreamingCallbackT,
1870+
None] = None) -> dict[str, Any]
1871+
```
1872+
1873+
Asynchronously execute chat generators sequentially until one succeeds.
1874+
1875+
**Arguments**:
1876+
1877+
- `messages`: The conversation history as a list of ChatMessage instances.
1878+
- `generation_kwargs`: Optional parameters for the chat generator (e.g., temperature, max_tokens).
1879+
- `tools`: Optional Tool instances or Toolset for function calling capabilities.
1880+
- `streaming_callback`: Optional callable for handling streaming responses.
1881+
1882+
**Raises**:
1883+
1884+
- `RuntimeError`: If all chat generators fail.
1885+
1886+
**Returns**:
1887+
1888+
A dictionary with:
1889+
- "replies": Generated ChatMessage instances from the first successful generator.
1890+
- "meta": Execution metadata including successful_chat_generator_index, successful_chat_generator_class,
1891+
total_attempts, failed_chat_generators, plus any metadata from the successful generator.
1892+

0 commit comments

Comments
 (0)