Auto Recovery Logic for chat completion clients on different types of server errors

[ekzhu]

What feature would you like to be added?

It would be nice instead if we can support a model client that auto-recovers from various host-related errors with a configurable logic such as retries.

Here is a good example of an apparently transient error:

openai.APIStatusError: Error code: 424 - {'error': {'message': 'Error occurred while processing image(s).', 'type': 'failed_dependency', 'param': None, 'code': None}}

Moreover, even for rate limit errors like below, it doesn't always retry, there is some inconsistency happening

openai.RateLimitError: Error code: 429 - {'error': {'code': '429', 'message': 'Rate limit is exceeded. Try again in 1 seconds.'}}

Why is this needed?

Currently, the implementation of the chat completion client for OpenAI fails to recover and crashes for a host of errors (except for retries that are rate-limit related where it auto-retries), however, a host of other errors randomly occur and that forces the applications built on top of the client to have their own logic for auto-recovery.

[ekzhu]

@husseinmozannar Moved your issue here.