Support rate limiting via exponential backoff

[hempels]

All Open AI APIs potentially impose rate limiting. Ideally, any library designed to abstract the APIs should support exponential backoff.

https://beta.openai.com/docs/guides/production-best-practices/managing-rate-limits-and-latency

[kayhantolga]

Thanks for raising the issue. I converted this to a discussion so we can talk about it.
Since API doesn't return any rate-limiting at the header there is no way to determine the actual rate limit. We could add this according to the page you shared but I have some concerns about adding limitations because OpenAI team is changing the rules quite suddenly.

Let me explain this with an example:
If OpeanAI increases the limit, you wouldn't be getting this privilege without updating the library. Also, you may need to deploy your code again. Or because your very talented OpenAI team increased your rate limit, you wouldn't get this benefit because of library limitations.

But I think there is a thing that we can support. We can recognize the error and throw RateLimitException. That would be helpful to handle such a scenario. I need a sample error for that. If you can find a sample response that would be very helpful.

[hempels]

We can recognize the error and throw RateLimitException. That would be helpful to handle such a scenario.

Yes, I would consider this the minimum viable solution.

[hempels]

The exponential backoff algorithm can be applied if and only if the API returns a 429 status code. The library would not need to change with API changes since that will always be true:

"When you call the OpenAI API repeatedly, you may encounter error messages that say 429: 'Too Many Requests' or RateLimitError. These error messages come from exceeding the API's rate limits."
-- https://github.com/openai/openai-cookbook/blob/main/examples/How_to_handle_rate_limits.ipynb

That link includes example rate limit responses.

[hempels]

It would be helpful if the library can return information about retries when they happen, but I consider that a nice-to-have, whereas the ability to handle rate limit errors when using the library is a requirement.