Rate limit per key and model

[olad32]

Hi, is it planned to be able to apply rate limit tpm/rpm on key but with a different rate limit per model ? the same way we can already apply a budget for a key and a specific model as describe here https://litellm.vercel.app/docs/proxy/users#add-model-specific-budgets-to-keys
As of today a key can grant access to multiple model, but the rate limit applied globally to multiple model does not makes a lot of sense in practical use.
Thanks

[krrishdholakia]

hi @olad32 that's interesting

when would you do it this way?

[krrishdholakia]

if not - why not just set it globally and let them use across models?

Context - trying to understand your scenario better

[salman-lilly]

@krrishdholakia when i am applying rpm limit per key to 10 then i am getting response of more than 10 request per minute.Some request are failing with error that is expected but why i am still getting more than 10 request responses any idea? ({"error":{"message":"Max parallel request limit reached Hit limit for api_key). Note-(Model deployment rpm is more than 100. i am setting rpm on key basis to control number of request per key)....(https://litellm.vercel.app/docs/proxy/users#add-model-specific-budgets-to-keys)why this behaviour?

[krrishdholakia]

@salman-lilly can you please file a ticket with steps to repro?

if you have multiple instances running without redis cache, then it could be due to a difference in in-memory cache values across instances.

[salman-lilly]

@krrishdholakia i am using aws elasticache redis cluster everything working fine but whenever i am putting hard limit of rpm for the keys it gives response to more than rpm limit and then started failing(eg if i set key rpm to 5 and send 20 unique request in a minute then it will give 8 to 12 responses and rest is failing in a single as well as multiple instance...means in both same behaviour. please let me know this might be a bug or i am missing something. if you confirm i will raise issues with all the details. Thanks and really appreciate your quick response on this because it is a blocker right now

[salman-lilly]

@krrishdholakia raised issue- #6938 please let me know if i am missing anything will be helpful.