why not have a llm load balance infrustucture?

[dl942702882]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

why not have a llm load balance infrustucture?

Motivation

cause where i build an application and depoly into production enviroment, it always need kind of llm and some account, that because one account exist RPM or TPM, so i need multi account to serve my customers；

there is already an open source project https://github.com/BerriAI/litellm
however it is not very good to use

Proposal (If applicable)

No response

[krrishdholakia]

Hi @dl942702882 - litellm maintainer here - what is litellm missing to be useful here?

there's also a langchain/litellm load balancer integration https://python.langchain.com/docs/integrations/chat/litellm_router/

[sepiatone]

@dl942702882 See fallbacks. The examples are still pre-ver 1.0.0, but they give you a flavor of what can be done.