Python server used for LiteLLM server

[dhawalkp]

I was curious to understand what server is being used by LiteLLM when deployed with Docker container as a Proxy. I read the DockerFile and code under /docker --> prod_entrypoint.sh. I see it executes the litellm cmd and replace the parent bash process. I am however not sure what server gets spawned as part of "litellm" execution. So far after reading the code, I see uvicorn and fastAPI. I also see proxy_cli.py having ollama serve as well. Can anyone confirm which server is used? Where is the code for executable "litellm"?

[krrishdholakia]

We use FastAPI.

[krrishdholakia]

proxy_cli.py is the entrypoint.

proxy_server.py is the server file

[dhawalkp]

Thanks! What do you use ollama server for?

[krrishdholakia]

We don't.

Can you share what you're looking at?

[dhawalkp]

litellm/litellm/proxy/proxy_cli.py

Line 269 in be35c9a

def _run_ollama_serve():

[dhawalkp]

Fine tuning the LiteLLM server is important for getting high performance (throughput) and hence we at AWS are working on a deep dive content on how LiteLLM works. I am leading that effort and hence asking this question on how things work under-the-hood and how to performance tune it. The litellm docs are not good enough to understand the serving architecture and hence these questions. Thanks for answering the questions quickly!