v1.0.0 Beta

[krrishdholakia]

Starting a discussion to track potential refactoring / breaking changes we need to make for a v1.

Motivation
Bundle near-term breaking changes into 1 release.

Driven by the discussion on #636.

Additional Thoughts
I haven't had a chance to explore the new openai sdk, but I anticipate that's going to cause some issues too. openai/openai-python#631

Timeline
We should aim to have a locked in list by 30th October.

[krrishdholakia]

#636 -
"I would suggest to make get_max_tokens return only an integer representig the max_tokens of a given model, and maybe add a get_model_info function returning the currenct dict with all the info!" - @mistrello96

[mistrello96]

Hi, has this been addressed in the 1.0.0 release? It seems the function still returns a dict

[ishaan-jaff]

yes @mistrello96 - addressed here: bd82559

[ishaan-jaff]

migration guide: https://docs.litellm.ai/docs/migration

[krrishdholakia]

remove old debugger dashboard variables -

litellm/litellm/__init__.py

Line 10 in cb100d1

email: Optional[

[krrishdholakia]

[krrishdholakia]

@shuther let me know what was missing in the docs - https://docs.litellm.ai/docs/simple_proxy

Would appreciate a quick chat to understand your scenario better - https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

[shuther]

I am based in europe, in which timezone are you so we can orgwnize this call if it is easier?
Few of my questions:

• there are multiple DockerFile/.env files within the project; what are the differences? As a general rule I am confused because there are "sub" packages called proxy and litlellm_server but the documentation states we should run: litellm for everything (so we should never call proxy_cli directly). is it correct? I refer here to a dev environment.

• there are some references to a file called litellm.secrets.toml but I can't find an example.

• I would like litlellm to behave as such: - is it how it is expected to work (should I refer to the proxy or router documentation)?

  1. Ignore the API key from the request, keep the model or work with a fallback one (it should retry in case we face rate limitations...);
  2. identify an API key based on a configuration file/strategy
  3. forward the request to OpenAI (i.e.)
  4. retry if needed

• the examples consider I install litlellm through pip but I had the feling that poetry is used now; I tried in one terminal:
  poetry run litellm --model huggingface/bigcode/starcoder -c litellm_server/router_config.yaml --debug
  and in a different one:
  poetry run litellm --model gpt-3.5-turbo --test
  but I am getting errors on both screens; the error message recommends to change the url to connect to https://my_ollama_api; the debug screen is large as some exceptions are fired from models.py:

raise MissingSchema(requests.exceptions.MissingSchema: Invalid URL 'my_ollama_api_base/api/generate': No scheme supplied. Perhaps you meant https://my_ollama_api_base/api/generate?

[krrishdholakia]

I'm in PST - but happy to do whenever works best for you @shuther

Also happy to set up a 1:1 support channel.

Here's my details:
LinkedIn: https://www.linkedin.com/in/krish-d/
Discord: Krrish#8748

[krrishdholakia]

we use pip

Here's the docs: https://docs.litellm.ai/docs/simple_proxy

Let me know if this works:
pip install litellm
export OPENAI_API_KEY = sk..
litellm --model gpt-3.5-turbo --test

[shuther]

so one issue was that poetry did not load the .env by default, so poetry self add poetry-dotenv-plugin was required.
for the 2nd issue I will share it through discord

[ishaan-jaff]

we should deprecate our gpt cache support: https://github.com/BerriAI/litellm/blob/main/litellm/gpt_cache.py

• I have not heard anyone using it
• we now have our own caching support with redis

[krrishdholakia]

OpenAI v1 drops support for invalidrequesterror and now calls it badrequesterror.
openai.InvalidRequestError -> openai.BadRequestError

[krrishdholakia]

OpenAI v1 changes how exception works

• All exceptions inherit from API Error (same as before)
• APIError now requires the request object (an httpx client) to be passed in