Moving daily releases to -nightly

[krrishdholakia]

We're planning on moving our daily releases to be pre-releases with a -nightly tag.

This is so we can separate releases which haven't gone through a qa checklist, and therefore might have some instability (e.g. not being able to see keys on UI as an internal user).

Normal releases will therefore be more stable as they would have gone through manual QA (in addition to our ci/cd) before being published.

[krrishdholakia]

Stable Release QA Checklist

Proxy Admin

• create a team
• Invite a user to the team
• Create a key in the team
• Regenerate key
• Update key with new tpm/rpm limits
• delete key
• Perform a chat in playground with at least 2 different models
• Review logs page - verify playground requests got logged
• Review Usage Page, see Top API Keys Table. Click into any API Key view Key Info. Click Chart View Toggle on Top Keys, see chart
• Adding new models -> add specific model -> Click Test Connection with Valid credentials. Should work
• Adding new models -> add specific model -> Click Test Connection with Bad credentials. Should show raw curl request sent + error
• Adding new models -> add specific model -> Add model. Confirm it's added under Models Page
• Adding new models -> add wildcard route
• Add new model w/ reusable credential -> test if it works
• Add Azure AI Foundry mode, select custom model name, make test request on UI
• Add Azure Openai model, set custom public model name
• [TODO] Add VertexAI model with reusable credentials
• Add OpenAI-compatible model (perplexity/fireworks ai/xai) on UI , test request, confirm it works
• add team-only model → assign to team
• call model via anthropic pass-through route -> confirm spend tracked
• call bedrock anthropic on test key pane -> sanity check token counts
• Edit team model
• Edit team member for team - proxy admin does not belong to
• delete team
• Refresh team page -> teams should load
• Create key -> test if it works
• Open create key modal -> confirm all values match /key/generate on swagger
• [TODO] Create Key, Select Team, The models page should show the models for selected team.
• [TODO] Create Key, Select a different Team, The models page should show the models for newly selected team.
• search for user email on the team member add
• see all internal user created keys in team
• Create a team that's within an Organization. Add a team member that's not in any org. Edit the models the team has access to, verify you can actually edit the team models.
• [TODO] Update RPM limit for org
• Should be able to edit Max Budget and Budget Duration in Settings for an internal user
• Create a key, got to logging settings, select "Langfuse OTEL", then enter credentials for a specific langfuse project. Make a request with key and check langfuse. The log should have landed in the correct langfuse project.

Router Settings QA

• Go to settings -> router settings -> fallbacks. Click add fallback, add one primary one secondary model. Verify the fallback got added
• Test the fallback using the follow curl, the response should be from fallback model

curl -i -X POST http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{
    "model": "anthropic/claude-3-5-haiku-latest",
    "mock_testing_fallbacks": true,
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }

Guardrails QA

• Set store_prompts_in_spend_logs: true
• Click on Guardrails, Create a guardrail, Select Presidio as provider, enter presidio api bases. Click Next, on configure PiI Select all & Mask
• Test Key, Select the new guardrail in dropdown and paste this text "My credit card number is 4111-1111-1111-1111, my email is [email protected], and my phone is 555-123-4567". Hit enter
• Check Logs for completed request. The request in Logs page has masked <CREDIT_CARD>, the real credit card should not be in logs.
• Go to teams, try to edit a team, click on guardrails. Ensure you can see the list of guardrails that are active on this dropdown.
• In guardrail information see dropdown of number of entities and their score
• Go to guardrails, select Delete. Refresh page, ensure you don't see guardrail.

Logs Page

• Go to Logs. Expand a log, verify you can view request details. Collapse, verify it collapses
• With store_prompts_in_spend_logs: true send a request. Ensure you can see both request / response content for /chat
• Click "Copy" icon for Request, paste it in a doc. Verify the request content matches
• Click "Copy" icon for Response, paste it in a doc. Verify the request content matches
• Expand a log, send 10 requests. Verify that the Expanded log does not change as you get traffic
• Make two /chat/completions requests with a different 'user' param. Go to Logs and verify you see the new fields in the "End User" Filter. Then filter by end user = "ishaan1". Verify you see just logs with this end user

You can use this curl

curl -i -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{
    "model": "claude-3-5-sonnet-20241022",
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ],
    "user": "ishaan1"
  }'

Use this to generate 10 requests, just run on your terminal

#!/bin/bash

for i in {1..10}; do
  echo "Request $i:"
  curl -i http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-PIp1h0RekR" \
  -d '{
    "model": "us-gpt-4",
    "messages": [
      {
        "role": "user",
        "content": "hi who is this."
      }
    ]
  }'
  echo -e "\n\n------------------------\n"
done

Other Proxy Admin

• Invite a user, make them Admin and give them no access to models
• Create a Key, with no models
• Create a Key, with specific proxy model
• Use the new key to call proxy model
• Use the new key for the following API request. Expect it to pass

curl -X GET "http://localhost:4000/team/list" \
  -H "Authorization: Bearer sk-<key-you-just-created>"

Team Admin

• able to view all team keys
• able to add/remove member from team
• able to add / edit / delete model in team
• able to create team key with all team models
• able to see all team models in test key dropdown

Internal User

• Create key -> test if it works
• Refresh key page → check if created key still shown
• Click on teams tab → click on team → should be able to view team info
• Create new user -> doesn't belong to any teams -> create key -> test if it works
• check keys page -> shouldn't see litellm-dashboard keys
• As ‘user’ role within team → try creating key → ensure it fails

Internal User Viewer

• Admin, invite an internal user viewer
• Internal Viewer should only see the following pages, nothing out of this set
• Only the follow pages are visible
  - Keys, Usage, Teams, API Ref, Model Hub, Logs, New usage, API Playground
  - Nothing outside this set
• Should not Test Key page
• Keys Page: Should not see create key button
• Keys Page: Click key, should not see edit delete or regenerate on root or on the key specific page
• Teams Page: should not see Members or Team settings page

Security Tests

• Sign in as Admin, Invite internal user, follow onboarding link, sign in as internal user. Hover over user info -> This should be the new user (Note: Remember the password your entered here)
• Click Logout
• Sign in as Admin, hover over User Sign in info (corner right), current user should be proxy admin
• Sign in as the new user with your credentials from step 1, on sign in the ui should load only info for the new internal user (not Proxy Admin)
• Set PROXY_LOGOUT_URL="https://www.google.com" in your .env, start LiteLLM, sign in, click logout, confirm you got directed to www.google.com

SSO Tests

• Set MICROSOFT_CLIENT_SECRET, MICROSOFT_CLIENT_ID, MICROSOFT_TENANT. Start Proxy, sign in. Hover over user tab and confirm your email is the correct email
• Sign in as Admin, On UI go to Admin Settings > SCIM > Create SCIM token

Server Tests

• Enable prometheus -> make request with key + team id -> check for any errors in server logs
• Confirm redis op/s is same as previous version

1. Enable redis on local instance

model_list:
 - model_name: fake-openai-endpoint
    litellm_params:
      model: openai/fake
      api_key: fake-key
      api_base: https://exampleopenaiendpoint-production.up.railway.app/

litellm_settings: 
     cache: true
     cache_params:
        type: redis
        ttl: 600
        password: os.environ/REDIS_PASSWORD
        supported_call_types: ["acompletion", "completion"]

2. Setup locust test

10 users, 1 ramp up, 2 minutes, 2 instances (port 4000, port 4001)

from locust import HttpUser, task, between
from large_file import text
import uuid
import os


class RealProxyUser1(HttpUser):
    host = "http://localhost:4000"  # First base URL
    # wait_time = between(1, 10)

    @task
    def post_chat_completions(self):
        headers = {
            "Content-Type": "application/json",
            "Authorization": "Bearer sk-1234",
            # "Authorization": "Bearer sk-P1zJMdsqCPNN54alZd_ETw",
        }
        data = {
            "model": "fake-openai-endpoint",
            "max_tokens": 1024,
            "messages": [{"role": "user", "content": "Hey, how's it going?" * 1024}],
        }
        self.client.post("/chat/completions", json=data, headers=headers)


class RealProxyUser2(HttpUser):
    host = "http://localhost:4001"  # First base URL
    # wait_time = between(1, 10)

    @task
    def post_chat_completions(self):
        headers = {
            "Content-Type": "application/json",
            "Authorization": "Bearer sk-1234",
            # "Authorization": "Bearer sk-P1zJMdsqCPNN54alZd_ETw",
        }
        data = {
            "model": "fake-openai-endpoint",
            "max_tokens": 1024,
            "messages": [{"role": "user", "content": "Hey, how's it going?" * 1024}],
        }
        self.client.post("/chat/completions", json=data, headers=headers)

3. Monitor redis dashboard

---

Please let me know if there's any scenarios we should add to this.

[krrishdholakia]

PR with change is live here - #8499

[krrishdholakia]

This is now live from today onwards

[krrishdholakia]

Rollout plan:

1. Stage 1: Move daily docker releases to -nightly tag
2. Stage 2: Move pypi releases to -nightly or .dev tag (depending on what pypi supports)
3. Stage 3: Establish regular release cadence post QA (either daily or weekly)

[krrishdholakia]

Update pypi releases are now marked with a .post1

[krrishdholakia]

Stable releases are done on a weekly basis (every Sunday) - https://docs.litellm.ai/docs/proxy/release_cycle