Proxy Mode Issues with Together.ai as the Backend Service #2027

vitalii-lobanov · 2024-02-17T00:55:08Z

vitalii-lobanov
Feb 17, 2024

I'm encountering an error when attempting to use Litellm in proxy mode, with Together.ai serving as the backend service. Here is the curl query:

export ENDPOINT_URL="http://0.0.0.0:8000/chat/completions"
export TOGETHERAI_API_KEY="******here is a valid API key******"
curl -X POST "$ENDPOINT_URL" \
-H "Authorization: Bearer $TOGETHERAI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
    "model": "together_ai/Mixtral-8x7B-Instruct-v0.1",
    "messages": [{
    "content": "Hello, tell me a story about a cat.",
    "role": "user"
    }]
}'

The response is the following:

{"error":{"message":"Together_aiException -
<!DOCTYPE html>
<html lang=\"en\">
  <head>
    <meta charSet=\"utf-8\" />
    <link data-next-font=\"\" rel=\"preconnect\" href=\"/\" crossorigin=\"anonymous\" />
    <link rel=\"preload\" as=\"style\" href=\"/_next/static/css/c887091bb50810ff.css\" />
    <link rel=\"preload\" as=\"script\" href=\"/_next/static/chunks/webpack-0213c00736f30f16.js\" />
    <link rel=\"preload\" as=\"script\" href=\"/_next/static/chunks/main-915862165dbce86d.js\" />
    <link rel=\"preload\" as=\"script\" href=\"/_next/static/chunks/pages/_app-45acb72122cca2f0.js\" />
    <link rel=\"preload\" as=\"script\" href=\"/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js\" />
    <link rel=\"preload\" as=\"script\" href=\"/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js\" />
    <link rel=\"preload\" as=\"script\" href=\"/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js\" />
    <link rel=\"preload\" href=\"/_next/static/css/c887091bb50810ff.css\" as=\"style\" />
    <title>TOGETHER</title>
    <meta name=\"description\" content=\"Together Computer\" />
    <link rel=\"icon\" href=\"/favicon.png\" />
    <meta name=\"viewport\" content=\"initial-scale=1, width=device-width\" />
    <meta name=\"next-head-count\" content=\"5\" />
    <meta name=\"theme-color\" content=\"#fff\" />
    <link rel=\"shortcut icon\" href=\"/favicon.png\" />
    <link rel=\"apple-touch-icon\" href=\"/favicon.png\" />
    <meta name=\"emotion-insertion-point\" content=\"\" />
    <style data-emotion=\"mui-style-global 15gmyd0\">
      html {
        -webkit-font-smoothing: antialiased;
        -moz-osx-font-smoothing: grayscale;
        box-sizing: border-box;
        -webkit-text-size-adjust: 100%;
      }

      *,
      *::before,
      *::after {
        box-sizing: inherit;
      }

      strong,
      b {
        font-weight: 900;
      }

      body {
        margin: 0;
        color: #333333;
        font-size: 1rem;
        font-family: -apple-system, BlinkMacSystemFont, \"Segoe UI\", Roboto, \"Helvetica Neue\", Arial, sans-serif, \"Apple Color Emoji\", \"Segoe UI Emoji\", \"Segoe UI Symbol\", Mplus 1p, Hiragino Sans, Hiragino Kaku Gothic Pro, 游ゴシック, 游ゴシック体, YuGothic, Yu Gothic, ＭＳ ゴシック, MS Gothic;
        font-weight: 400;
        line-height: 1.5;
        background-color: #fff;
      }

      @media print {
        body {
          background-color: #fff;
        }
      }

      body::backdrop {
        background-color: #fff;
      }
    </style>
    <style data-emotion=\"mui-style \"></style>
    <link rel=\"stylesheet\" href=\"/_next/static/css/c887091bb50810ff.css\" data-n-g=\"\" />
    <noscript data-n-css=\"\"></noscript>
    <script defer=\"\" noModule=\"\" src=\"/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js\"></script>
    <script src=\"/_next/static/chunks/webpack-0213c00736f30f16.js\" defer=\"\"></script>
    <script src=\"/_next/static/chunks/main-915862165dbce86d.js\" defer=\"\"></script>
    <script src=\"/_next/static/chunks/pages/_app-45acb72122cca2f0.js\" defer=\"\"></script>
    <script src=\"/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js\" defer=\"\"></script>
    <script src=\"/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js\" defer=\"\"></script>
    <script src=\"/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js\" defer=\"\"></script>
  </head>
  <body>
    <div id=\"__next\">
      <div class=\"bg-dotgrid\"></div>
    </div>
    <script id=\"__NEXT_DATA__\" type=\"application/json\">
      {
        \
        "props\":{\"pageProps\":{\"statusCode\":404,\"_sentryTraceData\":\"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1\",\"_sentryBaggage\":\"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true\"},\"session\":null,\"rootDehydratedState\":{\"mutations\":[],\"queries\":[]}},\"page\":\"/_error\",\"query\":{},\"buildId\":\"xc6W_gssf6ttfYKLA0zfV\",\"runtimeConfig\":{\"NEXT_PUBLIC_STRIPE_PUBLIC_KEY\":\"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE\",\"NEXT_PUBLIC_WSAPI_URL\":\"wss://api.together.xyz/api/websocket\",\"NEXT_PUBLIC_AUTH_PROVIDERS\":[\"cilogon\",\"github\",\"google\",\"linkedin\"],\"NEXT_PUBLIC_AUTH0_CLIENT_ID\":\"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV\",\"NEXT_PUBLIC_AUTH0_DOMAIN\":\"auth.together.ai\"},\"isFallback\":false,\"customServer\":true,\"gip\":true,\"appGip\":true,\"locale\":\"en-US\",\"locales\":[\"en-US\"],\"defaultLocale\":\"en-US\",\"scriptLoader\":[]}
    </script>
  </body>
</html>","type":null,"param":null,"code":404}}

Litellm's logs:

19:50:12 - LiteLLM:DEBUG: litellm.cache: None
litellm.cache: None
19:50:12 - LiteLLM:DEBUG: kwargs[caching]: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None
19:50:12 - LiteLLM:DEBUG: kwargs[caching]: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None
19:50:12 - LiteLLM:DEBUG:
LiteLLM completion() model= mistralai/Mixtral-8x7B-Instruct-v0.1; provider = together_ai
19:50:12 - LiteLLM:DEBUG:
LiteLLM: Params passed to completion() {'functions': None, 'function_call': None, 'temperature': None, 'top_p': None, 'n': None, 'stream': None, 'stop': None, 'max_tokens': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'custom_llm_provider': 'together_ai', 'response_format': None, 'seed': None, 'tools': None, 'tool_choice': None, 'max_retries': 0, 'logprobs': None, 'top_logprobs': None}
19:50:12 - LiteLLM:DEBUG:
LiteLLM: Non-Default params passed to completion() {'max_retries': 0}
19:50:12 - LiteLLM:DEBUG: self.optional_params: {'extra_body': {}}
self.optional_params: {'extra_body': {}}
19:50:12 - LiteLLM:DEBUG: PRE-API-CALL ADDITIONAL ARGS: {'headers': {'Authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96'}, 'api_base': ParseResult(scheme='https', userinfo='', host='api.together.xyz', port=None, path='/inference/', query=None, fragment=None), 'acompletion': True, 'complete_input_dict': {'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'extra_body': {}}}
19:50:12 - LiteLLM:INFO:

POST Request Sent from LiteLLM:
curl -X POST \
https://api.together.xyz/inference/ \
-H 'Authorization: Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225e********************' \
-d '{'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'extra_body': {}}'


19:50:12 - LiteLLM:DEBUG: Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details: logger_fn - None | callable(logger_fn) - False
19:50:12 - LiteLLM:DEBUG: Logging Details LiteLLM-Failure Call
Logging Details LiteLLM-Failure Call
19:50:12 - LiteLLM Router:DEBUG: Attempting to add 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown list. updated_fails: 1; self.allowed_fails: 0
19:50:12 - LiteLLM:DEBUG: get cache: cache key: 19-50:cooldown_models; local_only: False
get cache: cache key: 19-50:cooldown_models; local_only: False
19:50:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:50:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
19:50:12 - LiteLLM Router:DEBUG: adding 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown models
19:50:12 - LiteLLM:DEBUG: set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
19:50:12 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS

19:50:12 - LiteLLM Router:DEBUG: {'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1': ['<class \'litellm.exceptions.NotFoundError\'>Status: 404Message: NoneTogether_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>Full exceptionTogether_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>']}
19:50:12 - LiteLLM Router:DEBUG: Model together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1 had 1 exception
19:50:12 - LiteLLM:DEBUG: Custom Logger - final response object: None
Custom Logger - final response object: None
19:50:12 - LiteLLM Router:DEBUG: Attempting to add 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown list. updated_fails: 1; self.allowed_fails: 0
19:50:12 - LiteLLM:DEBUG: get cache: cache key: 19-50:cooldown_models; local_only: False
get cache: cache key: 19-50:cooldown_models; local_only: False
19:50:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:50:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
19:50:12 - LiteLLM Router:DEBUG: adding 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown models
19:50:12 - LiteLLM:DEBUG: set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
19:50:12 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS

Below is the complete Litellm test configuration that I utilized for this experiment:

model_list:

#Openrouter auto selection
  - model_name: openai/openrouter-auto # user-facing model alias
    litellm_params: # model-specific parameters
      model: openrouter/auto # the model's identifier
      api_base: https://openrouter.ai/api/v1 # OpenAI's API base URL

#Together AI Mixtral-8x7B-Instruct-v0.1
  - model_name: together_ai/Mixtral-8x7B-Instruct-v0.1 # user-facing model alias
    litellm_params: # model-specific parameters
      model: together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1 # the model's identifier
      api_base: https://api.together.xyz/inference


litellm_settings: # module level litellm settings
  drop_params: True
  set_verbose: True

Litellm version:

litellm --version
LiteLLM: Current Version = 1.23.16

I am able to access Together.ai's API from the same machine where Litellm is running. Here is an example of the query:

export ENDPOINT_URL="https://api.together.xyz/inference"
export TOGETHERAI_API_KEY="1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96"
curl -X POST "$ENDPOINT_URL" \
-H "Authorization: Bearer $TOGETHERAI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
    "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
    "messages": [{
    "content": "Hello, tell me a story about a cat.",
    "role": "user"
    }]
}'

The response is:

{
  "id": "8566e50e8e3a959d-TBS",
  "status": "finished",
  "prompt": [
    "<s>[INST] Hello, tell me a story about a cat. [/INST]"
  ],
  "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
  "model_owner": "",
  "num_returns": 1,
  "args": {
    "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
    "messages": [
      {
        "content": "Hello, tell me a story about a cat.",
        "role": "user"
      }
    ]
  },
  "subjobs": [],
  "output": {
    "finish_reason": "eos_token",
    "usage": {
      "prompt_tokens": 19,
      "completion_tokens": 395,
      "total_tokens": 414
    },
    "result_type": "language-model-inference",
    "choices": [
      {
        "text": " Once upon a time, in a small town nestled between rolling hills and a sparkling river, there lived a small calico cat named Daisy. Daisy was not your ordinary house cat – she had a mind of her own and a strong sense of adventure.\n\nDaisy spent her days exploring the town, making friends with the local dogs, birds, and even the occasional mouse. She was a friendly and curious feline, and she loved nothing more than discovering new sights, sounds, and smells.\n\nOne day, Daisy stumbled upon a beautiful garden filled with colorful flowers, shady trees, and a sparkling pond. She had never seen anything so lovely, and she couldn't resist exploring every inch of the garden.\n\nAs she was sniffing around the pond, Daisy heard a soft mewling sound. She followed the sound and found a small, shivering kitten hiding under a bush. The kitten was all alone, and Daisy could tell that she was scared and hungry.\n\nDaisy's heart went out to the little kitten, and she knew she had to help. She brought the kitten back to her own home, where she fed her warm milk and wrapped her in a soft blanket.\n\nOver the next few days, Daisy took care of the kitten, who she named Lily. She taught Lily how to hunt, how to climb trees, and how to play. And in return, Lily gave Daisy the companionship and love that she had been missing in her adventurous life.\n\nFrom that day on, Daisy and Lily were inseparable. They explored the town together, made new friends, and had many exciting adventures. And they always knew that no matter what challenges they faced, they would always have each other.\n\nThe end.",
        "finish_reason": "eos"
      }
    ],
    "prompt": [
      {
        "text": "<s>[INST] Hello, tell me a story about a cat. [/INST]",
        "tokens": [],
        "token_logprobs": []
      }
    ]
  }
}

Additionally, I've verified Litellm's functionality by testing it with the Openrouter API, which worked correctly. Here is the query used for testing:

export ENDPOINT_URL="http://0.0.0.0:8000/chat/completions"
export OPENROUTER_API_KEY="*******Valid OPEN AI API key here********"
curl -X POST "$ENDPOINT_URL" \
-H "Authorization: Bearer $TOGETHERAI_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
    "model": "openai/openrouter-auto",
    "messages": [{
    "content": "Hello, tell me a story about a cat.",
    "role": "user"
    }]
}'

The response is correct:

{
  "id": "gen-IA95sNsZCgqt0uStTdT9EfMoIBI2",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Once upon a time in the quiet town of Meadowgrove, there was a peculiar cat named Jasper. Unlike other cats, Jasper had a particular fondness for adventures and the great outdoors. This inquisitive feline had stunningly bright, emerald green eyes that seemed to sparkle with curiosity.\n\nOne sunny morning, Jasper found himself lounging on the warm windowsill of his human companion's home. The gentle breeze played with his soft <***skipped****>",
        "role": "assistant"
      }
    }
  ],
  "created": 1708098480,
  "model": "mistralai/mistral-medium",
  "object": "chat.completion",
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 765,
    "prompt_tokens": 18,
    "total_tokens": 783
  }
}

Finally, I will provide the complete Litellm logs corresponding to the error encountered when Litellm is used as a proxy for Together.ai's API:

HUGGINGFACE_API_KEY="******VALID KEY HERE*******" TOGETHERAI_API_KEY='******VALID KEY HERE*******' REPLICATE_API_TOKEN='******VALID KEY HERE*******' OPENAI_API_KEY='******VALID KEY HERE*******' OPENROUTER_API_KEY='******VALID KEY HERE*******' litellm --config /workspace/litellm_config.yaml  --detailed_debug
LiteLLM: Proxy initialized with Config, Set models:
    openai/openrouter-auto
    together_ai/Mixtral-8x7B-Instruct-v0.1
set cache: key: 352ae2c8-1706-4b3e-8b12-fa9f5f8c9477_async_client; value: <openai.AsyncOpenAI object at 0x105572250>
set cache: key: 352ae2c8-1706-4b3e-8b12-fa9f5f8c9477_client; value: <openai.OpenAI object at 0x103f6eb10>
set cache: key: 352ae2c8-1706-4b3e-8b12-fa9f5f8c9477_stream_async_client; value: <openai.AsyncOpenAI object at 0x10578e690>
set cache: key: 352ae2c8-1706-4b3e-8b12-fa9f5f8c9477_stream_client; value: <openai.OpenAI object at 0x105799510>
INFO:     Started server process [90346]
INFO:     Waiting for application startup.

#------------------------------------------------------------#
#                                                            #
#               'A feature I really want is...'               #
#        https://github.com/BerriAI/litellm/issues/new        #
#                                                            #
#------------------------------------------------------------#

 Thank you for using LiteLLM! - Krrish & Ishaan



Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new


19:50:00 - LiteLLM Proxy:DEBUG: Loaded config YAML (api_key and environment_variables are not shown):
{
  "model_list": [
    {
      "model_name": "openai/openrouter-auto",
      "litellm_params": {
        "model": "openrouter/auto",
        "api_base": "https://openrouter.ai/api/v1"
      }
    },
    {
      "model_name": "together_ai/Mixtral-8x7B-Instruct-v0.1",
      "litellm_params": {
        "model": "together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1",
        "api_base": "https://api.together.xyz/inference"
      }
    }
  ],
  "litellm_settings": {
    "drop_params": true,
    "set_verbose": true
  }
}
19:50:00 - LiteLLM Proxy:DEBUG:  setting litellm.drop_params=True
19:50:00 - LiteLLM Proxy:DEBUG:  setting litellm.set_verbose=True
LiteLLM: Proxy initialized with Config, Set models:
    openai/openrouter-auto
    together_ai/Mixtral-8x7B-Instruct-v0.1
19:50:00 - LiteLLM Router:DEBUG: Initializing OpenAI Client for together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1, Api Base:https://api.together.xyz/inference, Api Key:1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96
19:50:00 - LiteLLM:DEBUG: set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_async_client; value: <openai.AsyncOpenAI object at 0x1057a09d0>
set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_async_client; value: <openai.AsyncOpenAI object at 0x1057a09d0>
19:50:00 - LiteLLM:DEBUG: set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_client; value: <openai.OpenAI object at 0x1057c25d0>
set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_client; value: <openai.OpenAI object at 0x1057c25d0>
19:50:00 - LiteLLM:DEBUG: set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_stream_async_client; value: <openai.AsyncOpenAI object at 0x1057cd450>
set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_stream_async_client; value: <openai.AsyncOpenAI object at 0x1057cd450>
19:50:00 - LiteLLM:DEBUG: set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_stream_client; value: <openai.OpenAI object at 0x1057d0410>
set cache: key: 8669507c-fe05-410e-a1e9-93b49d27d445_stream_client; value: <openai.OpenAI object at 0x1057d0410>
19:50:00 - LiteLLM Router:DEBUG:
Initialized Model List [{'model_name': 'openai/openrouter-auto', 'litellm_params': {'model': 'openrouter/auto', 'api_base': 'https://openrouter.ai/api/v1'}, 'model_info': {'id': '73196f04-9a7e-40da-94df-2b9e8f293ebc'}}, {'model_name': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'litellm_params': {'model': 'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', 'api_base': 'https://api.together.xyz/inference'}, 'model_info': {'id': '8669507c-fe05-410e-a1e9-93b49d27d445'}}]
19:50:00 - LiteLLM Router:DEBUG: Intialized router with Routing strategy: simple-shuffle

19:50:00 - LiteLLM Proxy:DEBUG: INITIALIZING LITELLM CALLBACKS!
LiteLLM Proxy: INITIALIZING LITELLM CALLBACKS!
19:50:00 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x105568850>
callback: <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x105568850>
19:50:00 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x103f8c850>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x103f8c850>>
19:50:00 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x103f3d110>
callback: <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x103f3d110>
19:50:00 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x103f6e350>
callback: <litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x103f6e350>
19:50:00 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x1055cb2d0>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x1055cb2d0>>
19:50:00 - LiteLLM Proxy:DEBUG: prisma client - None
19:50:00 - LiteLLM Proxy:DEBUG: custom_db_client client - None
19:50:00 - LiteLLM Proxy:DEBUG: custom_db_client client None. Master_key: None
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
19:50:12 - LiteLLM Proxy:DEBUG: Request Headers: Headers({'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'})
19:50:12 - LiteLLM Proxy:DEBUG: receiving data: {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'proxy_server_request': {'url': 'http://0.0.0.0:8000/chat/completions', 'method': 'POST', 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'}, 'body': {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]}}}
19:50:12 - LiteLLM Proxy:DEBUG: Inside Proxy Logging Pre-call hook!
LiteLLM Proxy: Inside Proxy Logging Pre-call hook!
19:50:12 - LiteLLM Proxy:DEBUG: Inside Max Parallel Request Pre-Call Hook
Inside Max Parallel Request Pre-Call Hook
Inside Max Budget Limiter Pre-Call Hook
19:50:12 - LiteLLM:DEBUG: get cache: cache key: None_user_api_key_user_id; local_only: False
get cache: cache key: None_user_api_key_user_id; local_only: False
19:50:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:50:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
Inside Cache Control Check Pre-Call Hook
19:50:12 - LiteLLM Proxy:DEBUG: final data being sent to completion call: {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'proxy_server_request': {'url': 'http://0.0.0.0:8000/chat/completions', 'method': 'POST', 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'}, 'body': {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]}}, 'metadata': {'user_api_key': '1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '157'}, 'endpoint': 'http://0.0.0.0:8000/chat/completions'}, 'request_timeout': 600}
LiteLLM Proxy: final data being sent to completion call: {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'proxy_server_request': {'url': 'http://0.0.0.0:8000/chat/completions', 'method': 'POST', 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'}, 'body': {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]}}, 'metadata': {'user_api_key': '1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '157'}, 'endpoint': 'http://0.0.0.0:8000/chat/completions'}, 'request_timeout': 600}
19:50:12 - LiteLLM Router:DEBUG: Inside async function with retries: args - (); kwargs - {'proxy_server_request': {'url': 'http://0.0.0.0:8000/chat/completions', 'method': 'POST', 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'}, 'body': {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]}}, 'metadata': {'user_api_key': '1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '157'}, 'endpoint': 'http://0.0.0.0:8000/chat/completions', 'model_group': 'together_ai/Mixtral-8x7B-Instruct-v0.1'}, 'request_timeout': 600, 'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'original_function': <bound method Router._acompletion of <litellm.router.Router object at 0x1055cb2d0>>, 'num_retries': 0}
19:50:12 - LiteLLM Router:DEBUG: async function w/ retries: original_function - <bound method Router._acompletion of <litellm.router.Router object at 0x1055cb2d0>>
19:50:12 - LiteLLM Router:DEBUG: Inside _acompletion()- model: together_ai/Mixtral-8x7B-Instruct-v0.1; kwargs: {'proxy_server_request': {'url': 'http://0.0.0.0:8000/chat/completions', 'method': 'POST', 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'}, 'body': {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]}}, 'metadata': {'user_api_key': '1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '157'}, 'endpoint': 'http://0.0.0.0:8000/chat/completions', 'model_group': 'together_ai/Mixtral-8x7B-Instruct-v0.1'}, 'request_timeout': 600}
19:50:12 - LiteLLM Router:DEBUG: initial list of deployments: [{'model_name': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'litellm_params': {'model': 'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', 'api_base': 'https://api.together.xyz/inference'}, 'model_info': {'id': '8669507c-fe05-410e-a1e9-93b49d27d445'}}]
19:50:12 - LiteLLM:DEBUG: get cache: cache key: 19-50:cooldown_models; local_only: False
get cache: cache key: 19-50:cooldown_models; local_only: False
19:50:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:50:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
19:50:12 - LiteLLM Router:DEBUG: retrieve cooldown models: []
19:50:12 - LiteLLM Router:DEBUG: cooldown deployments: []
19:50:12 - LiteLLM Router:DEBUG: healthy deployments: length 1 [{'model_name': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'litellm_params': {'model': 'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', 'api_base': 'https://api.together.xyz/inference'}, 'model_info': {'id': '8669507c-fe05-410e-a1e9-93b49d27d445'}}]
19:50:12 - LiteLLM:DEBUG: get cache: cache key: 8669507c-fe05-410e-a1e9-93b49d27d445_async_client; local_only: True
get cache: cache key: 8669507c-fe05-410e-a1e9-93b49d27d445_async_client; local_only: True
19:50:12 - LiteLLM:DEBUG: in_memory_result: <openai.AsyncOpenAI object at 0x1057a09d0>
in_memory_result: <openai.AsyncOpenAI object at 0x1057a09d0>
19:50:12 - LiteLLM:DEBUG: get cache: cache result: <openai.AsyncOpenAI object at 0x1057a09d0>
get cache: cache result: <openai.AsyncOpenAI object at 0x1057a09d0>
19:50:12 - LiteLLM:DEBUG:



19:50:12 - LiteLLM:DEBUG: Request to litellm:
Request to litellm:
19:50:12 - LiteLLM:DEBUG: litellm.acompletion(model='together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', api_base='https://api.together.xyz/inference', messages=[{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], caching=False, client=<openai.AsyncOpenAI object at 0x1057a09d0>, timeout=6000, proxy_server_request={'url': 'http://0.0.0.0:8000/chat/completions', 'method': 'POST', 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'}, 'body': {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]}}, metadata={'user_api_key': '1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '157'}, 'endpoint': 'http://0.0.0.0:8000/chat/completions', 'model_group': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'deployment': 'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', 'model_info': {'id': '8669507c-fe05-410e-a1e9-93b49d27d445'}, 'caching_groups': None}, request_timeout=600, model_info={'id': '8669507c-fe05-410e-a1e9-93b49d27d445'}, max_retries=0)
litellm.acompletion(model='together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', api_base='https://api.together.xyz/inference', messages=[{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], caching=False, client=<openai.AsyncOpenAI object at 0x1057a09d0>, timeout=6000, proxy_server_request={'url': 'http://0.0.0.0:8000/chat/completions', 'method': 'POST', 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'content-type': 'application/json', 'content-length': '157'}, 'body': {'model': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]}}, metadata={'user_api_key': '1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96', 'user_api_key_user_id': None, 'user_api_key_metadata': {}, 'headers': {'host': '0.0.0.0:8000', 'user-agent': 'curl/8.2.1', 'accept': '*/*', 'content-type': 'application/json', 'content-length': '157'}, 'endpoint': 'http://0.0.0.0:8000/chat/completions', 'model_group': 'together_ai/Mixtral-8x7B-Instruct-v0.1', 'deployment': 'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1', 'model_info': {'id': '8669507c-fe05-410e-a1e9-93b49d27d445'}, 'caching_groups': None}, request_timeout=600, model_info={'id': '8669507c-fe05-410e-a1e9-93b49d27d445'}, max_retries=0)
19:50:12 - LiteLLM:DEBUG:



19:50:12 - LiteLLM:DEBUG: Initialized litellm callbacks, Async Success Callbacks: [<litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x103f6e350>, <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x103f3d110>, <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x105568850>]
Initialized litellm callbacks, Async Success Callbacks: [<litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x103f6e350>, <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x103f3d110>, <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x105568850>]
19:50:12 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x105568850>
callback: <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0x105568850>
19:50:12 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x103f8c850>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x103f8c850>>
19:50:12 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x103f3d110>
callback: <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0x103f3d110>
19:50:12 - LiteLLM:DEBUG: callback: <litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x103f6e350>
callback: <litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0x103f6e350>
19:50:12 - LiteLLM:DEBUG: callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x1055cb2d0>>
callback: <bound method Router.deployment_callback_on_failure of <litellm.router.Router object at 0x1055cb2d0>>
19:50:12 - LiteLLM:DEBUG: self.optional_params: {}
self.optional_params: {}
19:50:12 - LiteLLM:DEBUG: litellm.cache: None
litellm.cache: None
19:50:12 - LiteLLM:DEBUG: kwargs[caching]: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None
19:50:12 - LiteLLM:DEBUG: kwargs[caching]: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None
19:50:12 - LiteLLM:DEBUG:
LiteLLM completion() model= mistralai/Mixtral-8x7B-Instruct-v0.1; provider = together_ai
19:50:12 - LiteLLM:DEBUG:
LiteLLM: Params passed to completion() {'functions': None, 'function_call': None, 'temperature': None, 'top_p': None, 'n': None, 'stream': None, 'stop': None, 'max_tokens': None, 'presence_penalty': None, 'frequency_penalty': None, 'logit_bias': None, 'user': None, 'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'custom_llm_provider': 'together_ai', 'response_format': None, 'seed': None, 'tools': None, 'tool_choice': None, 'max_retries': 0, 'logprobs': None, 'top_logprobs': None}
19:50:12 - LiteLLM:DEBUG:
LiteLLM: Non-Default params passed to completion() {'max_retries': 0}
19:50:12 - LiteLLM:DEBUG: self.optional_params: {'extra_body': {}}
self.optional_params: {'extra_body': {}}
19:50:12 - LiteLLM:DEBUG: PRE-API-CALL ADDITIONAL ARGS: {'headers': {'Authorization': 'Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225eb49b6b758604a3eb8a96'}, 'api_base': ParseResult(scheme='https', userinfo='', host='api.together.xyz', port=None, path='/inference/', query=None, fragment=None), 'acompletion': True, 'complete_input_dict': {'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'extra_body': {}}}
19:50:12 - LiteLLM:INFO:

POST Request Sent from LiteLLM:
curl -X POST \
https://api.together.xyz/inference/ \
-H 'Authorization: Bearer 1fa5ac268c6ef1a77b8526c6ac8f97fe616c969e225e********************' \
-d '{'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'messages': [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}], 'extra_body': {}}'


19:50:12 - LiteLLM:DEBUG: Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details: logger_fn - None | callable(logger_fn) - False
19:50:12 - LiteLLM:DEBUG: Logging Details LiteLLM-Failure Call
Logging Details LiteLLM-Failure Call
19:50:12 - LiteLLM Router:DEBUG: Attempting to add 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown list. updated_fails: 1; self.allowed_fails: 0
19:50:12 - LiteLLM:DEBUG: get cache: cache key: 19-50:cooldown_models; local_only: False
get cache: cache key: 19-50:cooldown_models; local_only: False
19:50:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:50:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
19:50:12 - LiteLLM Router:DEBUG: adding 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown models
19:50:12 - LiteLLM:DEBUG: set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
19:50:12 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS

19:50:12 - LiteLLM Router:DEBUG: {'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1': ['<class \'litellm.exceptions.NotFoundError\'>Status: 404Message: NoneTogether_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>Full exceptionTogether_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>']}
19:50:12 - LiteLLM Router:DEBUG: Model together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1 had 1 exception
19:50:12 - LiteLLM:DEBUG: Custom Logger - final response object: None
Custom Logger - final response object: None
19:50:12 - LiteLLM Router:DEBUG: Attempting to add 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown list. updated_fails: 1; self.allowed_fails: 0
19:50:12 - LiteLLM:DEBUG: get cache: cache key: 19-50:cooldown_models; local_only: False
get cache: cache key: 19-50:cooldown_models; local_only: False
19:50:12 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:50:12 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
19:50:12 - LiteLLM Router:DEBUG: adding 8669507c-fe05-410e-a1e9-93b49d27d445 to cooldown models
19:50:12 - LiteLLM:DEBUG: set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
set cache: key: 19-50:cooldown_models; value: ['8669507c-fe05-410e-a1e9-93b49d27d445']
19:50:12 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS

19:50:12 - LiteLLM Router:DEBUG: {'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1': ['<class \'litellm.exceptions.NotFoundError\'>Status: 404Message: NoneTogether_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>Full exceptionTogether_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>']}
19:50:12 - LiteLLM Router:DEBUG: Model together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1 had 1 exception
19:50:12 - LiteLLM:DEBUG: Custom Logger - final response object: None
Custom Logger - final response object: None
19:50:12 - LiteLLM Proxy:DEBUG: Inside Max Parallel Request Failure Hook
Inside Max Parallel Request Failure Hook
19:50:12 - LiteLLM Router:INFO: litellm.acompletion(model=together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1) Exception Together_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>
19:50:12 - LiteLLM Router:DEBUG: TracebackTraceback (most recent call last):
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 274, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/llms/openai.py", line 424, in acompletion
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/llms/openai.py", line 409, in acompletion
    response = await openai_aclient.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 1330, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/_base_client.py", line 1725, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/_base_client.py", line 1428, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/_base_client.py", line 1519, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 965, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1101, in async_function_with_retries
    raise original_exception
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1059, in async_function_with_retries
    response = await original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 444, in _acompletion
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 423, in _acompletion
    response = await litellm.acompletion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2908, in wrapper_async
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2751, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 287, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 7520, in exception_type
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 6381, in exception_type
    raise NotFoundError(
litellm.exceptions.NotFoundError: Together_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>

19:50:12 - LiteLLM Router:DEBUG: Trying to fallback b/w models
Traceback (most recent call last):
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 274, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/llms/openai.py", line 424, in acompletion
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/llms/openai.py", line 409, in acompletion
    response = await openai_aclient.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 1330, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/_base_client.py", line 1725, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/_base_client.py", line 1428, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/openai/_base_client.py", line 1519, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 2378, in chat_completion
    response = await llm_router.acompletion(**data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 368, in acompletion
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 364, in acompletion
    response = await self.async_function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1042, in async_function_with_fallbacks
    raise original_exception
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 965, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1101, in async_function_with_retries
    raise original_exception
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1059, in async_function_with_retries
    response = await original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 444, in _acompletion
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 423, in _acompletion
    response = await litellm.acompletion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2908, in wrapper_async
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2751, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 287, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 7520, in exception_type
    raise e
  File "/workspace/litellm/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 6381, in exception_type
    raise NotFoundError(
litellm.exceptions.NotFoundError: Together_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>
19:50:12 - LiteLLM Proxy:DEBUG: An error occurred: Together_aiException - <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><link data-next-font="" rel="preconnect" href="/" crossorigin="anonymous"/><link rel="preload" as="style" href="/_next/static/css/c887091bb50810ff.css"/><link rel="preload" as="script" href="/_next/static/chunks/webpack-0213c00736f30f16.js"/><link rel="preload" as="script" href="/_next/static/chunks/main-915862165dbce86d.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_app-45acb72122cca2f0.js"/><link rel="preload" as="script" href="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js"/><link rel="preload" as="script" href="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js"/><link rel="preload" href="/_next/static/css/c887091bb50810ff.css" as="style"/><title>TOGETHER</title><meta name="description" content="Together Computer"/><link rel="icon" href="/favicon.png"/><meta name="viewport" content="initial-scale=1, width=device-width"/><meta name="next-head-count" content="5"/><meta name="theme-color" content="#fff"/><link rel="shortcut icon" href="/favicon.png"/><link rel="apple-touch-icon" href="/favicon.png"/><meta name="emotion-insertion-point" content=""/><style data-emotion="mui-style-global 15gmyd0">html{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;box-sizing:border-box;-webkit-text-size-adjust:100%;}*,*::before,*::after{box-sizing:inherit;}strong,b{font-weight:900;}body{margin:0;color:#333333;font-size:1rem;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol",Mplus 1p,Hiragino Sans,Hiragino Kaku Gothic Pro,游ゴシック,游ゴシック体,YuGothic,Yu Gothic,ＭＳ ゴシック,MS Gothic;font-weight:400;line-height:1.5;background-color:#fff;}@media print{body{background-color:#fff;}}body::backdrop{background-color:#fff;}</style><style data-emotion="mui-style "></style><link rel="stylesheet" href="/_next/static/css/c887091bb50810ff.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" noModule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0213c00736f30f16.js" defer=""></script><script src="/_next/static/chunks/main-915862165dbce86d.js" defer=""></script><script src="/_next/static/chunks/pages/_app-45acb72122cca2f0.js" defer=""></script><script src="/_next/static/chunks/pages/_error-7574ce9b2ec4e3e8.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_buildManifest.js" defer=""></script><script src="/_next/static/xc6W_gssf6ttfYKLA0zfV/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="bg-dotgrid"></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"statusCode":404,"_sentryTraceData":"79eb767bcbb943ddb7a648e775a28f50-be31d525e609b0b1-1","_sentryBaggage":"sentry-environment=production,sentry-release=xc6W_gssf6ttfYKLA0zfV,sentry-public_key=0f923fae2c344083925a1e9eb3d3db8b,sentry-trace_id=79eb767bcbb943ddb7a648e775a28f50,sentry-sample_rate=1,sentry-transaction=%2F_error,sentry-sampled=true"},"session":null,"rootDehydratedState":{"mutations":[],"queries":[]}},"page":"/_error","query":{},"buildId":"xc6W_gssf6ttfYKLA0zfV","runtimeConfig":{"NEXT_PUBLIC_STRIPE_PUBLIC_KEY":"pk_live_51MtEUEAyA3QpeTaBx08Uwerv6JtYmcZ47n1OoilSpII4vmOEZbHVLzpkWzMMw6egesD3chydKE52x0NXLUE7kgSA00cDaU4ieE","NEXT_PUBLIC_WSAPI_URL":"wss://api.together.xyz/api/websocket","NEXT_PUBLIC_AUTH_PROVIDERS":["cilogon","github","google","linkedin"],"NEXT_PUBLIC_AUTH0_CLIENT_ID":"4HvDdeJ32mErcbYHLAQflC13oVXjQ1cV","NEXT_PUBLIC_AUTH0_DOMAIN":"auth.together.ai"},"isFallback":false,"customServer":true,"gip":true,"appGip":true,"locale":"en-US","locales":["en-US"],"defaultLocale":"en-US","scriptLoader":[]}</script></body></html>

 Debug this by setting `--debug`, e.g. `litellm --model gpt-3.5-turbo --debug`
19:50:12 - LiteLLM Proxy:DEBUG: Results from router
19:50:12 - LiteLLM Proxy:DEBUG:
Router stats
19:50:12 - LiteLLM Proxy:DEBUG:
Total Calls made
19:50:12 - LiteLLM Proxy:DEBUG: together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1: 1
19:50:12 - LiteLLM Proxy:DEBUG:
Success Calls made
19:50:12 - LiteLLM Proxy:DEBUG:
Fail Calls made
19:50:12 - LiteLLM Proxy:DEBUG: together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1: 1
INFO:     127.0.0.1:49363 - "POST /chat/completions HTTP/1.1" 404 Not Found

The question arises as to what might have been configured incorrectly, and how can I correctly set up Litellm to serve as a proxy for Together.ai's service?

ishaan-jaff · 2024-02-17T17:08:08Z

ishaan-jaff
Feb 17, 2024
Maintainer

Hi @vitalii-lobanov your settings look good to me, can we hop on a quick call and debug this together ? Sharing a link to my cal for your convenience:
https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat?month=2024-02

0 replies

vitalii-lobanov · 2024-02-22T15:34:32Z

vitalii-lobanov
Feb 22, 2024
Author

I've identified and corrected an error in my configuration. Initially, I had set the API base URL for Together.ai to https://api.together.xyz/inference, which was incorrect. After updating it to https://api.together.xyz/v1, everything is now functioning properly. Here is my updated configuration:

model_list:
#Openrouter auto selection
  - model_name: openai/openrouter-auto # user-facing model alias
    litellm_params: # model-specific parameters
      model: openrouter/auto # the model's identifier
      api_base: https://openrouter.ai/api/v1 # OpenAI's API base URL

#Together AI Mixtral-8x7B-Instruct-v0.1
  - model_name: together_ai/Mixtral-8x7B-Instruct-v0.1 # user-facing model alias
    litellm_params: # model-specific parameters
      model: together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1 # the model's identifier
      api_base: https://api.together.xyz/v1


litellm_settings: # module level litellm settings
  drop_params: True
  set_verbose: True

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Proxy Mode Issues with Together.ai as the Backend Service #2027

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Proxy Mode Issues with Together.ai as the Backend Service #2027

Uh oh!

vitalii-lobanov Feb 17, 2024

Replies: 2 comments

Uh oh!

ishaan-jaff Feb 17, 2024 Maintainer

Uh oh!

vitalii-lobanov Feb 22, 2024 Author

vitalii-lobanov
Feb 17, 2024

ishaan-jaff
Feb 17, 2024
Maintainer

vitalii-lobanov
Feb 22, 2024
Author