Configuring fallbacks in config.yaml

[zegerius]

My fallbacks don't seem to work, with errors. I opened this issue (#11197), but I am curious if it might just be a configuration error.

My config:

model_list:
  - model_name: anthropic.claude-sonnet-4-20250514-v1:0
    litellm_params:
      model: bedrock/eu.anthropic.claude-sonnet-4-20250514-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: os.environ/AWS_REGION_EU_1
      rpm: 200
      tpm: 200000

  - model_name: anthropic.claude-sonnet-4-20250514-v1:0
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: os.environ/AWS_REGION_EU_2
      rpm: 200
      tpm: 200000

  # Anthropic Claude Sonnet 3.7 20250219 (v1)
  - model_name: anthropic.claude-3-7-sonnet-20250219-v1:0
    litellm_params:
      model: bedrock/eu.anthropic.claude-3-7-sonnet-20250219-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: os.environ/AWS_REGION_EU_1
      rpm: 250
      tpm: 1000000
  - model_name: anthropic.claude-3-7-sonnet-20250219-v1:0
    litellm_params:
      model: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name:  os.environ/AWS_REGION_EU_2
      rpm: 250
      tpm: 1000000

router_settings:
  fallbacks:
    - anthropic.claude-sonnet-4-20250514-v1:0:
        - anthropic.claude-3-7-sonnet-20250219-v1:0

The error:

litellm.RateLimitError: BedrockException - {"message":"Too many tokens, please wait before trying again."}. Received Model Group=anthropic.claude-sonnet-4-20250514-v1:0
Available Model Group Fallbacks=None
Error doing the fallback: list index out of range LiteLLM Retried: 1 times, LiteLLM Max Retries: 2

The documentation on this topic is incomplete, with different sections of the documentation using different config keys. I have also tried to use the "JSON"-approach from the docs ([{"model": ["fallback1"]}]), but (obviously) that makes no difference.

I prefer my config as code, so I do not store model definitions in the database. Am I doing something wrong?

[zegerius]

@ishaan-jaff would you have a second to help me out here? I have tried everything but keep getting this error.

[krrishdholakia]

Your fallback looks misconfigured, as Available Model Group Fallbacks=None

Please try setting fallbacks like this:

router_settings:
  fallbacks: [{"anthropic.claude-sonnet-4-20250514-v1:0": ["anthropic.claude-3-7-sonnet-20250219-v1:0"]}]

[zegerius]

That stopped working @krrishdholakia, with just the config you provide I still get:

% curl -X POST 'http://litellm:4041/chat/completions' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $LITELLM_PROXY_API_KEY" \
  -d '{
    "model": "anthropic.claude-sonnet-4-20250514-v1:0",
    "messages": [
      {
        "role": "user",
        "content": "ping"
      }
    ],
    "mock_testing_fallbacks": true
  }'
{"error":{"message":"litellm.InternalServerError: This is a mock exception for model=anthropic.claude-sonnet-4-20250514-v1:0, to trigger a fallback. Fallbacks=[{}]. Received Model Group=anthropic.claude-sonnet-4-20250514-v1:0\nAvailable Model Group Fallbacks=None\nError doing the fallback: list index out of range","type":null,"param":null,"code":"500"}}%

I use the model groups quite extensively, can that be a contributing factor?

edit: while not a viable option for me (IaC), I tested it in the UI. Same problem:

side quest: that selection dialogue is not very useful when using model groups: