Vertex / Google Gemini error in proxy mode: "unexpected keyword argument 'max_retries', 'type': null, 'param': null, 'code': 429."

[vitalii-lobanov]

I'm encountering an error while attempting to use Litellm in proxy mode with Gemini (Vertex) from Google as the backend provider for the LLM-API. The specific error message I receive is: "unexpected keyword argument 'max_retries', 'type': null, 'param': null, 'code': 429."

Below is the query I used:

export ENDPOINT_URL="http://0.0.0.0:8000/chat/completions"
export API_KEY="No key needed"
curl -X POST "$ENDPOINT_URL" \
-H "Authorization: Bearer $API_KEY" \
-H 'Content-Type: application/json' \
-d '{
    "model": "vertex_ai/gemini-pro",
    "messages": [{
    "content": "Hello, tell me a story about a cat.",
    "role": "user"
    }]
}'

The response is:

{"error":{"message":"VertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'","type":null,"param":null,"code":429}}       

Gemini-related section in config file is:

model_list:
  - model_name: vertex_ai/gemini-pro
    litellm_params:
      model: vertex_ai/gemini-pro

litellm_settings: # module level litellm settings
  drop_params: True
  set_verbose: True
  vertex_project: "vertex-project-id-0000000000000000000000" # 
  vertex_location: "europe-west4" # 

The log from Litellm reveals the following errors:

Making VertexAI Gemini Pro Vision Call
19:58:19 - LiteLLM:DEBUG:
Processing input messages = [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]

Processing input messages = [{'content': 'Hello, tell me a story about a cat.', 'role': 'user'}]
19:58:19 - LiteLLM:DEBUG: PRE-API-CALL ADDITIONAL ARGS: {'complete_input_dict': {'max_retries': 0}, 'request_str': "llm_model = GenerativeModel(gemini-pro)\nresponse = llm_model.generate_content(['Hello, tell me a story about a cat.'])\n"}
19:58:19 - LiteLLM:INFO:
Request Sent from LiteLLM:
llm_model = GenerativeModel(gemini-pro)
response = llm_model.generate_content(['Hello, tell me a story about a cat.'])


19:58:19 - LiteLLM:DEBUG: Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details: logger_fn - None | callable(logger_fn) - False
19:58:19 - LiteLLM:DEBUG: Logging Details LiteLLM-Failure Call
Logging Details LiteLLM-Failure Call
19:58:19 - LiteLLM Router:DEBUG: Attempting to add 6ac30624-4e22-4252-a23a-1d010569726b to cooldown list. updated_fails: 1; self.allowed_fails: 0
19:58:19 - LiteLLM:DEBUG: get cache: cache key: 19-58:cooldown_models; local_only: False
get cache: cache key: 19-58:cooldown_models; local_only: False
19:58:19 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:58:19 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
19:58:19 - LiteLLM Router:DEBUG: adding 6ac30624-4e22-4252-a23a-1d010569726b to cooldown models
19:58:19 - LiteLLM:DEBUG: set cache: key: 19-58:cooldown_models; value: ['6ac30624-4e22-4252-a23a-1d010569726b']
set cache: key: 19-58:cooldown_models; value: ['6ac30624-4e22-4252-a23a-1d010569726b']
19:58:19 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS

19:58:19 - LiteLLM Router:DEBUG: {'vertex_ai/gemini-pro': ["<class 'litellm.exceptions.RateLimitError'>Status: 429Message: NoneVertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'Full exceptionVertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'"]}
19:58:19 - LiteLLM Router:DEBUG: Model vertex_ai/gemini-pro had 1 exception
19:58:19 - LiteLLM:DEBUG: Custom Logger - final response object: None
Custom Logger - final response object: None
19:58:19 - LiteLLM Router:DEBUG: Attempting to add 6ac30624-4e22-4252-a23a-1d010569726b to cooldown list. updated_fails: 1; self.allowed_fails: 0
19:58:19 - LiteLLM:DEBUG: get cache: cache key: 19-58:cooldown_models; local_only: False
get cache: cache key: 19-58:cooldown_models; local_only: False
19:58:19 - LiteLLM:DEBUG: in_memory_result: None
in_memory_result: None
19:58:19 - LiteLLM:DEBUG: get cache: cache result: None
get cache: cache result: None
19:58:19 - LiteLLM Router:DEBUG: adding 6ac30624-4e22-4252-a23a-1d010569726b to cooldown models
19:58:19 - LiteLLM:DEBUG: set cache: key: 19-58:cooldown_models; value: ['6ac30624-4e22-4252-a23a-1d010569726b']
set cache: key: 19-58:cooldown_models; value: ['6ac30624-4e22-4252-a23a-1d010569726b']
19:58:19 - LiteLLM Router:DEBUG:
EXCEPTION FOR DEPLOYMENTS

19:58:19 - LiteLLM Router:DEBUG: {'vertex_ai/gemini-pro': ["<class 'litellm.exceptions.RateLimitError'>Status: 429Message: NoneVertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'Full exceptionVertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'"]}
19:58:19 - LiteLLM Router:DEBUG: Model vertex_ai/gemini-pro had 1 exception
19:58:19 - LiteLLM:DEBUG: Custom Logger - final response object: None
Custom Logger - final response object: None
19:58:19 - LiteLLM Proxy:DEBUG: Inside Max Parallel Request Failure Hook
Inside Max Parallel Request Failure Hook
19:58:19 - LiteLLM Router:INFO: litellm.acompletion(model=vertex_ai/gemini-pro) Exception VertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'
19:58:20 - LiteLLM Router:DEBUG: TracebackTraceback (most recent call last):
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/llms/vertex_ai.py", line 706, in async_completion
    generation_config=GenerationConfig(**optional_params),
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 277, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/llms/vertex_ai.py", line 869, in async_completion
    raise VertexAIError(status_code=500, message=str(e))
litellm.llms.vertex_ai.VertexAIError: GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 977, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1164, in async_function_with_retries
    raise original_exception
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1071, in async_function_with_retries
    response = await original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 456, in _acompletion
    raise e
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 435, in _acompletion
    response = await litellm.acompletion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2969, in wrapper_async
    raise e
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2812, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 290, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 7551, in exception_type
    raise e
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 6786, in exception_type
    raise RateLimitError(
litellm.exceptions.RateLimitError: VertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'

19:58:20 - LiteLLM Router:DEBUG: Trying to fallback b/w models
Traceback (most recent call last):
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/llms/vertex_ai.py", line 706, in async_completion
    generation_config=GenerationConfig(**optional_params),
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 277, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/llms/vertex_ai.py", line 869, in async_completion
    raise VertexAIError(status_code=500, message=str(e))
litellm.llms.vertex_ai.VertexAIError: GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 2579, in chat_completion
    responses = await asyncio.gather(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 380, in acompletion
    raise e
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 376, in acompletion
    response = await self.async_function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1054, in async_function_with_fallbacks
    raise original_exception
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 977, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1164, in async_function_with_retries
    raise original_exception
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 1071, in async_function_with_retries
    response = await original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 456, in _acompletion
    raise e
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/router.py", line 435, in _acompletion
    response = await litellm.acompletion(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2969, in wrapper_async
    raise e
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 2812, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/main.py", line 290, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 7551, in exception_type
    raise e
  File "/workspace/litellm-venv/lib/python3.11/site-packages/litellm/utils.py", line 6786, in exception_type
    raise RateLimitError(
litellm.exceptions.RateLimitError: VertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'
19:58:20 - LiteLLM Proxy:DEBUG: An error occurred: VertexAIException - GenerationConfig.__init__() got an unexpected keyword argument 'max_retries'

 Debug this by setting `--debug`, e.g. `litellm --model gpt-3.5-turbo --debug`
19:58:20 - LiteLLM Proxy:DEBUG: Results from router
19:58:20 - LiteLLM Proxy:DEBUG:
Router stats
19:58:20 - LiteLLM Proxy:DEBUG:
Total Calls made
19:58:20 - LiteLLM Proxy:DEBUG: vertex_ai/gemini-pro: 1
19:58:20 - LiteLLM Proxy:DEBUG:
Success Calls made
19:58:20 - LiteLLM Proxy:DEBUG:
Fail Calls made
19:58:20 - LiteLLM Proxy:DEBUG: vertex_ai/gemini-pro: 1
INFO:     127.0.0.1:59758 - "POST /chat/completions HTTP/1.1" 429 Too Many Requests

What might be causing the errors detailed above, and what steps can I take to resolve them?

[vitalii-lobanov]

After updating Litellm to version 1.27.1, everything is functioning smoothly with the same configuration parameters and queries.

Thank you!

[ishaan-jaff]

Awesome ! it's resolved on main latest

[ishaan-jaff]

Hi @vitalii-lobanov thanks for using LiteLLM - can we hop on a call sometime this week ? I'd love to learn how we can improve litellm for you. Sharing a link to my cal for your convenience:

https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat