LiteLLM x gemini-cli Integration

[ishaan-jaff]

Starting this thread for support for using litellm with gemini-cli. Send your questions/feature requests here

Get Started Doc

https://docs.litellm.ai/docs/tutorials/litellm_gemini_cli

Demo video

View here

cc @jgowdy-godaddy @dotmobo @zkcpku @ffreemt @Macmee @artdent

[dotmobo]

hi thanks !

Can we use a non-gemini model like qwen 3 hosted with vllm, with GEMINI_MODEL="qwen3" for example ?

I have some error with that in the debug console cli :

✕ [API Error: {"error":{"message":"litellm.APIConnectionError: APIConnectionError: Hosted_vllmException - Generate content streaming is not supported for 
  hosted_vllm","type":null,"param":null,"code":"500"}}]

at async GeminiClient.sendMessageStream                                                                                                               │
│    (file:///usr/local/lib/node_modules/@google/gemini-cli/node_modules/@google/gemini-cli-core/dist/src/core/client.js:179:38)                               │
│        at async file:///usr/local/lib/node_modules/@google/gemini-cli/dist/src/ui/hooks/useGeminiStream.js:276:26                                            │
│        at async file:///usr/local/lib/node_modules/@google/gemini-cli/dist/src/ui/hooks/useGeminiStream.js:343:38 

Thanks !

[ishaan-jaff]

did gemini work for you ?

[dotmobo]

sure i will test it.
gemini works yes, it's just about hosted vllm models.

[dotmobo]

same error with ollama models

✕ [API Error: {"error":{"message":"litellm.APIConnectionError: Generate content streaming is not supported for ollama\nTraceback (most recent call last):\n  File 
  \"/usr/lib/python3.13/site-packages/litellm/google_genai/main.py\", line 339, in agenerate_content_stream\n    setup_result = 
  GenerateContentHelper.setup_generate_content_call(\n        model=model,\n    ...<4 lines>...\n        **kwargs\n    )\n  File 
  \"/usr/lib/python3.13/site-packages/litellm/google_genai/main.py\", line 135, in setup_generate_content_call\n    raise ValueError(\n        f\"Generate content 
  {operation} is not supported for {custom_llm_provider}\".strip()\n    )\nValueError: Generate content streaming is not supported for 
  ollama\n","type":null,"param":null,"code":"500"}}]

[ishaan-jaff]

on it !

[ishaan-jaff]

ollama, openai, azure added here: #12081

[Tombar]

hey, do we need to have gemini models named some way to get this to work? do we need to expose all models?

we have for example a model named gemini_25_pro and gemini-cli isnt working

[dotmobo]

I saw the following parameters in the gemini-cli repo :

export const DEFAULT_GEMINI_MODEL = 'gemini-2.5-pro';
export const DEFAULT_GEMINI_FLASH_MODEL = 'gemini-2.5-flash';
export const DEFAULT_GEMINI_EMBEDDING_MODEL = 'gemini-embedding-001';

So the tool use an embedding model too

[ishaan-jaff]

I tried with the env vars, it did not change anything for me

[dotmobo]

DEFAULT_GEMINI_MODEL, DEFAULT_GEMINI_FLASH_MODEL and DEFAULT_GEMINI_EMBEDDING_MODEL are hardcoded, it's not env vars. The only en var is GEMINI_MODEL.

But i suppose we can use alias for the embedding model in litellm too.

[Tombar]

I saw the following parameters in the gemini-cli repo :

export const DEFAULT_GEMINI_MODEL = 'gemini-2.5-pro'; export const DEFAULT_GEMINI_FLASH_MODEL = 'gemini-2.5-flash'; export const DEFAULT_GEMINI_EMBEDDING_MODEL = 'gemini-embedding-001';

So the tool use an embedding model too

based on this, we need to alias those constants to our models, right?

[dotmobo]

Yes

[sj-rtl]

I don't think this is working as it should/expected.

I'm using the latest available LiteLLM version (1.73.2-nightly) and while only with this version I'm at least able to connect to Gemini via LiteLLM at all (prior versions aren't working), the gemini-cli isn't able to run any tools.
I can chat with Gemini just fine, however if I'm trying to use the cli as an agent it fails. For example if I tell it to

"create a textfile with the sentence 'hello world'"

I get the error:

✖ Failed to parse JSON response from generateJson
  Full report: /var/folders/.../gemini-client-error-generateJson-parse-2025-06-27T12-14-41-305Z.json

✖ Error generating JSON content via API  
  Full report: /var/folders/.../gemini-client-error-generateJson-api-2025-06-27T12-14-41-305Z.json
  
  ⚠ Failed to talk to Gemini endpoint when checking if conversation should continue

Error: Failed to generate JSON content: Failed to parse API response as JSON: 
Unexpected token '`', "```

at GeminiClient.generateJson (file:///.../gemini-cli-core/dist/src/core/client.js:239:19)
at async checkNextSpeaker (file:///.../gemini-cli-core/dist/src/utils/nextSpeakerChecker.js:98:33)  
at async GeminiClient.sendMessageStream (file:///.../gemini-cli-core/dist/src/core/client.js:179:38)
at async file:///.../gemini-cli/dist/src/ui/hooks/useGeminiStream.js:276:26
at async file:///.../gemini-cli/dist/src/ui/hooks/useGeminiStream.js:343:38

So I'm not even able to allow the shell command it has generated.
All works fine when using the Gemini API directly.

[ishaan-jaff]

@sj-rtl free to do a quick debug call ?

[Tombar]

working for us too, after we set the 3 alias properly

[dotmobo]

there is a new nightly docker image with the latest fixes ?

[sj-rtl]

@ishaan-jaff : thanks for taking a look! that is strange. it may sound stupid, but did you actually check that your file was created? because when I ask gemini, it tells me that the files exists, but in reality it does not. did you have to accept/confirm the action?

@dotmobo : no it still does not work for me with the latest available image (v1.73.6.rc.1).

The response that is being returned cannot be parsed by the gemini-cli, as according to the cli error logs there is an unexpected token '`'

I saw that you @ishaan-jaff were using gemini-cli 0.1.3 so I tried that out too, without luck.

[akhilbelide]

@ishaan-jaff
Even in your screenshot, it prompts you to create the file using the provided command. Shouldn’t the expectation be that the Gemini CLI creates the file automatically?

[grepsuzette]

With GOOGLE_GEMINI_BASE_URL=http://localhost:4000 gemini, does Gemini-cli still require to authenticate with Google?

[sj-rtl]

The Gemini-CLI integration still isn't working properly imo. Tested with the latest images (v1.73.6-stable.patch.1, v1.74.0.rc.2)
@ishaan-jaff could you maybe briefly double check, that it actually creates the files for you? Because Gemini is telling me that it created the file, it even shows me the "current directory structure" which is just wrong though.

Again, it's working correctly, when I use the Gemini API directly. The difference is, that when using the native Gemini API, I will immediately get asked to confirm the action.

[kenn20]

Hello! Thank you so much for providing the tutorial!

Currently I am attempting to use a locally hosted Ollama LLM model as the LLM for Gemini CLI, since I have sensitive documents that I prefer not to send to 3rd party.

When I tried the tutorial, I have successfully called LiteLLM's proxy with Gemini CLI, but then the response from the LiteLLM's proxy does not seem to work as expected.

When I send Gemini CLI a simple query, this is what was returned :

Then, I checked the LiteLLM server and found this response:

Last time I've checked from Ollama's side, Deepseek r1 should support tool calling. According to the tutorial here, https://docs.litellm.ai/docs/tutorials/litellm_gemini_cli#use-anthropic-openai-bedrock-etc-models-on-gemini-cli, I set the model_alias to use DeepSeek-r1 rather than Gemini-Pro-2.5.

Is this a problem regarding how LiteLLM handles some of the Gemini tool calling, or a problem regarding the model itself?

[retanoj]

I meet this problem, too. And I did some tests.
Firstly, I used LiteLLM proxy gemini-2.5-pro for gemini-cli. It works ok!
Secondly, I used LiteLLM proxy openai/gpt-4.1-mini. It throws Exception like below:

09:27:25 - LiteLLM Router:INFO: router.py:2634 - ageneric_api_call_with_fallbacks(model=gemini-2.5-pro) Exception litellm.APIConnectionError: APIConnectionError: OpenAIException - Error calling litellm.acompletion for generate_content: litellm.BadRequestError: OpenAIException - Invalid schema for function 'read_many_files': '1' is not of type 'integer'.

So I think this is about how LLM does model proxy.