I don't belive GEMINI CLI runs gemini-2.5-pro. Sorry, Google, but it performs so poor

[konashevich]

I checked the qaulity of return in Aistudio and Gemini chat, Gemini CLI and Code Assits extension in VS Code.

Aistudio and Gemini - solve code problems just fine.
The same problems Gemini CLI and Code Assits CANNOT resolve. It does not seems Code Assits even uses PRO, but there is no blame - you never advertised it.

But when GEMINI CLI returns garbage being kinda "PRO" it gets so obvious, especially when I throw the same task to PRO in Gemini Chat or Google AI studio.

But why?

[Manamama]

Run formal tests then, from any AI bench 🤖👨🏽‍🏭and FYI us (and Google) with harder proofs.

[bdmorgan]

Yes please provide additional details. Feel free to create an issue with a specific example. There is the raw Gemini 2.5 Pro model and then there is the application logic (i.e. Gemini CLI) built on top of it. Understanding where there are issues will allow us to continue to improve the CLI logic and system prompt.

[Manamama]

Or, interim, use some MCP: https://github.com/modelcontextprotocol/servers/tree/main/src/sequentialthinking

[IgorAherne]

I believe google itself is throttling Pro 2.5 heavily ...why wouldn't they? Nobody could prove it.

2.5 Pro recently started suggesting really bad code to me, so it's not the CLI issue.
I think it's the way of coping with huge number of users. Or as means to swap-out for a smaller, distilled model so that users vote its answers good/bad.

Here it completely missed that it needs static keyword for variables to persist across invocations.
The older 2.5 Pro Experimental never made such mistakes, given such tiny snippets.

[RichardMVM]

I believe google itself is throttling Pro 2.5 heavily ...why wouldn't they? Nobody could prove it.

2.5 Pro recently started suggesting really bad code to me, so it's not the CLI issue. I think it's the way of coping with huge number of users. Or as means to swap-out for a smaller, distilled model so that users vote its answers good/bad.

Here it completely missed that it needs static keyword for variables to persist across invocations. The older 2.5 Pro Experimental never made such mistakes, given such tiny snippets.

As for me, I don't use Gemini 2.5 Flash via the API, but rather Gemini 2.5 Flash experimental, because the former doesn't understand prompts as well as the latter.

[jrobinson01]

without hard evidence, I can attest to something being "off" in gemini-cli. Over the past week, I've been hammering on a decently complex react/typescript app on aistudio. Requests take a decent amount of time (Ran for 131s, Thought for 79s) and in the app builder in studio, you can see all(?) the "thinking" tokens. Overall, the experience has been amazing. Very rarely did the model not get the code right. After adding probably a dozen or so features, the app had one or two small regressions that it was able to remedy with very simple prompts. Kicker, as best as I can tell, this whole week's worth of development (about an hour or 2 a day) has cost me zero dollars.

I'm now at a point where I want to deploy the app but I don't trust the "deploy" button in aistudio, so I've pushed the code up to github and have checked it out locally. Editing in vscode, with gemini-cli. Gemini-cli is responding much quicker than aistudio, but also flubbing nearly every request, leaving the code a broken mess of either syntax or basic typescript errors, hallucinating package names, forgetting to install packages, etc. Kicker, gemini-cli is auth'd with my vertex account, and I believe actually costing me money.

If there is a way I don't know about to gather logs from either tool please do let me know.

[jrobinson01]

follow up - I had originally installed gemini-cli 0.1.0 a while ago, via homebrew. I had updated it to 0.4.0 yesterday evening. That's when things started feeling off. I ended up uninstalling it and re-installing via npm and now things seem much better. I don't know that it feels as solid as the aistudio app builder. Will report back in a few days with a yay or nay.