Architectural observations (nextSpeakerChecker, tool calls, parallel tool calls, flash-lite and codeassist vs full endpoint)

[acoliver]

So I forked gemini-cli https://github.com/acoliver/llxprt-code/ and keep up to main by an llm-back auto-cherrypicker script

1. Many of the problems have come from the nextSpeakerChecker -- moving to flash lite made it worse (the looping things reported on reddit). I ripped this out and now Gemini-2.5-pro is a. faster b. more reliable
2. None of the tools are guarded. A model can ask for 700 files from readManyFiles.
3. Parallel tool calls are a good idea in theory, but the async isn't guarded to make sure everything always stays in the correct order. (This was exposed by another model, and by speeding things up by removing the nextSpeakerChecker)
4. I understand the multi-model hybrid desire, but user choice would go a long way towards making the tool feel more reliable. The summaries for session compression produced by Flash Lite are not of the same level as those produced by Flash. Not every user will hit this with the big context window -- just the most dedicated using it for the biggest jobs.
5. Some instability comes from hitting the API on one hand and the codeAssist api on the other. There seem to be different limits. I hit that in gemini-cli and others have commented on similar issues.

The nextSpeakerChecker is just a bad idea - sending the whole conversation potentially multiple times per request to ask whose turn it is feels wasteful (of tokens, money, time). It is also why Gemini 2.5-pro feels so slow. After the request is done it says something like "I fixed the problem and it should work now.." then you sit for several seconds while nextSpeakerChecker sends the whole thing to flash-lite.

So far I haven't hit a case where pro stops like GPT-4o and just forgets whose turn it is. It seems smart enough to know what to do. In the case it isn't a heuristic could do most of the work rather than requiring a full trip to flash.

Aside from that, I realize this is the code-assist team but just using the full endpoint when the user has it enabled instead of some code-assist and some full endpoint would probably clear up the issues for the paid/power users.

I'd like these things worked through a bit because if I fix them downstream my merges get more expensive. However, I think it would make the software more robust and stable for everyone.

[acoliver]

Another Observation:

LoopDetector added another call to flash. The fun thing is that sometimes Flash actually causes a loop where there wasn't one. A better solution would be to inform the model at hand (pro) in response to its request that it seems to be in a loop, and to evaluate its approach. Also sometimes pro reacts to whatever flash is doing by really abbherent behavior. those times where it deletes and reimpleents or does generally "weird stuff" seem to be after a loop detection. But...often it wasn't looping.

It also seems that repeated tool calls with different parameters can trigger the loop (including 10 shell commands in a row) but often these aren't loops.