Feature: more than completion (deletion, jumping etc.)

[ilan-schemoul]

I tried a lil bit of Cursor and it's incredibly good. Here are some suggestions (especially the first one) inspired by Cursor that could be beneficial to minuet in my opinion. For added context Fusion, their technology behind tab, is proprietary and it might be a custom LLM (hard to verify tbh) so we might never hit the same level as them but can still improve things.

Deletion, moving around, modifications

If you try tab feature of Cursor it's very impressive. It dosn't just insert code, it can also suggest to modify/delete current line and even to "jump somewhere" and if you accept to jump to the position it then offers you a completion (so you press tab to jump then tab to accept new proposition). It's outstandingly impressive and productive.

Multiline partial suggestions

Another thing is that if there's a multiline proposition as such:
if (xyz) {
toto();
}
It first only offers to insert the if statement if (xyz) {
then once you accept it it immediately offers you to insert the rest (previously hidden suggestion). It's cool because it happens so often you want to accept a first part of proposition and then write some more code then accept again. You can manually accept line by line interesting proposition with minuet.

[milanglacier]

This is a duplication of #26.

Before closing this, I want to briefly introduce the status of quo.

The core competitive advantage of Cursor Tab lies in its private model and private inference stack. It's difficult for open-source software that relies on publicly available APIs to compete in this area. The open-source software Zed has released the weight of its own specially trained model, but as a commercial company, they have also deployed a private inference stack.

For more details, you can refer to the article published by Fireworks, Cursor's service provider: https://fireworks.ai/blog/cursor

In short, the reason why functionalities like Cursor Tab can achieve multi-location completion and predict the next cursor position is because the model generates the entire code block surrounding the cursor, including both the code that doesn't need to be modified and the code that does. The model will continue to output until it generates a <cursor_position> token. Once the model outputs this token, the editor compares the AI-generated code block with the current code file, generates ghost text or diff, and identifies the cursor position.

Given the nature of this task, you can imagine that extremely fast generation speed is required. Cursor's proprietary model achieves speeds of 1000+ tokens/s. Currently, the fastest publicly available model (excluding clumsy ones) is Gemini 2.0 Flash (non-lite), which only reaches 170-180 tokens/s.

A generation speed of 170-180 tokens/s is adequate for code completion at cursor position, as what Minuet is offering now, where the AI only needs to generate new code. However, this speed is insufficient for generating a complete code block.

With that being said, I do still have plan for experimenting with the entire code block generation plan for providing something similar to cursor tab. The only thing is that the speed cannot even remotely match cursor. As I don't have the resource for fine-tuning a model (like what the zed team did, they fine-tuned a qwen-2.5-coder model with RL). So all I can do will be just prompt engineering with a public LLM model, say gemini-flash as it is the fastest model on cloud. But the user can choose their preferred LLM to work with as I am not using any fine-tuned model.

[milanglacier]

Side note, the perceived speed of local LLMs, even when leveraging powerful GPUs like the M3 Max, can be misleading. While Ollama / Llama.cpp might seem fast in chat scenarios with streaming output, it may not be fast enough for entire code generation.

For a more detailed analysis, refer to the llama.cpp discussion: ggml-org/llama.cpp#4167. Observe the Q4_0 PP [t/s] metric for latency and Q4_0 TG [t/s] for generation speed.

For example, using a 7B model on an M3 Max yields a generation speed of approximately 65 tokens/s. Other than token generation speed (throughput), the time to the first token (latency) is also crucial. Before generating the first token, the model must process the prompt and compute the KV cache. The token processing speed for the M3 Max is over 800 tokens/s, implying a delay of 2-3 seconds before the first token appears when the prompt exceeds 2000 tokens.

[ilan-schemoul]

Thanks for the very thorough answer, that's very interesting. I didn't know Cursor had such a leading advantage in speed. It kinda saddens me that open source is like a year behind closed source but maybe progress will slow down and we'll eventually catch up.

[ilan-schemoul]

The open-source software Zed has released the weight of its own specially trained model, but as a commercial company, they have also deployed a private inference stack.

But just to be clear: if zed's llm is available why not build the support for next edit suggestion ? I understand their inference stack is private but I'm sure at some point some provider will be fast enough to have 1000 speed (if zed has it, why not others ?). Maybe currently no one has high speed like that but if no one implements client side for next edit suggestion but zed then there's no incentive for LLM providers to supply very fast servers with the LLM in question ?

I think maybe you should decorellate the question of hardware support and software support.

You say you don't have the ressources to fine tune a LLM but using zed's one would provide the same result, wouldn't it ?

[milanglacier]

The Zeta model requires recent edit history as a prompt for the LLM.

I’m unsure how to implement recent edit history in Vim. It’s a cumbersome task—I attempted using vim.fn.execute "changes", but that still far way from capturing the necessary edit history for prompt preparation.

[milanglacier]

You say you don't have the ressources to fine tune a LLM but using zed's one would provide the same result, wouldn't it ?

Honestly, I don't have a strong incentive to develop an extension for the Zed model.

One primary challenge is that I haven't identified an effective method to access recent edit history.

Additionally, I'm unable to use the model myself. I don't have a sufficiently powerful GPU to run Zeta at a reliable speed. Since Zeta is an entire code generation model in order to have next cursor prediction, it requires higher speed than a standard FIM model.

Please understand that a developer's motivation becomes questionable when they cannot personally use what they're creating. As this is a FOSS project that I make no money from, my primary motivation for plugin development is for my personal use.

[ilan-schemoul]

One primary challenge is that I haven't identified an effective method to access recent edit history.

You might find this discussions interesting then Davidyz/VectorCode#45 (comment). It's a bit harder than it seems indeed.

Additionally, I'm unable to use the model myself. I don't have a sufficiently powerful GPU to run Zeta at a reliable speed. Since Zeta is an entire code generation model in order to have next cursor prediction, it requires higher speed than a standard FIM model.

Oh you typically run LLMs locally gotcha. Yeah I couldn't even find online provider anyway.

Please understand that a developer's motivation becomes questionable when they cannot personally use what they're creating. As this is a FOSS project that I make no money from, my primary motivation for plugin development is for my personal use.

I understand that, that's also how I work when I do open source I do if it's useful to me. That's how it always goes.

My fever dream would be someone implementing next cursor edit with Codestral and then someone making a PR to implement it in minuet. But it's daydreaming :p

[milanglacier]

Oh you typically run LLMs locally gotcha. Yeah I couldn't even find online provider anyway.

There seems to be no public online provider offering Zeta inference. While it's technically possible to deploy Zeta on an H100 VPS, the cost is not acceptable, around $2 per hour.

someone implementing next cursor edit with Codestral

Regarding implementing the next cursor edit with Codestral, it's reported by OpenRouter that Codestral achieves an inference speed of 180 tokens per second, similar to the Gemini-2.0-flash. Therefore, generating code for a small context (say around 20 lines) should be not too slow (hopefully, I guess)...

If we employ Codestral or Gemini-2.0-flash to handle entire code generation within a small context surrounding the cursor (say 20 lines), we can achieve multiple addition, deletion, and next cursor prediction within this scope.

However, I do admit that recent edit history is really useful for an AI to accurately predict the next cursor location. Unfortunately, there seems to be no straightforward way in vim to provide this information to LLMs.

Plus, without a large chat model or a dedicated fine-tuned model, understanding recent edit history might also be challenging for the model.

[ilan-schemoul]

However, I do admit that recent edit history is really useful for an AI to accurately predict the next cursor location. Unfortunately, there seems to be no straightforward way in vim to provide this information to LLMs.

I think that maybe we should ask neovim core for such an implentation. It seems that moving forward it's an essential tool to have more useful LLMs as I assume it shouldn't be hard to do (as vim already implements an undo list all the change information are somewhere). Actually I am wondering if we could just reuse the undolist 🤔

Plus, without a large chat model or a dedicated fine-tuned model, understanding recent edit history might also be challenging for the model.

Yes I've tried for many hours and codestral and qwen absolutely fail to understand what I say. I tried to say that everything after <user_edit> is one single edit but they don't get it. However recent edit is a separate topic than next cursor edit suggestion I believe ?

Anyway it seems like we're missing both on tooling (list of recent changes) and LLM (that understand context and that can do region changes) to easily implement the feature we're talking about. Let's hope that things move forward and some LLM similar to zeta is released soon (or zeta is deployed on a cheap LLM provider) and that tooling follows.

Thanks for the insights.

[GitMurf]

If you end up pursuing this and need help testing or bouncing ideas off, let me know as I am very interested in this! I also played around with this concept of trying to "track recent changes" and the best I came up with was to use :changes info to develop the "history" / chronology of the recent changes and then match it up with git hunks of changes. So this requires being in a git repo and would only work against current hunks of uncommitted changes. It worked decent-ish but was still pretty brittle.