On-Device Zeta

[chrisvander]

Putting Zeta on servers and relying on data center compute, plus network transfer speed, is expensive in both latency and literal server cost. Why not, for the hardware that can reasonably support it, put Zeta on-device? This would potentially allow for more reliable latency and offline use, in addition to being better for privacy.

Either that, or an option to do completions on-device through Ollama.

It's baffling to me that companies continue to not utilize local compute. I don't want to pay yet another subscription to access tools like this. I paid for a powerful local system - use it!

[baldwindavid]

I don't really disagree though this is an OS model (as is Zed itself) and Zed does interface with Ollama and LM Studio so maybe that's coming. Other than that, I would very much like to see Zed continue on its own. Presumably at a certain point they'll need to monetize somehow or the logical conclusion is getting acquired by a large company with varied motivations and priorities.

[chrisvander]

Monetization via charging for collaboration features makes the most sense to me. On device completion as part of the open source solution just improves the product experience for a individual developers, who in turn will bring Zed to their teams. Maybe remote completion models are more powerful - that makes sense to charge for.

[baldwindavid]

Yep, I believe they've publicly stated collaboration and AI being the focus in terms of profitability. I think they've done a good job with transparency and interoperability thus far and have no reason to think that won't continue within the constraints of, uh, survivability.

[antran22]

I'm going to chime in here and give some of my perspective.
The current scene of AI model for editor, especially Copilot-like predictive editing is mainly revolving around propriety model and over-the-network communication. Remote LLM may make sense in the case of chat/reasoning models that are hundreds of billion parameters large, but for smaller models, it is already quite feasible to run locally efficiently. This allows me to utilize the feature in an air-gapped environment.

I respect Zed team's decision of releasing Zeta fully opensource & open-data, and I guess that on-device zeta is a direction that they will take in the near future. After all, Zed already allow fully customizable LLM source for the assistant, it would make no sense that they will paywall this feature.

I would love to get the Zed team's input on this.

[isterin]

Same, on device would be nice. I don't mind using remote for personal stuff, but lots of work restrictions about remote LLM models, makes using it not possible. I have local LM Studio and it would be great if I can use it with that.

[jeapostrophe]

Similarly, it would be nice to have the inline edit predictions with other models like a fat Qwen 32b model. I can manually make calls myself with Ollama, but I want it integrated into my awesome editor

[praktiskt]

I've been working on a personal project called zedex to which I recently added a crude implementation of edit prediction for Zed using any OpenAI-compatible backend. Some of the bits in the implementation are just winged / didn't really look too close at the Zed implementation but it seems to work OK for the small things I use it for. At the moment is uses some serious prompt engineering (lol) as glue to work well with both Llama 3.3 70B and the newer Llama 4 models. I haven't tried the Zeta fined-tuned model.

At the moment it requires you to give up a lot of other Zed features including (and not limited to) login and collaboration as you have to override server_url in your settings, but perhaps it's useful for other people than me.

[avdept]

would be nice to get it to work locally. Recently it seem like I hit a limit of free predictions on my free plan and was wondering what's next. I already run qwen2.5 coder locally, and since zeta also based on same model, I think it would be nice to run it all locally

[Mayeu]

@phromo are you able to run local prediction using ZED_PREDICT_EDITS_URL? I tried this approach with llama.cpp but I'm getting a 500 error:

got exception: {"code":500,"message":"[json.exception.out_of_range.403] key 'messages' not found","type":"server_error"}
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 500

Can you describe how you made it work if you did? Thanks :)

Edit:
I launched Zed with: ZED_PREDICT_EDITS_URL=http://localhost:8080/v1/chat/completions zed .

[phromo]

@Mayeu edit predictions API/URL rely on a non-standardised API (dont think there is any industry standard for wide edit predictions yet, at all, so this is not dramatic). To my best knowledge, you will have to implement your own server for now since zed-industries has not open sourced their cloudflare worker that runs the model. Have a look at https://github.com/praktiskt/zedex/blob/main/src/zed/edit_predict.go and https://github.com/zed-industries/zed/blob/main/crates/zeta/src/zeta.rs for pointers. I never finished my server but will ping and open source it if I do :)

TLDR -- ZED_PREDICT_EDITS_URL does not expect the chat/completions "openai"-style API. It needs something else since the prompt is created serverside and not clientside.

[Mayeu]

@phromo I see, thanks for the pointer :) I found out that somebody created a server for the prediction: https://github.com/g0t4/zed-zeta-server, I tried that a bit, but I don't think my laptop is powerful enough for it to be usable anyway x)

[mspinelli]

@phromo I see, thanks for the pointer :) I found out that somebody created a server for the prediction: https://github.com/g0t4/zed-zeta-server, I tried that a bit, but I don't think my laptop is powerful enough for it to be usable anyway x)

I tried to use this locally on my Mac Mini M4 Pro. Each keystroke fires off the model, and there seems to be a delay of several seconds for the model to process the input (it adds up when typing as it queue the async server with ollama requests). Even accounting for the delay, I would expect that it would eventually show something in zed, but it never does. Not sure if this is because I'm doing something wrong with the setup, or if zed is timing out for predictive edits. Opened an issue over there about this here.

To be able to run Zeta locally for tab/predictive completions, I think that we'd need to train a smaller version of Zeta (e.g. 1.5B for the M4 Pro and maybe a 3B for the M4 Max). The training data and example notebooks are available on hugging face. It might be something I could attempt to try on my Mac if the Nvidia Cuda code is portable to Apple MLX.

I also understand that Zed uses Zeta with speculative decoding, and that probably would be needed as well, but that might present issues as well as I can't get speculative decoding to work properly on Mac (it's about 24% slower form me at least using GGUF running through LM Studio, I haven't tried ollama and don't know if they have enabled the feature yet.

That said, I know continue.dev can easily do fast predictive completions with a 1.5B model locally, though it's nothing like the kind of super-complete that Windsurf or Cursor offer. And I'm not even sure if Zeta is more like continue.dev or more like Cursor/Windsurf. Does anyone know? With continue.dev I was using this model: hf.co/prithivMLmods/Qwen2.5-Coder-1.5B-Instruct-GGUF:F16

[praktiskt]

@phromo This was always possible with zedex but not clear without reading the source, so I updated the README to include a section for it.

[formbook]

this is a no brainer, people are not paying 20 a month to get edit predictions from qwen2.5coder 7b, an open source, small and limited model, theyre paying for api access to the latest big time models with agenetic editing

zeta is open source, the community contributed to it, and any half decent hardware can run it locally, whatever it costs to run edit predictions for zed would be practically wiped out

windsurf has separated edit prediction from the main subscription, its unlimited on free tier and its a better model/ux than zeta, although not as good as Cursor Tab, zeta is in 3rd place at the minute for edit prediction and theres just no justification for locking it behind a 20 a month subscription

the people who want chat, agents and collab are going to subscribe regardless of whether zeta can be run locally or not, and people who only use edit prediction are not going to subscribe when better is available elsewhere for free

[marcochiodo]

I would add that locally a developer could also use qwen2.5coder 32b and have significant advantages.

[JonathanHHenson]

I’m personally not allowed to use Zed’s LLM API due to security policies. I can only use offline models and copilot. So I’m being locked out of a major feature in Zed.

[adamierymenko]

I would pay for Zed to get this feature, so to me it's not a $$$ question it's a flexibility and capability question. I'd like to be able to set up and configure arbitrary local or remote providers as edit predictors with a lot of flexibility.

[ooJaan]

+1 would also really appreciate this feature

[dwt]

I would really like to hack on this feature, try other models, see the difference, try their fine tuning and improve on it. Not being able to execute this locally - at least I couldn't yet find a way - makes this really difficult.

[marcochiodo]

This would be a much appreciated feature. Vscode allows code autocomplete via ollama with the continue plugin.

[raucheacho]

same opinion zed is an awesome code editor i switched from vscode but lost my local completion capability and want to go back🫢

[sergiomujica]

I might pay to have Gemini do my code predictions, not being able to do it is highly motivating to refuse to pay a subscription and move away from zed.

[van-sprundel]

I would +1 on the idea to support ollama for edit predictions. Currently every network option (yes, even gh copilot) is too slow. Supermaven was a great standout but they stopped updating their plugins (despite their blogpost saying the opposite).

Finding more alternatives would be amazing

[filipwiech]

Might be of interest: #33616, support for Ollama as inline completion provider, courtesy of @oliverbarnes. Perhaps could be even used to run the open source Zeta model (fine-tune of Qwen 2.5 Coder) in the future. 👍

[oliverbarnes]

I spoke too soon, unfortunately - sorry for raising expectations. During implementation I realized how different Zeta is from other coding models available on Ollama, it really needs a dedicated solution.

I've since restricted this PR's scope to support FIM models. Zeta support will have to wait for another iteration

[gordio]

And ollama pull hf.co/tensorblock/zeta-GGUF:Q4_K_M too?

[oliverbarnes]

Haven't tested it - how is it different from canonical Zeta?

[gordio]

Oh, I misinterpreted your previous message.

[filipwiech]

Will leave this here, since it might be of interest: Introducing Instinct: the world’s best open Next Edit model, built by Continue. 👍

It's open source, locally deployable in Ollama, they mention Zed's Zeta in the blog post as well. 🙂

[floinay]

Any answer/update from the Zed team?

[skorotkiewicz]

https://github.com/g0t4/zed-zeta-server

[everdrone]

Any news on this? Would love to use Zeta locally in Zed.

[probably-neb]

Hello all! For those interested in running Zeta locally, we recently added support for it through ollama as part of the release last week.
We didn't advertise it very loudly as unfortunately ollama doesn't have support for speculative decoding and therefore local zeta runs orders of magnitude slower locally via ollama on competent hardware than it does in our hosted offering (including network latency!). If you'd like to try it out - be my guest, but expect multiple seconds of delay.
This is an area where we'd happily welcome community contributions! We're focusing on other projects internally now, but if someone were to open a PR that made it so we used the OpenAI-compatible ollama endpoint instead of the ollama specific one (or just add support for OpenAI-compatible API support) then users could use runtimes like vllm to run zeta with speculative decoding locally.
If you're interested in taking this on, feel free to open a PR and @-mention me in the description and I can help answer questions and guide it in the right direction.

[everdrone]

afaik https://github.com/ggml-org/llama.cpp already supports speculative decoding: ggml-org/llama.cpp#10455