[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗

[Vaibhavs10]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - brew install llama.cpp works
2. For Linux (CUDA) - they need to clone and install directly from github
3. For Windows - winget (?)

This adds barrier for non technically inclined people specially since in all the above methods users would have to reinstall llama.cpp to get upgrades (and llama.cpp makes releases per commit - not a bad thing, but becomes an issue since you need to upgrade more frequently)

Opening this issue to discuss what could be done to package llama.cpp better and allow users to maybe download an executable and be on their way.

More so, are there people in the community interested in taking this up?

[KhazAkar]

From this, IMO it only misses Linux+CUDA bundle to be useable as download & use.

Click to expand

If we want better packaging on Linux, we can also work on snap/bash installer when trying to use pre-built packages.

[michaelgiba]

On the first point this was a pain point I faced as well (#15249)

[mitkox]

It’s high time HuggingFace to copy Ollama’s packaging and GTM strategy, but this time, give credit to llama.cpp. Ideally, we should retain llama.cpp as the core component.

[KhazAkar]

For HF - they should follow similar path as Jan.ai does for example, a.k.a raw llama.cpp, not some weird forks like ollama.

[mitkox]

On technology- yes, on marketing strategy, GTM, and partnership lock - there is nothing like Ollama's way of doing it. I can't open any app without ollama support to be the "local AI" support.

[slaren]

Is the barrier the installation process, or the need to use a complex command line to launch llama.cpp?

[ericcurtin]

A bit of both.

I think packaging has improved in recent times, windows as an example. I kinda like the containers approach for Linux, it makes sense, I'm not sure how many Linux distros are going to tolerate separate llama-cpp-cuda, llama-cpp-rocm, llama-cpp-vulkan packages as per their packaging guidelines.

I do think at a minimum llama.cpp should document upstream what flags should be used with each backend CUDA, Vulkan, Metal (--threads, --cache-reuse, --flash-attention, etc.), as discussed here:

ollama/ollama#11714 (comment)

People use sub-optimal flags often when using llama.cpp and assume performance isn't up to scratch.

[qnixsynapse]

[slaren]

Are you using the GGML_CPU_ALL_VARIANTS option?

[qnixsynapse]

Nope. Separate instructions builds for CPU instructions(For example: using -DGGML_AVX512=ON for AVX512 build). We even found people use processors which doesn't even have avx2.

[slaren]

You should look into GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS, it is designed to solve this problem. Most of our releases use it.

[qnixsynapse]

Thank you!

[simonw]

For me the biggest thing is llama-server. That tool is fantastic... but it has very low discoverability. Until a few months ago I still thought it was just a demo because it lived in the examples/ folder (I just noticed it moved from there to the tools/ folder in May).

I'd love to see more emphasis placed on llama-server. It's really good! I think it's probably the most end-user appropriate way to interact with this project.

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux, which on Mac and Windows work like a desktop application: you get an icon you can use to launch the tool which start the server running and launch a window that shows that web UI.

Maybe include systray integration and a simple UI for selecting and downloading models too.

At that point llama-server would feel like an alternative to Ollama and LM Studio and Jan. I think it deserves that - it has most of what those tools offer implemented already, what's missing is an installer and a think desktop shell.

[YavorGIvanov]

Agreed. If llama-server was nicely packaged on all OSes and could be easily distributed and run everywhere, then I don't see a reason to use any alternative. The benefit will also be that you get the latest version of llama.cpp immediately without waiting for other companies to package it with nice UI.

[t1u1]

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux,

An easy way to do this is to make it a PWA. Will work in most major browsers, including Safari, Chrome and its derivatives, Android and iOS, but not Firefox. It just takes a few lines of code to do this.

[allozaur]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

[calculatortamer]

i hope they won't deprecate llama-server CLI for a llama-server GUI, it would be too annoying to use on termux or over SSH

[red-co]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

I think I'm not that enough new to this, and I can compile llamacpp for a specific GPU, but I don't know how to use the llamacpp server for persistent chat-session data caching. Does it simply not have this feature? Or am I missing the parameter?

[yaronsumel]

It will be cool if 'llama-server' would have auto configuration option to the machine/model like 'ollama' does it.

[exxocism]

For windows maybe choco and windows store would be a good idea? 🤔

[acbits]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - `brew install llama.cpp` works

2. For Linux (CUDA) - they need to clone and install directly from github

I created a rpm spec to manage installation though I think flatpaks might be more user friendly and distribution agnostic.

[schnow265]

For Windows - winget (?)

The released Windows builds are available via Scoop.

Updates happen automatically. Old installed versions are kept, and current one symlinked into a folder „current“ which provides the executables on the path.

[damianofalcioni]

is it feasible to have a single release for OS including all the backend?

[slaren]

It's technically possible, but the size of the package would be quite big.

[SamuelMarks]

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

[tellsiddh]

For linux I just install the vulkan binaries and run the server from there. Maybe we can have a install script like ollama that detects the system and launches the server which can be controlled from an app as well as cli? The user then gets basic command line utillities like run start stop load list etc?

[leoxzhao]

On Mac, the easiest way (also arguably the safest way) from a user's perspective is to find it in App Store, and install from there. Because of apps from App Store are in a sandbox, so from a user's point of view, installing or uninstalling is simple and clean. Creating a build and passing the App Store review might take some efforts (due to the sandbox constraint), but it should be a one-time thing.

[O-J1]

Its my understanding that none of the automated installs support GPU acceleration. I might be wrong but its definitely the case for Windows, which makes it useless to install via winget.

[slaren]

The winget package comes with the Vulkan backend.

[henk717]

To me the biggest advantage ollama currently has is that the optimal settings for a model are bundled, the gguf spec would allow for this to since its versatile enough to make this a metadata field inside the model. It would allow people to load the settings from a gguf and frontends can extract them and adapt them as they see fit. I think that part is going to be more valuable than obtaining the binary since downloading the binary from github is not that hard.

[logan-markewich]

This is the killer feature right here. Id use llama.cpp directly over any wrapper if this was added

[digitalspaceport]

My personal wishlist

1. Llama-swap integration
2. Catchall compiled binaries for linux, win, osx
3. GUI improvements

[neutrinotek]

I feel like point number 1 is widely undermentioned in this thread. The ease that Llama-Swap adds to being able to bounce around models for various tasks is invaluable. If it weren't for that, I would probably still be using Ollama. It still needs a bit of upfront setup, but once you get a model in, it basically becomes "set it and forget it."

Edit I just realized who I was replying to and felt the need to come back and say I love the YT channel man. Keep up the good work!

[ericcurtin]

Docker Model Runner provides 1 and 2 FWIW... They are trying to grow the community also... It uses llama.cpp upstream, contributes any changes required back... Number 3... I recommend AnythingLLM, but there are many options

[SabinStargem]

You can probably borrow KoboldCPP for this purpose. The UX is two parts - a GUI for setting up what GPU, threads, context size, and so forth to use. The other bit is a web interface for prompting the AI. Probably should ask the people at Kobold whether they want to share their stuff for the cause, or even join LlamaCPP from the UX side of things.

KoboldCPP supports Windows, Linux, and Mac.

KoboldCPP

[O-J1]

Kobold uses customTkinter which is frankly dogshit. It (CTK) hasn’t been maintained in ages, there are countless bugs and the author has publicly stated he doesn’t remember how to maintain it anymore.

it’s not at all suitable.

[henk717]

Problem there is that our launcher is completely incompatible with llamacpp's project. The launcher UI is python based and for llamacpp you'd want something c++ based.

[alonsosilvaallende]

Today was announced PyTorch 2.8 and the possibility of just using a single command (uv pip install torch) to install PyTorch with optimal compatibility with your hardware and OS (Windows and Linux):

https://astral.sh/blog/wheel-variants

It is the result of a joint open source project called WheelNext:

https://wheelnext.dev/

In particular for Nvidia hardware uses this to detect best compatibility:

https://github.com/wheelnext/nvidia-variant-provider

This might be useful to use or to partially use just the detection of the user hardware.

[rowheel]

I believe a package for llama-vulkan would be the killer app..

[zereraz]

brew install is great but we also need a single curl script like in https://bun.sh/ and https://docs.astral.sh/uv/getting-started/installation/

Also https://x.com/zereraz/status/1956081559695745528 the DX here is not the best.
llama-cli could be renamed to llama

llama-cli -hf ggml-org/gemma-3-270m-it-GGUF -c 0 -fa --jinja

to

llama run ggml-org/gemma-3-270m-it-GGUF

Also even though the logs are cool, verbose mode can be separate.

Better TUI inspired by https://github.com/vadimdemedes/ink or https://github.com/sst/opentui

[ericcurtin]

llama-run does exist although the syntax to pull from huggingface is a little different (I wrote it). I recommend

docker model run hf.co/ggml-org/gemma-3-270m-it-GGUF

it uses llama.cpp backend.

I think command prompts are better off being done in golang... I used linenoise.cpp for the command prompt, C++ is just not a great language for the user facing stuff...

[ericcurtin]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - brew install llama.cpp works
2. For Linux (CUDA) - they need to clone and install directly from github

Docker works great, is simpler than this.

3. For Windows - winget (?)

This adds barrier for non technically inclined people specially since in all the above methods users would have to reinstall llama.cpp to get upgrades (and llama.cpp makes releases per commit - not a bad thing, but becomes an issue since you need to upgrade more frequently)

Opening this issue to discuss what could be done to package llama.cpp better and allow users to maybe download an executable and be on their way.

More so, are there people in the community interested in taking this up?

[ckastner]

TL;DR Within the next few weeks, you'll be able to just apt-get install llama.cpp from the official Debian archive, and ideally Ubuntu soon after.

Longer Version: With regards to installtion usability, within the next few weeks, llama.cpp will be made available in Debian via the official bookworm-backports and trixie-backports distributions.

There are already packages in the unstable distribution but these are obsolete and will be replaced by a more sophisticated layout. This is already finished and uploaded, and is only pending Archive master approval (they are swamped at the moment because we just released Debian 13). I also plan to provide official backports for Ubuntu but I need to clear upload rights for that first (shouldn't be much of an issue).

In the releases we prepared, we ship the CPU, BLAS, HIP, CUDA and Vulkan backends. More backends will be added on request. However this is also a matter of test infrastructure, as I do test all of the backends on actual hardware before a release. I'm currently setting up a CI to automate this.

The CPU backend is built with GGML_CPU_ALL_VARIANTS=ON and should work fine on any common CPU even if old. I've tested this for amd64, arm64, and ppc64el.

I would consider it a bug of the Debian package if its performance is not on par with upstream, hence the CI I'm setting up.

This is just some initial info, I'm going to submit more information once I've got the CI finished. Most importantly, I want upstream here to see this as a benefit, not a burden, so I need to work out some user documentation and also see if upstream can benefit from our CI in some way.

Edit: Forgot to say, upstream has been very accommodating in accepting changes that make shipping universally usable packages easier for us downstreams. Also @mbaudier's input on and testing of the Debian packages mentioned above were really helpful in finalizing this.

[ericcurtin]

I'd been using GGML_NATIVE=OFF to build native binaries without features specific to newer CPUs, at least that's what I thought it did 😅 What's the difference between GGML_CPU_ALL_VARIANTS=ON and GGML_NATIVE=OFF?

[ckastner]

GGML_NATIVE=ON probes your CPU features at build-time, and builds exactly one CPU backend with those exact features.

GGML_CPU_ALL_VARIANTS=ON builds multiple CPU backends all with different features, such that a maximally performant backend is available for any common CPU. The best backend is dynamically loaded at runtime, hence this requires GGML_BACKEND_DL=ON.

[gesrtewrtwr]

I was about to open a similar issue and ask when CUDA Linux builds, like text-generation-webui has them.
(Fortunately the newly introduced --cpu-moe (never tried the same over a regex) increased the tokens per second for my 4070+64GB RAM setup from:
Before: ~6 t/s tg: llama-server -m 'Qwen3-30B-A3B-Instruct-2507-BF16-00001-of-00002.gguf' --n_gpu_layers 7 --port 8080 --threads 3 --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0 -c 4096 (~85% VRAM usage).
After: ~10 t/s tg: llama-server -m 'Qwen3-30B-A3B-Instruct-2507-BF16-00001-of-00002.gguf' --n_gpu_layers 99 --port 8080 --threads 3 --cpu-moe --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0 -c 4096 (~40% VRAM usage).
Using text-generation-webui' CUDA build I get like 12 t/s, so not bad at all for the -bin-ubuntu-vulkan-x64.zip build so far.)

Another thing is to address the ..ubuntu.. in the release build's name and either rename it to ..linux.., or add linux to the ubuntu name, because the build works on my arch just fine.

[ericcurtin]

Well I guess it's built against Ubuntu libraries so Ubuntu makes sense. This is a Linux packaging problem in general, if using dynamic linking, you gotta build for each distro. But because things are also packaged as containers, you can just use the containers on pretty much any Linux distro.