[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗

[Vaibhavs10]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - brew install llama.cpp works
2. For Linux (CUDA) - they need to clone and install directly from github
3. For Windows - winget (?)

This adds barrier for non technically inclined people specially since in all the above methods users would have to reinstall llama.cpp to get upgrades (and llama.cpp makes releases per commit - not a bad thing, but becomes an issue since you need to upgrade more frequently)

Opening this issue to discuss what could be done to package llama.cpp better and allow users to maybe download an executable and be on their way.

More so, are there people in the community interested in taking this up?

[KhazAkar]

From this, IMO it only misses Linux+CUDA bundle to be useable as download & use.

Click to expand

If we want better packaging on Linux, we can also work on snap/bash installer when trying to use pre-built packages.

[michaelgiba]

On the first point this was a pain point I faced as well (#15249)

[maxwell-kalin]

"snap" the thing from ubuntu?

[mitkox]

It’s high time HuggingFace to copy Ollama’s packaging and GTM strategy, but this time, give credit to llama.cpp. Ideally, we should retain llama.cpp as the core component.

[KhazAkar]

For HF - they should follow similar path as Jan.ai does for example, a.k.a raw llama.cpp, not some weird forks like ollama.

[mitkox]

On technology- yes, on marketing strategy, GTM, and partnership lock - there is nothing like Ollama's way of doing it. I can't open any app without ollama support to be the "local AI" support.

[slaren]

Is the barrier the installation process, or the need to use a complex command line to launch llama.cpp?

[ericcurtin]

A bit of both.

I think packaging has improved in recent times, windows as an example. I kinda like the containers approach for Linux, it makes sense, I'm not sure how many Linux distros are going to tolerate separate llama-cpp-cuda, llama-cpp-rocm, llama-cpp-vulkan packages as per their packaging guidelines.

I do think at a minimum llama.cpp should document upstream what flags should be used with each backend CUDA, Vulkan, Metal (--threads, --cache-reuse, --flash-attention, etc.), as discussed here:

ollama/ollama#11714 (comment)

People use sub-optimal flags often when using llama.cpp and assume performance isn't up to scratch.

[qnixsynapse]

[slaren]

Are you using the GGML_CPU_ALL_VARIANTS option?

[qnixsynapse]

Nope. Separate instructions builds for CPU instructions(For example: using -DGGML_AVX512=ON for AVX512 build). We even found people use processors which doesn't even have avx2.

[slaren]

You should look into GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS, it is designed to solve this problem. Most of our releases use it.

[qnixsynapse]

Thank you!

[simonw]

For me the biggest thing is llama-server. That tool is fantastic... but it has very low discoverability. Until a few months ago I still thought it was just a demo because it lived in the examples/ folder (I just noticed it moved from there to the tools/ folder in May).

I'd love to see more emphasis placed on llama-server. It's really good! I think it's probably the most end-user appropriate way to interact with this project.

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux, which on Mac and Windows work like a desktop application: you get an icon you can use to launch the tool which start the server running and launch a window that shows that web UI.

Maybe include systray integration and a simple UI for selecting and downloading models too.

At that point llama-server would feel like an alternative to Ollama and LM Studio and Jan. I think it deserves that - it has most of what those tools offer implemented already, what's missing is an installer and a think desktop shell.

[YavorGIvanov]

Agreed. If llama-server was nicely packaged on all OSes and could be easily distributed and run everywhere, then I don't see a reason to use any alternative. The benefit will also be that you get the latest version of llama.cpp immediately without waiting for other companies to package it with nice UI.

[t1u1]

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux,

An easy way to do this is to make it a PWA. Will work in most major browsers, including Safari, Chrome and its derivatives, Android and iOS, but not Firefox. It just takes a few lines of code to do this.

[allozaur]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

[calculatortamer]

i hope they won't deprecate llama-server CLI for a llama-server GUI, it would be too annoying to use on termux or over SSH

[red-co]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

I think I'm not that enough new to this, and I can compile llamacpp for a specific GPU, but I don't know how to use the llamacpp server for persistent chat-session data caching. Does it simply not have this feature? Or am I missing the parameter?

[yaronsumel]

It will be cool if 'llama-server' would have auto configuration option to the machine/model like 'ollama' does it.

[exxocism]

For windows maybe choco and windows store would be a good idea? 🤔

[acbits]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - `brew install llama.cpp` works

2. For Linux (CUDA) - they need to clone and install directly from github

I created a rpm spec to manage installation though I think flatpaks might be more user friendly and distribution agnostic.

[schnow265]

For Windows - winget (?)

The released Windows builds are available via Scoop.

Updates happen automatically. Old installed versions are kept, and current one symlinked into a folder „current“ which provides the executables on the path.

[damianofalcioni]

is it feasible to have a single release for OS including all the backend?

[slaren]

It's technically possible, but the size of the package would be quite big.

[SamuelMarks]

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

[tellsiddh]

For linux I just install the vulkan binaries and run the server from there. Maybe we can have a install script like ollama that detects the system and launches the server which can be controlled from an app as well as cli? The user then gets basic command line utillities like run start stop load list etc?

[leoxzhao]

On Mac, the easiest way (also arguably the safest way) from a user's perspective is to find it in App Store, and install from there. Because of apps from App Store are in a sandbox, so from a user's point of view, installing or uninstalling is simple and clean. Creating a build and passing the App Store review might take some efforts (due to the sandbox constraint), but it should be a one-time thing.

[O-J1]

Its my understanding that none of the automated installs support GPU acceleration. I might be wrong but its definitely the case for Windows, which makes it useless to install via winget.

[slaren]

The winget package comes with the Vulkan backend.

[O-J1]

Does that work reliably with CUDA cards and does it support the same kinds of models (Qwen2.5VL for instance)

[anderspitman]

My experience with the Vulkan backend on a RTX 3060 has been excellent.

[henk717]

To me the biggest advantage ollama currently has is that the optimal settings for a model are bundled, the gguf spec would allow for this to since its versatile enough to make this a metadata field inside the model. It would allow people to load the settings from a gguf and frontends can extract them and adapt them as they see fit. I think that part is going to be more valuable than obtaining the binary since downloading the binary from github is not that hard.

[logan-markewich]

This is the killer feature right here. Id use llama.cpp directly over any wrapper if this was added

[digitalspaceport]

My personal wishlist

1. Llama-swap integration
2. Catchall compiled binaries for linux, win, osx
3. GUI improvements

[neutrinotek]

I feel like point number 1 is widely undermentioned in this thread. The ease that Llama-Swap adds to being able to bounce around models for various tasks is invaluable. If it weren't for that, I would probably still be using Ollama. It still needs a bit of upfront setup, but once you get a model in, it basically becomes "set it and forget it."

Edit I just realized who I was replying to and felt the need to come back and say I love the YT channel man. Keep up the good work!

[ericcurtin]

Docker Model Runner provides 1 and 2 FWIW... They are trying to grow the community also... It uses llama.cpp upstream, contributes any changes required back... Number 3... I recommend AnythingLLM, but there are many options

[06kellyjac]

llama.cpp on NixOS is very easy in my experience.

Simply adding the following to your config gets you llama-server etc.

environment.systemPackages = [
  pkgs.llama-cpp
];

Depending on your config it may already have CUDA enabled. If not you can do

environment.systemPackages = [
  (pkgs.llama-cpp.override { cudaSupport = true; })
];

For rocm

environment.systemPackages = [
  (pkgs.llama-cpp.override { rocmSupport = true; })
];

Vulkan

environment.systemPackages = [
  (pkgs.llama-cpp.override { vulkanSupport = true; })
];

Etc etc.

You can also try combo multiple but your mileage may vary

[SteelPh0enix]

Note that nix flake used for building llama.cpp is currently outdated and lacks some configuration options for ROCm and CUDA builds; i've made a PR to fix that: #15747
(it will have to be synced with nixos repository too)

[06kellyjac]

IDK about the in-repo flake but I'm using the nixpkgs build.

https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/by-name/ll/llama-cpp/package.nix

[jjerphan]

Note that llma.cpp is available on conda-forge, and can be installed with conda, mamba, pixi, etc.

CUDA builds exists for Windows and Linux, and macOS builds are optimized for accelerate and Metal on x86_64 and arm64 respectively.

This distribution can easily be updated by anyone to newer versions and refined for specific targets if needed.

[3Simplex]

I have a good idea for a self configuring front end, just been a little consumed with my new code companion project to start on it.

For Windows users my Llama.Cpp-Toolbox is probably a good place to get up and running fast. I'm going to be updating it soon with more functionality for power users.

The new idea basically would map out the functions after each build, generate a GUI from them and should never go out of date. I haven't thought hard about how it would look but pulling the options and the executables seems easy enough. The scripts and examples could be annoying but if I can determine how to pinpoint them and the instructions or options it could work. I'll get around to it when my new Ai-Code-Companion can scan it all and I'll see what's possible.

eye rolls
Crap that name was already taken, time to rename the project.

[3Simplex]

I solved CURL and dependency management. The script now also installs ccache since that build takes so long without it. The only manual thing you have to do is install either cuda or vulkan but that is said in the GUI...
I have not yet added personal fork source for a toolbox repo/branch but that is on the way, changing llama.cpp repo/branch was always available.

[Rotatingxenomorph]

I had to install extra stuff for compiling with curl to work when compiling CUDA version on ubuntu 25.04 today, it was pretty obscure: sudo apt-get install curl libssl-dev libcurl4-openssl-dev

[ericcurtin]

This is pretty normal and standard for a C/C++ application that uses libcurl as a dependency, you need the headers and library.

[Rotatingxenomorph]

It's a trap for young players, though. You just get told you need curl, so you get curl but that doesn't fix it. Then you google and find a random 2 year old web page to find what you need to do.

[SteelPh0enix]

the thing is that the environment setup process will depend on the specific Linux distribution, keeping a guide for major distros is technically possible, but since most people who'd want to manually compile llama.cpp already know how to set it up it seems unnecessary, considering that you can get the packaged versions in plenty of distributions already...

an alternative is to use docker/nix to build, but that's probably not something a new guy would know how to use either.

[Rotatingxenomorph]

the thing is that the environment setup process will depend on the specific Linux distribution, keeping a guide for major distros is technically possible, but since most people who'd want to manually compile llama.cpp already know how to set it up it seems unnecessary, considering that you can get the packaged versions in plenty of distributions already...

an alternative is to use docker/nix to build, but that's probably not something a new guy would know how to use either.

It is true that by the time you've got to compiling it you can find out how to fix it. I was a bit worried because it was some old web page, but I checked later to see if Claude Opus would fix it and it did. Have to get used to asking AI to solve things as the web kind of dies off.

I still don't know if I even needed to compile it with curl at all, but I just assumed it might be important for the llama-server to work.

[ericcurtin]

@slaren if you build with GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS on. And have all backends installed, is there a way at runtime to switch between say vulkan and ROCm or vulkan and CUDA?

[slaren]

You can use --device to select the devices to use.

[ericcurtin]

Thanks, interestingly my integrated graphics aren't listed as Vulkan-capable:

$ ./build/bin/llama-server --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon RX 7600, gfx1102 (0x1102), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx90c:xnack- (0x90c), VMM: no, Wave Size: 64
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Available devices:
  ROCm0: AMD Radeon RX 7600 (8176 MiB, 8142 MiB free)
  ROCm1: AMD Radeon Graphics (15726 MiB, 15669 MiB free)
  Vulkan0: AMD Radeon RX 7600 (RADV NAVI33) (7936 MiB, 7936 MiB free)

[slaren]

I believe you need to enable them explicitly by setting GGML_VK_VISIBLE_DEVICES=0,1 etc.

[ericcurtin]

Yup, that worked:

$ GGML_VK_VISIBLE_DEVICES=0,1 ./build/bin/llama-server --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon RX 7600, gfx1102 (0x1102), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx90c:xnack- (0x90c), VMM: no, Wave Size: 64
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 7600 (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Available devices:
  ROCm0: AMD Radeon RX 7600 (8176 MiB, 8142 MiB free)
  ROCm1: AMD Radeon Graphics (15726 MiB, 15669 MiB free)
  Vulkan0: AMD Radeon Graphics (RADV RENOIR) (10825 MiB, 10825 MiB free)
  Vulkan1: AMD Radeon RX 7600 (RADV NAVI33) (7936 MiB, 7936 MiB free)

[0cc4m]

This should be fixed by #15947. Can you give it a try?

[Windsage63]

I just went through this, so it's fresh in my mind. I had not used Linux since about 2012, when I changed our in-house servers to Windows server. Running local llm's I have used Text-Generation-WebUI, Ollama, and LM Studio. With the release of GPT-OSS-120b, most of the wrappers did not keep up with the new changes as quickly as I wanted. I run a 5090, and I also wanted to move to the newer torch release. I was doing some small training stuff that needed Triton anyway, so, Linux.

OP lists using winget, in Windows, but the recompiled llama.cpp is Vulcan. The binary distributions stop at 12.4, so if you need the newer copies of torch >12.4, it's source code. Having been away from Linux a long time, setting up WSL and creating everything for a build was a bit of a struggle. I actually had GPT-OSS write me a step by step walk through with copy and paste commands so I wouldn't screw it up.

Once everything was running, it's great, and a significant speedup. I moved my dev work to Linux now. Then I wrote a script (really me and GPT-OSS wrote a script) to fire up llama.cpp, so I could just pick models from an old fashioned script menu. It's crude, but it's never out of date. I posted the script creation script that GPT wrote below because I thought the idea of a self creating script was nice.

make_l-server.md

[am17an]

IMHO the UI and the packaging can be a dedicated separate repo under the ggml umbrella. Llama.cpp should be remain true to its manifesto of "inference at the edge" and continue to be a fast and feature-full backend for edge devices. The impedance mismatch of being both a developer and a consumer product is probably not easy to manage.

[l29ah]

For Linux (CUDA) - they need to clone and install directly from github

llama.cpp is packaged well in Gentoo, no need to do anything fancy.

[MineloaderGuo]

When will llama.cpp have the ability to search on the internet? Without the ability to search for information online and relying solely on existing training data, answering questions is always outdated or biased, and can only be considered a toy with no practical value

[SteelPh0enix]

I recommend setting up OpenWebUI with llama.cpp as back-end for that. You can plug the llama-server via OpenAI API.

[cheery]

I keep my llama.cpp in a podman container. I've setup scripts to build it in a container as well. And I'm using cuda.

Before I picked up llama.cpp I used ollama, found it bit too excessive and non-suited for me. Also saw that most libraries just package up llama.cpp and run my load there. I ended up just using llama.cpp directly through C API with python bindings.

I figure it's nice to build llama.cpp but a bit difficult to pick up because there are so many options and things to look at. Non technical people might have difficulty picking it up just for that. But I personally am here for it.

I find llama.cpp could use improvement on variety of places. One major place is the chat format interfaces. I think the current way chat templates work is too generic because the way this works is that you have an append-only log and you're paying high cost if you splice it. Chat templates on other hand pretend you can add stuff in the middle and they're essentially for-loops over messages list. I'd suggest you would completely change this design.

• have a toplevel prompt.
• Then have message templates for system, user, assistant prompts.
• Likewise have messages for text in middle completion.
• Finally have an assistant-starts-talking-predule...
• ...and a way to parse the assistant output back to corresponding messages.
• and assistant-stops-talking -token sequence.
• Just have a simple templating format that includes&leaves stuff from message if it is present or not.

In practice that covers all message formats because they're forced to be constrained in same way as long as they generate tokens in sequence. Even if the constraint was lifted some day, I think this makes a good format.

[bugparty]

I think we should focus on what it is now, non-tech users will use Ollama or LMStudio anyway.

[angt]

Hey everyone!

I’ve been working on some scripts and tools to make it super easy to download and run llama-server on any Linux machine (not just Ubuntu) and for both aarch64 and x86_64.

Feature detection is done before downloading to reduce download time, and CUDA is supported as well, though not fully optimized yet. This is also the only way to provide optimized builds for aarch64.

Right now, models are downloaded with hf download, to make things as easy, like Ollama.
That means the hf CLI is required for now.

On Linux, you can try it with:

curl -s https://angt.github.io/installama.sh | sh -s -- qwen3-4b:Q4_0

This should start a server running the requested model.

I’d love to hear your thoughts on this kind of packaging approach for Linux.
I’m currently exploring macOS with Metal support as the next step.

[cheery]

The scripts you've written look super clean and easy to understand. The script probes with target-features that is super simple. It's got a similar program called cuda-probe that does the same for cuda. And it's necessary that they're coming first. You've created lot of binaries for different platforms and likely have the scripts in place to update them automatically.

It looks like bare minimum to get it up and running. I personally like bare minimums. I would even throw away the 'hf' and let people decide how they download their huggingface models. You've already done the bulk difficult job when you've compiled it all for them and feature-recognize which version needs to be downloaded.

Then if the community creates frontends, maybe, for instance for linux with python rich & argparse -modules so that it's nice and colorful and running in terminal. That might make it full turn-key solution. Or if you really insist writing more complex bash scripts, then that stuff.

TL;DR I like it!

[0cc4m]

Please keep in mind that piping scripts directly from a web page into your shell is inherently unsafe and you (the user) are responsible for checking the contents of the script before running it. It's very easy to do something unwanted this way.

[rodrigovimieiro]

For Linux (CUDA) - they need to clone and install directly from github

It would be nice to have binaries for Ubuntu with CUDA, even though the one with Vulkan works pretty well

[angt]

This exactly what installama provides: https://huggingface.co/datasets/angt/installamacpp-cuda/tree/main.
They should work on all recent Linux distributions.

[pandruszkow]

It would be great if GGML_CPU_ALL_VARIANTS=ON binaries with CUDA support were included in this. I have an old Ivy Bridge machine for llama.cpp inference, and it would make my life so much easier if I could just download a ready-to-go binary that will work with Nvidia cards and not require AVX2 or FMA, but take advantage of AVX and F16C.

[angt]

Yes for CUDA i let the llama.cpp defaults to wait for comments and check if the current values were the good ones :)
I believe I can reduce to AVX without causing significant performance degradation (need to check).

[angt]

@pandruszkow I've updated the Linux CUDA binaries compiled with -march=x86-64-v2. To test them, please delete your ~/.installama directory and rerun the command. It would be awesome if you could check that they work on your setup :)

[angt]

Hey!!

Apple M1 through M4 are now supported with Metal. It turned out to be much easier than expected.
As before, just run:

curl -s https://angt.github.io/installama.sh | sh -s -- qwen3-4b:Q4_0

and you’re good to go 🚀

Again hf is still required to download the models, i'm going to work on that.

[ggerganov]

Do you mean use libhttp for the file downloads?

Generally, dropping libcurl dependency would be great. But last time I looked into this, we still need libssl dependency which ironically seems even more difficult to provide compared to libcurl. But if you have ideas how to improve this, feel free to give it a try. Less dependencies is better.

[angt]

Yes, this is exactly what I had in mind. I've opened a PR to explore this: #16185

[angt]

All builds have been updated with the changes from PR #16185.
Now you can run:

curl -s https://angt.github.io/installama.sh | sh -s -- -hf unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_0

[ggerganov]

Interesting approach! I'll take a look at the PR soon.

Btw, having an installer that does not require model to be specified would be also useful. I suppose this is currently like this for ease of use, but I think eventually the installer should run without setting a model.

[angt]

True! I've updated the script, so now you can run it without any arguments:

$ curl -s https://angt.github.io/installama.sh | sh
Run ~/.installama/llama-server to launch the llama.cpp server

It's still an early PoC, so everything can be changed or improved :)