Skip to content

hosseinal/llama.cpp

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5,418 Commits
.devops
.devops
 
 
.github
.github
 
 
ci
ci
 
 
cmake
cmake
 
 
common
common
 
 
docs
docs
 
 
examples
examples
 
 
ggml
ggml
 
 
gguf-py
gguf-py
 
 
grammars
grammars
 
 
include
include
 
 
licenses
licenses
 
 
media
media
 
 
models
models
 
 
pocs
pocs
 
 
prompts
prompts
 
 
requirements
requirements
 
 
scripts
scripts
 
 
src
src
 
 
tests
tests
 
 
tools
tools
 
 
.clang-format
.clang-format
 
 
.clang-tidy
.clang-tidy
 
 
.dockerignore
.dockerignore
 
 
.ecrc
.ecrc
 
 
.editorconfig
.editorconfig
 
 
.flake8
.flake8
 
 
.gitignore
.gitignore
 
 
.gitmodules
.gitmodules
 
 
.pre-commit-config.yaml
.pre-commit-config.yaml
 
 
AUTHORS
AUTHORS
 
 
CMakeLists.txt
CMakeLists.txt
 
 
CMakePresets.json
CMakePresets.json
 
 
CODEOWNERS
CODEOWNERS
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
SECURITY.md
SECURITY.md
 
 
build-xcframework.sh
build-xcframework.sh
 
 
convert_hf_to_gguf.py
convert_hf_to_gguf.py
 
 
convert_hf_to_gguf_update.py
convert_hf_to_gguf_update.py
 
 
convert_llama_ggml_to_gguf.py
convert_llama_ggml_to_gguf.py
 
 
convert_lora_to_gguf.py
convert_lora_to_gguf.py
 
 
flake.lock
flake.lock
 
 
flake.nix
flake.nix
 
 
mypy.ini
mypy.ini
 
 
poetry.lock
poetry.lock
 
 
pyproject.toml
pyproject.toml
 
 
pyrightconfig.json
pyrightconfig.json
 
 
requirements.txt
requirements.txt
 
 

Repository files navigation

llama.cpp

llama

License: MIT Server

Roadmap / Project status / Manifesto / ggml

Inference of Meta's LLaMA model (and others) in pure C/C++

Recent API changes

Hot topics

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads MTT GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

Multimodal

Bindings
UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

Tools
  • akx/ggify – download PyTorch models from HuggingFace Hub and convert them to GGML
  • akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp
  • crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
  • gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage
  • Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)
Infrastructure
  • Paddler - Stateful load balancer custom-tailored for llama.cpp
  • GPUStack - Manage GPU clusters for running LLMs
  • llama_cpp_canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
  • llama-swap - transparent proxy that adds automatic model switching with llama-server
  • Kalavai - Crowdsource end to end LLM deployment at any scale
  • llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes.
Games
  • Lucy's Labyrinth - A simple maze game where agents controlled by an AI model will try to trick you.

Supported backends

Backend Target devices
Metal Apple Silicon
BLAS All
BLIS All
SYCL Intel and Nvidia GPU
MUSA Moore Threads MTT GPU
CUDA Nvidia GPU
HIP AMD GPU
Vulkan GPU
CANN Ascend NPU
OpenCL Adreno GPU
RPC All

Building the project

The main product of this project is the llama library. Its C-style interface can be found in include/llama.h. The project also includes many example programs and tools using the llama library. The examples range from simple, minimal code snippets to sophisticated sub-projects such as an OpenAI-compatible HTTP server. Possible methods for obtaining the binaries:

Obtaining and quantizing models

The Hugging Face platform hosts a number of LLMs compatible with llama.cpp:

You can either manually download the GGUF file or directly use any llama.cpp-compatible models from Hugging Face or other model hosting sites, such as ModelScope, by using this CLI argument: -hf <user>/<model>[:quant].

By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable MODEL_ENDPOINT. For example, you may opt to downloading model checkpoints from ModelScope or other model sharing communities by setting the environment variable, e.g. MODEL_ENDPOINT=https://www.modelscope.cn/.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

To learn more about model quantization, read this documentation

Running a Small Model with llama.cpp

This guide will walk you through the steps to download a small model from Hugging Face, convert it to the GGUF format, and run it using llama-cli. We'll use microsoft/phi-2 as an example, which is a good small model to start with.

1. Choosing a Model

You can find many models on Hugging Face. For this example, we'll use microsoft/phi-2. It's a 2.7B parameter model that offers a good balance between performance and resource requirements. You can browse other models on the Hugging Face models page.

2. Downloading and Converting the Model

llama.cpp uses the GGUF (GPT-Generated Unified Format) for its models. This format is optimized for rapid loading and saving of models, and for ease of reading. It's a single-file format that includes all the necessary information to run the model.

To use a Hugging Face model like phi-2 with llama.cpp, you first need to download its weights and then convert them to the GGUF format.

a. Download the model from Hugging Face:

It's recommended to clone the model repository using git-lfs. First, ensure you have git-lfs installed. You can usually install it with:

git lfs install

Then, clone the model repository. For microsoft/phi-2:

git clone https://huggingface.co/microsoft/phi-2

This will download the model files into a directory named phi-2.

b. Convert the model to GGUF:

llama.cpp provides a Python script convert_hf_to_gguf.py (often found in the examples or tools directory, or you might need to ensure it's in your Python path if you installed llama.cpp differently) to convert Hugging Face models to GGUF.

Navigate to your llama.cpp directory and run the conversion script. The exact path to the script might vary based on your llama.cpp setup.

# Assuming you are in the root of the llama.cpp repository
# and the phi-2 model was cloned into a directory named 'phi-2' sibling to the llama.cpp directory.
# Adjust paths as necessary.

# Example for older script name 'convert-hf-to-gguf.py'
python3 ./examples/convert-hf-to-gguf.py ../phi-2 --outfile phi-2.gguf

# Or, if the script is named 'convert_hf_to_gguf.py' and located in 'tools'
python3 ./tools/convert_hf_to_gguf.py ../phi-2 --outfile phi-2.gguf

# Some models might require specifying the output type, e.g., f16 or q8_0
python3 ./tools/convert_hf_to_gguf.py ../phi-2 --outfile phi-2-f16.gguf --outtype f16

This command will create a phi-2.gguf (or phi-2-f16.gguf) file in your current directory. This is the file you'll use with llama-cli.

Note on convert_hf_to_gguf.py: This script has evolved. Older versions were named convert-hf-to-gguf.py and might have been located in examples. Newer versions are typically convert_hf_to_gguf.py and located in the tools directory. Always check your local llama.cpp repository for the correct script name and location. The script itself often has options to control the quantization of the output GGUF file (e.g., f16 for 16-bit floating point, q8_0 for 8-bit quantization, etc.). Smaller, quantized models run faster and use less memory but might have slightly reduced accuracy.

3. Understanding GGUF

GGUF (GPT-Generated Unified Format) is a binary format designed for llama.cpp. Key features include:

  • Single File: All model data (weights, metadata, vocabulary) is contained in one file.
  • Extensibility: Allows adding new information to models without breaking compatibility with older versions.
  • Quantization Support: Natively supports various quantization types, allowing for smaller model sizes and faster inference.
  • Memory Mapping (mmap): Designed to be efficiently loaded using memory mapping, which speeds up model loading times significantly.

4. Running the Model with llama-cli

Once you have your phi-2.gguf file, you can run it using the llama-cli tool. llama-cli is the primary command-line interface for interacting with llama.cpp models.

# Assuming 'llama-cli' is built and in your PATH, or you are in the directory where it's located.
# And 'phi-2.gguf' is in the current directory.

./llama-cli -m phi-2.gguf -p "Once upon a time" -n 50

Explanation of the command:

  • ./llama-cli: Executes the llama-cli program.
  • -m phi-2.gguf: Specifies the path to your GGUF model file.
  • -p "Once upon a time": Provides a prompt to the model.
  • -n 50: Specifies the number of tokens (roughly, parts of words) the model should generate.

You should see the model start generating text based on your prompt.

This covers the basic workflow of getting a small model like phi-2 up and running with llama.cpp. You can explore further options for llama-cli to control aspects like temperature, top-k sampling, context size, and more for different generation behaviors.

5. Running on GPU (NVIDIA CUDA with GTX 3060 Example)

If you have an NVIDIA GPU like the GTX 3060, you can offload some computation layers to it for significantly faster inference.

Prerequisites:

  • NVIDIA CUDA Toolkit: You'll need the NVIDIA CUDA Toolkit installed.
  • Compatible Drivers: Ensure you have NVIDIA drivers that are compatible with the CUDA Toolkit version you install.
  • Build with CUDA Support: llama.cpp must be compiled with CUDA support enabled. You can do this using CMake: 
    mkdir build
cd build
cmake .. -DLLAMA_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="<arch>"
cmake --build . --config Release
    Replace <arch> with the appropriate compute capability for your GPU (e.g., "86" for a GTX 3060 which is Ampere generation). You can find the correct compute capability for your GPU on NVIDIA's CUDA GPUs website: https://developer.nvidia.com/cuda-gpus. For more detailed instructions, refer to the CUDA build documentation.

Troubleshooting CUDA Build Issues

If you encounter errors during the build process with CUDA enabled, here are some common issues and recommendations:

  • std_function.h Parameter Pack Error: Errors like "std::function does not support variadic template arguments" or similar messages related to parameter packs in <functional> (often seen as std_function.h in error paths) can indicate an incompatibility between your C++ compiler (e.g., GCC version) and the installed NVIDIA CUDA Toolkit version (nvcc).
    • Verify Compatibility: Check NVIDIA's official CUDA Toolkit documentation for a list of supported host C++ compilers for your specific CUDA Toolkit version. Using a newer GCC with an older CUDA Toolkit (or vice-versa) is a common source of such problems.
    • Community Solutions: Search the llama.cpp GitHub issues and discussions. Other users might have faced and solved similar issues with specific compiler/CUDA version combinations on similar operating systems.
    • C++ Standard (Advanced): Sometimes, explicitly setting a compatible C++ standard for CUDA compilation can help. You might try adding -DCMAKE_CUDA_STANDARD=17 (or another version like 14) to your CMake command. However, do this cautiously as it can affect other parts of the build.
    • Keep Software Updated: Generally, ensure your NVIDIA drivers, CUDA Toolkit, and C++ compiler are up-to-date, but always prioritize official compatibility matrices provided by NVIDIA.

Running with GPU Offload:

Once llama.cpp is built with CUDA support, you can use the -ngl (number of GPU layers) flag with llama-cli to specify how many layers of the model you want to offload to your GPU.

Here's an example using a hypothetical model.gguf and offloading 35 layers to a GTX 3060:

./llama-cli -m model.gguf -ngl 35 -p "Running this model on my GTX 3060!" -n 50

Important Notes for GTX 3060 (and other NVIDIA GPUs):

  • -ngl Value: The optimal number for -ngl depends on the specific model you are using and the amount of VRAM on your GPU (the GTX 3060 typically comes with 6GB or 12GB VRAM). If you set -ngl too high for your VRAM, you might encounter out-of-memory errors. If it's too low, you might not get the best possible performance. You may need to experiment with this value to find the sweet spot for your setup. Start with a moderate number and increase it gradually.
  • Driver and Toolkit Versions: It's crucial to have a matching set of NVIDIA drivers and CUDA Toolkit. For a GTX 3060, always check the official NVIDIA website for the latest recommended drivers for your operating system and the compatible CUDA Toolkit versions.

By offloading layers to your GPU, you can significantly speed up prompt processing and token generation.

llama-cli

A CLI tool for accessing and experimenting with most of llama.cpp's functionality.

  • Run in conversation mode

    Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding -cnv and specifying a suitable chat template with --chat-template NAME

    llama-cli -m model.gguf

# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!
  • Run in conversation mode with custom chat template 
    # use the "chatml" template (use -h to see the list of supported templates)
llama-cli -m model.gguf -cnv --chat-template chatml

# use a custom template
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'
  • Run simple text completion

    To disable conversation mode explicitly, use -no-cnv

    llama-cli -m model.gguf -p "I believe the meaning of life is" -n 128 -no-cnv

# I believe the meaning of life is to find your own truth and to live in accordance with it. For me, this means being true to myself and following my passions, even if they don't align with societal expectations. I think that's what I love about yoga – it's not just a physical practice, but a spiritual one too. It's about connecting with yourself, listening to your inner voice, and honoring your own unique journey.
  • Constrain the output with a custom grammar 
    llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'

# {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}

    The grammars/ folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.

    For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/

llama-server

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

  • Start a local HTTP server with default configuration on port 8080 
    llama-server -m model.gguf --port 8080

# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions
  • Support multiple-users and parallel decoding 
    # up to 4 concurrent requests, each with 4096 max context
llama-server -m model.gguf -c 16384 -np 4
  • Enable speculative decoding 
    # the draft.gguf model should be a small variant of the target model.gguf
llama-server -m model.gguf -md draft.gguf
  • Serve an embedding model 
    # use the /embedding endpoint
llama-server -m model.gguf --embedding --pooling cls -ub 8192
  • Serve a reranking model 
    # use the /reranking endpoint
llama-server -m model.gguf --reranking
  • Constrain all outputs with a grammar 
    # custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf

llama-perplexity

A tool for measuring the perplexity 12 (and other quality metrics) of a model over a given text.

  • Measure the perplexity over a text file 
    llama-perplexity -m model.gguf -f file.txt

# [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
# Final estimate: PPL = 5.4007 +/- 0.67339
  • Measure KL divergence 
    # TODO

llama-bench

Benchmark the performance of the inference for various parameters.

  • Run default benchmark 
    llama-bench -m model.gguf

# Output:
# | model               |       size |     params | backend    | threads |          test |                  t/s |
# | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |
#
# build: 3e0ba0e60 (4229)

llama-run

A comprehensive example for running llama.cpp models. Useful for inferencing. Used with RamaLama 3.

  • Run a model with a specific prompt (by default it's pulled from Ollama registry) 
    llama-run granite-code

llama-simple

A minimal example for implementing apps with llama.cpp. Useful for developers.

  • Basic text completion 
    llama-simple -m model.gguf

# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of

Contributing

  • Contributors can open PRs
  • Collaborators can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Collaborators will be invited based on contributions
  • Any help with managing issues, PRs and projects is very appreciated!
  • See good first issues for tasks suitable for first contributions
  • Read the CONTRIBUTING.md for more information
  • Make sure to read this: Inference at the edge
  • A bit of backstory for those who are interested: Changelog podcast

Other documentation

Development documentation

Seminal papers and background on the models

If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:

XCFramework

The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS, and macOS. It can be used in Swift projects without the need to compile the library from source. For example:

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(
            name: "MyLlamaPackage",
            dependencies: [
                "LlamaFramework"
            ]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
            checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
        )
    ]
)

The above example is using an intermediate build b5046 of the library. This can be modified to use a different version by changing the URL and checksum.

Completions

Command-line completion is available for some environments.

Bash Completion

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

Dependencies

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • minja - Minimal Jinja parser in C++, used by various tools/examples - MIT License
  • linenoise.cpp - C++ library that provides readline-like line editing capabilities, used by llama-run - BSD 2-Clause License
  • curl - Client-side URL transfer library, used by various tools/examples - CURL License

Footnotes

  1. tools/perplexity/README.md ↩

  2. https://huggingface.co/docs/transformers/perplexity ↩

  3. RamaLama ↩

About

LLM inference in C/C++

Resources

Readme

License

MIT license

Contributing

Contributing

Security policy

Security policy
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • C++ 63.3%
  • C 11.5%
  • Python 7.8%
  • Cuda 6.0%
  • Objective-C 2.2%
  • Metal 2.0%
  • Other 7.2%