llama.cpp — Compile-Once CPU/GPU Inference

llama.cpp is a high-performance LLM inference engine written in pure C/C++. Originally created by Georgi Gerganov as a port of Meta's LLaMA inference code, it has grown into a full inference ecosystem supporting 50+ model architectures, multiple hardware backends, and the GGUF quantization format. Its defining advantage: a single compiled binary with zero runtime dependencies can load and run billion-parameter models on commodity hardware.

llama.cpp can be installed via package managers or compiled from source. Compiling from source is required for cutting-edge model support and GPU backends on Linux. On macOS, Homebrew provides a convenient pre-built option with Metal acceleration.

brew install llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)

llama-server --version

sudo apt update && sudo apt install -y \
  build-essential cmake git curl \
  libcurl4-openssl-dev

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

cmake -B build
cmake --build build --config Release -j$(nproc)

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

cmake -B build -DGGML_HIPBLAS=ON -DAMDGPU_TARGETS="gfx1100"
cmake --build build --config Release -j$(nproc)

echo 'export PATH="$HOME/llama.cpp/build/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

docker pull ghcr.io/ggerganov/llama.cpp:server

mkdir -p ~/llama-models

docker run -p 8080:8080 \
  -v ~/llama-models:/models \
  ghcr.io/ggerganov/llama.cpp:server \
  -m /models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 4096 -np 4

docker run --gpus all \
  -p 8080:8080 \
  -v ~/llama-models:/models \
  ghcr.io/ggerganov/llama.cpp:server-cuda \
  -m /models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 4096 -ngl 99 -np 4

GGUF models are hosted on HuggingFace. The bartowski user maintains high-quality quantized builds of most popular open-weight models. Install the HuggingFace CLI to download specific quantization files without pulling the entire repository:

# Install HuggingFace CLI
pip install huggingface_hub

# Download a specific quantization (Q4_K_M recommended for balance)
huggingface-cli download \
  bartowski/Llama-3.2-3B-Instruct-GGUF \
  --include "Llama-3.2-3B-Instruct-Q4_K_M.gguf" \
  --local-dir ~/llama-models

# Download Mistral 7B Q5_K_M
huggingface-cli download \
  bartowski/Mistral-7B-Instruct-v0.3-GGUF \
  --include "Mistral-7B-Instruct-v0.3-Q5_K_M.gguf" \
  --local-dir ~/llama-models

Or download directly via wget using the URL from the HuggingFace file page:

# Download directly via wget (get the URL from the HuggingFace file page)
wget -P ~/llama-models \
  "https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf"

The suffix in a GGUF filename indicates quantization level. Lower quantization = smaller file + faster inference + lower quality. Q4_K_M is the recommended default for most use cases, offering a practical balance of size, speed, and output quality.

llama-server exposes an OpenAI-compatible HTTP API. It handles concurrent requests, streaming, and chat completions — compatible with any client that supports the OpenAI API format, including the openai Python SDK.

Basic launch:

# Basic server on port 8080
llama-server \
  -m ~/llama-models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 4096        # context size in tokens \
  -np 4          # parallel request slots

With GPU offloading:

# NVIDIA: offload all layers to GPU
llama-server \
  -m ~/llama-models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 8192 -np 4 \
  -ngl 99        # number of GPU layers (99 = all)

# Partial offload (model too big for full VRAM)
llama-server -m model.gguf -ngl 24 -c 4096

Test with curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-cpp",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

llama-cpp-python wraps llama.cpp in Python, supporting the OpenAI SDK interface and direct integration with llama_index and LangChain. It compiles the C++ library at install time, so GPU flags must be passed via CMAKE_ARGS.

# Install (CPU)
pip install llama-cpp-python

# Install with CUDA support
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

# Install with Metal (macOS)
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="~/llama-models/Llama-3.2-3B-Instruct-Q4_K_M.gguf",
    n_gpu_layers=-1,   # -1 = all layers on GPU
    n_ctx=4096,        # context window
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain attention in transformers."},
    ]
)
print(response["choices"][0]["message"]["content"])