llama.cpp
A portable C/C++ inference engine that underpins much of the GGUF local-model ecosystem.
By ggml-org · reviewed 2026-08-07
What llama.cpp does
llama.cpp provides efficient inference across CPUs and many GPU backends, with quantization tooling and an OpenAI-compatible server. It offers fine-grained runtime control and broad portability, but expects more technical setup than desktop-first applications.
Pricing model
Free and open source
Position in the stack
Where llama.cpp fits
llama.cpp is a low-level inference engine and toolchain. It loads GGUF models, runs quantized inference across many CPU and GPU backends, provides conversion and quantization utilities, and can expose an API server. Many higher-level local AI products build on it.
Typical deployment flow
A practical sequence for evaluating llama.cpp before making it part of a permanent stack.
- STEP 1
Select a build
Use a prebuilt binary or compile with the hardware backend required by the target CPU or GPU.
- STEP 2
Prepare a GGUF model
Choose or create a quantization that fits memory while preserving acceptable quality for the intended task.
- STEP 3
Tune and expose
Set context, GPU offload, threads, and batching, then run the CLI or the included API server.
Best for
- Maximum portability and runtime control
- CPU and mixed CPU/GPU inference
- Embedding a local inference engine
Not the right layer for
- Users who want a complete model catalog and polished desktop UI
- Native cluster scheduling
Capabilities
Capabilities refer to the tool's application layer. Hardware and model support can still depend on a connected inference engine.
Deployment and compatibility
- Deployment
- Desktop / localSelf-hosted
- Platforms
- macOS · Windows · Linux
- Hardware backends
- CPU · Apple Silicon · NVIDIA GPU · AMD GPU · Intel GPU
- Install methods
- Build from source · Prebuilt binaries · Docker
Models and integrations
- Model formats
- GGUF
- Common integrations
- OpenAI-compatible clientsKoboldCppLocalAI
Strengths
- Wide hardware backend support
- Mature GGUF quantization ecosystem
- Lightweight server and command-line tools
Limitations
- Configuration choices can be complex
- Model discovery and multi-user management are external concerns
Planning checklist
Before you choose llama.cpp
Answer these questions with the exact models, hardware, users, and data you expect to operate.
- Which compute backend is stable on the target operating system?
- How many model layers can be offloaded without exhausting VRAM?
- What quality loss is acceptable from quantization?
- Do you need a separate UI, model catalog, or authentication proxy?
llama.cpp FAQ
Why is llama.cpp important for local AI?
It provides the portable GGUF inference foundation used directly or indirectly by many desktop and server tools, especially on consumer hardware.
Does llama.cpp include an API server?
Yes. The project includes a lightweight server with OpenAI-compatible endpoints, although user management and application workflows remain separate concerns.
Is llama.cpp only for CPUs?
No. It supports CPU inference and multiple GPU backends, including Apple Silicon, NVIDIA, AMD, and Intel paths, with support varying by platform and build.
Official sources
Use these links to confirm current compatibility and installation requirements.
Related tools
A self-hosted OpenAI-compatible API that runs language, image, audio, and multimodal models through modular local backends.
Apple Silicon-native tooling for generating, quantizing, fine-tuning, and serving language models with MLX.
A straightforward local model runtime with a CLI, model library, and local API.
Comparisons featuring llama.cpp
Use a reviewed comparison when the choice is between two adjacent tools.
Execution evidence
Known working recipes using llama.cpp
Recipes connect hardware, a model artifact, tools, settings, verification, and a reportable result.
No verified recipe is linked to this record yet.
Compatibility estimates remain available in the planner. A recipe appears here only after its exact stack and verification protocol are documented.