The lightweight C/C++ inference engine that powers Ollama, LM Studio, and most local-LLM tools run it directly if you want raw speed and full control.
Get prebuilt binariesOpens the official ggml-org releases page · CPU, CUDA, Vulkan & SYCL builds available
Maintainer
ggml-org
Cost
Free
Interface
CLI + local server
Format
GGUF
| NVIDIA GPU | Grab the cuda build matching your driver's CUDA version for GPU acceleration |
| AMD GPU | Vulkan builds work broadly; ROCm/HIP builds give better performance on supported cards |
| Intel Arc | SYCL build targets Intel GPUs specifically |
| No GPU / unsure | The plain CPU build runs anywhere, just slower on larger models |
Open the Releases page and find the newest build
Click the button above, open the top release, and expand Assets to see the Windows zip files by backend (cpu, cuda, vulkan, etc).
Download and extract the matching zip
No installer — just unzip it anywhere. You'll see llama-cli.exe and llama-server.exe inside.
Download a GGUF model
Grab a quantized model file (Q4_K_M is a common balance of size/quality) from Hugging Face and place it in the same folder.
Run it
From Command Prompt in that folder: llama-server -m model.gguf starts a local OpenAI-compatible server at localhost:8080.
llama.cpp itself is command-line and server-only. If you'd rather have a point-and-click chat window, tools like LM Studio and Jan use llama.cpp underneath but wrap it in a full desktop app.