llama.cpp

Use llama.cpp when you want to choose an exact model file, control how it uses your computer, or provide a local connection for another application. It provides the inference engine that runs the model's calculations, along with terminal and server tools. Its server also includes a browser chat interface.

This guide takes you from installation to one short response and a local API request. If you mainly want to browse a model library and chat without maintaining launch commands, Ollama or LM Studio may be a simpler starting point. Direct llama.cpp use becomes worthwhile when its controls solve a specific problem, such as comparing quantizations or reproducing a configuration.

Choose one installation route

The Llama installation page offers a companion desktop app for Mac and Windows, command-line installation, and links to packages and prebuilt archives. The managed app and a server you launch in a terminal are separate processes; closing a terminal does not necessarily stop the app.

Use one installation channel first. The project's package instructions include:

winget install llama.cpp

That command is for Windows with Winget available. On a Mac with Homebrew, use:

brew install llama.cpp

Open a fresh terminal after installation. The unified command is llama cli; standalone distributions may instead expose llama-cli and llama-server. Check what your installation supplies rather than mixing names from different guides. For the unified tool, run:

llama cli --version
llama cli --help

If using standalone binaries, run llama-cli --version and llama-cli --help. In Windows PowerShell, an executable in the current folder normally needs a path such as .\llama-cli.exe. In the examples below, replace llama cli with your standalone CLI and llama serve with your standalone server when applicable. If an option is absent from that binary's help, use a release supporting it rather than assuming rolling documentation describes an older package.

For a manually downloaded archive, match the operating system, processor architecture, and graphics backend. Extract the entire archive and retain its companion libraries. An x64 archive is not an ARM64 build; a Linux archive built for one distribution is not a universal Linux package.

These are choices to verify, not promises for every card from a manufacturer. The build guide gives dependencies for each path. After installation, run llama cli --list-devices to list the devices llama.cpp can use. If your GPU does not appear in the output or you encounter errors, it may not be supported or may require updated drivers. A CPU-only executable does not gain GPU support merely because you change a model setting.

If you need a source build, install the documented compiler, CMake, and backend dependencies first. For a CUDA build on an appropriate NVIDIA system:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

On Windows, use the developer shell specified in the build instructions. These commands do not install the prerequisites and are not a CUDA setup for a Mac. Other backends use different configuration options.

Download an exact model file

GGUF is the model-file format used here. A family name alone does not identify the download: check the publisher, variant, numerical compression, and required companion files. Quantization stores weights at reduced precision, which changes storage and memory use and may change answers.

For a modest loading exercise, use Qwen3.5-0.8B-Q4_0.gguf from ggml-org/Qwen3.5-0.8B-GGUF. The exact file page lists a 563 MB download. That is the stored file size, not the total running memory. The conversion identifies Qwen's original Qwen3.5-0.8B checkpoint as its source. This small model is a setup exercise, not a recommendation for important document analysis.

Create a working folder and a models subfolder. Download that exact file into models; do not download the whole repository unless you intend to obtain its other variants. Run the following from the working folder:

llama cli -m "./models/Qwen3.5-0.8B-Q4_0.gguf" -c 4096

The -m option selects the local file. The -c 4096 option selects a modest context allowance. Context includes instructions, conversation, supplied text, and room for the answer. If you saved the file elsewhere, replace the path with its actual location.

Alternatively, the tool can download the exact file from Hugging Face. This is one command, which may wrap on the page:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF -hff Qwen3.5-0.8B-Q4_0.gguf -c 4096

Here, -hff identifies the filename. Do not assume an unsuffixed repository request selects Q4_K_M: this repository has other quantizations, and the resolver can choose another file when its usual preference is unavailable. Downloads require internet access; a local -m path makes the file used for later offline work explicit.

Get one short response working

At the chat prompt, enter:

Rewrite "The meeting will commence at 10:30" in plain English. Keep the time unchanged. Return one sentence.

Check for an intelligible sentence preserving 10:30 without adding a date or location. The wording can vary. This is an expected check on an invented input, not a report that the model produced a particular answer. Press Ctrl+C to leave the process.

To send one prompt from the command line, current unified CLI instructions support -st for a single turn and -n for the output-token limit. For example:

llama cli -m "./models/Qwen3.5-0.8B-Q4_0.gguf" -c 4096 -n 128 -st -p "Rewrite: The meeting will commence at 10:30. Keep the time. Return one sentence."

A short output limit is useful for setup, but can cut off an answer or be consumed by a reasoning model's intermediate output. Inspect the result; do not treat an empty or truncated answer as a successful task. Increase the allowance deliberately if necessary, after confirming loading works.

Start a server for another application

Close the terminal chat before loading another copy. From the same working folder, start:

llama serve -m "./models/Qwen3.5-0.8B-Q4_0.gguf" -c 4096 --host 127.0.0.1 --port 9931 --alias local-test

Leave the terminal open and visit http://127.0.0.1:9931 in your browser. The explicit loopback address limits the listener to this computer. The explicit port avoids dependence on a version's default: v0.7.0 changed the native server default from 8080 to 9931. local-test is the alias by which a client can request the loaded model.

Save this example as request.json in the working folder:

{
  "model": "local-test",
  "messages": [
    {
      "role": "user",
      "content": "Rewrite: The meeting will commence at 10:30. Keep the time."
    }
  ],
  "max_tokens": 128,
  "stream": false
}

On macOS or Linux, open a second terminal in that folder and send:

curl --fail-with-body http://127.0.0.1:9931/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @request.json

On Windows PowerShell, use:

$reply = Invoke-RestMethod `
  -Uri 'http://127.0.0.1:9931/v1/chat/completions' `
  -Method Post -ContentType 'application/json' `
  -InFile '.\request.json'
$reply | ConvertTo-Json -Depth 10

The PowerShell backticks continue the command onto the next line; keep them at the line ends without trailing spaces. Look at the returned JSON and inspect the generated assistant message. An HTTP error, empty answer, or connection failure needs investigation. No exact response wording is promised.

A client asking for an OpenAI-compatible base URL normally needs http://127.0.0.1:9931/v1. Compatibility covers particular endpoints and features, not every capability of the OpenAI service. A client inside Docker or on another computer interprets localhost differently; changing the server to listen everywhere isn't the first fix for an address error. For remote access, choose the listening address deliberately. The --host option chooses which network interfaces accept connections; 0.0.0.0 means all IPv4 interfaces, not a particular public address. A specific LAN address can narrow that choice. The --api-key option adds authentication for protected API requests; neither option encrypts traffic. For example: llama serve -m "./models/Qwen3.5-0.8B-Q4_0.gguf" -c 4096 --host 0.0.0.0 --port 9931 --api-key YOUR_SECRET_KEY

Replace YOUR_SECRET_KEY with a strong secret before use. This example alone is not a secure internet deployment. Limit access with a firewall, and use encrypted transport before sending requests over an untrusted network. With an HTTPS reverse proxy, prevent clients from bypassing it to reach the plain HTTP server directly. Protect API keys and review access logs for unauthorized attempts.

Plan memory and check acceleration separately

Keep disk storage, system RAM and dedicated GPU memory distinct. On Apple Silicon, the operating system and GPU share unified memory. Weights are only part of the running workload: conversation memory, temporary buffers and other programs also need room.

Increase context gradually while watching memory and responsiveness. Current llama.cpp can adjust unset options with its --fit on behavior; inspect the settings selected in the loading output. Automatic fitting is not proof that the chosen workload will remain responsive.

Run llama cli --list-devices, then inspect the loading log for the selected backend and the layers or buffers assigned to it. Detecting a GPU is different from using it for the model. If allocation fails, reduce context, close another loaded model, or choose a smaller file before forcing more work onto the GPU.

For an advanced comparison, the standalone benchmark tool separates prompt processing from text generation:

llama-bench -m "./models/Qwen3.5-0.8B-Q4_0.gguf" -p 512 -n 128 -r 3

Those are example test settings, not measured results. Record the exact file, build and acceleration settings. The benchmark excludes tokenization and sampling, so also time a complete representative task in the client you intend to use. A faster benchmark does not establish that the answer preserves the source's meaning.

Troubleshoot from the actual error

The server's /health endpoint reports HTTP 503 while loading and HTTP 200 when ready under the documented default behavior. Readiness is a service check; the short prompt still needs to produce a usable response.

Keep the operating boundary clear

For an offline check, obtain the full model file first, disconnect networking and repeat the local -m example. Current CLI help also provides --offline for cached operation. Image or audio support requires a suitable model and, where specified, a companion projector supplied through --mmproj. A text response does not establish multimodal support.

Keep tools disabled while establishing the basic setup. A command-execution tool can read or change files, and an external tool can send information elsewhere. A locally running model does not limit those additional components.

Stop a foreground server with Ctrl+C. Quit or unload a separately installed app through its own controls.

Before updating, note your launch command and model paths. Update through the same channel you installed from (winget, Homebrew or the same archive source), then rerun the short chat and the API request. Not every channel offers a rollback, so keep the previous archive if you might need it.

Deleting a model file removes that copy, not client conversations, request files, logs, or backups. Check whether another app shares the same cache before removing it.

If you later share the server, choose the listening address, authentication and transport protection deliberately. An API key does not encrypt HTTP traffic, so use an HTTPS reverse proxy and a firewall rule that lists the computers allowed to connect. The application's MIT License and the example model's Apache 2.0 license govern different components; neither grants permissions for an unrelated model or service.

Related pages

• What is local AI?
• RAM and VRAM calculator
• Browse models

Sources

• llama.cpp project, package installation and v0.7.0 release.

• Llama CLI, server setup and API guide; full CLI reference and server reference.

• Benchmark method, performance troubleshooting and multimodal requirements.

• Original Qwen3.5-0.8B model card.

• Speculative decoding and MTP