Quick diagnosis
“Slow” can mean at least five different failures: the model takes too long to load, the first token is delayed, generation speed is low, performance collapses during a long chat, or the machine slows after sustained load. Diagnose which one you have before changing random settings.
The most common cause is simple: the model or context is too large for the fastest available memory pool. A smaller model that stays on the GPU or within comfortable unified memory can feel dramatically better than a larger model that spills into system RAM.
Name the slow part
| What feels slow | What it usually points to | First check |
|---|---|---|
| Model startup | Storage speed, loading, conversion, or memory pressure | Time a second load after a clean restart |
| Delay before the first token | Prompt processing, long context, model size, or cold start | Test a new short chat |
| Low tokens per second | CPU-only execution, poor GPU offload, oversized model, or backend limits | Confirm the active backend and GPU use |
| Speed degrades during a long chat | Context growth, swapping, or thermal pressure | Start a fresh chat and compare |
| The whole machine becomes slow | Memory exhaustion and operating-system swapping | Watch memory pressure while generating |
| Speed drops after several minutes | Thermal throttling or sustained power limits | Compare a cold run with a later run |
First six checks
- Run one short prompt in a new chat. This separates context growth from base-model speed.
- Confirm the runtime is using the intended accelerator. Check actual runtime diagnostics, not the presence of a GPU alone.
- Try a smaller model. Move down a model class and compare before adjusting sampling controls.
- Reduce context length. Long system prompts, chat history, and retrieved passages increase prompt-processing work and memory demand.
- Close competing applications. Games, browsers, virtual machines, video tools, and other AI processes can consume memory and compute.
- Restart and compare a cold run with a sustained run. A large difference can point to thermal or retained-process problems.
Check whether the GPU is actually active
Owning a supported GPU does not guarantee that the runtime is using it. The installed backend, driver, runtime build, model format, and offload settings all matter.
Look for the runtime’s own device or backend output. On a discrete-GPU system, also watch dedicated GPU-memory use while the model runs. If the GPU remains idle and the CPU is saturated, diagnose the backend before blaming the model family.
On Apple Silicon, confirm that the native Apple acceleration path is active and that the process is not under sustained memory pressure. Unified memory still has a bandwidth and capacity limit.
Model size and active parameters
Larger dense models generally require more work per generated token. Mixture-of-experts models can have a large total parameter count with a smaller active set, but memory and runtime behavior still depend on the exact architecture and implementation.
Compare exact artifacts, not family names. Quantization, format, active parameters, and runtime support can change both speed and memory use.
Context can dominate the experience
A long conversation requires the model to process more tokens before producing the next response. Document retrieval can insert large passages into the prompt. Agent tools can add schemas, logs, and repeated history.
Test a blank chat with a short prompt. If it is fast, then gradually add the system prompt, context target, documents, and tools until the slowdown returns. That identifies the layer creating the cost.
Swapping and partial offload
A model may technically run while constantly moving data between GPU memory and system RAM or between RAM and disk. This often produces an experience that is far worse than the successful launch suggests.
Watch for:
- high system-memory pressure;
- dedicated GPU memory at or near its limit;
- growing shared-GPU-memory use;
- heavy disk activity during generation;
- pauses that get worse as the chat grows.
The correct fix is normally a smaller model, shorter context, or more suitable hardware—not a cosmetic interface change.
CPU-only expectations
CPU inference is useful for small models, learning, offline experiments, and machines without a supported GPU. It can also be too slow for the intended daily workflow even when the model fits in RAM.
Use a small model first. If that remains too slow, adding RAM alone will not solve the compute bottleneck. A supported GPU or higher-bandwidth Apple Silicon machine changes the experience more materially.
Thermal and power limits
Laptops and compact systems can reduce sustained performance as they heat up. Battery mode, quiet mode, power limits, blocked cooling, and background work can all matter.
Compare a short cold run with a later run under the same prompt and model. If speed degrades over time while memory use is stable, check thermals and power settings before changing the model configuration.
What success looks like
The model starts within a tolerable time, produces the first token without an unexplained delay, generates at a pace suitable for the task, and does not make the rest of the machine unusable. Long chats and document workflows should remain stable at the context size you actually need.
When to use a smaller model or cloud service
Switch paths when:
- the smallest model capable of the task remains too slow on the available hardware;
- GPU acceleration is unsupported or operationally impractical;
- the workload requires context or throughput beyond the machine’s practical limit;
- a larger model offers little task improvement relative to its speed cost;
- the machine must remain available for other work while inference runs.
Using a smaller local model or a controlled remote service is often the honest answer. The goal is a usable workflow, not proving that the machine can eventually emit tokens.
Next step
Recheck what model class the machine can run, then use the model directory to choose a smaller task-appropriate model rather than changing five runtime settings at once.