Skip to main content
Local LLM Guide

Hardware

NVIDIA 16GB VRAM GPU

Start around 14B 4-bit. Test context growth before moving into larger model classes.

Example: 32GB system memory16GB GPU memory

Model sizes to check

14B class estimates

Likely bottleneck

Dedicated GPU memory, then context length

Good first workload

Chat, coding, and moderate document workflows

Upgrade trigger

Move to more GPU memory when the model or context repeatedly spills into system RAM.

What this machine is suited for

Start around 14B 4-bit. Test context growth before moving into larger model classes.

  • 14B class estimates

What will limit it

Fast local inference usually depends on keeping most model work in GPU memory. If the app moves work into system RAM, responses can become slower.

  • The memory values describe an example configuration; check the actual computer before choosing a model.
  • The estimate changes with the exact model file, context length, runtime, GPU offload, drivers, thermals, and other running apps.
  • A model that technically loads may still be constrained, unstable, or memory-limited for ordinary use. Treat “fits” and “feels good” as different questions.

Before moving to a larger model

  1. Use a common 4-bit build.
  2. Keep context modest until the first run is stable.
  3. Close memory-heavy applications and watch actual memory pressure.
  4. Increase model size only when the smaller model is not solving the task.

What to upgrade

Move to more GPU memory when the model or context repeatedly spills into system RAM.