What this machine is suited for
Start with 14B–32B 4-bit models. Larger experiments still depend on context, offload, system memory, and the exact runtime.
- 14B to 32B estimates
- 70B requires CPU offload or a smaller-than-Q4 file
What will limit it
Fast local inference usually depends on keeping most model work in GPU memory. If the app moves work into system RAM, responses can become slower.
- The memory values describe an example configuration; check the actual computer before choosing a model.
- The estimate changes with the exact model file, context length, runtime, GPU offload, drivers, thermals, and other running apps.
- A model that technically loads may still be constrained, unstable, or memory-limited for ordinary use. Treat “fits” and “feels good” as different questions.
Before moving to a larger model
- Use a common 4-bit build.
- Keep context modest until the first run is stable.
- Close memory-heavy applications and watch actual memory pressure.
- Increase model size only when the smaller model is not solving the task.
What to upgrade
Move to more GPU memory when the model or context repeatedly spills into system RAM.