Model compression, also called quantization, stores weights using fewer bits. It can make a model's download and memory requirement smaller, with possible changes to answer quality. For a supported text model, Q4_K_M is often a useful GGUF candidate to investigate first; choose a higher-precision download when it fits and improves the work you actually do.
This guide suits someone facing several download files for the same model. It is not a claim that four-bit models always beat smaller higher-precision models, or that one compression label determines speed and quality across different architectures.
What Q4, Q5 and Q8 mean
The number suggests the approximate precision used for stored weights, but modern formats can mix precisions across different parts of the model. Keep the full label, such as Q4_K_M or Q8_0, rather than shortening every file to “Q4” or “Q8.”
| Download choice | Practical reason to investigate it | What to check |
|---|---|---|
| Q4-class GGUF | Smaller weight file when memory is limited | Exact file size and whether errors increase on your task |
| Q5-class GGUF | Another balance between precision and size | Whether it improves your answer enough to justify the extra memory |
| Q8-class GGUF | More precise stored weights than lower-bit alternatives | Weight size, context, and remaining memory |
| MLX quantized download | A supported path for an app using MLX | Bit setting, group configuration, model support, and companion files |
These are representation choices. GGUF and MLX downloads are not interchangeable just because both say four-bit. See GGUF vs MLX before choosing a format for the app.
File size is only the start
As a rough arithmetic illustration, 8 billion weights at exactly four bits each would occupy 4 billion bytes, or 4GB. Real files can be larger because of metadata, mixed precision, scales, and other tensors. This is not the size of any named download and does not include the memory needed to run it.
The operating system, app, conversation memory, and temporary calculations also need space. A mixture-of-experts model still loads the required total weights even when fewer parameters are active for an individual token. Use the exact file in the memory calculator, including the intended context.
Choose a download in five steps
- Find the exact model and checkpoint. Do not compare different generations under a shared family name.
- Confirm the app supports its architecture and format.
- Select a quantized file from the publisher or a clearly identified conversion repository. Read the original model license and any relevant download terms.
- Check file size and memory at your intended context, leaving room for the desktop and other apps.
- Load one file and test your task before downloading several more.
For example, the official Qwen3 4B GGUF repository contains different compressed downloads. In LM Studio, inspect the exact filename and repository. An Ollama tag is a packaged download with its own file choices; inspect its library details rather than assuming it equals a particular Hugging Face file.
Split downloads require all the parts the model runner expects. A vision model may need a separate projector or other companion file. A small file selected in isolation is not necessarily a complete model.
Compare precision without changing everything else
If two downloads of the same checkpoint fit, try the same prompt in new chats with the same context and model-appropriate generation settings. Include a task with a known answer and a realistic task you can inspect. For writing, check that dates and conditions survive the rewrite. For coding, run meaningful tests. For documents, check the quoted source.
A useful lower-bit file loads, leaves acceptable memory, and produces answers adequate for that job. If the higher-precision file improves the answer materially, keep it when resources permit. More precision cannot repair missing source text, a wrong chat template, or a model that lacks the required capability.
Keep conversation compression separate
Some model runners also support compressed conversation memory, or KV-cache quantization. This can reduce a different part of the memory requirement, especially for long context, but can introduce its own quality tradeoffs and compatibility conditions.
Do not change weight compression and cache precision at the same time while diagnosing a regression. Return to a supported baseline, change one setting, and check the actual answer. See context windows for that memory distinction.