Who should use this model?
- Apache-2.0 MoE local model
- strong classic open-weight chat baseline
- systems that benefit from sparse active parameters
- benchmarking against newer dense local models
Choose something else when
- you have less than 32 GB RAM
- you need easy 24 GB GPU full-offload deployment
- you need a current-generation small model with simpler deployment
What hardware makes sense?
The download is only part of the memory needed. Leave room for the app, the input it processes, and other open programs. For chat models, context includes the instructions, conversation, and answer. Longer context needs more memory.
- CPU only
- Memory or response time may limit use. Try a smaller model first.
- 8GB system memory
- Use a smaller model for this memory limit.
- 16GB system memory
- Use a smaller model for this memory limit.
- 32GB system memory
- Memory or response time may limit use. Try a smaller model first.
- 8GB GPU memory
- Use a smaller model for this memory limit.
- 12GB GPU memory
- Use a smaller model for this memory limit.
- 24GB GPU memory
- Some model work must use system memory, which can slow responses.
- Apple unified memory
- Limited on 32GB unified memory
Model facts that change the decision
- Parameters
- 46.7B total / 12.9B active
- Input
- Text
- Published context limit
- 32,768 tokens
- Model format
- safetensors · GGUF · Ollama
- Suggested compression
- Q4_K_M
- Typical Q4 file
- 26.44GB before runtime and context overhead
First setup
Ollama
ollama run mixtral:8x7bLM Studio: Use TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF and select Q4_K_M only on machines with enough memory.
llama.cpp
Download the Q4_K_M GGUF from TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF and run llama-server with a context length appropriate to available memory.
Conditions that change the answer
- RAM and VRAM estimates are conservative local-inference estimates based on GGUF file size plus runtime and conversation memory overhead; TheBloke's table reports 28.94 GB max RAM for Q4_K_M before additional deployment margin.
- Total and active parameter counts are MoE-specific; the catalog stores total parameters separately from approximate active parameters per token.
- Check that this exact model can use your app’s connected tools before relying on that feature.
- Language coverage is not confirmed in the available model information. Test a representative prompt in the language you need before choosing this model.
Read the controlling license text →