Who should use this model?
- Local code explanation and code generation with an instruction/chat prompt format
Choose something else when
- You need a modern function-calling/tool-calling model with an official tool schema
What hardware makes sense?
The download is only part of the memory needed. Leave room for the app, the input it processes, and other open programs. For chat models, context includes the instructions, conversation, and answer. Longer context needs more memory.
- CPU only
- Possible but slow
- 8GB system memory
- Use a smaller model for this memory limit.
- 16GB system memory
- Use a smaller model for this memory limit.
- 32GB system memory
- Memory or response time may limit use. Try a smaller model first.
- 8GB GPU memory
- Use a smaller model for this memory limit.
- 12GB GPU memory
- Use a smaller model for this memory limit.
- 24GB GPU memory
- Memory or response time may limit use. Try a smaller model first.
- Apple unified memory
- 32GB unified memory or more
Model facts that change the decision
- Parameters
- 34B
- Input
- Text
- Published context limit
- 100,000 tokens
- Model format
- safetensors · transformers · GGUF · Ollama
- Suggested compression
- Q4_K_M
- Typical Q4 file
- 20.22GB before runtime and context overhead
- Languages
- English · Python · C++ · Java · PHP · TypeScript · JavaScript · C#
First setup
Ollama
ollama run codellama:34b-instructLM Studio: Search for `TheBloke/CodeLlama-34B-Instruct-GGUF` and start with the Q4_K_M file for balanced quality and size.
llama.cpp
Use llama-cli -hf TheBloke/CodeLlama-34B-Instruct-GGUF:Q4_K_M with the Code Llama instruct prompt template
Conditions that change the answer
- Memory needs include the model file, the app, and conversation memory. Longer conversations and other open apps need more space.
- Meta describes stable generations up to 100,000 tokens, while common local runtimes and the Ollama catalog may default to smaller contexts such as 16K.
Read the controlling license text →