Skip to main content
Local LLM Guide

Model

StarCoder2 3B

Start with a supported compressed build and short context. 8GB RAM estimate. Best use: Local code completion and fill-in-the-middle workflows

3BCode

Best use

Local code completion and fill-in-the-middle workflows

Memory planning

8GB RAM estimate

Expected pace

Depends on your hardware, app, and conversation length

License

BigCode OpenRAIL-M v1; read the model’s terms before use

Who should use this model?

  • Local code completion and fill-in-the-middle workflows

Choose something else when

  • You need an instruction/chat model; the official model card says the base model is not an instruction model.

What hardware makes sense?

The download is only part of the memory needed. Leave room for the app, the input it processes, and other open programs. For chat models, context includes the instructions, conversation, and answer. Longer context needs more memory.

CPU only
A possible starting point with a compressed model and short conversation; check the exact file size.
8GB system memory
A possible starting point with a compressed model and short conversation; check the exact file size.
16GB system memory
A possible starting point with a compressed model and short conversation; check the exact file size.
32GB system memory
A possible starting point with a compressed model and short conversation; check the exact file size.
8GB GPU memory
A possible starting point with a compressed model and short conversation; check the exact file size.
12GB GPU memory
A possible starting point with a compressed model and short conversation; check the exact file size.
24GB GPU memory
A possible starting point with a compressed model and short conversation; check the exact file size.
Apple unified memory
A possible starting point with a compressed model and short conversation; check the exact file size.

Model facts that change the decision

Parameters
3B
Input
Text
Published context limit
16,384 tokens
Model format
safetensors · transformers · GGUF · Ollama
Suggested compression
Q4_K_M
Typical Q4 file
1.85GB before runtime and context overhead
Languages
17 programming languages

First setup

Ollama

ollama run starcoder2:3b

LM Studio: Search for `second-state/StarCoder2-3B-GGUF` and choose Q4_K_M for the first local run.

llama.cpp

Use llama-cli -hf second-state/StarCoder2-3B-GGUF:Q4_K_M -p '<code prefix>'

Conditions that change the answer

  • The official StarCoder2 base checkpoints are code-focused but not instruction-tuned; choose an instruction-tuned version if you need chat.
  • RAM and VRAM estimates are conservative estimates from GGUF file sizes plus local runtime and conversation memory overhead.

Read the controlling license text →