Small language models

TL;DR

• Small language models (1B-14B parameters) run on consumer hardware at $0.10-0.50 per million tokens versus $15-30 for GPT-4, a 30-150x cost difference at volume.
• A fine-tuned 7B model regularly beats a general-purpose 70B on narrow tasks. Specialization concentrates capacity where you need it, scale spreads it everywhere.
• Phi-4 (14B) matches GPT-4-class performance on STEM benchmarks, proving that quality training data outweighs raw parameter count for specific domains.
• On-device deployment via Ollama runs Mistral 7B on a MacBook Pro M3 at 20-30 tokens per second with one command: no cloud, no API key, no data egress.
• Privacy is the clearest non-cost win: regulated data (PHI, PII) never leaves your infrastructure when you run on-premise.

---

The problem it solves

At low volume, cloud LLM APIs are great. You call an endpoint, you get a response, and your team ships features fast without thinking about infrastructure. The problem is what happens at scale.

A startup processing 80,000 support tickets per day through GPT-4 is spending $35,000-$48,000 per month on API costs alone. Add 100-500ms network round-trip latency per request, and the fact that every customer document transits a third-party system, and the calculus changes fast.

Small language models trade the top 5-10% of general capability for dramatically lower cost, lower latency, and privacy by default. For many tasks, that tradeoff is overwhelmingly one-sided in your favor.

---

What is it?

A small language model (SLM) is a transformer-based language model in roughly the 1B-14B parameter range that fits on consumer or single-GPU hardware. The architecture is identical to larger models: embedding layer, stacked self-attention and feedforward blocks, output projection. The differences are in parameter count and training approach.

Think of it like a specialist versus a generalist doctor. A cardiologist has more focused expertise in heart conditions than a general practitioner, even though a GP covers more ground overall. An SLM fine-tuned for your task is the specialist: narrower, but often better at the specific thing you need.

The key shift in recent years is that scale is no longer the only path to quality. Phi-4 (14B) and Gemma 3 (9B) have shown that carefully curated training data, including synthetic data generated by stronger models to demonstrate rigorous reasoning, can close most of the capability gap with much larger models on specific task categories.

---

How it works

Architecture: identical to large models, just smaller

SLMs are full transformer decoders. If you understand how GPT-4 works architecturally, Mistral 7B works the same way. The difference is in layer count, hidden dimension size, and attention head count.

Both use multi-head attention, rotary position embeddings (RoPE), and grouped-query attention (GQA) for inference efficiency. The 70B model is not a fundamentally different technology, just a larger one.

Why training data quality matters more than scale

The "Phi hypothesis" from Microsoft Research changed how the field thinks about small models. Instead of training Phi-4 on raw internet text, they generated synthetic examples specifically constructed to exhibit high-quality reasoning steps, then used those as the primary training corpus. The result: a 14B model that outperforms many 70B models on math and science benchmarks.

For production use, this has a direct implication. Fine-tuning a small model on 1,000-3,000 high-quality examples of your specific task routinely outperforms a general large model zero-shot. The fine-tuned model has concentrated its capacity on your task's patterns. The large model is spreading capacity across everything.

Inference hardware requirements

Without quantization, a 7B model requires roughly 14GB of VRAM in fp16. With 4-bit quantization, the same model drops to 4-5GB, fit for most modern consumer GPUs and all Apple M-series chips.

GGUF is the standard format for CPU and Metal inference via Llama.cpp and Ollama. GPTQ and AWQ are GPU-optimized formats. For most users, Q4_K_M is the right starting point: it applies different quantization levels to different weight components, preserving quality where it matters most.

Deployment pipeline