Distillation

In distillation you take a big, expensive, smart "teacher" model — say Claude Opus 4.5 or GPT-5 — and use its outputs to train a much smaller "student" model. The student can either match the teacher's exact next-token probability distributions (classical distillation) or simply train on millions of teacher-generated example responses (response distillation, the dominant flavor in 2026). The student ends up punching far above its parameter count because it's compressing a smarter model's behavior rather than learning from scratch. This is how the modern small-model tier exists. Claude Haiku, GPT-5 mini, Gemini Flash, Llama 4 8B Instruct — all of them are distilled, in some way, from larger sibling models. They run cheaper and faster than the flagship while capturing 70-90% of the quality on most tasks. For high-volume applications (chat assistants, classification, summarization at scale), distilled small models are usually the right answer over calling the big model directly. Distillation has a quietly transformative second-order effect: it lets a lab amortize a $100M pre-training run across a whole product family. You pre-train one giant base, post-train it into the flagship, then distill that flagship into mini, nano, and edge variants. The same investment ships five products. It also creates legal questions when the teacher belongs to someone else — DeepSeek and others have been credibly accused of distilling from closed frontier APIs, which is one of the live ToS battlegrounds in 2026.