Pre-training

Pre-training is where the raw model is born. You take a randomly initialized neural network and feed it trillions of tokens of text — books, code, web pages, papers, transcripts — and have it predict the next token over and over until it has internalized grammar, facts, reasoning patterns, and the statistical shape of human language. Everything the model "knows" about the world traces back to this phase. It is also the most expensive single thing AI labs do. A frontier pre-training run costs $100M+ in compute alone for GPT-5, Claude Opus 4.5, or Gemini 3 Ultra-class models, runs across tens of thousands of GPUs for months, and consumes enough electricity to power a small city. This is why only a handful of labs in the world actually pre-train frontier models from scratch — most teams start from someone else's pre-trained checkpoint. Pre-training produces a "base model": fluent but raw. It will happily continue any text in any direction, with no concept of being a helpful assistant, no refusals, no "user vs assistant" turns. To turn that base model into something like ChatGPT or Claude, you need post-training (SFT, RLHF/DPO, safety tuning). Pre-training sets the ceiling on what the model can know; post-training shapes how it behaves. The two phases reward different things, and that is why one cannot replace the other. Pre-training scores the model on how well it predicts the next token of real text, so a continuation that is wrong, rude or evasive still scores well if that is what the source text said. Post-training scores responses on whether a person (or a reward model trained on people's ratings) judges them helpful, honest and safe. A base model asked a question may reply with three more questions, because on many web pages a question is followed by more questions. Post-training is what teaches it to answer.