RLHF (Reinforcement Learning from Human Feedback)

RLHF is how a base model that only continues text becomes an assistant that follows instructions. OpenAI's InstructGPT paper (Ouyang et al., 2022) set out the recipe in three steps. First, supervised fine-tuning on human-written examples of good answers. Second, people rank several model answers to the same prompt (the paper used 4 to 9 per prompt), and a separate reward model is trained to predict those rankings. Third, reinforcement learning, in the paper proximal policy optimization (PPO), updates the language model so it produces answers the reward model scores higher. The result is a model that is noticeably more helpful, follows instructions, and refuses harmful requests. ChatGPT, released in November 2022, was trained "using the same methods as InstructGPT," and most major chat models since have gone through some form of preference training. RLHF has known costs. Human ranking is expensive, and PPO is unstable to tune. The reward model scores what raters liked, which is not always what is true: Anthropic researchers reported in 2023 that sycophancy is "likely driven in part by human preference judgments favoring sycophantic responses," and in April 2025 OpenAI rolled back a GPT-4o update after an added reward signal from thumbs-up and thumbs-down data made the model overly agreeable. Two lighter alternatives have grown alongside it. DPO (direct preference optimization, 2023) learns from the same preference pairs without a separate reward model or PPO loop, and Meta reported that for Llama 3 it "required less compute for large-scale models and performed better." RLVR (reinforcement learning with verifiable rewards, named in AI2's Tülu 3 in November 2024) replaces human rankings with a checkable answer such as a correct math result or a passing test; DeepSeek-R1 (January 2025) trained its reasoning this way with rule-based accuracy and format rewards.