RLAIF (RL from AI Feedback)

RLAIF takes the basic shape of RLHF — collect preference comparisons, train a reward signal, optimize the policy — but swaps the human labelers for a strong AI judge. You give a teacher model (often the lab's flagship) two candidate answers and a rubric, and have it pick the better one. Those AI-generated preferences become training data for the smaller model being aligned. The scale advantage is enormous. Human preference labeling tops out at thousands of comparisons per labeler per week, costs dollars per comparison, and is bottlenecked on hiring. A capable LLM can produce millions of comparisons per day at fractions of a cent each. That difference is why RLAIF and synthetic preference pipelines are now the default at every lab — even at frontier labs that can afford humans, AI judges are used for the bulk of preferences and humans are reserved for high-leverage edge cases and final calibration. The catch is bias inheritance. The AI judge has its own preferences, blind spots, and pet peeves, and any model trained against it inherits those. If your judge is sycophantic, your student becomes sycophantic. If your judge over-refuses, your student over-refuses. This is one of the leading hypotheses for why models from different labs are converging in style — they're increasingly trained against AI judges that share lineage. Constitutional AI is essentially RLAIF with an explicit, written rubric instead of an implicit one.