Eval / evaluation

An "eval" (engineer slang for evaluation) is your private benchmark. You collect 50-500 examples of real tasks your app handles, define what "good" looks like (often via a rubric scored by another LLM, sometimes by humans), and run your candidate models against the set. The output is a number per model per task, so you can decide whether GPT-5.1 is actually better than Claude Sonnet 4.6 on your customer support flow. Custom evals beat public benchmarks for production work for one reason: they measure your distribution. MMLU tells you a model knows random facts; your eval tells you whether the model handles your customers' weirdly-phrased refund requests. Frameworks like OpenAI Evals, Promptfoo, Braintrust, LangSmith, and Anthropic's Inspect framework make this accessible — most teams can stand up a basic eval suite in a day. The evolution is to LLM-as-judge eval: you don't write deterministic graders, you have GPT-5 score the output against a rubric. This is faster than human grading and (with calibration) often correlates well with human judgment. Watch for systematic biases — judge models prefer their own family's outputs, prefer longer answers, and miss subtle failures. Cross-check with human spot-checks on a 5-10% sample. The teams shipping good AI products in 2026 all have eval pipelines they ship as part of CI.