Turn your hand review of an AI feature into a grader

Help me turn the way I review this AI feature by hand into an LLM-as-judge grader I can run on every change.

The feature: {{what it does, what goes in, what comes out}}
What I currently check by hand: {{the things you look for before you ship — be honest, including the vague ones}}
Example outputs I would call good: {{paste 2-3}}
Example outputs I would call bad, and why: {{paste 2-3 with your reason for rejecting each}}
What I grade against: the final output only / the full transcript including tool calls / both: {{pick}}

Work through this in order.

**1. Split my vague check into separate dimensions.** Take what I said I check and break it into dimensions that can be judged independently — for example groundedness in the source, coverage of the required facts, format compliance, tone. Name each one. If two of my checks collapse into one dimension, say so; if one of my checks is really three, split it. Grade each dimension with its own judge call rather than one judge scoring everything at once, because a single judge conflates criteria and the score stops telling you which thing broke.

**2. Write the rubric for each dimension.** For each one produce: the question the judge answers, a narrow scale (binary pass/fail where the dimension allows it, otherwise 1-3 or 1-5), and a concrete description of what each score level looks like on THIS feature, written from my examples rather than in the abstract. A level description a stranger could not apply to my examples is not finished.

**3. Write the judge prompt itself.** One per dimension. Each should take the input, the output, and the reference material where one exists, and return a structured verdict with a short explanation and the score. Include an explicit escape hatch: the judge returns Unknown when the material does not contain enough to decide, so it stops inventing confident verdicts on cases it cannot see.

**4. Give me the calibration procedure.** How many outputs I should label myself before trusting the judge, how to measure agreement between my labels and the judge, and what level of disagreement means the rubric is wrong rather than my labels. Say what to do in each failure direction: the judge marks good outputs down (usually a vague rubric or a level description the output cannot satisfy) versus the judge passes bad outputs (usually a dimension I never wrote down).

**5. Name what this grader will not catch.** The failures that live outside every dimension we wrote, and the cheapest non-judge check for each — a deterministic assertion, a regex, a schema validation, a human spot-check on a sample.

Do not propose a scoring scheme that averages the dimensions into one number unless I asked for one. A blended score hides which dimension regressed.

Tip: Run step 4 before you wire the grader into CI. An uncalibrated judge produces a number that moves for reasons you cannot trace.