AI Output Verifier — catch hallucinated APIs before they ship

Today's post is a manual review checklist for AI-generated PRs. Useful, but slow when you're staring at a 600-line diff at 4pm on a Friday.

The AI Output Verifier skill automates the most expensive part: checking that the imports, function calls, and API signatures the AI used exist in the version of the library you're running. It catches hallucinated APIs, made-up signatures, and silently-broken examples — the bugs that pass CI and break in prod.

What it does

You hand it the diff (or the function the AI just wrote). It scans for:

• Imports that don't exist in your installed package version
• Function signatures that drift from the real API
• Method calls that look right but aren't (findManyAndCount when only findMany exists)
• TypeScript-coerced ghost APIs — calls that compile because TS finds a near-match, but crash at runtime

Then it returns a ranked list of items to verify by hand, ordered by likelihood of hallucination.

Why this pairs with the post

Item 2 on the post's checklist — "Does every import resolve to a real version?" — is the highest-yield 30-second check on any AI PR. The skill turns it from a manual grep into a structured pass. Use the checklist for the judgment calls; use the skill for the lookups.

Open the skill →