Ship an AI Feature on Claude

A Claude-powered feature is easy to demo and hard to ship. The demo works once on a happy-path input; production gets weird inputs, costs that creep, hallucinations users see, and no way to tell when quality drifts. This is the path from "it works in my chat" to a feature that holds up — cost-controlled, grounded, observable, and verified.

The first Claude call you write works, then doesn't hold up. The system prompt gets re-billed on every request, a long generation times out at the SDK's HTTP boundary, structured output comes back as prose you have to regex, and a transient 529 takes down a user action. This is the integration done so it survives real traffic: caching the stable prefix, streaming long output, current model defaults, tool use, and retries that fire on the right errors.

You know your LLM feature costs too much, so you reach for the obvious levers — a cheaper model, a shorter prompt, more caching. Each one cuts the bill. But you have no way to tell whether it also quietly broke the answers, so you ship the cut on faith and find out from a user. This builds a small eval set first, then applies the cost levers one at a time and re-scores after each — so every cut is one you can see held quality, not one you hope did.

Your Claude feature needs to answer from your own data — internal docs, a knowledge base, product specs — and the model doesn't have it. So it either makes things up or says nothing useful. RAG (retrieval-augmented generation) fixes that by fetching the relevant chunks of your data and putting them in the prompt at answer time. This is the full path: decide whether you even need it, chunk and embed your docs, store them in pgvector on Supabase, retrieve and rerank for a query, and build a grounded prompt that cites sources and admits when it doesn't know.

A Claude-powered feature that ran fine in the demo will fail in ways a normal app doesn't. It won't throw — it'll return a confident wrong answer, drift in quality as inputs shift, or quietly triple its token cost on a bad prompt. None of that shows up in your error tracker, because nothing errored. This wires the two things that catch it: a log of every call (input, output, cost, latency, traced to a user) and a guardrail on the output *before* the user acts on it.

The demo worked. That's the trap. A Claude feature that answers one clean question in your chat will get hit by empty inputs, 50k-token pastes, prompt-injection attempts, off-topic noise, and hostile users — none of which the demo covered. This is the gate between "it works for me" and "it's safe for a stranger." You run the eval set, attack the feature on purpose, confirm the guardrails actually catch what they're supposed to, verify any AI-generated code in the path, and confirm you'll see what it does in production before you ship it.