Getting structured output: JSON, lists, and formats that hold

Asking a model for output in a specific format — JSON, a table, a numbered list, a code block — should be simple. It often isn't. Models drift. They add commentary when you wanted just the data. They output YAML when you asked for JSON. They wrap results in a markdown code block when you're feeding the output to a parser that doesn't expect one.

Format control is a small skill with outsized practical value. Here's what actually works.

Why models drift from requested formats

Models are trained to be helpful, which often means being conversational. A helpful response includes context, explanation, and caveats — not just a bare JSON object. The model's default behavior works against strict format compliance.

Every format instruction you give is fighting that default. The way to win is to make the format constraint unambiguous, place it at the right point in the prompt, and — for high-stakes cases — validate the output programmatically.

The prompt placement problem

Format instructions at the end of long prompts are often ignored. Instructions earlier in the prompt get more attention, especially for complex tasks.

Effective order:

1. Context (who is this for, what is it for)
2. Format specification (exact output shape)
3. Task instructions
4. Input data

Most prompts put format instructions last ("...and respond in JSON"). Moving them earlier helps.

Less reliable:

Analyze these five customer reviews and identify the main themes.
Extract: main theme, sentiment (positive/negative/neutral), most common complaint, and any feature requests.
Respond in JSON format.

More reliable:

Output a JSON array. No other text.

Each element should have these exact keys:
- "theme": string
- "sentiment": "positive" | "negative" | "neutral"
- "main_complaint": string or null
- "feature_requests": array of strings (empty array if none)

Analyze these five customer reviews and extract the above for each:
[reviews]

The second version specifies the schema before describing the task. The model generates into that constraint.

JSON specifically

JSON is the most common format in production pipelines, and the most common place format control breaks.

Schema matters more than the word "JSON." Saying "output JSON" is less reliable than showing the exact structure you expect.

Explicit null handling. If a field might be absent, say what to do: "complaint": null rather than omitting the key entirely. Inconsistent field presence breaks parsers.

No trailing commentary. Models often follow a JSON block with a sentence explaining what they did. If you're parsing programmatically, say: "Output only the JSON. No explanation before or after it."

Avoid deeply nested structures. The more complex the schema, the higher the error rate. If you need deeply nested JSON, consider splitting the task.

Template approach — give the model a template to fill:

Fill in the following JSON template. Output the completed template only.
Do not add or remove keys.

{
  "product_name": "",
  "category": "",
  "price_usd": 0,
  "in_stock": true,
  "tags": []
}

Product description to extract from:
[description]

Tables

Markdown tables are readable by humans and parseable by most markdown processors. They're also easy to mess up — column counts change, pipes get dropped, alignment characters disappear.

For clean markdown tables:

Output a markdown table with these exact columns: Name | Role | Status
Include a header row and a separator row.
No text outside the table.

If you need to parse the table programmatically, consider requesting CSV or JSON instead — markdown tables are harder to parse reliably than either.

Lists

Numbered lists and bullet lists are the most forgiving format — models handle them well by default. The main failure mode is inconsistency: sometimes a list, sometimes prose, depending on subtle prompt variations.

For pipelines that need consistent list output:

Output a numbered list. Each item is one line. No introductory sentence. No closing commentary.

Code blocks

If you're asking for code output, models usually default to wrapping it in markdown code blocks (```). This is correct for display, but wrong if you're piping the output directly to a file or interpreter.

If you need raw code: "Output only the code. No markdown formatting, no explanation, no code fences."

If you want the code fence with a language tag: "Wrap the output in a ```python code block."

Native structured output APIs

Several model APIs now support structured output as a first-class feature — you pass a schema, the model is constrained to emit valid JSON matching it:

• OpenAI: response_format: { type: "json_schema", json_schema: {...} }
• Anthropic (Claude): structured output via tool use (pass a tool definition, force a specific tool call)
• Google Gemini: response_mime_type: "application/json" with response_schema

This is more reliable than prompt-based format control for production use. The model can't deviate from the schema because the sampling is constrained at the API level.

When to use native structured output:

• Production pipelines where a format error causes downstream failures
• High volume (many calls per day) where manual review isn't feasible
• Schemas with required fields, specific types, or enum constraints

When prompt-based format control is fine:

• Interactive use where you can verify the output visually
• Development and prototyping
• Cases where the API doesn't support your target format

The validation layer

For production use, treat format control as a two-layer problem: prompt instructions + output validation.

Even well-formatted prompts will occasionally produce malformed output. A JSON parser that throws on bad output is more reliable than a prompt that says "always output valid JSON." The right posture is: write the prompt to minimize failures, and validate the output to catch the ones that get through.

Retry on parse failure with a modified prompt ("the previous response was not valid JSON — try again") works reliably for most cases and doesn't require complex fallback logic.