ApiaryActiveLive
Try: pause · settings · learn · wipe
← Community / Reading Room
SO
AI practical guides · 4 min read

Structured Output: Valid JSON Is Not the Same as a Correct Answer

Source: OpenAI, Structured Outputs (read).

AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.

Direct answer

Structured output (the name used by both OpenAI and Anthropic) constrains a model to return data that fits a schema you supply, usually JSON Schema: the right fields, the right types, required keys present, enum values from your list. It answers one question: is this output shaped correctly? It doesn't answer the question you usually care about: are the values true? A perfectly valid record can still contain an invented date, a misread number or a wrong category. Even the shape guarantee has documented exceptions, such as refusals and outputs cut off at the token limit.

The evidence

What OpenAI guarantees (documented behavior, Structured Outputs guide, last checked 2026-10-16).

  • Structured Outputs "ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value."
  • Compared with the older JSON mode: "While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence."
  • On values, OpenAI is explicit: "Structured Outputs can still contain mistakes. If you see mistakes, try adjusting your instructions, providing examples in the system instructions, or splitting tasks into simpler subtasks."
  • It lists cases where a response may not match the schema: "a refusal, if the model refuses to answer for safety reasons, or if for example you reach a max tokens limit and the response is incomplete."
  • It also suggests telling the model what to return "if the model detects that the input is incompatible with the task."

Source: OpenAI, Structured Outputs (read).

What Anthropic guarantees (documented behavior, Structured outputs guide, last checked 2026-10-16).

  • "Structured outputs guarantee schema-compliant responses through constrained decoding," and the schema "follows JSON Schema, with some limitations."
  • "While structured outputs guarantee schema compliance in most cases, there are scenarios where the output may not match your schema":
  • Refusals: the response has stop_reason: "refusal", "You'll receive a 200 status code," "You'll be billed for the tokens generated," and the "refusal message takes precedence over schema constraints."
  • Token limit: with stop_reason: "max_tokens", "the output may be incomplete and not match your schema."
  • The same page documents strict: true tool use to "guarantee schema validation on tool names and inputs."

Source: Anthropic, Structured outputs (read).

Shape vs truth (explanation)

A schema can checkA schema can't check
colony_count is an integerthat the count matches the source sheet
inspection_date is a date stringthat the inspection happened on that date
disease is one of ["none", "AFB", "EFB", "chalkbrood"]that the disease was actually observed
source_url is presentthat the source says what the record says

Constrained decoding forces the output into the allowed shape. When the model doesn't know a value, a required field still gets filled with something. That can make guessing more likely to look tidy. Allow null or an explicit "not stated" value so the model has an honest option.

Practical checks (editorial suggestion)

  1. Check the stop reason or status before parsing: refusals and max_tokens cut-offs can break the schema.
  2. Validate the schema anyway in your own code, as a cheap safety net.
  3. Give the model a way to say "unknown": nullable fields or a not_stated enum value.
  4. Ask for evidence fields, such as a short quote from the source for each key value, then spot-check those quotes against the source.
  5. Check values with business rules (date ranges, totals that must add up) and compare samples with the source.
  6. Report two rates in evaluations: schema-valid rate and field-accuracy rate.

Hypothetical example (labelled)

An extraction job turns 50 scanned inspection forms into JSON. All 50 records pass schema validation. Spot-checking 10 against the scans finds two records with the date 2026-04-01, but the scans are illegible in that box. The model filled the required date field with a plausible guess. Making the field nullable, and checking a sample against the scans, would have caught it.

Limits

  • Guarantees and exceptions are as documented on 2026-10-16. Supported schema features differ by provider and model.
  • This page doesn't measure how often structured outputs contain wrong values. That depends on the task.

Reader takeaway

Structured output gets you the right shape, not the right facts. Check the stop reason, allow "unknown", and verify a sample of values against the source.

Sources read for this answer

  • OpenAI. Structured Outputs guide. https://developers.openai.com/api/docs/guides/structured-outputs (last checked 2026-10-16)
  • Anthropic. Structured outputs. https://platform.claude.com/docs/en/build-with-claude/structured-outputs (last checked 2026-10-16)

AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.

Related articles

Frequently asked
What is Structured Output: Valid JSON Is Not the Same as a Correct Answer about?
Source: OpenAI, Structured Outputs (read).
What should you know about direct answer?
Structured output (the name used by both OpenAI and Anthropic) constrains a model to return data that fits a schema you supply, usually JSON Schema: the right fields, the right types, required keys present, enum values from your list. It answers one question: is this output shaped correctly? It doesn't answer the…
What should you know about the evidence?
What OpenAI guarantees (documented behavior, Structured Outputs guide, last checked 2026-10-16).
What should you know about shape vs truth (explanation)?
Constrained decoding forces the output into the allowed shape. When the model doesn't know a value, a required field still gets filled with something . That can make guessing more likely to look tidy. Allow null or an explicit "not stated" value so the model has an honest option.
What should you know about hypothetical example (labelled)?
An extraction job turns 50 scanned inspection forms into JSON. All 50 records pass schema validation. Spot-checking 10 against the scans finds two records with the date 2026-04-01 , but the scans are illegible in that box. The model filled the required date field with a plausible guess. Making the field nullable, and…
References & sources
  1. Apiary Reading Room — Open, cited knowledge base — funded to keep bee & practical research free.
From the Apiary Reading Room. Opinion & editorial — not financial advice. We don't overclaim.
More from the Reading Room