Context Wire
Tool Reviews

JSON Mode vs Structured Outputs: Guarantees and Implementation

Compare JSON mode and schema-constrained outputs, then implement schemas, validation, retries, versioning, and safe production handoffs.

Published 13 min read
  • #JSON mode
  • #structured outputs
  • #JSON Schema
  • #LLM APIs

JSON mode asks a model API for syntactically valid JSON. Structured Outputs adds a schema constraint, so the response must also have the keys, types, and permitted values your application expects; this guide compares those guarantees and shows how to carry the stronger contract into production code.

This article uses “Structured Outputs” as the general name for provider features that constrain a final response to a supplied schema. Product names and request fields vary. Feature, limit, and compatibility statements were checked against official OpenAI, Microsoft Azure, Google Gemini, and Anthropic documentation on 2026-07-24.

Quick answer

Use Structured Outputs when downstream code depends on a stable contract: extracting records, rendering a known UI, or feeding a typed pipeline. Use JSON mode when you need parseable JSON but cannot use the provider’s schema-constrained feature or must support an older model.

The practical distinction is simple:

QuestionJSON modeStructured Outputs
Is the response intended to parse as JSON?YesYes
Are required keys and value types enforced by the model API?NoYes, within the provider’s supported schema subset
Can values still be factually or semantically wrong?YesYes
Do you still need application validation?Yes: syntax edge cases, shape, and meaningYes: refusal or truncation handling plus meaning and business rules
Does the request depend on a provider-supported schema subset?NoYes
Is it the right interface for requesting a tool action?No; use function or tool callingNo; use function or tool calling
Is it broadly available on older models?Often broaderUsually narrower and provider-specific
Is it a good default for a typed production integration?Usually a fallbackUsually the first choice

OpenAI explicitly recommends Structured Outputs over JSON mode when supported. Its official comparison says both produce valid JSON, but only Structured Outputs provides schema adherence (OpenAI Structured Outputs documentation). Microsoft gives the same recommendation for Azure OpenAI (Azure JSON mode documentation).

For tool arguments, use the provider’s function or tool-calling interface. Use a structured response format when the model’s final answer needs a fixed shape. The schema machinery may be similar, but the control flow is different.

Four levels of output control

Teams often describe several different techniques as “structured output.” They provide different guarantees.

TechniqueWhat it controlsTypical failureBest use
Prompt-only formattingThe model is asked to follow a shapeExtra prose, missing keys, invalid syntaxPrototypes and human-reviewed drafts
JSON modeOutput is restricted to valid JSON syntaxValid JSON with the wrong fields or typesFlexible machine-readable responses
Schema-constrained generationTokens are limited to paths allowed by a schemaSemantically wrong values inside valid fieldsStable application interfaces
Post-generation validationCode checks syntax, types, ranges, and rulesRejected output requires repair or fallbackEvery production boundary

Constrained decoding removes token paths that cannot produce a schema-valid result. If a field permits only "low", "medium", or "high", the decoder can block a value such as "urgent-ish". It still cannot determine whether "high" is the factually correct label. Structural validity is a transport guarantee, not a truth guarantee.

Worked example: turn an email into a proposed refund

Suppose an assistant reads this customer message but is not allowed to execute refunds:

I was charged twice for order A1049. One charge was $48.50. Please fix it.

The application can require this candidate-object schema:

{
  "type": "object",
  "properties": {
    "order_id": { "type": "string" },
    "reason": {
      "type": "string",
      "enum": ["duplicate_charge", "missing_item", "damaged_item", "other"]
    },
    "requested_amount": { "type": ["number", "null"] },
    "evidence_quote": { "type": "string" }
  },
  "required": ["order_id", "reason", "requested_amount", "evidence_quote"],
  "additionalProperties": false
}

A schema-compliant result might be:

{
  "order_id": "A1049",
  "reason": "duplicate_charge",
  "requested_amount": 48.5,
  "evidence_quote": "I was charged twice for order A1049."
}

That result is structured, but it is not authorized. The application still has to confirm account ownership, settled charges, currency, refund eligibility, and the operator’s authority. A safe workflow keeps interpretation and action separate:

  1. The model extracts a proposed request.
  2. Schema validation checks its shape.
  3. Deterministic code looks up the order and payment records.
  4. Business rules calculate the eligible amount.
  5. A human or authorized policy approves consequential cases.
  6. A payment service performs the refund and returns a transaction ID.
  7. The system records both the proposal and the actual outcome.

The LLM translates messy language into a candidate object; it does not become the payment ledger.

Design and version schemas as application contracts

Expose the distinctions the software genuinely needs instead of placing an entire answer in one text field. If a missing value differs from an uncertain value, represent both states rather than forcing the model to invent a field value.

Useful design rules include:

  • use enums only for genuinely closed sets of downstream states;
  • name units explicitly, such as duration_seconds or amount_minor_units;
  • separate source evidence from model interpretation;
  • disallow unexpected properties when the interface must remain stable;
  • use the smallest schema that supports the next deterministic decision; and
  • attach a schema version when outputs may outlive the current release.

Treat schema changes like API changes. Give breaking versions distinct identifiers, test stored examples against new consumers, and monitor failures by schema version, model, and prompt revision. Generated compile-time types help code you control; runtime validation still protects the boundary where untrusted bytes enter that code.

What JSON mode guarantees

JSON mode removes a common failure class: prose around an object or another response that cannot be parsed as JSON. You normally enable a response format such as json_object, then describe the desired object in the prompt.

The guarantee stops at JSON syntax. If you ask for:

{
  "ticket_id": "string",
  "priority": "low | medium | high",
  "needs_human": "boolean"
}

JSON mode can still return valid JSON with ticketId instead of ticket_id, omit needs_human, encode the Boolean as "yes", or invent a priority outside the allowed set. Every example is parseable; none necessarily satisfies your contract.

OpenAI says JSON mode requires an explicit instruction to generate JSON; without it, generation may continue with whitespace until the token limit. The API checks that “JSON” appears in the context. Azure also tells clients to inspect a length-related finish reason because a response can be cut off (OpenAI JSON mode notes, Azure JSON mode troubleshooting).

Treat JSON mode as a transport-format control, not a data contract. Pair it with a runtime schema validator, output limit, refusal and truncation handling, and a bounded repair strategy.

What schema-constrained output guarantees

Structured Outputs uses a supplied JSON Schema, or a provider-specific subset of it, to constrain generation. A successful normal response should therefore contain the required properties, use the declared types, obey supported enums, and reject unspecified properties when the schema requires that behavior.

That stronger contract lets typed services and UIs consume a known shape without guessing which keys arrived. It can also eliminate repair calls caused by the wrong shape.

Schema compliance does not prove that the content is correct. A model can emit a valid priority: "high" for a low-priority ticket, put an incorrect date in a correctly typed string, or produce a made-up identifier that matches a pattern. Google’s official guidance says structured output guarantees syntactically correct JSON but not semantically correct values, and tells developers to validate those values in application code (Gemini Structured Outputs documentation).

OpenAI documents refusals and token-limit truncation as cases requiring explicit handling even with Structured Outputs. Anthropic documents invalid-output and schema-complexity cases. Your integration therefore needs two branches:

  1. a normal, schema-compliant result that proceeds to semantic validation; and
  2. a non-result state such as refusal, truncation, provider error, or schema-compilation error.

Before any side effect, check authorization, resource ownership, ranges, state transitions, idempotency, and policy. The guide to machine gates for AI output covers this boundary.

Provider support and schema limits

“Supports JSON Schema” rarely means every keyword in every draft of the specification. Check the exact model, endpoint, request field, supported subset, and incompatibilities before designing a shared abstraction.

Provider pathDocumented position on 2026-07-23Important constraints
OpenAI APIStructured Outputs is available on newer models starting with GPT-4o; older models may use JSON modeSupports a subset of JSON Schema. Unsupported strict schemas return an error. OpenAI lists supported types including objects, arrays, enums, and anyOf, while excluding some composition keywords.
Azure OpenAIJSON mode and Structured Outputs are separate response formats; Microsoft recommends Structured Outputs when possibleAzure documents model and API-version compatibility, requires every field to be listed as required for its supported subset, requires additionalProperties: false on objects, and does not support Structured Outputs in bring-your-own-data scenarios. Its documented maximum is five nesting levels and 100 total object properties.
Google Gemini APIStructured output accepts a schema and JSON response MIME type through provider-specific request fieldsGemini supports a subset of JSON Schema and warns that very large or deeply nested schemas may be rejected. Supported keywords and models vary by API path.
Anthropic Claude APIJSON outputs constrain final responses; strict tool use applies schemas to tool inputsSchema grammars are compiled and cached. Anthropic documents limits including 20 strict tools, 24 optional parameters, and 16 union-type parameters across strict schemas in one request. Citations and message prefilling have documented incompatibilities with JSON outputs.

Sources: OpenAI supported schemas, Azure Structured Outputs, Gemini Structured Outputs, and Anthropic Structured Outputs.

For a multi-provider system, keep a canonical application schema, then generate and test an adapter for each endpoint. Fail during deployment if an adapter uses unsupported keywords rather than discovering the mismatch in production.

Version schemas independently from prompts, store a schema ID with each response, and test old fixtures against migrations.

Validation and retry behavior

A production pipeline should distinguish four failure layers:

  1. Transport: timeout, rate limit, authentication error, or provider outage.
  2. Completion: refusal, truncation, cancellation, or another non-normal finish state.
  3. Structure: invalid JSON or mismatch with the application schema.
  4. Semantics: structurally valid data that violates business rules or lacks evidence.

JSON mode mainly reduces layer three’s syntax failures; you still validate the whole schema. Structured Outputs reduces both syntax and supported-schema failures on normal completions, but layers one, two, and four remain.

Use a local validator even when the provider promises schema adherence. It catches adapter mistakes, regressions, and cached responses produced under an older schema. Record the provider, model identifier, schema version, finish state, validator result, and retry reason without unnecessarily logging sensitive content.

Retries should be narrow. Retry transient transport failures with backoff. For JSON mode, a repair attempt can include concise validator errors, but cap the loop. After a deterministic schema rejection, correct the schema rather than repeating the request. Do not retry refusals as malformed JSON. Make side effects idempotent.

A provider fallback also needs contract tests. If the fallback supports only JSON mode, the application must re-enable full shape validation and perhaps a repair path. The LLM fallback strategy explains why a provider switch must preserve behavior, not merely return HTTP 200.

Latency and cost trade-offs

Structured Outputs can lower end-to-end cost by preventing repair calls and downstream exceptions. It can also add request tokens, provider-side processing, or grammar compilation. The net result depends on schema complexity, failure rate, traffic, and provider implementation.

OpenAI documents additional latency on the first request with a new schema, followed by no such added latency for later requests using the same schema; one section limits the note to fine-tuned models while another states it more generally, so recheck the exact scope for the chosen model before making a latency commitment. Anthropic is more explicit: the first use compiles a grammar, compiled grammars are cached for 24 hours after last use, and changes to schema structure or the set of tools invalidate that cache. Anthropic also says its injected format instruction slightly increases input-token usage (OpenAI latency notes, Anthropic compilation and token-cost notes).

Do not assume JSON mode is cheaper because it has fewer controls. Measure the complete transaction:

effective cost per accepted record
= (model calls + repair calls + validator compute + failed-work handling)
  / accepted records

Benchmark representative schemas and inputs. Separate cold- and warm-schema latency, track p50 and p95 time to an accepted result, and count retries, refusals, truncations, rejections, tokens, and operator interventions. Per-token price alone cannot identify the cheaper mode.

Decision by use case

Choose according to the strictest downstream consumer:

  • Data extraction into a database: Prefer Structured Outputs, then enforce domain rules and database constraints.
  • Classification into a fixed taxonomy: Prefer Structured Outputs with enums; still test labeling accuracy.
  • A final response rendered by a typed UI: Prefer Structured Outputs, with a safe renderer for every permitted variant.
  • Tool or function arguments: Use strict function calling when available, not a final JSON response that your code treats as an implicit tool request.
  • Exploratory analysis with a flexible object: JSON mode can be sufficient if the object is inspected by a person or validated before use.
  • Older-model or endpoint compatibility: Use JSON mode plus local schema validation and a bounded repair path.
  • Multi-provider fallback: Use Structured Outputs where each provider supports the canonical contract; otherwise place a full validator at the shared boundary and test every adapter.
  • High-risk side effects: Neither mode is an authorization control. Require deterministic checks and, where appropriate, human approval after validation.

For a new typed integration, start with Structured Outputs and the smallest schema that expresses the real contract. Flatten unnecessary nesting, encode optional values in the provider-supported way, and keep response schemas separate from tool schemas. Add JSON mode only when compatibility requires it.

For an existing JSON-mode integration, migration is worthwhile when repair retries, missing keys, or type errors are measurable. Run both paths against a fixed evaluation set, compare accepted-result latency and cost, then switch behind a feature flag. If you are also moving OpenAI endpoints, use the Responses API migration guide.

The durable rule is: use generation constraints for shape, application code for meaning, and authorization controls for action. Structured Outputs narrows the model’s freedom at the right boundary; it does not turn probabilistic content into trusted business data.

Before connecting a structured response to application logic, verify that:

  1. the required structure is defined outside the prompt;
  2. the selected generation mode enforces the schema rather than merely encouraging it;
  3. runtime code validates every response;
  4. unknown, missing, and ambiguous values have honest representations;
  5. source evidence and model interpretation remain separate;
  6. deterministic systems enforce permissions and business rules;
  7. validation failure can end safely without invented defaults;
  8. producers and consumers record the schema version; and
  9. consequential actions are logged with their actual outcomes.

Frequently asked questions

Does JSON mode guarantee the same keys every time?

No. JSON mode targets valid JSON syntax, not a specific schema. The response can still omit a required key, change a key name, return the wrong type, or add an unexpected property. Validate the full object locally and use a bounded repair path if Structured Outputs is unavailable.

Do Structured Outputs eliminate validation code?

No. They reduce shape failures for supported schemas on normal completions, but your application must still handle refusals, truncation, provider errors, and business-rule failures. A schema can prove that amount is a number; it cannot prove that the amount is authorized or correct.

Should function calling or Structured Outputs be used for tool arguments?

Use the provider’s function- or tool-calling interface when the model needs to request an action. Use a structured response format when the final answer itself must have a fixed shape. If tool execution is failing despite valid-looking JSON, see how to fix LLM tool-calling errors.

Is Structured Outputs always slower or more expensive than JSON mode?

Not necessarily. Schema transmission and first-use grammar compilation can add overhead, while fewer repair calls can reduce total latency and cost. Compare cold and warm schemas, accepted-result latency, retry rate, and total tokens on your own workload; the LLM API benchmarking guide provides a repeatable measurement framework.

Can one JSON Schema be reused unchanged across providers?

Usually not. Providers support different JSON Schema subsets, request fields, model combinations, and complexity limits. Keep one canonical application contract, but compile and test a provider-specific adapter for each API.

Primary sources

The following first-party documentation was rechecked on 2026-07-24: