Structured output

TL;DR

• Structured output forces the LLM to produce valid JSON, XML, or any schema-compliant format by constraining the token sampler at generation time.
• OpenAI's structured output mode and Anthropic's tool-use with input schemas achieve this via constrained decoding: the model physically cannot emit an invalid token.
• For open-source models, libraries like Outlines and Instructor intercept the logit distribution and mask out tokens that would violate the schema.
• Structured output eliminates the "parse and retry" anti-pattern that adds 200-500ms latency and wastes tokens on malformed responses.
• JSON mode alone is not enough for production. You need schema enforcement (specific fields, types, enums) not just "some valid JSON."

The problem it solves

Your invoice extraction pipeline looks clean in staging. You prompt the model to return JSON with invoice_number, amount, and currency, and it does. Then you deploy and JSONDecodeError starts appearing in your logs.

The model returns something like "Here is the JSON you requested:\n```json\n{...}\n```" for about 2% of inputs. Another 1.5% return amount as the string "149.99" rather than a float, so downstream arithmetic throws a TypeError. Your response.choices[0].message.content parser expected raw JSON and received a markdown code block.

# What you asked for (and received in staging):
{"invoice_number": "INV-2024-001", "amount": 149.99, "currency": "USD"}

# What production returned at 2am on a Friday:
"Here is the JSON you requested:\n```json\n{\n  \"invoice_number\": \"INV-2024-001\",\n  \"amount\": \"149.99\",\n  \"currency\": \"USD\"\n}\n```\nLet me know if you need any adjustments!"

The textbook fix is a parse-and-retry loop: catch the error, append "please return valid JSON only" to the prompt, and call the model again. At 100K documents per day with a 3% failure rate, that is 3,000 retries generating roughly 300K wasted tokens daily. Each retry adds 200-500ms of latency.

The root cause is that prompt instructions operate at the semantic level but token generation is stochastic. No matter how carefully you word "output only valid JSON", some fraction of generations will include markdown fences, prose preambles, or type coercions. You cannot fix this with better prompting alone.

What is it?

Structured output is a generation-time constraint that forces the model to emit only tokens that keep the partial output schema-valid at every step.

Think of it like the difference between a blank essay and a tax form. A blank page lets the model say anything, but the reader has to parse structure from prose. A fill-in-the-blank form guarantees the fields exist in exactly the right places and in the right format. Structured output turns the model's blank page into a fill-in-the-blank form at the token level, not after the fact.

The constraint is enforced by a finite-state machine that tracks which schema states are open at each step. Any token that would advance the output into an invalid state is masked to negative infinity before sampling. The model still applies all of its learned knowledge, but only within the token paths that keep the schema valid.

How it works

Token-level constrained decoding

The model generates output one token at a time. Without constraints, the sampler picks from all ~100,000 vocabulary tokens according to the model's probability distribution. Constrained decoding adds a gating step between the logit computation and the sampler.

Before sampling each token, a finite-state machine (FSM) derived from the JSON schema computes the set of valid next tokens: tokens that, if appended to the partial output so far, would keep it on a valid path toward a schema-complete output. Any token outside that set is masked:

logits'[k] = logits[k] if token k is valid per FSM, else -∞

The softmax then operates only over valid tokens. The model's relative preferences among valid tokens are preserved. If three tokens are all valid next steps, the model still picks the one it finds most likely given the context.

The FSM compilation happens once per schema and is typically cached at startup. The per-token masking step adds 10-40ms of total overhead per full generation, not per individual token.

JSON mode vs schema-enforced mode

These two terms are routinely conflated, and the distinction matters in production.

JSON mode (available from OpenAI, Anthropic, and most providers) guarantees that the output is syntactically valid JSON. The model will not emit markdown fences, prose preambles, or malformed brackets. It can still return {}, omit required fields, return numbers as strings, or include extra fields you did not ask for. JSON mode solves the "garbage text" problem but not the "wrong schema" problem.

Structured output (OpenAI's strict: true, Anthropic tool_use with a defined input schema, Google's response_schema) enforces your exact schema: specific field names, specific types, enum values, and required vs. optional distinctions. If you define amount: float, the model cannot return "149.99". If you define currency: Literal["USD", "EUR", "GBP"], the model cannot return "Canadian dollars".

I have seen teams ship JSON mode to production, watch the parse failure rate drop from 3% to 0.5%, and declare victory. That remaining 0.5% is worse: it fails silently when invoice_number is missing, because the response is technically valid JSON and no exception fires.

Open-source: Outlines, Instructor, and SGLang

For self-hosted models, constrained decoding is implemented in the inference library rather than the model weights. Three tools are in common use today.

Outlines converts a Pydantic schema (or a regex pattern) into a token-level FSM, then injects the logit mask at each generation step. It works with any HuggingFace-compatible model and adds roughly 15-40ms of overhead per generation call.

import outlines
from pydantic import BaseModel
from typing import Literal

class Invoice(BaseModel):
    invoice_number: str
    amount: float
    currency: Literal["USD", "EUR", "GBP"]
    line_items: list[str]

# Load any HuggingFace-compatible model
model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

# Outlines compiles the Invoice schema to a token-level FSM automatically
generator = outlines.generate.json(model, Invoice)

prompt = "Extract invoice data: INV-2024-001, $149.99 USD, office supplies"
invoice: Invoice = generator(prompt)
# invoice is a fully typed Pydantic object. No parsing, no try/except.
print(invoice.amount)    # 149.99 (float, not string)
print(invoice.currency)  # "USD" (validated enum, not "US dollars")

Instructor wraps the OpenAI and Anthropic clients and adds retry-and-validate logic. It does not use constrained decoding: it calls the model, tries to parse the response against your Pydantic schema, and retries with the validation error appended if parsing fails. Instructor is the right choice when you need structured output from a hosted model that lacks native strict mode and when you can absorb the retry latency cost.

SGLang is a high-throughput inference engine that implements constrained decoding for production serving. Its regex-guided generation is particularly fast because it pre-compiles constraint masks at model-load time rather than per request.