Correction, October 2026: An earlier version of this article claimed numerical results from a six-model benchmark and pointed to a test harness that I cannot provide or verify. I have withdrawn those figures and the comparative recommendations built on them. This version contains a runnable example and distinguishes what a schema proves from what it does not.

If an agent sends its output to code, valid JSON matters. But the phrase structured output hides three separate questions: Can you parse the bytes? Do the fields satisfy the contract? And are the values actually supported by the input? A model can pass the first two and fail the third.

Start with the contract

Imagine a program extracting information from a short report. It needs a priority, a count of affected users, and a summary. Here is a small Python example using Pydantic 2; install it with python -m pip install 'pydantic>=2,<3' if you want to run the snippet:

from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, ValidationError

class Extraction(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")
    priority: Literal["low", "medium", "high"]
    affected_users: int = Field(ge=0)
    summary: str

source = "The report says no users were affected. Priority is low."
shape_valid_but_wrong = (
    '{"priority":"low","affected_users":12,'
    '"summary":"No users were affected."}'
)
record = Extraction.model_validate_json(shape_valid_but_wrong)
print(record.affected_users)              # 12: the schema accepts it
print(record.affected_users == 0)         # False: it contradicts the report

try:
    Extraction.model_validate_json(
        '{"priority":"low","affected_users":"twelve",'
        '"summary":"No users were affected."}'
    )
except ValidationError:
    print("Type validation rejected the string count")

The first object has the required keys and types, yet 12 contradicts the sentence it was supposedly extracted from. Pydantic is doing its job: validating the declared contract, not reading the report. The second object fails because its count is a string rather than an integer. With extra="forbid", unexpected keys fail too. Pydantic’s strict-mode documentation explains why the default coercion rules deserve attention.

This code is an illustration, not a result from a language-model benchmark. There is no model in the loop.

Where a grammar fits

A constrained decoder can restrict the next token so the generated text fits a grammar or schema. Outlines documents JSON-schema output types, and llama.cpp documents its grammar facilities. Those are useful tools when a parser expects a particular shape. They do not verify that an extracted number is present in the source, that a summary is faithful, or that a priority is justified.

There is another distinction: a prompt saying “return JSON” is not the same as constrained decoding. A prompt is a request to the model; a grammar restricts which tokens the decoder can emit. You should record which mechanism you tested. You should also validate the completed output in application code rather than treating the generation mechanism as the final authority.

What an actual model comparison would need

To compare models fairly, publish the input prompts, expected answers, schema, exact model files and quantizations, inference engine and version, decoding settings, validation code, and raw outputs. State the number of trials and how ambiguous answers are scored. Separate at least these failure classes:

  1. Parse failure: the output is not valid JSON.
  2. Contract failure: it parses, but required fields, types, ranges or allowed values fail validation.
  3. Evidence failure: it passes the contract, but a value is unsupported or contradicted by the source.
  4. Operational failure: the request times out, truncates, or otherwise fails before a complete answer.

A first-try pass rate is useful only when its numerator, denominator and failure definition are explicit. A schema-valid rate is not the same metric as a factually correct extraction rate. If constrained decoding eliminates most syntax errors, evaluation should move to evidence quality instead of celebrating braces and commas.

A practical pipeline

Use a small, explicit schema; constrain generation where your chosen backend supports it; parse and validate again at the application boundary; then check consequential fields against the source or a trusted system. Make a failed check visible and retry or escalate according to its impact. Never silently convert “well-formed” into “true.”

I cannot recommend one local model over another for this task without a reproducible comparison. The useful takeaway is narrower and sturdier: structure is a property of an output contract; correctness is a claim about the world. Test both.