Your LLM Returns Almost Valid JSON, and Almost Is Breaking You

There is a specific kind of production incident that only happens with LLMs. Everything is green. No exception, no stack trace, no failed request in the dashboard. But a downstream job quietly wrote a null where a customer's plan tier should have been, and three days later billing is wrong for a few hundred accounts. You trace it back and find the cause: the model returned JSON that parsed perfectly and meant nothing useful.

This is the trap of structured output. The moment an LLM stops talking to a human and starts feeding a real workflow, its response is no longer prose you can eyeball. It is data that another system consumes without thinking. And "mostly valid" data is worse than clearly broken data, because clearly broken data fails loudly and mostly valid data fails silently, later, somewhere else.

I want to walk through why this happens and how to stop it, with the pattern I reach for on every project that pipes model output into code: a validation gateway that owns the schema, parses the output, repairs what it can, and refuses to pass anything downstream that is not the exact shape you asked for.

The business problem hiding inside a formatting bug

Teams treat structured output as a formatting concern. It is not. It is a data integrity concern, and the cost lands in places that have nothing to do with the model.

Say you are extracting fields from uploaded invoices: vendor, total, currency, due date. The extraction works in your tests. In production, one invoice in fifty comes back with the total as "1,240.00" instead of 1240.0, because the model helpfully formatted it like a human would. Your parser accepts the string, your database column is a string too, and now your finance report sums a column that is half numbers and half formatted text. Nobody gets an error. Somebody gets a wrong number.

The failure modes stack up fast once you look:

• Right syntax, wrong shape. Valid JSON, but the field is named amount instead of total, or nested one level deeper than your code expects.
• Right shape, wrong type. The number arrives as a string, the boolean as "yes", the array as a single object because there was only one item.
• Right type, invalid value. An enum field comes back as "USDollar" when your system only knows "USD".
• Everything valid, answer invented. The cleanest failure of all. Perfect JSON, plausible values, entirely made up because the source document did not actually contain a due date.

Only the last one needs a smarter model. The first three are your problem to solve in code, and they are the ones that cause most of the pain because they slip through anything that only checks whether the JSON parses.

Why the obvious fixes are not enough

The first instinct is to ask nicely. Put "respond only with valid JSON matching this shape" in the prompt, add an example, and hope. This gets you to maybe eight or nine times out of ten on a good model. That sounds high until you run it ten thousand times a day and do the arithmetic on the failures.

The second instinct is JSON mode, the provider flag that forces syntactically valid JSON. This is a real improvement and you should use it. But read what it actually promises. It promises the output parses. It says nothing about field names, types, enum values, or whether the content is true. I have watched teams turn on JSON mode, see the parse errors vanish, and conclude the problem is solved, right up until the first null where a required value should have been.

The third instinct, native structured output with schema-constrained decoding, is genuinely strong. When the provider constrains the tokens the model can emit so the result has to match your schema, the shape and type failures largely disappear. Use it wherever you can. But it still does not cover two things. It does not judge whether the values are correct, and it does not cover the providers and older models where the strict schema flag is best effort rather than enforced. Function calling has the same asterisk. The schema you attach to a tool is a strong hint to the model, and on many providers it is explicitly not a hard constraint on the arguments you get back.

So the honest summary is this. Every layer helps, and no single layer is a contract. The contract has to live in your code, because your code is the only place you fully control.

The pattern: a validation gateway

The move that makes this reliable is to stop scattering json.loads and hopeful field access across your codebase, and instead route every structured call through one module. That module owns the provider call, the schema, parsing, validation, repair, and metrics. Downstream code gets back a typed object or an explicit failure. It never sees raw model text.

Start with the schema as the single source of truth. In Python, Pydantic is the natural home for this, because the same definition documents the shape, generates the JSON Schema you send to the model, and validates what comes back.

from enum import Enum
from datetime import date
from pydantic import BaseModel, Field

class Currency(str, Enum):
    usd = "USD"
    eur = "EUR"
    gbp = "GBP"

class Invoice(BaseModel):
    vendor: str = Field(min_length=1)
    total: float = Field(gt=0)          # a float, not a formatted string
    currency: Currency                  # must be one of the three, nothing else
    due_date: date | None = None        # optional, but typed when present

# This is what you send to the model as the schema, and the exact
# same object is what you validate the response against later.
schema = Invoice.model_json_schema()

Notice what the schema is already enforcing before the model runs. The total has to be a positive number, so a formatted string fails validation instead of sneaking into your database. The currency has to be one of three literal values, so "USDollar" is rejected at the door. The due date is allowed to be absent, which is honest, and typed as a date when it is present. You have encoded the rules once, in the place that also checks them.

Now the gateway itself. The job is narrow: call the model, parse, validate, and return a typed result or a structured failure. No downstream caller should ever have to think about any of this.

import json
from pydantic import ValidationError

class ParseFailure(Exception):
    def __init__(self, reason: str, raw: str):
        self.reason = reason
        self.raw = raw            # keep the raw text for logging, never return it

def parse_invoice(raw_text: str) -> Invoice:
    # Step 1: does it even parse as JSON?
    try:
        data = json.loads(raw_text)
    except json.JSONDecodeError as e:
        raise ParseFailure(f"not valid json: {e}", raw_text)

    # Step 2: does it match the schema, with the right types and values?
    try:
        return Invoice.model_validate(data)
    except ValidationError as e:
        # e carries the exact field-level errors, which we reuse for repair
        raise ParseFailure(e.json(), raw_text)

Two failure gates, two different reasons. The first catches broken syntax. The second catches everything JSON mode lets through: the wrong field name, the string total, the invalid currency. And crucially, the ValidationError is not a dead end. It tells you exactly which field failed and why, which is the raw material for the step most people skip.

Repair beats retry

When validation fails, the naive response is to call the model again with the same prompt and hope for a better roll. Sometimes that works. But you already have something far more useful than another roll: a precise description of what was wrong. Hand that back to the model and ask it to fix only that, and the success rate on the second attempt is dramatically higher than a blind retry, at the cost of exactly one more call.

def get_invoice(document: str, model_call, max_repairs: int = 1) -> Invoice:
    prompt = build_extraction_prompt(document, schema)
    raw = model_call(prompt)

    for attempt in range(max_repairs + 1):
        try:
            return parse_invoice(raw)
        except ParseFailure as fail:
            if attempt == max_repairs:
                # out of repair budget: fail loudly, do not guess
                log_parse_failure(fail)
                raise

            # feed the exact validation error back and ask for a targeted fix
            repair_prompt = (
                f"Your previous response did not match the required schema.\n"
                f"The errors were:\n{fail.reason}\n\n"
                f"Return corrected JSON for this schema only:\n{json.dumps(schema)}\n\n"
                f"Your previous response was:\n{fail.raw}"
            )
            raw = model_call(repair_prompt)

The repair budget is the important detail. One or two attempts, then a hard stop. Without a cap, a document the model simply cannot extract correctly will loop and burn tokens forever, which is the same failure I described in the piece on tool call reliability: treating a permanent failure as if another attempt will fix it. Bound the loop, and when the budget runs out, fail honestly rather than returning a half-built object.

That last point deserves emphasis. When repair fails, the gateway raises. It does not return an Invoice with empty fields to keep the pipeline moving. A loud failure that a human sees beats a silent one that corrupts data three systems away. The whole value of the gateway is that it is the one place allowed to say no.

The same idea in TypeScript

None of this is Python-specific. If your stack is TypeScript, Zod plays the exact role Pydantic does, schema and validator in one definition, and the gateway shape is identical.

import { z } from "zod";

const Invoice = z.object({
  vendor: z.string().min(1),
  total: z.number().positive(),                 // rejects "1,240.00"
  currency: z.enum(["USD", "EUR", "GBP"]),      // rejects anything else
  dueDate: z.string().date().nullable(),
});

type Invoice = z.infer<typeof Invoice>;

export function parseInvoice(raw: string): Invoice {
  let data: unknown;
  try {
    data = JSON.parse(raw);
  } catch (e) {
    throw new ParseFailure(`not valid json: ${e}`, raw);
  }

  const result = Invoice.safeParse(data);        // no throw, inspect the result
  if (!result.success) {
    // result.error.issues is the field-level list you feed back for repair
    throw new ParseFailure(JSON.stringify(result.error.issues), raw);
  }
  return result.data;
}

Same two gates, same typed result, same repair material in result.error.issues. The point is the architecture, not the library. One schema, one gateway, two validation steps, a bounded repair, and a loud failure.

Tradeoffs and the things that bite

Latency is real, so match the effort to the stakes. Every layer you add costs milliseconds, and a repair round costs a whole extra model call. A low stakes extraction that feeds a dashboard does not need the same paranoia as a field that drives billing. Put the repair loop and the strict validation where wrong data actually hurts, and keep the cheap path cheap.

Do not over-constrain the schema and cause your own failures. If you mark a field required that the source genuinely sometimes lacks, you will manufacture validation failures for documents that were fine. The invoice with no due date is not an error, it is an invoice with no due date. Model optional as optional, and reserve required for fields that truly must exist.

Log the raw text, never return it. When a parse fails, you want the exact model output in your logs to debug it. What you do not want is that raw text leaking downstream as a fallback. Keep it inside the gateway for observability and let the typed contract be the only thing that crosses the boundary.

Validation logic deserves tests like any other contract. Your schema and repair rules will be wrong about some edge case at first, and that is fine as long as they live in one testable place. Pin the behavior down with evals in your CI so a well meaning tweak to a field or an enum does not quietly start rejecting good data, or worse, start accepting bad data you had already learned to catch.

The takeaway

LLM output that feeds code is not text, it is a data contract, and a contract that only "mostly" holds is not a contract. JSON mode and native structured output are worth using, but neither one is the guarantee your downstream systems are quietly assuming. The guarantee has to live in your code, in one place, because that is the only boundary you fully own.

Route every structured call through a gateway. Let one schema define the shape, the types, and the valid values. Parse in two gates, syntax then schema. Repair with the actual validation error instead of a blind retry, and bound the repair so it cannot loop. And when it still fails, fail loudly, because a visible error is cheap and a silent one shipped straight into your billing system is not. The model does not have to get smarter for this to work. Your boundary has to get stricter.

If your pipeline works in testing and keeps letting almost-right data through in production, book a consultation call and we will make the boundary between your model and your systems something you can actually trust.