Shape ≠ Meaning

In 2024, OpenAI tested a top model on complex JSON. Asked nicely, it matched the schema less than 40% of the time. Today the shape is enforced, and the reply always parses. I still check every one.

The short version

• Code doesn't forgive. When an agent hands JSON to another program, a friendly sentence before it, a missing field, or "two weeks" where a number goes is a crash somewhere downstream.
• Move 1: a schema for the shape. Fields, types, what's required, no extra keys. With structured outputs, the API enforces it by constrained decoding: at every token, anything that can't fit is masked out, so the reply always parses.
• The twist: shape isn't meaning. {"mileage": -40} is a perfectly valid integer. The docs also list fine print: a refusal or a max_tokens cut-off can break the shape, and some keywords, like minimum, aren't enforced by the API.
• Move 2: validate in your code, every time: ranges, allowed values, fields that must agree.
• Moves 3 and 4: retry once, with the error, then fail loud. If the retry fails too, log it, alert, and fall back to something safe. In my Service Bay design, that's the next scripted question: bad data never reaches the conversation.

The failure comes from my design for a car-service intake system, Four Agents Walk Into a Service Bay, which is my own recreation, not Nissan's system. Its "Bad JSON" button plays the same retry-then-fallback.

Further reading

• Structured outputs (Claude docs) — constrained decoding, and the "Invalid outputs" fine print.
• OpenAI Launches Structured Outputs (PureAI, August 2024) — OpenAI's <40% and 100% results, reported the day after the launch.
• Understanding JSON Schema — how to write the contract.

Next episode: Rate limiting. Previously: Only the Neighbours Move. All episodes →