JSON mode isn't making your model dumber. Your schema might be.
I tested constrained decoding, schema visibility and field order separately. The tax comes from how you write the schema.
If you've built anything on top of an LLM, you've probably turned on structured output. You hand the server a JSON schema, it uses constrained decoding to block any token that would break it, and you get back JSON that always parses. No more regex, no more "Sure! Here's the JSON you asked for:".
There's a nagging worry that this makes the model worse. A 2024 paper reported that format restrictions hurt reasoning, a well-known rebuttal argued it was the prompts, and the debate never quite settled.
I think part of the reason is that "JSON mode" changes three things at once:
- the decoder is constrained,
- the prompt usually shows the schema,
- the schema fixes the order the model answers in.
So I tested them separately. Three Qwen3 models (1.7B, 4B and 8B), three tasks (math word problems, news topic classification and named-entity extraction), 300 examples each, 12,600 requests on my own GPU.
Constraining is nearly free
On GSM8K math problems, the 8B model scored 92.7% answering in plain text and 92.3% answering in schema-enforced JSON. The 4B model: 92.3% vs 89.7%. Both differences are within noise. The JSON answers also used less than half the tokens, because the reasoning inside a JSON string comes out much terser.
The smallest model, 1.7B, did lose about 12 points. So there's a tax, but only at the bottom of the size range.
Field order is the real tax
Here's the schema most people would write without thinking:
{"answer": 42, "reasoning": "..."}
Answer first, explanation after. With that order, accuracy fell from 92% to around 30%, at every model size. That's exactly as bad as asking for the answer with no reasoning at all, which makes sense: the model has already committed to a number before it starts "thinking", so the reasoning can't change anything.
It gets weirder. Some of those answers never ended. The model committed to a wrong number, then wrote the same justification over and over until it hit the length limit. 55 of the 56 answers that got cut off were loops like that.
This isn't a new discovery (Dylan Castillo showed the same thing on GPT-4o-mini in early 2025), but it's worth
repeating because it's so easy to get wrong. Put reasoning first. Always.
Show the model the schema
Constrained decoding will force the right keys even if the model has never seen the schema. You might think that's good enough. It isn't: enforcing a schema the model wasn't shown cost 9 to 15 points on math. The model doesn't know a reasoning field is coming, so it writes less of it.
Unconstrained JSON breaks small models
The worst option for small models was asking for JSON without enforcing it. The 1.7B model produced unparseable output for about half of its answers. Two failure modes stood out:
- It wrote its working in LaTeX math and then produced JSON with invalid backslash escapes.
- It copied the JSON Schema itself. Instead of
{"label": "Business"}it returned{"type": "object", "properties": {"label": "Business"}, ...}, the schema's own structure with the answer inside. It knew the answer; it just copied the wrong thing.
Constrained decoding removes both problems.
One exception
Showing the schema isn't always better. On entity extraction, the 4B and 8B models that could see the schema dutifully
filled the catch-all misc field with race times, scores and dates. Hidden schema, fewer false positives, about 10
points higher F1. The schema is part of the prompt, and an empty array in it reads like a request to fill it.
The short version
- Enforce your schema. It costs little or nothing at 4B and up, and it saves small models.
- Put reasoning fields before answer fields.
- Show the schema in the prompt, except maybe for extraction with catch-all fields, where you should measure both.
- If you parse JSON from a small model without constraints, expect it to hand you its own schema back.
Everything is open: the prompts, the 12,600 responses and the scoring, so you can check any of these numbers yourself.
Paper: PDF · DOI 10.5281/zenodo.23754532
Code and data: github.com/alleyxd/SchemaTax