Structured Output with Local LLMs: When Valid JSON Is Not Enough

Getting structured JSON out of a cloud LLM is straightforward — the provider handles schema enforcement server-side. Doing the same with a locally hosted model is where production pipelines quietly break. Shuai Guo's August 2026 Towards Data Science walkthrough covers the full implementation using Gemma 4's 4B-parameter variant (gemma4:e4b), Ollama as the serving runtime, and Pydantic for schema definition and validation — then exposes the failure mode that catches most developers off guard.

The Core Implementation Pattern

The stack is three components: Pydantic defines the expected schema, Ollama enforces it during generation via its format argument, and model_validate_json() parses the raw response back into a typed Python object. The bridge between Pydantic and Ollama is a single line — schema.model_json_schema() — which converts a Pydantic model into the JSON schema Ollama passes to the constrained decoding layer. The ollama.chat() call also sets think="medium", enabling Gemma 4's chain-of-thought reasoning during generation.

The schema in the case study is nested: a top-level SchedulingContext holds shared household facts (current time, concurrent device limit, electricity tariffs across multiple time bands) plus a list[DeviceToSchedule], where each entry carries duration_minutes, energy_kwh, earliest_start, and finish_by as typed fields. A custom ClockTime annotated type enforces HH:MM string format with min_length=5, max_length=5. The nesting is exactly what stresses small models.

The Failure Mode: Valid Shape, Wrong Content

When Gemma 4 receives the full household context and the complete SchedulingContext schema in a single call, it returns valid JSON that passes Pydantic validation — and includes the wrong device. The robot vacuum appears in devices_to_schedule despite the source notes stating it completed its kitchen pass at 16:10 and requires no further scheduling that day.

The schema was satisfied. The content was wrong. This is the central finding: constrained decoding enforces structure, not semantic accuracy. A 4B-parameter model asked to simultaneously determine scheduling scope, extract per-device facts, map them to the correct fields, and assemble a nested object is handling too many reasoning steps in one forward pass. Schema enforcement provides no protection against that.

This pattern applies broadly — any agentic workflow that relies on local models for structured data extraction will hit this ceiling when schemas grow complex enough to require multi-step reasoning in a single generation.

The Decomposition Fix

The practical remedy is task decomposition: split the single structured call into two sequenced calls, each targeting a simpler schema.

Step 1 uses a minimal SchedulingScope schema with only two fields — focus_device: str and device_names_to_schedule: list[str] — and asks the model only to identify which devices still require scheduling. With this reduced schema, Gemma 4 correctly returns the dishwasher, EV charger, and washing machine, excluding the robot vacuum entirely.

Step 2 passes the device list from Step 1 alongside the original source material and asks the model to fill the full SchedulingContext. Because scope determination is already resolved, the model's generation budget in Step 2 is spent entirely on fact extraction and field mapping.

The final output from the decomposed approach is fully correct:

The robot vacuum is absent, all numeric fields match the source notes, and personal scheduling details not needed downstream are dropped.

Practical Tradeoffs

Decomposition costs an additional inference round-trip, which matters when latency budgets are tight. The tradeoff is that each individual call becomes substantially more reliable, and the intermediate SchedulingScope output can be validated and logged — providing an auditable checkpoint before the more expensive fact-extraction pass. For pipelines where the structured output feeds a cloud model for downstream reasoning (as in this case study's intended architecture), the extra local call is almost always cheaper than debugging silent content errors in production. The approach also aligns with the prompt optimisation discipline of reducing per-call cognitive load rather than compensating for it with larger prompts.

The broader signal is that small local models are capable structured-output engines, but their effective schema complexity ceiling is lower than their JSON validity rate implies. Passing Pydantic validation is a necessary condition for a usable response — it is not a sufficient one. Developers building extraction pipelines on top of sub-10B-parameter models should treat decomposition as a first-class design pattern rather than a fallback, sizing each schema to match the reasoning budget of the model rather than the convenience of the caller.