Constrained Inference: Compare Accuracy, Coverage and Serving Conditions

Use a constrained inference design when its output contract fits the task, then measure the errors it can still make. A valid schema or reproducible routing decision does not establish that the returned information is correct.

Correction, 21 October 2026: The earlier article called learned extraction stages deterministic, overstated OpenCodeReview's cross-file limitations and omitted concurrency from Jina's throughput figure. It also presented an extraction score too broadly. Those claims are corrected below. AI Mastery has not reproduced these vendor evaluations.

OpenCodeReview combines engineering and model analysis

Alibaba's repository describes a hybrid review system with deterministic file handling and positioning around LLM analysis. Its documentation explicitly permits full-file reads, codebase searches and inspection of other changed files.

The project reports roughly one-ninth the tokens of a Claude Code comparison with the same underlying model, higher precision and F1, and lower recall. Its benchmark comprises 200 pull requests across 50 repositories and 10 languages. These are project-reported results for that evaluation, not a guarantee for a different model or repository.

The earlier claim that deterministic dispatch makes cross-file analysis impossible is therefore unsupported. Lower recall also does not identify its own cause. Establishing that a routing choice caused missed defects would require a comparison changing that choice while holding the rest of the system fixed.

For your own repository, record which defects were missed and whether the relevant context was available. Distinguish an intentionally suppressed comment from a failure to inspect a file or understand its effect.

Jina OCR still generates text

The jina-ocr-v1 card describes a DeepSeek-OCR-based parser with a 3B mixture-of-experts decoder, approximately 570M active parameters per token and a speculative FastMTP head. Its reported 2.57 pages per second uses an A100 at concurrency 32 on olmOCR-Bench.

FastMTP drafts up to three tokens, then uses greedy verification to preserve the verifier's output sequence. This accelerates an autoregressive system; it does not eliminate generation or prove the output matches the document.

Page throughput under concurrent load is a different measurement from the time an individual user waits for one document. Measure both, including document size, queueing, parsing and any repair after decoding. The card lists CC BY-NC 4.0; review the actual model terms before selecting a deployment.

GLiFormer separates extraction from record assembly

GLiFormer's card describes a shared encoder with task heads and a decoder assembling source-grounded fields and relationships. Predictions depend on the input, schema and thresholds. Learned span or relationship predictions should not be described as deterministic correctness checks.

The reported 91.10 multilevel-structuring result is order-free, boundary-tolerant JSON F1 on 500 examples. It allows the evaluator's record alignment and limited boundary repairs; it is not exact JSON match. Pydantic validation checks schema compliance rather than factual truth.

Copying a passage can prevent one kind of invented field content while still producing a misleading record. A date taken from the wrong invoice remains wrong even if the characters all appear in the input.

Choose measurements that match the failure

This is our proposed comparison procedure. It does not convert three unrelated vendor evaluations into a common leaderboard.

Keep the acceptance set representative of what users will submit, including documents with missing fields and changes involving several files. Count retries and manual repair in completed-task cost. Our bounded-output review explains why enforcing a format still leaves a separate semantic validation problem.