Jina-ocr-v1: 3.4B MoE Parses 2.57 Pages/sec on One A100

Jina AI has released jina-ocr-v1, a 3.4B-parameter mixture-of-experts visual document parser that converts PDFs, scans, tables, charts, and invoices into clean Markdown in a single forward pass. The model ships its speculative decoding head inside the same checkpoint, targets low-budget GPUs such as the NVIDIA L4, and posts the highest pages-per-second throughput among the 14 systems Jina AI benchmarked on a single A100. As our coverage of architectural specificity outperforming GPU scaling has documented, the gains here are structural rather than brute-force.

Architecture: DeepEncoder compression plus MoE decoding

The model post-trains DeepSeek-OCR and retains its two efficiency subsystems. The encoder, DeepEncoder, carries approximately 380M parameters and chains SAM, a 16× convolutional compressor, and CLIP-L. A 1024×1024 global page view enters as 4,096 patches and exits as 256 visual tokens — a 16× reduction that shortens the prefix the decoder must process. A dynamic-resolution mode called Gundam adds up to 9 local tiles at 100 tokens each, capping a densely typeset page at 1,156 visual tokens.

The decoder is DeepSeek-3B-MoE with 12 layers, 64 routed experts, and 2 shared experts. Top-6 routing fires approximately 570M parameters per token while all weights remain resident in memory. The context window extends to 32,768 positions. Output formats are Markdown prose, HTML tables, and LaTeX formulas. Total checkpoint size is approximately 6.8 GB in BF16.

FastMTP speculative decoding: K=3 lossless drafting

OCR output is near-deterministic and locally structured, which makes it an unusually clean target for speculative decoding. Jina AI adds a FastMTP head — a single dense draft block applied recursively for K=3 steps with constant parameters across all depths. The main MoE decoder then verifies all three drafts in one pass, accepts the longest matching prefix, and appends one token of its own. Because verification uses greedy matching against what the decoder would have chosen anyway, the committed output is provably identical to plain greedy decoding.

On an NVIDIA L4 at batch size 1 in eager mode, the baseline is 42.7 tokens per second. At K=3 the acceptance rate is 57.6%, yielding 2.73 tokens per step and 83.1 tokens per second — a 1.95× lift. With CUDA graphs the baseline rises to 158.3 tokens per second, and the optimal setting shifts to K=1, which achieves 185.6 tokens per second at an 82.6% acceptance rate. The asymmetry matters: a speculative step in eager mode costs roughly 1.4 plain steps, so the higher acceptance rate at K=1 is less important than at K=3, where each step amortizes more.

Benchmark performance and throughput

Post-training adds 7.4 points over the DeepSeek-OCR backbone on olmOCR-Bench. The accuracy picture is mixed: PaddleOCR-VL-1.6 leads OmniDocBench at 96.34, HunyuanOCR-1.5 reaches 94.74, and chandra-ocr-2 edges jina-ocr-v1 on olmOCR-Bench at 85.8 versus 83.4. Throughput is where jina-ocr-v1 separates itself: 2.57 pages per second on a single A100 against 1.22 for olmOCR-2 and 0.38 for chandra-ocr-2. Jina AI attributes part of that lead to output conciseness — 1,085 tokens per page, described as the shortest among systems scoring above 83 on olmOCR-Bench.

AI Mastery analysis

The FastMTP design exposes a deployment constraint operators should evaluate before committing to a hardware configuration. In eager mode on the L4, K=3 is the efficient choice because each plain step is expensive and the 57.6% acceptance rate still nets 2.73 tokens per verifier pass. Under CUDA graphs, static compilation eliminates most per-step overhead for the main decoder, compressing the marginal gain from additional draft depth and making K=1 the throughput-optimal setting. Teams running vLLM 0.21+ in a compiled graph regime will need to tune K explicitly — the checkpoint does not enforce a default.

The CC BY-NC 4.0 license is a harder constraint for commercial teams. Weights are freely available for research and non-commercial deployment, but production use requires a separate agreement with Jina AI. This makes jina-ocr-v1 a benchmark and development reference rather than a drop-in commercial component — a dependency worth factoring into pipeline architecture early, since production AI failures often trace back to integration architecture rather than model capability.

The GRPO post-training methodology also deserves attention. Every reward term — content, formula, table, structural validity, unit tests, repetition, format — runs as deterministic code against a reference transcription, and terms are multiplied rather than summed. Floors of 0.2 on structural, unit-test, and format terms, and 0.1 on the table term, prevent a single failed check from zeroing the signal for a largely correct page. The repetition term deliberately has no floor, because a loop can otherwise inflate the content score and mask the failure. The JinaOCRSynth synthetic dataset was built specifically to surface formula and table cases that are rare in natural-document corpora.

The broader signal is that throughput engineering on a 3.4B MoE model is now producing page-level parsing rates that dwarf much larger systems at batch scale, confirming a pattern where software extraction outpaces hardware acquisition at the AI frontier. For RAG pipeline engineers, the operative question is not whether jina-ocr-v1 leads every accuracy benchmark, but whether 2.57 pages per second on a single A100 under a negotiable commercial license fits their cost and throughput envelope.

Primary source

Jina AI Releases jina-ocr-v1: A 3.4B MoE Document Parser With Built-In Speculative Decoding for Low-Budget GPUs — MarkTechPost