NeoMME 260M Matches ColQwen2.5 3.75B on ViDoRe v3

H Company has released NeoMME, a family of bidirectional multimodal encoders at 262,937,906 and 793,715,032 parameters that discards the architectural conventions inherited from generative vision-language models. Where ColPali-style retrievers repurpose a causal LLM — carrying a pretrained vision tower and a decoder that never emits a token — NeoMME routes multilingual text tokens and raw 32×32 RGB image patches through a single shared Transformer trained from random initialization. The headline result: at 260M parameters, NeoMME-Retriever scores 0.523 nDCG@10 on ViDoRe v3, within 0.002 of ColQwen2.5-v0.2 at 3.75B parameters, and 26.1 points above the best other sub-300M competitor on that benchmark.

This follows the same logic documented in architectural specificity outperforms GPU scaling: purpose-designed encoders consistently close the gap on general-purpose stacks carrying task-irrelevant components.

Architecture

Text enters through an ALBERT-style factorized embedding: a 256-dimensional lookup projected to full model width. Images are split into non-overlapping 32×32 patches and projected by a two-layer MLP. There is no patch-merging module, no SigLIP2 tower. Both model sizes support a 16,384-token context — sufficient for two 3,840×2,160 4K UHD images after patching. Most layers use symmetric sliding-window attention; every sixth layer and the final layer attend globally. Additional components include grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs.

The tokenizer is a whitespace-unconstrained BPE trained from scratch with a 131,072-entry vocabulary. Across 14 target languages in the FLORES-200 devtest set, it produces 44.4% fewer tokens than ModernBERT. Pretraining runs discrete masked diffusion over text, optionally conditioned on visible image patches. Text-only segments draw a corruption rate uniformly from 0 to 1; multimodal segments draw from 0.30 to 1, removing the language-only shortcut. At 90% masking, visible page patches raise masked-token accuracy by 38.4 points for the 260M model and 40.5 points for the 800M model. Each pretraining run processes approximately 524 billion packed input tokens — roughly 290 billion text-only — on 16 and 32 H100 accelerators respectively.

Retrieval Benchmarks

NeoMME-Retriever adds two jointly trained heads on the shared backbone: a mean-pooled dense head with Matryoshka widths, and a late-interaction head that projects every token and patch to 128 dimensions. A single forward pass returns both.

The 800M model trails Vultron Retriever Flash by 0.9 points on ViDoRe v3. Text-only retrieval is a documented weakness: on BEIR-15, late interaction reaches 0.4881 at 260M and 0.5126 at 800M, against 0.5722 for LateOn at 149M parameters. The authors attribute this partly to supervision scale — NeoMME saw roughly 430,000 text query examples, against roughly 660 million contrastive examples for mLateOn.

Indexing Throughput and Storage

A 2,048×2,048 page in late-interaction mode yields 4,162 vectors, approximately 1,536.7 kB per document in float32. Three measured compression configurations:

• No pooling, float32: 1,536.7 kB per document; 100% of baseline nDCG@10 retained.
• Hierarchical token pooling at factor 10, int8 queries and documents: 39.0 kB per document; 39.4× compression; 99.16% of baseline nDCG@10 retained.
• Pool factor 8, int8 queries, binary documents: 6.0 kB per document; 255.5× compression; 95.19% of baseline nDCG@10 retained.

On a single NVIDIA L40S at 2,048×2,048 input, the 260M model encodes 51.3 pages per second against ColModernVBERT's 26.0 — a 1.97× throughput advantage despite outputting 4,162 vectors per page versus ColModernVBERT's 1,149. On CPU-only hardware, query encoding takes 78.3 ms. All checkpoints ship under Apache 2.0 with day-zero support in Hugging Face Transformers.

NeoMME is a concrete architectural bet: retrieval-specific design, trained from scratch rather than adapted from a generative stack, can match models more than 14× its size on document retrieval while remaining deployable on commodity hardware. The text-retrieval gap and limited transfer to natural images are real constraints that determine which production workloads it suits. For teams evaluating small language models on Hugging Face for document retrieval, the 260M variant's throughput and storage profile is a practical baseline before committing to heavier infrastructure.