MiniCPM5-2B: 2.5B Model Beats Qwen3.5-4B With 53.9 Avg Across 34 Benchmarks

OpenBMB has released MiniCPM5-2B, a dense causal language model with 2,516,756,480 total parameters — 1,981,982,720 of them outside the embeddings — built explicitly for on-device and edge deployment. The architecture is standard LlamaForCausalLM across 42 layers, with grouped-query attention using 16 query heads and 2 key/value heads, and a native context window of 131,072 tokens. Because the architecture is stock Llama, the Apache 2.0 weights load directly into vLLM, SGLang, Transformers, llama.cpp, Ollama, LM Studio, MLX, and FlagOS with no custom kernels and no model-code fork. For engineers tracking how architectural specificity outperforms GPU scaling, MiniCPM5-2B is a concrete data point: a sub-3B model that averages 53.9 across 34 benchmarks, ahead of Qwen3.5-4B at 51.1 despite roughly half the parameter count.

Benchmark Profile: Strengths and Gaps

OpenBMB's comparison set places MiniCPM5-2B against LFM2.5-2.6B, Qwen3.5-2B, and Gemma-4-E2B-it in the same size class, with Qwen3.5-4B, granite-4.2-3B, Nemotron-3-Nano-4B, Gemma-4-E4B-it, and LFM2.5-8B-A1B as larger reference points. The 53.9 average headline obscures a split performance profile.

Tool use carries the widest margins. MiniCPM5-2B scores 97.1 on τ²-Bench Telecom, 66.6 on BFCL v4, and 20.8 on τ³-Bench Banking against 6.8 for the next-best result. Code reasoning follows: 69.1 on LiveCodeBench v6 against 56.4, and 46.4 on SWE-bench Verified against 33.6. Long-context retrieval is split — 68.1 on NoLiMa against 43.5, but 59.0 on AA-LCR against 61.0, and 43.7 on LongBench v2 against 47.3. General knowledge is where the size gap is visible: 70.8 on MMLU-Pro against 78.0, and 8.9 on Humanity's Last Exam against 9.9.

Post-Training Pipeline: RL Teachers and On-Policy Distillation

After stable and decay base-training phases and a mid-training distribution-adaptation step, the pipeline runs 400 billion tokens of deep-thinking supervised fine-tuning. It then trains specialised RL teachers for math, code, agentic tasks, and writing using the critic-based JustRL II algorithm. The final stage is on-policy distillation (OPD), which merges 16 RL expert models — five of them agentic — into the single shipped checkpoint.

OPD replaces the standard verification-based advantage signal with full-vocabulary reverse KL divergence computed between student and teacher logits at each response position, and reuses the existing RL prompt corpus as distillation data — no new dataset construction is required. OpenBMB measures the combined RL-plus-OPD stage at a 10.96 average-point gain on reasoning and general benchmarks and a 6.96 average-point gain on agentic benchmarks. This multi-expert merge strategy reflects the broader pattern described in pipeline architecture driving AI gains in 2026, where post-training composition rather than pretraining scale determines deployed capability.

Open Data and Reproducibility

OpenBMB released training datasets alongside the weights: Ultra-FineWeb, Ultra-FineWeb-L3, UltraX, UltraData-Code with L0-through-L3 tiered management, UltraData-Math, UltraData-SFT-2605, UltraData-SFT-Agent-2609 with 500,000 agent samples, and UltraData-RL-2609 with over 80,000 RL samples. Intermediate checkpoints covering Base, Midtrain, and SFT-only stages are also published, making the per-stage contribution of the RL-plus-OPD gain directly verifiable. For practitioners evaluating best small language models on Hugging Face, this degree of training transparency is atypical at the sub-3B scale.

MiniCPM5-2B is narrowly optimised for tool invocation, agentic code execution, and long-context retrieval — not a general-purpose knowledge system. The tool-use and coding margins over models with roughly twice the parameter count suggest the OPD merge strategy is doing real work; the published intermediate checkpoints make that claim tractable to investigate on out-of-distribution tasks.