Baidu Releases 280B ERNIE 5.5 Flash Base With 14B Active Parameters

Baidu has released ERNIE 5.5 Flash Base, a sparse Mixture-of-Experts checkpoint with 280B total parameters and 14B active per token, stopped at the end of pre-training before any mid-training or post-training. That makes it a shared starting point for fine-tuning, instruction tuning, and RL comparisons. It was pre-trained from scratch on 33.5T tokens with a 128K training context; fine-tuning and inference support 1M tokens.

Architecture: hybrid attention, multi-router MoE, and ECHO

ERNIE 5.5 Flash Base has 43 layers: one dense FFN and 42 Multi-Router Latent MoE layers. Attention is split between 37 Heavily Compressed Attention layers, each caching a single entry per 128 tokens, and six Multi-head Latent Attention layers with sparse attention (MLA + DSA), so one layer in seven keeps token-level full-context access. HCA layers use 512-dim heads over the 128× compressed sequence; MLA layers use 256-dim heads. Baidu reports the attention output gate is computed from the low-rank query latent, cutting gate-projection parameters by 75%. Virtual Head Attention adds a lightweight low-rank post-mix with an identity skip.

Each MoE layer has 512 routed experts with top-10 selection plus one shared expert, running in a 2048-d latent space, half the 4096 hidden width. Several independent sigmoid routers replace a single router; Baidu says selection improves without changing activated expert count, expert FLOPs, or communication. Auxiliary-loss-free load balancing, RMSNorm before up-projection, and SiTU-GLU activations stabilize expert outputs. The final layer doubles as the ECHO future-token predictor and speculative draft model; MarkTechPost reports ECHO adds 0.36B parameters outside the headline counts. The model also uses manifold-constrained Hyper-Connections at expansion rate 4, a MultiMax output head, and a ≈200K-entry BPE tokenizer that tokenizes numbers right-to-left in groups of up to three digits.

Benchmarks and comparison

All scores below are Baidu-reported from its unified evaluation pipeline, using identical prompts and few-shot settings; no independent validation is cited. Baidu reports ERNIE 5.5 Flash Base leads on 24 of 30 base-model tasks. Representative scores include 78.4 MMLU-Pro, 93.3 MMLU, 82.3 HumanEval+, 48.7 LongBench-v2, and 85.9 HELMET-ICL. Its weakest reported result is 56.2 on HELMET-Rerank, behind DeepSeek-V4-Flash-Base at 60.0. In the same pipeline, DeepSeek-V4-Flash-Base leads on ChineseSimpleQA, INCLUDE-lite, HELMET-Rerank, and InfiniteBench; Hy3-preview-Base leads on BBH CoT and FullStackBench. MarkTechPost notes Baidu re-measures rivals: DeepSeek lists MMLU-Pro 68.3 but Baidu measures 69.2; Tencent lists Hy3 MMLU-Pro 65.76 but Baidu measures 66.9.

*Scores from Baidu's unified evaluation, self-reported.

Availability and licensing

FP8 and BF16 weights are on Hugging Face and ModelScope. Baidu documents SGLang with --tp 4 and vLLM 0.32.0 plus the vLLM-ERNIE plugin; both servers accept up to 1,048,576 tokens. As a base model, it uses the completions endpoint rather than chat. Training ran on the FlashMask-4 kernel for SM90/SM100 GPUs. Baidu reports up to 175% faster forward and 74% faster backward on SM100 versus FlashAttention-4 MaskMod. MarkTechPost describes the license as free for non-commercial research, education, and evaluation, and free for commercial use up to US$50M in related annual revenue. It excludes the EU, UK, and South Korea, forbids outputs that improve other AI models, bans military use, and selects Hong Kong law.

AI Mastery analysis

The 14B active count cuts per-token compute, but serving still requires all 280B weights plus expert dispatch traffic. Teams self-hosting should measure memory and communication before assuming a small-model footprint—the same reality covered in Choosing Local AI: Memory, Tools and Data Boundaries.

The attention split creates a long-context caveat. Only six of 43 layers have token-level full-context access; the other 37 compress to one cache entry per 128 tokens. Baidu reports strong long-context results in its own pipeline, but exact positional recovery and fine-grained retrieval may still hit compressed layers before the 1M limit. A base checkpoint also strips away instruction tuning and safety behavior, making alignment and refusal handling the adopter's job.

The release is a large pre-training artifact, not a turnkey assistant. Teams with the infrastructure to fine-tune and serve a 280B checkpoint gain a clean substrate; the region-limited license and aggressive compression make it a starting point for integration rather than a drop-in model.

Sources

• Primary: baidu/ERNIE-5.5-Flash-Base · Hugging Face
• Reported by: MarkTechPost: Baidu Releases ERNIE 5.5 Flash Base: A 280B MoE Base Model With 14B Active Parameters