Mixture of Experts
14 pieces on Mixture of Experts.
News & Analysis
JetBrains Mellum2.1: 2.5B Active Params Reach 47 SWE-bench Verified
JetBrains' 12B MoE model activates 2.5B parameters per token and lifts SWE-bench Verified from 2.0 to 47.0 after reinforcement learning in real repositories.
Reflection Beam: 501B Parameters, 23B Active, 3–4x Less Compute Claim
Reflection's text-only 501B-parameter MoE activates only 23B parameters and claims 3–4x lower inference compute than leading Western open models.
Kolibri Activates Just 3.46B of Its 78.1B Parameters
Aleph Alpha’s bilingual MoE offers a 1,048,576-token context and an FP8 checkpoint designed to run on one high-memory GPU.
Jina-ocr-v1: 3.4B MoE Parses 2.57 Pages/sec on One A100
Jina AI's 3.4B MoE document parser posts 2.57 pages/sec on a single A100 — highest of 14 systems benchmarked — via lossless speculative decoding.
DeepSeek-V4.1-Flash: 552B MoE at 890 Bytes per Token
DeepSeek-V4.1-Flash uses a Causal Encoder-Decoder architecture to cut prefill to 8B active params and compress global KV cache to 890 bytes per token.
Cohere's 218B MoE Scores 83.6 on WMT26, Beats DeepL and Google Translate
North Small Translate activates 25B of 218B parameters per token, scores 83.6 on WMT26 across 50 languages, and runs on a single B200 in 4-bit mode.
DeepSeek-V4.1-Flash: 890 Bytes Per Token, 437x Smaller KV Cache
DeepSeek's 552B MoE model cuts global KV cache to 890 bytes per token — 437x below V1 — using CED, CSA2, and FP4 quantization.
2.4T-Parameter Qwen3.8 Runs on One Node With vLLM and NVFP4
AWS documents a single-node serving path for Qwen3.8-2.4T-A95B on a p6-b300 instance using NVFP4 quantization, cutting TTFT by nearly 60%.
IFM K2 Horizon: Six Apache 2.0 Models, 0.9B to 375B, With Self-Audit
IFM releases six open-weight models from 0.9B to 375B, plus training corpus, code, and a self-published reward-hacking audit that corrects 70.2% to 66.9%.
FreeToken Runs 284B MoE Models on a Single Consumer GPU
FreeToken's q* policy and semantic anchor checkpointing deliver 3–4× faster decode and 6–30× faster prefill for frontier MoE models on RTX consumer hardware.
GLM-5.3-Flash and Qwen3.8-Flash-Next Independently Hit the Same 3:1 Attention Ratio
Two Chinese AI labs independently converged on a 3:1 linear-to-full attention ratio, a 2048-token sparse budget, four gated residual streams, and Muon.
Qwen3.8-Flash-Next: 125B MoE Runs at 6B Active Params, Previews Qwen4
Alibaba's Qwen team releases a 180B-on-disk multimodal MoE that activates only 6B parameters per token, trained at one-ninth the cost of Qwen3.7-Plus.
NVIDIA Nemotron 3.5 Lightning: 30B MoE with 3B Active Parameters
NVIDIA ships Nemotron 3.5 Lightning, a 30B MoE with 3B active parameters and 1M-token context, plus NeMo Switchyard for per-step agent routing.
GLM-5.2 Raises the Bar for Text-Only Open-Weights LLMs
Z.ai's GLM-5.2 arrives as a 753B-parameter open-weights text model with a 1M-token context window, strong benchmark results, and aggressive API pricing.