GLM-5.3-Flash and Qwen3.8-Flash-Next Independently Hit the Same 3:1 Attention Ratio

Two open-weight models shipped within a day of each other in late August: Z.ai's GLM-5.3-Flash, a 320B-parameter multimodal mixture-of-experts model with 18B active parameters, and Alibaba's Qwen3.8-Flash-Next, a 125B model with 6B active parameters that previews the Qwen4 architecture family. The teams designed these systems independently. Yet their architectural configs converge on four specific choices — a 3:1 ratio of linear to full attention, a compressed indexer capped at 2048 tokens, four gated residual branches, and the Muon optimizer with the same matrix-splitting procedure — with enough precision that the overlap cannot be dismissed as superficial resemblance. For ML practitioners evaluating how architecture choices now rival raw scaling in determining efficiency outcomes, this constitutes independent empirical validation at frontier scale.

The 3:1 Linear Hybrid and What It Buys

GLM-5.3-Flash stacks 45 layers: 34 linear-attention layers and 11 full-attention layers. Qwen3.8-Flash-Next stacks 48 layers in a repeating block of 3 Gated DeltaNet layers plus 1 Qwen Sparse Attention layer. Both land on a 3:1 ratio.

The linear layers carry the economics. Instead of a KV cache that grows with sequence length, they compress all history into a fixed-size recurrent state, so compute per token stays constant regardless of context length. GLM uses Kimi Delta Attention (KDA), which applies a per-channel decay gate. Qwen uses its own Gated DeltaNet (GDN), which gates at per-head granularity. Different gating granularity, same delta-rule family, same function.

The architectural impact on GLM is concrete: Z.ai reports the Flash design cuts attention compute by roughly 3x and KV cache size by 4.4x compared with the full GLM-5.3 model, while nearly halving active parameters (18B versus 32B) and layer count (45 versus 92). GLM-5.3-Flash was trained on a 30T-token multimodal corpus and serves a 1M-token context window, priced at $0.15 per million input tokens and $0.50 per million output tokens. Qwen3.8-Flash-Next carries a native context of 262,144 tokens extensible to 1M via YaRN, and the Qwen team reports training required approximately one-ninth the compute of Qwen3.7-Plus.

Compressed Indexers and the 2048-Token Budget

Neither model lets its full-attention layers attend over the entire context. Both attach a small learned indexer that scores chunks of history and retains only the top candidates, with the same numerical cap.

GLM's sparse layers use a 32-head lightning indexer with top-2048 token selection. To manage indexer cost at 1M-token contexts, Z.ai introduces IndexPool, which compresses four indexer key vectors into one through weighted pooling before scoring. Qwen's QSA operates at micro-block granularity: a compressed lightweight indexer scores 4-token blocks and keeps the top 512 blocks, which is exactly 2048 tokens. Both models therefore compress 4x before scoring and cap the attention budget identically. Qwen credits QSA with up to 7.6x prefill and 4.9x decoding speedups over full attention at 1M tokens. The pattern traces to DeepSeek's DSA, introduced in DeepSeek-V3.2-Exp.

Four Gated Residual Streams and the Muon Optimizer

Both models abandon the single residual stream standard since the original transformer. GLM adopts Manifold-Constrained Hyper-Connections (mHC), a DeepSeek-originated design configured with 4 branches. Qwen built its own Gated Residual variant, which modulates flow through 4 widened streams via an element-wise data-dependent read gate and a per-branch scalar write gate. Qwen's version removes the extra branch-mixing step in Hyper-Connections to reduce memory-access overhead, and the gate suppresses activation outliers well enough to enable FP8 residual storage. The Qwen team ablated both designs and found them roughly equal in quality — two independent implementations reaching the same answer.

Both models train with Muon, and both apply the same refinement: fused projection matrices are split into their independent transformations before Muon orthogonalizes them. Qwen documents splitting fused QKV, SwiGLU, and GDN projections, assigning Muon to genuine 2D linear maps and AdamW to embeddings, routers, and low-rank parameters. Qwen also refitted its scaling laws for the new architecture and eliminated batch-size warmup after measuring that warmup consumed 18.8% more optimizer steps without improving results.

Where They Split, and the MiniMax Dissent

The clean split is rotary positional embeddings inside the full-attention layers. GLM-5.3-Flash removes them entirely — the config sets qk_rope_head_dim = 0 — relying on the recurrent linear layers to carry positional information implicitly. Qwen tried the same thing. Pre-training loss showed no measurable difference. The failure appeared after post-training: the NoPE variant frequently failed to terminate generation. Qwen kept RoPE. That result is a direct caution to any lab assuming pre-training metrics fully characterize behavioral robustness through the post-training stack.

MiniMax represents the hardest dissent. During M2 development, the team ran scaled ablations of linear and sliding-window attention and found severe multi-hop reasoning deficits, particularly beyond 32K context after supervised fine-tuning. M2 shipped with full softmax attention. For M3, MiniMax adopted MiniMax Sparse Attention, which sparsifies softmax attention via block selection but contains no linear-attention layers at all.

The 3:1 linear hybrid has now been independently validated by Z.ai, Qwen, DeepSeek, and Kimi at production scale — but MiniMax's ablations demonstrate that the reasoning cost is real and reproducible. Which side of that tradeoff matters most depends heavily on how the surrounding pipeline is structured, not just the attention mechanism in isolation. The architectural consensus forming across Chinese open-model labs is striking, but MiniMax's empirical objection means the question is not yet closed.