FreeToken Runs 284B MoE Models on a Single Consumer GPU
In this article
Researchers from UC Berkeley and MIT, including Databricks co-founders Matei Zaharia and Ion Stoica alongside Song Han and Kurt Keutzer, have released FreeToken, an open-source inference engine targeting frontier Mixture-of-Experts models on consumer GPUs. The core problem it addresses is structural: sparse MoE architectures activate only a subset of parameters per token, but decoding still requires routing across hundreds of billions of inactive weights. In datacenter deployments, NVLink absorbs that expert-transfer overhead. On consumer hardware, PCIe throughput of 16–64 GB/s and host RAM latency create hard decode ceilings. Existing edge runtimes—Ollama, llama.cpp, KTransformers—handle this with static expert offloading: inactive weights sit in system RAM and stream synchronously to the GPU on activation, stalling execution on cache misses.
Dynamic Co-Scheduling with the q* Policy
FreeToken replaces static offloading with the q policy*, a closed-form computation that determines per-layer, per-token optimal splits between CPU cores and GPU tensor cores based on real-time interconnect throughput measurements. Rather than pausing the GPU during a cache miss, the scheduler routes token computation to whichever compute substrate can proceed immediately, with the split recalculated continuously as PCIe bandwidth fluctuates.
Two additional mechanisms support this. A fast weight format (FTW) combined with full-layer double buffering allows weight streaming over PCIe to overlap completely with active computation layers, eliminating the sequential dependency that makes static offloading costly. An elastic memory manager dynamically reallocates VRAM between KV cache entries and resident expert slots at runtime, without triggering model reloads—meaningfully different from KTransformers, which applies static CPU/GPU offloading rules decided at initialization.
Semantic Anchor Checkpointing
Agentic execution patterns—frequent prompt edits, tool-call injections, thinking blocks that mutate mid-sequence—are punishing for standard KV cache designs, which discard linear caches whenever a prefix changes and force full-sequence recomputation. FreeToken addresses this with semantic anchor checkpointing: intermediate attention states and recurrent activations are cached at logical task boundaries rather than at fixed token offsets. When an agent modifies tool arguments or injects external execution output mid-sequence, FreeToken reuses the relevant sub-sequence states instead of invalidating the entire prompt cache. This directly suits the agentic inference patterns discussed in what the agentic era means for data science.
Benchmark Results
The paper reports three headline results: Qwen3.6-35B running at approximately 39 tokens/sec on an 8 GB RTX 4060 laptop GPU; DeepSeek-V4-Flash at 284B parameters served on an RTX 5090 desktop; and GLM-5.2 at 753B parameters processed on a single workstation GPU. Against the most directly comparable runtimes, FreeToken achieves 3–4× faster decode and 6–30× faster prefill on equivalent MoE models versus Ollama and llama.cpp. vLLM and SGLang are out of scope for direct comparison because their PagedAttention and continuous-batching designs assume high-bandwidth datacenter interconnects rather than heterogeneous memory hierarchies.
| Runtime | Expert Scheduling | Interconnect Assumption | KV Cache Strategy | MoE Decode vs FreeToken |
|---|---|---|---|---|
| FreeToken | Dynamic q* closed-form split, per-layer real-time | Heterogeneous PCIe (16–64 GB/s), adaptive | Semantic anchor checkpointing | Baseline |
| Ollama / llama.cpp | Static layer-wise offloading, synchronous on cache miss | Consumer PCIe, fixed | Linear KV cache, prefix invalidation | 3–4× slower decode; 6–30× slower prefill |
| KTransformers | Static CPU/GPU rules set at initialization | Consumer PCIe, fixed | Standard | No per-layer adaptive splits |
| vLLM / SGLang | Continuous batching, PagedAttention | High-bandwidth datacenter (NVLink assumed) | Paged KV blocks | Not designed for heterogeneous memory hierarchies |
FreeToken is available via FlashML.ai and its GitHub repository, with CLI and desktop clients supporting NVIDIA RTX 30, 40, and 50 series GPUs on Linux and Windows.
Community Reception and Caveats
Reception on Hacker News and Reddit's LocalLLaMA forum has been enthusiastic but technically scrutinised. Engineers note that pairing FreeToken with used RTX 3090 or 4080 GPUs and standard DDR4/DDR5 RAM could let developers self-host 284B-scale models without recurring cloud API costs—directly addressing the calculus examined in software extraction beats hardware acquisition. The substantive critique is whether q*'s closed-form calculations hold under real-world CPU dispatch latency, memory contention, and variable expert residency when concurrent agent workloads run—conditions the controlled benchmark environment may not fully represent. Debates over baseline comparisons against hand-tuned llama.cpp configurations remain unresolved.
FreeToken is a concrete instance of the broader thesis that architectural specificity now outperforms GPU scaling alone in constrained, heterogeneous deployment environments. Whether q*'s real-time splits survive production agent workloads will determine if the benchmark numbers translate to practitioner adoption—but open-source availability on consumer RTX hardware means that answer will come from the community rather than a lab.