Lily Beats MLX-LM 1.35x on Decode: Perplexity's Rust+Metal Engine for Apple Silicon

Perplexity has open-sourced Lily, the local inference engine powering Hybrid Compute in Perplexity Computer, publishing it to the pplx-garden repository as a standalone Rust and Metal inference server. The engine exposes a minimal OpenAI-compatible chat-completions API, streams tokens from hand-written Metal kernels, and routes zero compute through PyTorch or MLX. That is the core design argument: narrow specialisation to one model — Qwen3.6-35B-A3B — on one hardware family, Apple Silicon, buys measurable throughput that a general-purpose stack cannot match.

For practitioners targeting local agent deployments where systems-level specificity drives gains, Lily is worth examining at the kernel level. Perplexity's own ablations show individual optimisations lifting prefill throughput by as much as 89% and decode throughput by 40.2% at long contexts — per-feature measurements against the same engine with that feature toggled off.

Model structure and three workload shapes

Qwen3.6-35B-A3B stores 35B parameters but activates roughly 3B per token. A router scores 256 experts and picks eight, plus one shared expert that processes every token regardless of routing. Alongside that MoE structure, the model mixes 10 full-attention layers using grouped-query attention (GQA) with 16 query heads and 2 KV heads, and 30 Gated DeltaNet layers. That architecture presents three fundamentally different computation patterns: uneven expert groups from sparse routing, attention over a growing KV cache, and fixed-size recurrence.

The checkpoint uses groupwise affine 4-bit quantisation, with every group of 64 weights sharing a bfloat16 scale and bias. Approximately 70 GB of bfloat16 weight data compresses to a 19.4 GB checkpoint. Metal 4 tensor operations consume bfloat16, so weights must be reconstructed before use. The realistic memory floor is a 32 GB unified-memory Mac; Perplexity's shipping Hybrid Compute product lists macOS 15+ with 24 GB minimum and 32 GB recommended.

Prefill: fusing dequantisation and keeping routing GPU-resident

Instead of expanding the 4-bit checkpoint to a full bfloat16 array in unified memory before feeding the grouped GEMM, Lily reconstructs weights one tile at a time inside threadgroup memory, accumulating in FP32 — the expanded array never materialises in unified memory. Perplexity's ablation measured this fusion raising end-to-end prefill throughput 77.4% at a 512-token prompt.

The second large gain addresses MoE routing latency. Computing the routing histogram, prefix scan, scatter, and block map inside a single GPU command buffer — removing CPU synchronisation from each MoE layer — added 89% at a 512-token prompt. Moving from 16-row to 32-row tiles with four simdgroups added 13.2% at a 2K prompt. A register-resident Gated DeltaNet scan, where each simdgroup carries one column of recurrent state through the full scan rather than moving an intermediate 256 MiB per layer, contributed a further 5.6% at 2K. Long prompts are processed in bounded chunks so temporary activations do not compete with weights and KV cache for unified memory bandwidth.

Decode: eliminating per-token CPU round trips

Batch-1 decode is almost entirely bandwidth-bound rather than compute-bound. One recorded decode step launched 795 kernels forming 555 sequential stages. Lily records real data dependencies in a concurrent Metal command pass so independent kernels can overlap, and the selected token writes directly into the next step's GPU-resident input slot, removing a per-token CPU round trip. Four kernel chains are fused to keep intermediates in registers.

Coalesced KV cache reads raised key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s. GQA packing — four query heads sharing one threadgroup so each KV row loads once — improved decode throughput 23.8% at a 32K context. A fixed-block attention layout, engaging above 32K, splits the cache into equal parallel pieces and improved decode 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K. Speculative decoding was measured as an 18% regression in batch-1 decode on this hardware because verification batches of two to five rows are an inefficient shape for Metal and pull in additional expert weights; Perplexity uses speculative decoding in its batched Qwen deployment on Blackwell but ships Lily without it.

Benchmark results against MLX-LM

All measurements were taken on a single M5 Max with a 40-core GPU and 128 GB of unified memory, batch 1, loading the identical 19.4 GB 4-bit checkpoint against MLX-LM's fastest direct-generation path across ten prompt and context lengths from 256 to 128K tokens.

Lily was faster at every individually recorded point, with prefill ratios ranging from 1.12× to 1.42× and decode ratios from 1.31× to 1.37×. A teacher-forced accuracy check across 192 positions placed Lily's perplexity just 0.04% higher than MLX-LM, with the same top-ranked token produced 96.35% of the time.

The result aligns with the pattern that software extraction from existing hardware increasingly rivals raw hardware acquisition: Lily runs the same checkpoint bytes on the same chip and recovers 23–35% throughput headroom purely through kernel fusion and data-flow restructuring. For teams building local agent pipelines on Apple Silicon, the engine's public availability in pplx-garden means these gains are immediately accessible without waiting for next-generation hardware.