Liquid AI's Pipette Benchmarks 1,000+ On-Device Configs, Not Just Models
In this article
Liquid AI released Pipette on August 25, 2026 — an Apache 2.0 benchmarking platform built in partnership with Artificial Analysis as an independent methodology validator. The core premise is a direct challenge to how model performance is normally communicated: model cards report quality under server-class, full-precision conditions, and those numbers rarely predict behaviour on a phone. Pipette reframes the unit of measurement as a deployment configuration — model + quantization + runtime + device — rather than a model in isolation. For teams working on best small language models or edge deployment, that distinction is immediately consequential.
What the Launch Dataset Covers
The initial release spans more than 1,000 model × quantization × runtime × device × context configurations, covering 30+ models and llama.cpp builds for macOS, iOS, Windows, and Android. Context lengths run from 256 to 8,192 tokens. Five on-device performance metrics are tracked per configuration. Verified results at launch come from three devices: a MacBook Pro with M5 Max, an iPhone 17 Pro, and a Galaxy S26 Ultra. AMD Ryzen AI Max+ 395 and Radeon 8060S results are listed as forthcoming.
Quality scores — IFBench, GPQA Diamond, and MATH-500 — are produced on NVIDIA H100 80GB reference systems and then matched to on-device runs sharing the same model and quantization. A quality number shown alongside phone throughput was not produced on the phone; the methodology is explicit about this. Performance runs use fixed token shapes, greedy decoding, one discarded warm-up, five measured repetitions, and platform-specific readiness gating that verifies thermal and load conditions before each timed repetition. Failing runs are not published.
Four Comparisons That Show Why Configuration Matters
The published dataset surfaces four comparisons demonstrating how much a deployment context can shift a procurement or architecture decision — the kind of variance that systems engineering at the stack level makes measurable but model cards obscure entirely.
| Comparison | Device | Quantization | Key Finding |
|---|---|---|---|
| Granite-4.0-H-350M vs Granite-4.0-350M, context scaling | Galaxy S26 Ultra | Q4_K_M | Hybrid retains 78.4% of decode throughput at 4,096 tokens; non-hybrid retains 33.8% |
| LFM2.5-8B-A1B vs Qwen3.5-4B and Ministral-3-3B-Instruct-2512, decode speed | Galaxy S26 Ultra | — | LFM2.5-8B-A1B decodes 2.4× faster than Qwen3.5-4B and 2.6× faster than Ministral at 2,048 input tokens; activates 1.5B of 8.5B parameters per token but peaks at 5.29 GiB RAM |
| MiniCPM5-1B vs LFM2.5-1.2B-Instruct, speed vs quality | iPhone 17 Pro | Q4_K_M | MiniCPM5-1B completes 2,048-in / 256-out in 3.47 s vs 4.12 s (15.8% faster); LFM2.5-1.2B-Instruct scores 9.0 points higher on MATH-500 |
| Granite-4.1-8B vs Ministral-3-8B-Instruct-2512 | MacBook Pro M5 Max | Q4_K_M | 2.4% difference in decode throughput, 1.2% difference in peak RAM; Granite leads IFBench by 7.3 points, Ministral leads GPQA Diamond by 14.0 points |
The sparse-activation finding is particularly instructive: LFM2.5-8B-A1B's speed profile resembles a small model, but its memory footprint does not, because all expert weights must remain resident even though only 1.5B of 8.5B parameters activate per token. A developer optimising for latency while assuming memory scales with active parameters would undersize the device.
Architecture: Three Repositories, One Auditable Record
Pipette ships as three open-source repositories under Apache 2.0. pipette-clients fetches versioned benchmark definitions, runs inference on the target device, and generates evaluation completions. pipette-mgmt serves the benchmark catalog, ingests submissions, and forwards completions for scoring. pipette-scores supplies evaluation prompts and scores completions without access to their generation provenance — enforcing blind scoring. Every submission records benchmark version, token shape, model artifact, quantization, runtime version and settings, and device hardware and OS.
This separation of concerns makes the pipeline auditable in a way that matters for the use cases Liquid AI targets: SoC procurement validation, runtime and OS regression testing, and independent verification of vendor performance claims — a concern that pipeline architecture increasingly drives rather than model capability alone. A hosted dashboard and native iOS and Android benchmark apps are available without waitlist. Community-submitted results publication is in beta.
Deployment Model and Practical Applications
The three-tier deployment model maps to organisational scale. Solo developers and seed-stage teams can use the dashboard and apps without infrastructure. Mid-market product teams can run pipette-clients across an internal device fleet. Large OEMs, chip vendors, and enterprises can operate the full pipeline behind their own firewall. The Apache 2.0 licence imposes no restrictions on internal commercial use.
Practical applications include model and quantization selection before a sprint commits, context-length capacity planning, and hardware procurement validation — domains where the 78.4% versus 33.8% throughput-retention gap between two nominally equivalent 350M models represents a real engineering decision, not a benchmark curiosity. As inference moves to hardware teams do not control — phones, embedded systems, edge nodes — the gap between server benchmark numbers and deployed behaviour becomes the actual engineering problem, and Pipette is positioned as neutral infrastructure for closing it.