Needle 2: 45M-Parameter Tool-Calling Model in a 14MB Binary

Cactus Compute has released Needle 2, an open-weight 45M-parameter model purpose-built for tool calling, device control, and structured field extraction. The entire model ships as a single 14MB binary — weights included — and runs a complete session in approximately 28MB of RAM. For developers evaluating best small language models on Hugging Face for on-device deployment, those two numbers define a meaningful new floor: Needle 2 eliminates the runtime installation step entirely, executes without a GPU or NPU, and targets hardware that most inference frameworks treat as out of scope.

The design premise is narrow by intent. Cactus frames tool calling as a function-signature mapping problem — translating a natural-language sentence onto a typed call — and argues that task requires no open-ended world knowledge. That framing is why 45M parameters are sufficient, and why reported decode throughput reaches 500 tokens/sec on a Raspberry Pi 5, 400–1,500 tokens/sec on Meta Quest 3S and Apple Vision Pro, and 300–700 tokens/sec on sub-$200 phones. Pebble already runs Needle 2 locally inside the Index 01 app for offline voice actions.

Architecture: What Replaces the FFN

Needle 2 uses what Cactus calls a Simple Attention Network, a 27-layer, 512-wide architecture documented on arXiv as A Controlled Study of Attention-Only Transformers. The feed-forward network is replaced by a Hadamard MLP. The model retains grouped-query attention, adds engram key-value memory derived from hashed n-gram tables, and uses multi-lane hyper-connections. Of the 45M total parameters, 8M live in engram tables accessed via gather operations — no matrix multiply — leaving 35M matmul-active parameters. That separation produces 70 MFLOPs per token, compared with 460 for LFM2.5 230M, 540 for FunctionGemma 270M, and approximately 6,000 for Apple FM.

Pretraining consumed a proprietary 115B-token corpus, followed by 38B tokens of post-training. Quantization to CQ2-bit using Cactus Quants is trained in from the start, not applied after the fact, meaning the deployed binary is the trained model rather than a compressed approximation. Weights never decompress into RAM; the 2-bit codes expand inside vector registers and fuse into integer dot products, keeping the arithmetic path at int8. At startup, one binary probes the CPU and selects a kernel tier — SDOT, NEON, AVX2, RISC-V vectors, wasm SIMD, or scalar — enabling the same binary to run across macOS, Linux (x86-64, ARM64, ARMv7, RISC-V, MIPS32el), Windows, Android, iOS/watchOS/tvOS, and WebAssembly.

Token Constraints, Tool Retrieval, and Confidence Gating

Three runtime mechanisms enforce correctness without adding memory. First, a byte-level grammar compiled from supplied JSON schemas constrains every emitted token; because the matcher resolves legal bytes before logits exist, the engine skips up to 98% of vocabulary projection on structural tokens. Second, when more than five tools are declared, a contrastive retrieval head embeds each schema once, scores the query per turn, and admits only the top five — unselected tools are unreachable at the grammar level, not merely deprioritised in probability. Third, every response carries a confidence value defined as the minimum of a calibrated post-hoc head and the decoding probability of the call tokens; off-topic or ambiguous requests return the empty call [], converting edge-to-cloud escalation into an explicit, threshold-driven product decision rather than a silent failure.

Attention runs a 256-token sliding window, with the system turn and tool declarations pinned as KV sinks. That pinning keeps RAM at a fixed 28MB ceiling regardless of conversation length.

Benchmark Results

Cactus evaluated Needle 2 on five public function-calling benchmarks using ordered strict exact match — names, call order, and every argument must be correct. Needle 2 ran end-to-end through its shipped engine at CQ2-bit with retrieval active; all baselines ran at f16 under vLLM. The team discloses both asymmetries upfront: the f16 baseline setup favours competing models on general benchmarks, while Needle 2's consumer device action corpus favours it on Seal-Tools.

Needle 2 leads both Seal-Tools splits and posts 98.3% function-name accuracy on Mobile Actions. It trails on BFCL v4 — Cactus attributes this to distribution mismatch, as its corpus targets consumer device actions rather than the general and enterprise API surface BFCL v4 covers. Well-formed output rate across the 3,641 BFCL rows is 93.4%.

As agentic deployments move toward constrained, autonomous runtimes, the case for a model that fits in firmware-class memory — with deterministic output formatting, explicit refusal behaviour, and zero external dependencies — grows stronger independently of leaderboard position. Needle 2's deficit on BFCL v4 is real, but the tradeoff is compute efficiency roughly two orders of magnitude below the next competitor, not parity with cloud-scale models. For teams shipping regulated or offline products where inference cannot reach an external API, that constraint changes the evaluation entirely.