Perplexity's Ivy, Tulip and ROSE Serve pplx-embed Without a Separate Engine

Perplexity's engineering team has published Fast Embeddings on GPUs, a detailed account of the serving infrastructure behind pplx-embed and the reranking models running across Perplexity Search, Computer, and its API Platform. The central claim: on mature Hopper and Blackwell hardware, embedding inference has largely converged across engines, and the remaining headroom sits in the runtime harness — CUDA graph management, asynchronous result tracking, and a Rust request path. This repositions GPU embedding serving as a systems-engineering problem rather than a kernel-selection problem, consistent with the argument that architectural specificity outperforms GPU scaling.

Ivy, Tulip, and ROSE

Every embedding request traverses three internal services. Ivy is a Rust HTTP gateway responsible for all CPU-side work: JSON parsing, tokenization, input templating, and batch splitting. It translates incoming requests into a custom gRPC protocol and load-balances large-batch requests by splitting them into chunks distributed across replicas — directly addressing the load imbalance that emerges when production payload sizes vary. Tulip is a gRPC inference server interface written in Rust using tokio and tonic; it accumulates requests, schedules and packs them into batches, then dispatches to the engine using a first-come, first-served policy. ROSE (Runtime-Optimized Serving Engine) is primarily Python and sits closest to the hardware: it provides kernels, layer definitions, and model architecture, manages CUDA graphs, and exposes a step() function to Tulip.

The deliberate simplicity of Tulip's scheduler is justified by a specific measurement. For small embedding models at the sequence lengths Perplexity actually serves, the linear cost of dense layers dominates the quadratic cost of attention — meaning latency is approximately proportional to token count, not sequence count. Once a batch reaches roughly 512 tokens on a sub-billion-parameter model, the GPU is saturated and packing in additional sequences yields no efficiency gain.

Kernel Reuse and Attention Backend Selection

Rather than building a dedicated embedding engine, Perplexity reuses the prefill and decode kernels from its LLM stack. Batch embedding — compute-bound like prefill — and online embedding of short queries — memory-bound like decode — map naturally onto existing infrastructure, avoiding duplicated kernel maintenance across two separate serving paths. This is an instance of the systems-engineering gains that increasingly rival scaling.

ROSE supports three attention backends for ragged inputs: FlashInfer 2, FlashInfer 3, and FlashAttention 4. FlashAttention 4 is generally faster, but FlashInfer 3 outperforms it on Qwen-based models at very long sequence lengths, making backend selection model- and length-dependent. When serving an embedding model specifically, ROSE skips KV cache instantiation entirely and dispatches to ragged attention variants to avoid padding overhead.

CUDA Graphs and the LazyTensor Abstraction

On small batches, CPU-side kernel launch overhead can exceed actual GPU execution time. Perplexity addresses this by building whole-model CUDA graphs for all embedding models, collapsing every individual kernel launch into a single driver call. The inflection point where GPU work exceeds launch cost occurs at batches of thousands of tokens and tens of sequences for sub-billion-parameter models. One engineering obstacle: certain attention implementations block full-model graph capture by depending on dynamic host-side inputs. Perplexity resolved this by upstreaming changes to FlashInfer to enable capture.

Graph capture must be repeated per configuration, with token counts padded to buckets that are multiples of 64 or 256 — producing thousands of graphs and multiple minutes of capture time per model. The solution is lazy capture: each configuration gets an eager warmup run on first hit, then triggers capture and replay on its second hit. This trades elevated p99 latency at startup for amortizing capture cost across hours of operation.

The LazyTensor eliminates blocking device synchronization. Instead of step() waiting on the device, it returns a LazyTensor that tracks a page-locked host buffer, a cudaMemcpyAsync, and a CUDA event. A Rust async task can then wait on batch N's results while the CPU is already enqueuing batch N+1, overlapping CPU batch preparation with in-flight GPU work.

Benchmark Setup

All benchmarks run against vLLM v0.22.0 in BF16 on real weights with eval-derived inputs. Warmup runs verify cosine similarity divergence within 0.1%, establishing that the optimizations preserve numerical fidelity. Ivy, Tulip, and ROSE remain internal infrastructure; the embedding capability is externally accessible via Perplexity's Embeddings API as pplx-embed.

The stack illustrates a pattern tracked in pipeline architecture driving AI gains in 2026: frontier products differentiating at the serving layer through runtime engineering — kernel reuse, lazy graph capture, and async tensor tracking — rather than at the model layer alone.