Shopify's Gisting Cuts 6,000-Token Prompts to 1,500, Drops Latency 38%

Shopify's engineering team has published details on Gisting, a technique that compresses lengthy LLM system prompts into a compact set of learned "gist" tokens, deployed in production against the Sidekick GraphQL agent. The technique is not a summarization heuristic — it trains token embeddings that minimize the KL divergence between the model's output distribution when given the original prompt and its output distribution when given only the gist tokens. The compressed representation is optimized to reproduce model behavior rather than human-readable meaning: you are distilling behavioral equivalence into fewer tokens, not trading prompt fidelity for brevity.

Applied to Sidekick's system prompt, Gisting reduced context size from roughly 6,000 tokens to 1,500 gist tokens — a 4:1 compression ratio — without measurable degradation in prediction quality. Fewer tokens processed on every request cascades into lower median latency, higher throughput, and a reduced GPU footprint. This places Gisting alongside other pipeline architecture improvements that deliver gains without requiring better underlying models.

Training Mechanics

The training procedure runs two forward passes. In the teacher pass, the model ingests the real system prompt and produces a distribution over the response vocabulary — the teacher logits. In the student pass, the same model ingests only the gist tokens and produces student logits. The gist token embeddings are updated to minimize KL divergence between the two distributions until the student's predictions closely match the teacher's across the relevant output space.

Once training converges, the gist embeddings are written directly into the model's embedding matrix, and the gist tokens are registered as special tokens in the tokenizer. No custom attention masking, no additional encoder, and no modified serving path are required at inference time — the model loads and runs identically to any standard checkpoint. This clean integration boundary avoids the operational complexity of maintaining a separate compression service or a forked inference stack. Shopify also uses an autosearch process to tune the Gisting procedure; the source does not expose the specific hyperparameters or search space involved.

The foundational approach traces to a 2022 paper on prompt compression and contrastive conditioning.

Measured Production Impact

At 350 requests per minute, Shopify recorded the following for the Sidekick GraphQL agent:

The 2.6-second end-to-end latency reduction is especially relevant in agentic workflows where multiple sequential LLM calls compound per user interaction. The throughput gain from 20.2 to 23.4 QPS at the same request rate allowed Shopify to reduce the number of allocated GPUs, making this a direct infrastructure cost lever rather than a purely latency-focused optimization.

Composability with Prefix Caching

Shopify frames Gisting as complementary to, not a replacement for, prefix caching. Prefix caching avoids recomputing KV tensors for repeated prompt prefixes, but the model still processes those cached tensors during decoding. Because Gisting shortens the token sequence itself rather than bypassing computation, the two techniques target different phases of the inference pipeline and their benefits compound. Shopify deploys both simultaneously on Sidekick.

This composability matters for teams that have already adopted prefix caching and assumed most prompt-related gains were captured. Gisting opens a second optimization axis — sequence length at the embedding level — that prefix caching leaves untouched. Teams looking to automate LLM prompt optimization in production now have a structurally grounded option beyond static prompt pruning or retrieval-augmented reduction.

The production numbers reported for Sidekick make a strong case for evaluating Gisting anywhere long, stable system prompts dominate context budgets. Whether Shopify's specific autosearch tuning and embedding injection approach generalizes across model families and diverse system prompt structures remains an open engineering question.