LLM Inference
8 pieces on LLM Inference, including 2 step-by-step guides.
Guides
Run SGLang Offline Batch Inference Directly from Python
Use SGLang's offline_batch_inference.py pattern to batch-generate text without an HTTP server.
Chain of Verification with SGLang: Reduce LLM Hallucinations
Build a factored Chain-of-Verification pipeline with SGLang that runs draft, verify, refine, and summarize as independent LLM calls.
News & Analysis
Architect's Liquid Inference Auctions Every LLM Request
Architect's Liquid Inference auctions every LLM request across providers, locking the max price before the first token and paying 20% referral fees.
SageMaker Prefix-Aware Routing Cuts LLM P50 TTFT by Up to 77%
Amazon SageMaker's new PREFIX_AWARE routing lifts KV cache hit rates from ~25% to ~82% on long-context workloads, cutting P50 TTFT by up to 77%.
Chunked Prefill Beats Disaggregation Below 1,000 GPUs
Every major inference framework now ships prefill-decode disaggregation — but for most teams it's the wrong default, and chunked prefill is the safer fix.
Shopify's Gisting Cuts 6,000-Token Prompts to 1,500, Drops Latency 38%
Shopify compressed Sidekick's system prompt from 6,000 to 1,500 tokens using learned gist embeddings, cutting end-to-end latency from 6.8s to 4.2s in production.
AWS Cuts RAG Token Costs 33% With a Two-Call Compression Pattern
A post-retrieval Lambda function routes chunks through Claude Haiku before Claude Sonnet, cutting tokens by 8.6× and costs by 33% across 500K documents.
PagedAttention vs RadixAttention: How LLMs Tame the KV Cache
Two architectures attack LLM KV cache inefficiency from opposite angles — memory fragmentation and redundant prefill. Here is how each works.