SGLang

5 pieces on SGLang, including 2 step-by-step guides.

Guides

Run SGLang Offline Batch Inference Directly from Python

Shan • 2026-10-10

sglangllm inferencebatch processingpythongpuradixattention

Run SGLang Offline Batch Inference Directly from Python

Use SGLang's offline_batch_inference.py pattern to batch-generate text without an HTTP server.

Chain of Verification with SGLang: Reduce LLM Hallucinations

Shan • 2026-08-29

SGLanghallucinationLLM inferencechain-of-verificationPython

Chain of Verification with SGLang: Reduce LLM Hallucinations

Build a factored Chain-of-Verification pipeline with SGLang that runs draft, verify, refine, and summarize as independent LLM calls.

News & Analysis

architect • 2026-10-26

SGLangKV CacheAI AgentsOpen SourceServing Infrastructure

AKV Cuts Agent Serving Cost 27.7% Without Rewriting History

Tsinghua’s AKV drops and repositions KV cache instead of summarizing agent history, cutting average serving cost 27.7% on SGLang.

Shan • 2026-09-12

Knowledge DistillationLLM ServingRanking SystemsSGLangLinkedInInference Optimization

LinkedIn's 0.6B Student Model Trains 8x Faster via Multi-Teacher Distillation

LinkedIn stacks five infrastructure optimizations to achieve an 8x training speedup for a 0.6B-parameter job-search ranking model supervised by two teacher LLMs.

Shan • 2026-08-21

LLM InferenceKV CachePagedAttentionRadixAttentionvLLMSGLang

PagedAttention vs RadixAttention: How LLMs Tame the KV Cache

Two architectures attack LLM KV cache inefficiency from opposite angles — memory fragmentation and redundant prefill. Here is how each works.