Chain of Verification with SGLang: Reduce LLM Hallucinations

Hallucinations are not random noise — they are confident, coherent, grammatically flawless lies. A model asked when the Titanic sank might answer 1911 with the same fluency it would use to answer 1912. Chain of Verification (CoVe) addresses this by treating the model's first answer not as a final output but as a hypothesis to be challenged. The technique, introduced in Dhuliawala et al. (2023), routes the initial draft through an independent verification pass that checks whether the answer actually addresses the question, then — when it fails — asks the verifier to produce a correction in the same session where it built its critique. The result is a multi-step pipeline that catches a meaningful proportion of factual errors before they reach the user, without requiring a second, external model.

This guide is adapted from SGLang's chain_of_verification.py, released under the Apache-2.0 licence. The implementation follows the factored variant of CoVe, which matters: in the factored approach the verification call starts a completely fresh chat session with no shared KV-cache from the draft. The original paper shows this consistently outperforms the joint variant, where the verifier can attend to the draft's token history and tends to simply repeat it. Engineers building retrieval-augmented pipelines, fact-sensitive chatbots, or any system where an undetected hallucination carries real cost — medical summaries, legal research, financial data extraction — will find this pattern worth the overhead.

The overhead is concrete. Each query now takes three LLM calls instead of one (four if you enable summarization), so latency and token costs multiply accordingly. This is not a pattern to apply unconditionally; it is a pattern to apply where the cost of a hallucination exceeds the cost of extra inference. Discussions around inference cost reduction with hardware like Nvidia MPS are worth reviewing if per-query budget is a hard constraint.

---

Prerequisites

• Hardware: A GPU with enough VRAM to serve your chosen model. Llama-3.1-8B-Instruct fits comfortably on a single 24 GB consumer GPU; larger models need proportionally more. CPU-only serving is possible but slow enough to make the three-call overhead painful in interactive settings.
• SGLang server: Install via pip install sglang[all] and confirm you can launch python -m sglang.launch_server. The OpenAI-compatible endpoint (/v1/chat/completions) must be reachable.
• Python packages: openai (the client library, used here to talk to SGLang's compatible endpoint, not OpenAI's cloud).
• Model weights: Any model served by SGLang that supports chat-completion format. The examples below use meta-llama/Llama-3.1-8B-Instruct.

---

Step 1: Launch the SGLang server

Before running the CoVe script, bring up the inference server. SGLang exposes an OpenAI-compatible API, which the pipeline uses as its transport layer.

python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000

Wait until the server prints a ready message before proceeding. The default base URL the script targets is http://127.0.0.1:30000/v1 — a localhost address that only resolves when the server is running on the same machine. If you run on a remote machine or a different port, pass --base-url accordingly at runtime. The api_key is set to the string "EMPTY" — SGLang does not validate it, but the OpenAI client library requires a non-null value.

---

Step 2: Define the prompts and the shared chat helper

The CoVe pipeline is built around two system-level strings and a thin wrapper around the OpenAI completion call. These constants drive the verifier's behaviour and are worth understanding before you touch them.

VERIFY_SYSTEM_PROMPT = (
    "You are a strict fact-checker. "
    "You will be given a user question and a candidate answer. "
    "Decide whether the answer is accurate and directly addresses the question. "
    "Reply with exactly one of: PASS or FAIL, followed by a brief reason."
)

REFINE_INSTRUCTION = (
    "The previous answer was flagged as inaccurate or off-topic. "
    "Please provide a corrected, accurate answer to the original question."
)

SUMMARIZE_INSTRUCTION = (
    "Please give a concise, one-paragraph version of the verified answer above."
)

def resolve_model(client: OpenAI, model: str | None) -> str:
    if model:
        return model
    models = client.models.list()
    if not models.data:
        raise RuntimeError("Server returned no models from /v1/models")
    return models.data[0].id


def chat(
    client: OpenAI,
    model: str,
    messages: list[dict[str, Any]],
    max_tokens: int,
    temperature: float,
) -> str:
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=max_tokens,
        temperature=temperature,
    )
    msg = response.choices[0].message
    # Reasoning models (e.g. Kimi-K2.5, Qwen3) may return the final answer in
    # `content` and the chain-of-thought in `reasoning_content`.  In some
    # configurations (greedy / temperature=0) `content` can be empty while the
    # useful text lives in `reasoning_content`, so fall back to it.
    content = msg.content or ""
    if not content.strip():
        content = getattr(msg, "reasoning_content", None) or ""
    return content

The VERIFY_SYSTEM_PROMPT constrains the verifier to emit a binary verdict (PASS or FAIL) plus a reason. That structure is what makes the downstream passed = verdict.strip().upper().startswith("PASS") check reliable. If you change the wording, you must also revisit the parsing logic — an unconstrained verifier frequently produces multi-sentence verdicts that make prefix-matching ambiguous. The chat helper also handles reasoning models that put their useful output in reasoning_content rather than content, a behaviour you will encounter with Qwen3 and similar architectures.

---

Step 3: Implement the four-stage CoVe pipeline

The core function wires together the draft, verify, refine, and optional summarize steps. The factored design — fresh verify_messages list with no overlap with draft_messages — is the structural decision that separates this from a naive self-critique loop.

def chain_of_verification(
    client: OpenAI,
    model: str,
    user_query: str,
    max_tokens: int = 1024,
    temperature: float = 0.6,
    summarize: bool = False,
    verbose: bool = True,
) -> str:

    def _log(title: str, text: str) -> None:
        if verbose:
            print(f"\n{'=' * 60}")
            print(f"[{title}]")
            print(text)

    # ------------------------------------------------------------------
    # Step 1 — Draft: generate the initial answer
    # ------------------------------------------------------------------
    draft_messages: list[dict[str, Any]] = [
        {"role": "user", "content": user_query},
    ]
    draft_answer = chat(client, model, draft_messages, max_tokens, temperature)
    _log("Step 1 · Draft answer", draft_answer)

    # ------------------------------------------------------------------
    # Step 2 — Verify (Factored): fresh session, no shared history/cache
    #
    # The verification session deliberately starts from scratch so the
    # model cannot attend to the draft answer's token embeddings.  This
    # mirrors the "factored" variant in the CoVe paper, which consistently
    # outperforms the joint variant.
    # ------------------------------------------------------------------
    verify_messages: list[dict[str, Any]] = [
        {"role": "system", "content": VERIFY_SYSTEM_PROMPT},
        {
            "role": "user",
            "content": (
                f"Question: {user_query}\n\n"
                f"Candidate answer:\n{draft_answer}\n\n"
                "Does this answer accurately and completely address the question?"
            ),
        },
    ]
    verdict = chat(client, model, verify_messages, 256, temperature)
    _log("Step 2 · Verification verdict", verdict)

    passed = verdict.strip().upper().startswith("PASS")

    if passed:
        final_answer = draft_answer
        _log("Result", "Verification PASSED — using draft answer as final answer.")
    else:
        # ------------------------------------------------------------------
        # Step 3 — Refine: ask the verifier to correct its own critique
        #
        # We continue in the *verify* session (not the draft session) so the
        # model has context about *why* the draft failed.
        # ------------------------------------------------------------------
        verify_messages.append({"role": "assistant", "content": verdict})
        verify_messages.append({"role": "user", "content": REFINE_INSTRUCTION})
        refined_answer = chat(client, model, verify_messages, max_tokens, temperature)
        _log("Step 3 · Refined answer", refined_answer)
        final_answer = refined_answer

    # ------------------------------------------------------------------
    # Step 4 — Summarize (optional)
    # ------------------------------------------------------------------
    if summarize:
        summarize_messages: list[dict[str, Any]] = [
            {"role": "user", "content": user_query},
            {"role": "assistant", "content": final_answer},
            {"role": "user", "content": SUMMARIZE_INSTRUCTION},
        ]
        summary = chat(client, model, summarize_messages, 256, temperature)
        _log("Step 4 · Summary (optional)", summary)
        final_answer = summary

    return final_answer

Notice that the refinement step appends to verify_messages, not draft_messages. This is intentional: the model that produced the critique now also owns the correction, giving it full context for why the draft was wrong. The summarization step, by contrast, opens yet another fresh session — it only needs the final answer, not the critique.

---

Step 4: Wire up the CLI and run

The entry point parses arguments and calls the pipeline. The CLI --max-tokens default is 10240 (set in parse_args); the function signature default of 1024 applies only when calling chain_of_verification directly from Python. The api_key="EMPTY" value is required by the OpenAI client library but is not validated by SGLang.

def main() -> None:
    args = parse_args()
    client = OpenAI(api_key="EMPTY", base_url=args.base_url)

    try:
        model = resolve_model(client, args.model)
    except Exception as exc:
        print(f"Failed to connect to SGLang server: {exc}", file=sys.stderr)
        print(f"  Check server at {args.base_url}", file=sys.stderr)
        sys.exit(1)

    print(f"Model  : {model}")
    print(f"Query  : {args.prompt}")

    final = chain_of_verification(
        client=client,
        model=model,
        user_query=args.prompt,
        max_tokens=args.max_tokens,
        temperature=args.temperature,
        summarize=args.summarize,
        verbose=not args.quiet,
    )

    if args.quiet:
        print(final)

Run it against a live server:

python chain_of_verification.py --prompt "Who invented the telephone and in what year?"

To suppress intermediate step output and see only the final answer:

python chain_of_verification.py --prompt "What year did the Titanic sink?" --quiet

To enable the optional summarization step:

python chain_of_verification.py \
    --prompt "Explain the causes of World War I" \
    --summarize

---

Parameter tradeoffs

---

What to watch out for

The verifier can be sycophantic. Smaller models in the 7B–13B parameter range serving as both drafter and verifier tend to PASS their own answers at a high rate, especially when temperature is low. The pipeline structure helps, but if the verifier never flags anything, try raising temperature to 0.6–0.8 or route the verification call to a larger model by pointing --base-url at a separate server.

Verdict format drift under load. The VERIFY_SYSTEM_PROMPT requests a response starting with exactly PASS or FAIL. Under high temperature or with models that have not been strongly instruction-tuned, you will occasionally receive verdicts like "The answer PASSES" or "I would say FAIL" — the latter parses correctly, but the former will be misread as FAIL because the check is startswith("PASS"). Monitor the raw verdict strings in production and add a secondary regex check if your use case is sensitive to false negatives.

Three calls means three failure modes. A network timeout, context-length overflow, or model error on any of the three calls aborts the pipeline without a partial result. The current implementation does not retry. For production use, wrap each chat(...) call in a retry loop with exponential backoff, and consider returning the draft answer as a fallback when verification fails structurally rather than logically.

KV-cache savings disappear. One of SGLang's headline advantages is aggressive KV-cache reuse across requests. The factored design deliberately discards that cache for the verification pass, which is correct for accuracy but means you lose the latency benefit for that call. If you are running a high-QPS service, batch verification calls together or pre-warm a common system-prompt prefix to recover some cache utility.

Token cost accumulates non-linearly. A 500-token question-and-answer pair becomes roughly 500 (draft) + 800 (question + draft answer in verify prompt + verdict) + 1200 (verify context + refined answer) tokens across three calls. Enabling summarization adds a fourth. Budget accordingly; this is not a 3× multiplier but closer to 4–5× when you account for the prompt overhead of passing the draft into the verify call.

Reasoning-model quirks. As noted in the chat helper, some models (Qwen3, certain Kimi variants) return empty content with the actual response in reasoning_content. The fallback in the helper handles this, but if you swap in a new model and see blank outputs at any step, this is the first place to check. This kind of architectural variation across model families is increasingly common — see coverage of next-architecture convergence trends for context on why this will become more prevalent.

---

Where to go next

The natural extension of this pipeline is specialization: replace the generic VERIFY_SYSTEM_PROMPT with a domain-specific fact-checking rubric, or route the verification call to a different, larger model than the one producing drafts. SGLang's multi-endpoint support makes the latter straightforward — instantiate a second OpenAI client pointing at a second server and pass it to a modified chain_of_verification that accepts separate draft and verify clients. From there, structured output constraints (JSON schema enforcement via SGLang's --constrained-decoding flag) can make the PASS/FAIL parsing robust without relying on prompt engineering. The SGLang repository contains additional examples for constrained generation, batched inference, and multi-turn agent patterns that pair naturally with what you have built here.