Benchmark vLLM Speculative Decoding with EAGLE, n-gram, and MTP

Adapted from the vLLM project's offline speculative decoding benchmark in the vLLM repository under the Apache-2.0 licence. The code blocks below reproduce the source script and are attributed to the vLLM contributors. This guide adds a source-supported reading of the five method branches and the metric loop; it does not re-run an independent benchmark.

Speculative decoding attacks the main bottleneck in local LLM serving: autoregressive generation produces one token per forward pass, which is memory-bandwidth bound and leaves compute idle. Instead of waiting for one token, a small draft model proposes a short sequence, and the target model verifies the entire draft in a single forward pass. Accepted tokens are kept; rejected tokens are discarded. With proper rejection sampling, speculative decoding is designed to preserve the target model's output distribution while reducing target-model steps per request, which can lower latency and raise throughput on GPU-bound deployments.

This guide walks through vLLM's offline benchmark for speculative decoding. It covers five methods: EAGLE, EAGLE3, n-gram prompt lookup, a user-supplied draft model, and MTP. You will see how to configure each method, run generation, extract acceptance metrics, and interpret whether speculative decoding is actually helping your workload.

The costs are real. A separate draft model consumes additional GPU memory, and if acceptance is low, you pay extra compute for rejected tokens without much speedup. The source comment puts the built-in self-test at roughly 30 seconds on one H100 for 80 prompts, but production tuning can take much longer. You also need enough VRAM for both target and draft model weights plus KV cache; the default gpu_memory_utilization is 0.9, which can OOM when a large draft model is added.

Prerequisites

What the source actually establishes is straightforward:

• Hardware: an NVIDIA GPU with enough memory for the target model, the draft model when present, and the KV cache. The built-in test asserts tp=1, and the source comment records the test on one H100.
• Software: a Python environment with vLLM installed, plus transformers for the tokenizer. The script imports vllm.v1.metrics.reader, so you need a vLLM build that exposes V1 metrics.
• Models: the default target is meta-llama/Llama-3.1-8B-Instruct. The default EAGLE drafter is yuhuili/EAGLE-LLaMA3.1-Instruct-8B, and the default EAGLE3 drafter is yuhuili/EAGLE3-LLaMA3.1-Instruct-8B. Depending on Hugging Face access, gated models may require a token.
• Dataset: the script uses vLLM's dataset parser. The test run uses --dataset-name hf --dataset-path philschmid/mt-bench with 80 prompts.

Accounts, API keys, and distributed training infrastructure are not required by this benchmark.

Step 1: Understand the script's prompt handling

The source starts by importing vLLM's LLM class, sampling parameters, dataset utilities, and V1 metric readers. It also defines a set of image URLs for the built-in multimodal prompt mode.

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

from transformers import AutoTokenizer

from vllm import LLM, SamplingParams
from vllm.benchmarks.datasets import add_dataset_parser, get_samples
from vllm.utils.argparse_utils import FlexibleArgumentParser
from vllm.v1.metrics.reader import Counter, Vector

QUESTION = "What is the content of each image?"
IMAGE_URLS = [
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/flycatcher.jpeg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/somefish.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/starfish.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/snail.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/thistle.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/husky.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/orangetabbycat.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/guineapig.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/rabbit.jpg",
    "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/horsepony.jpg",
]


def get_custom_mm_prompts(num_prompts):
    prompts = []
    for url in IMAGE_URLS:
        prompts.append(
            [
                {"type": "image_url", "image_url": {"url": url}},
                {"type": "text", "text": QUESTION},
            ]
        )
    if num_prompts > len(IMAGE_URLS):
        prompts = prompts * (num_prompts // len(IMAGE_URLS) + 1)

    return [[{"role": "user", "content": prompt}] for prompt in prompts[:num_prompts]]

There are two prompt paths. With --custom-mm-prompts, the script cycles through the fixed image URLs and asks the model to describe each image. Without it, the script loads samples from the dataset configured through add_dataset_parser. For non-chat backends, it tokenizes each prompt with add_special_tokens=False to avoid inserting a beginning-of-sequence token twice once the chat template is applied.

Step 2: Configure the method and generation parameters

Every knob that affects speculative decoding and sampling lives in parse_args(). The default method is eagle, but you can switch to ngram, eagle3, mtp, or draft_model.

def parse_args():
    parser = FlexibleArgumentParser()
    add_dataset_parser(parser)
    parser.add_argument("--test", action="store_true")
    parser.add_argument(
        "--method",
        type=str,
        default="eagle",
        choices=["ngram", "eagle", "eagle3", "mtp", "draft_model"],
    )
    parser.add_argument("--backend", type=str, default="openai")
    parser.add_argument("--num-spec-tokens", type=int, default=2)
    parser.add_argument("--prompt-lookup-max", type=int, default=5)
    parser.add_argument("--prompt-lookup-min", type=int, default=2)
    parser.add_argument("--tp", type=int, default=1)
    parser.add_argument("--enforce-eager", action="store_true")
    parser.add_argument("--enable-chunked-prefill", action="store_true")
    parser.add_argument("--max-model-len", type=int, default=16384)
    parser.add_argument("--temp", type=float, default=0)
    parser.add_argument("--top-p", type=float, default=1.0)
    parser.add_argument("--top-k", type=int, default=-1)
    parser.add_argument("--print-output", action="store_true")
    parser.add_argument("--output-len", type=int, default=256)
    parser.add_argument("--model-dir", type=str, default=None)
    parser.add_argument("--eagle-dir", type=str, default=None)
    parser.add_argument("--draft-model", type=str, default=None)
    parser.add_argument("--custom-mm-prompts", action="store_true")
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.9)
    parser.add_argument("--disable-padded-drafter-batch", action="store_true")
    parser.add_argument("--max-num-seqs", type=int, default=None)
    parser.add_argument("--parallel-drafting", action="store_true")
    parser.add_argument("--allowed-local-media-path", type=str, default="")
    parser.add_argument("--use-heterogeneous-vocab", action="store_true")
    return parser.parse_args()

The most important flags for speculative decoding are --method, --num-spec-tokens, --eagle-dir, --draft-model, --prompt-lookup-max, and --prompt-lookup-min. Sampling is deliberately greedy by default: --temp 0, --top-p 1.0, and --top-k -1. Higher sampling temperatures usually reduce draft acceptance and therefore reduce the speedup.

Here is how the supported methods compare at configuration level:

Step 3: Build the speculative configuration and the vLLM engine

Inside main(), the script resolves the model directory, builds prompts, and creates a speculative_config dict whose shape depends on the selected method.

def main(args):
    model_dir = args.model_dir
    if args.model_dir is None:
        if args.custom_mm_prompts:
            raise ValueError(
                "custom_mm_prompts requires mm based models"
                "default llama3.1-8b-instruct is not mm based"
                "please specify model_dir to give a mm based model"
            )
        model_dir = "meta-llama/Llama-3.1-8B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_dir)

    if args.custom_mm_prompts:
        prompts = llm_prompts = get_custom_mm_prompts(args.num_prompts)
    else:
        prompts = get_samples(args, tokenizer)
        if args.enable_multimodal_chat:
            llm_prompts = [p.prompt for p in prompts]
        else:
            # add_special_tokens is False to avoid adding bos twice
            # when using chat templates
            llm_prompts = [
                {
                    "prompt_token_ids": tokenizer.encode(
                        prompt.prompt, add_special_tokens=False
                    ),
                    "multi_modal_data": prompt.multi_modal_data,
                }
                for prompt in prompts
            ]
    if args.method == "eagle" or args.method == "eagle3":
        eagle_dir = args.eagle_dir
        if args.method == "eagle" and eagle_dir is None:
            eagle_dir = "yuhuili/EAGLE-LLaMA3.1-Instruct-8B"

        elif args.method == "eagle3" and eagle_dir is None:
            eagle_dir = "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B"
        speculative_config = {
            "method": args.method,
            "model": eagle_dir,
            "num_speculative_tokens": args.num_spec_tokens,
            "disable_padded_drafter_batch": args.disable_padded_drafter_batch,
            "parallel_drafting": args.parallel_drafting,
        }
    elif args.method == "ngram":
        speculative_config = {
            "method": "ngram",
            "num_speculative_tokens": args.num_spec_tokens,
            "prompt_lookup_max": args.prompt_lookup_max,
            "prompt_lookup_min": args.prompt_lookup_min,
        }
    elif args.method == "draft_model":
        assert args.draft_model is not None and args.draft_model != ""
        speculative_config = {
            "method": args.method,
            "model": args.draft_model,
            "num_speculative_tokens": args.num_spec_tokens,
            "enforce_eager": args.enforce_eager,
            "max_model_len": args.max_model_len,
            "parallel_drafting": args.parallel_drafting,
            "use_heterogeneous_vocab": args.use_heterogeneous_vocab,
        }
    elif args.method == "mtp":
        speculative_config = {
            "method": "mtp",
            "num_speculative_tokens": args.num_spec_tokens,
        }
    else:
        raise ValueError(f"unknown method: {args.method}")

    llm = LLM(
        model=model_dir,
        trust_remote_code=True,
        tensor_parallel_size=args.tp,
        enable_chunked_prefill=args.enable_chunked_prefill,
        enforce_eager=args.enforce_eager,
        gpu_memory_utilization=args.gpu_memory_utilization,
        speculative_config=speculative_config,
        disable_log_stats=False,
        max_model_len=args.max_model_len,
        limit_mm_per_prompt={"image": 5},
        disable_chunked_mm_input=True,
        max_num_seqs=args.max_num_seqs,
        allowed_local_media_path=args.allowed_local_media_path,
    )

For EAGLE and EAGLE3, the script supplies sensible default draft models when none is specified. For draft_model, you must pass --draft-model explicitly or the assertion fails. For ngram, no extra model is loaded; the engine uses prompt lookup over request context. For mtp, no separate drafter is configured because the target model itself contains the multi-token heads.

The LLM call passes speculative_config through to vLLM's engine. Note that trust_remote_code=True is enabled by default in this script, which means remote model code will be executed. Use it only with models you trust.

Step 4: Run generation and extract acceptance metrics

After the engine is built, the script creates greedy sampling parameters, runs either chat or generate, and then reads counters from llm.get_metrics().

    sampling_params = SamplingParams(temperature=args.temp, max_tokens=args.output_len)
    if args.backend == "openai-chat":
        outputs = llm.chat(llm_prompts, sampling_params=sampling_params)
    else:
        outputs = llm.generate(
            llm_prompts,
            sampling_params=sampling_params,
        )

    # print the generated text
    if args.print_output:
        for i, output in enumerate(outputs):
            print("-" * 50)
            if not args.custom_mm_prompts:
                print(f"prompt: {prompts[i].prompt}")
            else:
                print(f"prompt: {prompts[i]}")
            print(f"generated text: {output.outputs[0].text}")
            print("-" * 50)

    metrics = llm.get_metrics()

    total_num_output_tokens = sum(
        len(output.outputs[0].token_ids) for output in outputs
    )
    num_drafts = 0
    num_draft_tokens = 0
    num_accepted_tokens = 0
    acceptance_counts = [0] * args.num_spec_tokens
    for metric in metrics:
        if metric.name == "vllm:spec_decode_num_drafts":
            assert isinstance(metric, Counter)
            num_drafts += metric.value
        elif metric.name == "vllm:spec_decode_num_draft_tokens":
            assert isinstance(metric, Counter)
            num_draft_tokens += metric.value
        elif metric.name == "vllm:spec_decode_num_accepted_tokens":
            assert isinstance(metric, Counter)
            num_accepted_tokens += metric.value
        elif metric.name == "vllm:spec_decode_num_accepted_tokens_per_pos":
            assert isinstance(metric, Vector)
            for pos in range(len(metric.values)):
                acceptance_counts[pos] += metric.values[pos]

    print("-" * 50)
    print(f"total_num_output_tokens: {total_num_output_tokens}")
    print(f"num_drafts: {num_drafts}")
    print(f"num_draft_tokens: {num_draft_tokens}")
    print(f"num_accepted_tokens: {num_accepted_tokens}")
    acceptance_length = 1 + (num_accepted_tokens / num_drafts) if num_drafts > 0 else 1
    print(f"mean acceptance length: {acceptance_length:.2f}")
    print("-" * 50)

    # print acceptance at each token position
    for i in range(len(acceptance_counts)):
        acceptance_rate = acceptance_counts[i] / num_drafts if num_drafts > 0 else 0
        print(f"acceptance at token {i}: {acceptance_rate:.2f}")

    return acceptance_length

The key number is mean acceptance length: the average number of draft tokens accepted per speculative step, including the guaranteed target token. An acceptance length of 1 means no drafts are accepted, so speculative decoding is pure overhead. Values above 2 are often worthwhile, but the exact break-even depends on the relative cost of the draft model versus the target model.

The per-position acceptance rates show how far into the draft sequence tokens survive. If acceptance at token 0 is high but token 2 collapses, increasing --num-spec-tokens may not help much because the later drafts rarely pay off.

Step 5: Run the built-in self-test

The script includes a test mode that validates acceptance length against known values when run with the specified test configuration. It is useful as a smoke test after changing vLLM versions or GPU memory settings.

if __name__ == "__main__":
    args = parse_args()
    args.enable_multimodal_chat = args.backend == "openai-chat"

    acceptance_length = main(args)

    if args.test:
        # takes ~30s to run on 1xH100
        assert args.method in ["eagle", "eagle3"]
        assert args.tp == 1
        assert args.num_spec_tokens == 3
        assert args.dataset_name == "hf"
        assert args.dataset_path == "philschmid/mt-bench"
        assert args.num_prompts == 80
        assert args.temp == 0
        assert args.top_p == 1.0
        assert args.top_k == -1
        assert args.enable_chunked_prefill

        # check acceptance length is within 2% of expected value
        rtol = 0.02
        expected_acceptance_length = 2.296 if args.method == "eagle" else 2.811

        assert (
            acceptance_length <= (1 + rtol) * expected_acceptance_length
            and acceptance_length >= (1 - rtol) * expected_acceptance_length
        ), (
            f"acceptance_length {acceptance_length} is not "
            f"within {rtol * 100}% of {expected_acceptance_length}"
        )

        print(
            f"Test passed! Expected AL: "
            f"{expected_acceptance_length}, got {acceptance_length}"
        )

To reproduce the test, run the script with --test --method eagle --num-spec-tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --enable-chunked-prefill. The expected acceptance length is 2.296 for EAGLE and 2.811 for EAGLE3. The assertion allows only 2% deviation, so if GPU memory pressure changes batch sizes or you alter the prompt set, the test will fail even though the setup may still be correct.

What to watch out for

• Acceptance length is temperature-sensitive. The script defaults to greedy sampling with temp=0. As soon as you raise temperature, draft acceptance drops. A method that looks great at temperature 0 may be slower than standard generation at temperature 1.
• Drafter-target mismatch. EAGLE drafters are tied to a specific base model. If you change --model-dir but keep the default EAGLE drafter, you will get garbage tokens or a crash. Always match --eagle-dir to your target.
• Vocabulary mismatches with custom draft models. draft_model paths may use a different tokenizer. If the vocabularies do not match exactly, you need --use-heterogeneous-vocab. Otherwise the target model cannot evaluate the draft logits correctly.
• Memory pressure. The default gpu_memory_utilization of 0.9 is optimistic when both a target and a draft model are loaded. If you hit an out-of-memory error, lower this value before touching max_model_len or max_num_seqs.
• Multimodal prompt validation. --custom-mm-prompts requires a multimodal target model. The default Llama-3.1-8B-Instruct is not multimodal, and the script raises a ValueError if you try.
• The self-test is not a general benchmark. It asserts very specific flags. Passing it tells you the script works in a known configuration; it does not tell you whether speculative decoding is faster for your real traffic. You still need to compare tokens/s against a non-speculative baseline.

For production decisions, run this script with your own dataset, at your serving temperature, and compare both acceptance length and end-to-end throughput against standard generation. If latency in long-context prefill is also a constraint, the techniques in Tokengate's 11.3x prefill improvement may complement speculative decoding on the decode side.

Where to go next

vLLM's documentation covers speculative decoding configuration in much more depth, including per-method tradeoffs and advanced options. If you are tuning multimodal or video serving latency, see Kyutai's CADRE video autoencoders and latency work. For broader throughput work, benchmark accepted-token count per second rather than raw tokens per second; that metric often reveals whether speculative decoding is actually paying for its extra memory.