Benchmark Local Models with llama.cpp's llama-eval.py

When you change something in a local model's serving stack — a new quantization, a different batch size, a patched sampler, a rebuilt GGUF — the question is always the same: did the change make the model better, worse, or just different? For many practitioners the answer comes from vibes, a handful of manual prompts, or a leaderboard number that does not reproduce on their hardware. llama-eval.py, shipped in the llama.cpp repository, is the better answer: a single-file evaluation harness that turns a running llama.cpp server into a benchmark target for mathematics and science suites — AIME (three vintages), GSM8K, and GPQA — and reports accuracy with Wilson confidence intervals rather than a misleading raw percentage.

This guide is adapted from llama.cpp's llama-eval.py, which is MIT-licensed and reproduced here with attribution to the llama.cpp contributors. The code excerpts below are copied verbatim from that source; a few long methods are shown only in part, with the omission noted where it occurs.

The audience is engineers who already run llama.cpp in server mode and want a reproducible baseline before committing a change, plus teams publishing local-model numbers that need to be defensible. Costs depend on model size, hardware, and generation length, which the harness exposes as n_predict. Disk costs are modest — datasets cache under ~/.cache/huggingface/datasets — and the only network dependency after first run is GPQA, which is fetched from a remote CSV. With evals increasingly gated or pushed off public infrastructure (see our coverage of Anthropic cutting evals off the internet), a local harness means your regression signal does not depend on anyone else's uptime.

Prerequisites

Everything that follows comes from the source's own imports and assumptions. You need:

• A running llama.cpp server with the OpenAI-compatible /v1/models and /v1/chat/completions endpoints mounted. The harness never launches the server itself; it only reaches it over HTTP. You can run one or several instances, because each becomes a ServerConfig.
• Python 3 with requests, datasets, pandas, and tqdm. pandas is only used by GPQA, and datasets only by the HuggingFace-backed suites.
• Hardware with enough VRAM — or enough patience, for CPU inference — to serve the model under evaluation. If you opt into the LLM grader later, reserve separate capacity for a grader model.
• Disk for the HuggingFace cache, plus a one-time network fetch for GPQA's CSV from openaipublic blob storage.

Step 1: Bring up a server and prove it is reachable

The harness models every inference target with the ServerConfig dataclass, and the first thing it does per target is probe /v1/models. This fail-fast check means a typo in the port or a server that crashed during startup surfaces immediately, not after twenty prompts have silently failed.

@dataclass
class ServerConfig:
    url: str
    threads: int
    name: str = ""

The check itself is a static method on the Processor class:

    @staticmethod
    def _check_server(server_config: ServerConfig) -> List[str]:
        url = f"{server_config.url}/v1/models"
        try:
            response = requests.get(url)
            response.raise_for_status()
            models = [m["id"] for m in response.json().get("data", [])]
            return models
        except Exception as e:
            print(f"Error: Cannot reach server {server_config.name} ({server_config.url}): {e}", file=sys.stderr)
            sys.exit(1)

The name field is cosmetic but load-bearing for multi-server runs: every stored result carries a server_name, so when you compare a Q4_K_M against a Q6_K quant side by side, the report tells you which number came from which server.

Step 2: Load a benchmark and assemble the task list

EvalState owns the dataset, the task list, and the results dictionary that gets dumped to JSON. load_dataset() is a registry: it maps a dataset_type string to a dataset class. setup_tasks() then turns the dataset into a list of (index, task_id) pairs, with two behaviours worth noting. First, the list is seeded-shuffled so runs are reproducible given the same seed. Second, if you request more cases than the dataset contains, the script tiles the dataset into chunks — so you can run every problem multiple times and measure per-problem pass@k or stability across repeated generations.

    def load_dataset(self, seed: int = 1234):
        if self.dataset_type == "aime":
            self.dataset = AimeDataset()
        elif self.dataset_type == "aime2025":
            self.dataset = Aime2025Dataset()
        elif self.dataset_type == "aime2026":
            self.dataset = Aime2026Dataset()
        elif self.dataset_type == "gsm8k":
            self.dataset = Gsm8kDataset()
        elif self.dataset_type == "gpqa":
            self.dataset = GpqaDataset(variant="diamond", seed=seed)
        else:
            raise ValueError(f"Unknown dataset type: {self.dataset_type}")

    def setup_tasks(self, n_cases: Optional[int] = None, seed: int = 1234):
        if self.dataset is None:
            raise ValueError("Dataset not loaded. Call load_dataset() first.")

        if n_cases is None:
            n_cases = len(self.dataset)

        dataset_size = len(self.dataset)
        rng = random.Random(seed)

        self.tasks = []
        for chunk_idx in range((n_cases + dataset_size - 1) // dataset_size):
            chunk_size = min(dataset_size, n_cases - chunk_idx * dataset_size)
            indices = list(range(dataset_size))
            rng.shuffle(indices)
            chunk_indices = indices[:chunk_size]

            for i in chunk_indices:
                task_id = f"{self.dataset_type}_{chunk_idx:03d}_{i:03d}"
                self.tasks.append((i, task_id))

        self.all_tasks = list(self.tasks)

The task_id format encodes both the chunk index and the problem index, which is what lets the resume logic later distinguish "already scored in run one" from "pending in run two" without a separate metadata table.

Step 3: Choose a grader

The grader is the most consequential decision you will make, because it decides what "correct" means. Three grader types are available. The regex grader stores per-dataset patterns; for the original AIME and AIME 2025 it first looks for \boxed{...} matches and takes the last one, treating that as the final answer, then falls back to scanning for numbers from the end of the response. AIME 2026 uses the numeric fallback only, because the source special-cases only aime and aime2025.

GRADER_PATTERNS = {
    "aime": r'\boxed{(\d+)}|\b(\d+)\b',
    "aime2025": r'\boxed{(\d+)}|\b(\d+)\b',
    "aime2026": r'\boxed{(\d+)}|\b(\d+)\b',
    "gsm8k": r'\b(\d+)\b',
}

    def _extract_answer_regex(self, pred: str) -> Optional[str]:
        """Extract answer using regex pattern"""
        if not self.pattern:
            return None

        # For AIME datasets, prioritize boxed answers
        if self.dataset_type in ["aime", "aime2025"]:
            boxed_pattern = r'\\boxed{([^}]+)}'
            boxed_matches = re.findall(boxed_pattern, pred, re.IGNORECASE)
            if boxed_matches:
                # Return the last boxed answer found (most likely the final answer)
                return boxed_matches[-1].strip()

        # For other datasets, search for numbers from the end of the text
        # This prioritizes numbers that appear later in the response
        matches = re.findall(self.pattern, pred, re.IGNORECASE)
        if not matches:
            return None

        # Process matches from end to start
        for match in reversed(matches):
            if isinstance(match, tuple):
                match = match[0] if match[0] else match[1]
            answer = match.strip()
            if answer:
                return answer
        return None

    def _grade_regex(self, gold: str, pred: str) -> Tuple[bool, Optional[str]]:
        """Grade using regex pattern matching"""
        answer = self._extract_answer_regex(pred)
        if answer is None:
            return False, None
        is_correct = answer.strip() == gold.strip()
        return is_correct, answer

The SAMPLE_ANSWERS registry only matters if you select the LLM grader, which uses them as few-shot extraction examples:

SAMPLE_ANSWERS = {
    "aime": [
        "42",
        "-123",
        "999"
    ],
    "aime2025": [
        "42",
        "-123",
        "999"
    ],
    "aime2026": [
        "42",
        "-123",
        "999"
    ],
    "gsm8k": [
        "42",
        "-123",
        "999"
    ],
    "gpqa": [
        "A",
        "D",
        "C"
    ],
}

The tradeoffs are real, and the wrong grader will quietly inflate or deflate your numbers:

Step 4: Drive prompts through the server

The Processor holds the list of server configs, the grader, and the generation-length cap. n_predict maps to llama.cpp's generation limit: -1 means "use the server default", which in practice can mean unbounded generation, so pin it for cost control. Note the fallback model name — if you omit one, the harness asks for "llama":

class Processor:
    def __init__(
        self,
        server_configs: List[ServerConfig],
        grader: Grader,
        model_name: Optional[str] = None,
        n_predict: int = -1
    ):
        self.server_configs = server_configs
        self.grader = grader
        self.model_name = model_name
        self.n_predict = n_predict

The request construction is strict about sampling: temperature is only injected into the payload when the sampling config contains a non-None value. This means your run's sampling is whatever you set, not whatever the server defaults to — but it also means an empty sampling config silently inherits server defaults, so pin sampling explicitly rather than assuming greedy:

    def _make_request(
        self, server_config: ServerConfig, eval_state: EvalState, prompt: str
    ) -> Tuple[Dict[str, Any], int, Optional[float], Optional[float], str]:
        url = f"{server_config.url}/v1/chat/completions"
        headers = {"Content-Type": "application/json"}
        data = {
            "model": self.model_name if self.model_name else "llama",
            "messages": [{"role": "user", "content": prompt}],
            "n_predict": self.n_predict
        }
        if eval_state.sampling_config.get("temperature") is not None:
            data["temperature"] = eval_state.sampling_config["temperature"]

The method then performs the HTTP POST and parses the chat-completions response; the repository contains the full implementation. The surrounding runner uses ThreadPoolExecutor and as_completed to saturate the server while collecting per-task token counts, generation time in milliseconds, and throughput in tokens per second, all of which land in the JSON state.

Step 5: Read the summary with confidence intervals

A raw percentage is the least useful number you can report. Ten correct out of ten looks identical to a leaderboard-topping 100%, but the sample is tiny. That is why the script's summary path goes through a Wilson score interval rather than a naive proportion:

def wilson_interval(correct: int, total: int, z: float = 1.96) -> Tuple[float, float]:
    """Wilson score confidence interval for a proportion."""
    if total == 0:
        return (0.0, 1.0)
    p = correct / total
    z2 = z * z / total
    center = (p + z2 / 2) / (1 + z2)
    margin = z * sqrt((p * (1 - p) + z2 / 4) / total) / (1 + z2)
    return (center - margin, center + margin)

The interval is computed only over tasks whose status is ok, not over pending or errored tasks — a deliberate choice worth remembering when you resume a partial run:

    def accuracy_ci(self) -> Tuple[float, float]:
        """Compute Wilson score confidence interval from completed cases."""
        cases = self.task_states.get("cases", {})
        completed = {tid: c for tid, c in cases.items() if c.get("status") == "ok"}
        correct = sum(1 for c in completed.values() if c.get("correct", False))
        total = len(completed)
        return wilson_interval(correct, total)

    def print_summary(self):
        if self.total == 0:
            print(f"\n{'='*60}")
            print(f"Results: 0/0 correct (0.0%)")
            print(f"{'='*60}")
        else:
            ci_lower, ci_upper = self.accuracy_ci()
            print(f"\n{'='*60}")
            print(f"Results: {self.correct}/{self.total} correct ({self.correct/self.total*100:.1f}%) [{ci_lower*100:.1f}%, {ci_upper*100:.1f}%]")
            print(f"{'='*60}")

Beyond the console, dump() writes the full state to a JSON file and generates a self-contained HTML report with per-problem summary statistics and click-to-expand prompts and responses. Inspect that HTML before quoting any number — it is where grader mistakes become visible.

Step 6: Extend to embedding or model-validation checks

The dataset abstraction in this harness is deliberately thin, and that thinness is the extension point. To add a benchmark, you subclass BaseDataset, implement four methods — get_question, get_question_text, get_answer, get_prompt — and register the class in load_dataset() plus a template in TEMPLATE_REGISTRY. Here is the full set for the three prompt shapes the harness already knows, and the complete AIME dataset class as a worked example:

TEMPLATE_REGISTRY = {
    "aime": """Solve the following math problem step by step. Put your answer inside \\boxed{{}}.

{question}

Remember to put your answer inside \\boxed{{}}.
""",
    "aime2025": """Solve the following math problem step by step. Put your answer inside \\boxed{{}}.

{question}

Remember to put your answer inside \\boxed{{}}.
""",
    "aime2026": """Solve the following math problem step by step. Put your answer inside \\boxed{{}}.

{question}

Remember to put your answer inside \\boxed{{}}.
""",
    "gsm8k": """{question}
Please reason step by step, and put your final numeric answer within \\boxed{{}} without any extra characters.
""",
    "gpqa": """Answer the following multiple choice question. The last line of your response should be in the following format: 'Answer: A/B/C/D' (e.g. 'Answer: A').

{Question}

A) {A}
B) {B}
C) {C}
D) {D}
""",
}

class AimeDataset(BaseDataset):
    def __init__(self, split: str = "train"):
        self.split = split
        self.questions = []
        self._load_dataset()

    def _load_dataset(self):
        print(f"Loading AIME dataset (split: {self.split})...")
        from datasets import load_dataset

        cache_path = cache_dir / "AI-MO___aimo-validation-aime" / "default" / "0.0.0"
        if cache_path.exists():
            print(f"Using cached dataset from {cache_path}")
            ds = load_dataset("AI-MO/aimo-validation-aime", split=self.split, cache_dir=str(cache_path))
        else:
            ds = load_dataset("AI-MO/aimo-validation-aime", split=self.split)

        self.questions = []
        for row in ds:
            question = dict(row)
            question["dataset_type"] = "aime"
            self.questions.append(question)

        print(f"AIME dataset loaded: {len(self.questions)} questions")

    def get_question(self, index: int) -> Dict:
        """Get question by index"""
        return self.questions[index]

    def get_question_text(self, question: Dict) -> str:
        """Get question string"""
        return question["problem"] if "problem" in question else question["question"]

    def get_answer(self, question: Dict) -> str:
        answer = question["answer"]
        if isinstance(answer, str):
            normalized = normalize_number(answer)
            return str(normalized) if normalized is not None else answer
        return str(answer)

    def get_prompt(self, question: Dict) -> str:
        """Get formatted prompt for the question"""
        return TEMPLATE_REGISTRY[question["dataset_type"]].format(
            question=self.get_question_text(question),
        )

For embedding checks, the same seam applies: a BaseDataset subclass that returns text pairs (or triplets) and a grader that scores cosine similarity against a threshold or a held-out positive/negative split. You inherit the concurrency, the resume state, the Wilson intervals, and the HTML report for free — only the four dataset methods and one grading rule change. If you are evaluating embedding models specifically, our coverage of Perplexity's PPLX Embed v2 is a useful point of comparison for what downstream numbers to expect. This is an architectural extrapolation, not a pre-built feature of the source; the value is that you do not need to rebuild the plumbing.

What to watch out for

The regex grader will inflate your numbers and you will not notice unless you inspect. For AIME, a model that writes \boxed{42} but then rambles into a wrong answer is scored correct, because the extractor takes the last boxed match. For GSM8K, anything that matches \b(\d+)\b at the end of a response can count — a model that says "the answer is not 7, it is 14" may get extracted as 14 and marked correct even if the reasoning defends 7. Read the HTML report before quoting.

Sampling is another silent trap. The request only sends temperature when the sampling config sets a non-None value. If you think you are running greedy but your server's default is 0.8, your numbers will not match a peer's run. Pin every sampling parameter you care about.

Resumption is opportunistic about what counts as pending. The resume path retries anything whose status is not ok, including errored tasks. That is usually what you want, but if one server in a multi-server comparison is mid-deploy, a failed task gets retried against whichever server answers — which can quietly mix results across servers. The server_name field saves you here; use it.

The LLM grader is a second inference budget. It runs at temperature 0, but on the same machine it competes with the model under test for memory bandwidth, and on a separate machine it adds per-sample latency. If you run it, budget for it as a first-class cost and not a freebie. The CLI grader's 30-second timeout is hard-coded in the source, so a slow external validator will false-negative rather than hang.

Two dataset-specific gotchas: GPQA is fetched fresh from a hard-coded URL on every load — there is no HuggingFace cache fallback for it — so offline runs fail unless you mirror the CSV. And GPQA's answer options are shuffled using the seed, which means the "correct letter" changes if you change the seed. Cross-seed comparisons are only valid if you keep the corresponding task-state files.

Finally, n_predict=-1 is not "the sensible default". On servers configured for unbounded generation, it can burn minutes per problem with no extra evidence. Cap it.

Where to go next

The source file is the best next stop: read the full _make_request implementation and the dump_html method at https://github.com/ggml-org/llama.cpp to understand the response parsing and the interactive report. For CI integration, diff the accuracy and ci_lower/ci_upper fields from the JSON state across commits rather than eyeballing console output. If your work touches evaluation integrity more broadly, our note on Sakana's review of LLM-judge claim errors is a useful caution about trusting any grader — including regex — without adversarial checks.