Claude Code Gets Plugin Evals: 6 Grader Types and a CI Gate

Anthropic has shipped a structured evaluation framework for Claude Code plugins, surfacing three previously unmeasurable properties of a plugin's behaviour: whether a skill triggers on natural phrasing, whether it survives model or schema changes, and whether it actually outperforms a bare Claude response. Available in Claude Code v2.1.269 and later, the feature lands as claude plugin eval and integrates directly into CI/CD via a documented invocation pattern that enforces a score threshold, caps spend, and exits non-zero on regressions. For teams shipping tool-use plugins or agent skills, this closes the gap that claude plugin validate cannot reach — manifest syntax checking tells you nothing about whether the model will actually choose your skill.

The timing matters: as agent plugin ecosystems mature, capability verification becomes the hard problem, not capability development.

Eval suite structure and the Δ metric

Each eval suite lives in an evals/ directory inside the plugin. Individual cases are subdirectories, each containing a prompt.md and a graders/ folder. The prompt.md body is passed to Claude verbatim — @path mentions are not expanded — with frontmatter controlling max_turns (default 10), timeout_seconds (default 300), model, tags, and allowed_tools. Graders are markdown files whose frontmatter specifies a type, an optional weight, and an optional arm.

By default, every case executes twice: a with-arm run where the plugin is loaded, and a without-arm run where it is not. The difference between their scores — Δ — is the operative metric. A case that scores 1.0 in both arms proves only that Claude could handle the prompt without the plugin. The docs example shows a single case at WITH 1.00, W/OUT 0.33, Δ +0.67, across 6 total runs, at an estimated cost of $0.41, completing in 74 seconds.

The most common pathological first finding is a Δ near zero paired with a failing tool_used: Skill grader — meaning Claude is not selecting the skill when the prompt is phrased the way a real user would phrase it.

The six grader types

Four graders are computed from the transcript or filesystem and carry no incremental cost. Two call a judge model and add to the billing account.

A grader carrying arm: with-only is excluded from the numeric score in a two-arm run and reported separately as an indicator — the without-arm has no skill to fire, so including it would corrupt the denominator.

CI gate mechanics and cost model

The documented CI invocation uses claude-sonnet-5 as the eval model and claude-haiku-4-5 as the judge, with --threshold 0.8, --max-cost-usd 20, and --no-publish to suppress the claude.ai report upload. --trust-plugin is required; without it, an untrusted checkout exits with code 1 when no terminal is attached. --json results.json suppresses progress output and writes machine-readable results. Report-level problems — warn-only findings — do not alter the exit code.

Cost scales as cases × runs × arms agent invocations, plus 3 short judge calls per llm or baseline grader per run. Every invocation is a real model call billed to the plan or API account, which makes --max-cost-usd a guard against unexpectedly large bills during a CI spike. Usage-limit errors are a documented failure mode that can produce false regression signals.

claude plugin eval init bootstraps a first suite interactively: it reads the plugin, prompts for success criteria, proposes cases and graders, tests them, and writes the files. In CI, --bare writes an empty template instead.

AI Mastery analysis

The two-arm design is the framework's most defensible architectural choice. Measuring only the with-plugin score conflates model capability with plugin contribution — precisely the error that lets a well-written plugin mask a skill that never fires. Forcing the without-arm run as default means Δ is always available without developer discipline.

The cost model deserves scrutiny. An llm or baseline grader adds 3 judge calls per run, multiplied across cases × runs × arms. A suite with 10 cases, 3 runs, 2 arms, and 1 llm grader per case reaches 60 agent runs plus 180 judge calls per CI trigger. The --max-cost-usd cap creates its own failure mode: a budget ceiling hit mid-suite produces an incomplete result that could be misread as passing if the reporting layer does not surface it clearly. Teams should calibrate --max-cost-usd to full-suite cost, not treat it as a safety floor.

Relying on claude-haiku-4-5 as the default judge also introduces a model-quality dependency: judge consistency varies with judge capability, and a weaker judge on an llm grader adds variance to scores that should be deterministic. Teams running regression detection across model versions need to pin both --model and --judge-model explicitly — otherwise a score change could reflect judge drift rather than plugin regression. This is the same architectural specificity problem that surfaces across agentic systems broadly: instrumentation that depends on model behaviour inherits model variance.

The LLM-assisted claude plugin eval init bootstrapper lowers authoring cost enough that teams who would not otherwise write evals might ship a usable first suite — but the quality of the first graders depends on how well the model understood the developer's intent during init, which is worth auditing manually before committing. As production AI failures increasingly trace to architectural and integration failures rather than raw model capability, formalising the measurement of tool-use behaviour in CI is the correct layer to instrument.

Primary source

Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills — MarkTechPost