Reading AI Benchmarks: Keep the Dataset, Metric and Baseline

A benchmark result needs its dataset, metric, model configuration and comparator. Losing one of those can turn a valid narrow result into a false claim.

Correction, 21 October 2026: The earlier article incorrectly said Voice of Reason's reported 54% overlap concerned GSM8K. It concerned four other arithmetic benchmarks that the authors excluded. It also overstated what different reasoning-effort labels prove and treated several unverified comparisons as settled. This revision retains source-supported examples and a practical reading procedure.

Keep contamination findings attached to the right data

Appendix D of Voice of Reason checks paraphrase-level overlap with Orca-Math for AddSub, MultiArith, SingleEQ and SVAMP. It reports 366 flagged questions out of 678 checked across those sets and excludes them from the main results. The paper instead uses GSM8K for its main mathematical reasoning evaluation.

That is not evidence that 54% of GSM8K was contaminated. It is also not proof that GSM8K is entirely free of overlap. The correction is to preserve the scope of the test rather than infer a conclusion about an untested population.

When reviewing a contamination claim, retain the matching method as well as the total. Candidate retrieval, a model-based paraphrase judgment and exact string matching measure different relationships. A flagged example needs the meaning assigned by that procedure.

Keep component counts separate from deployment memory

Qwen's model card describes 7B parameters in the visual generation component. Its pipeline also contains other components, and its memory-optimization section discusses offloading.

A component count is therefore insufficient for a full memory estimate. Conversely, saying every component must reside on the GPU simultaneously is too strong when the chosen implementation supports offload. Record which parts reside where, precision, image size, batch size and peak observed memory.

This is the same distinction used in our local deployment review: an architecture label cannot replace a measured operating configuration.

Preserve the aggregation rule

ThinkingCap's card defines its 37.2% headline as the equal-weight mean of per-benchmark thinking-token reductions. It is not the ratio between the two macro-average token counts.

For example, consider hypothetical tasks with reductions of 50% and 10%. Their equally weighted mean is 30%. A production mix dominated by the second task should not be budgeted at a 30% saving.

The card's AIME accuracy entries, 98.13 and 94.27, differ by 3.86 percentage points. Keep that result with its task and uncertainty rather than declaring the accuracy cost acceptable for an entire industry.

Compare operating points explicitly

An effort label such as medium or maximum is a provider-defined setting. Identical labels do not establish identical compute, and different labels do not automatically invalidate a comparison. A cost-quality curve can use different operating points if it states them clearly.

For a deployment decision, compare the points you could actually run within the application's budget and latency requirements. Separately test sensitivity to an increased token cap, a longer timeout or a changed tool harness.

These checks are an editorial procedure, not an independent reproduction of the releases. Keep the primary report available beside the extracted figures so readers can inspect what the comparison actually measured.