ThinkingCap-Qwen3.8-27B Cuts Thinking Tokens 37.2% for 0.86pp Accuracy

BottleCap AI's ThinkingCap-Qwen3.8-27B is a fine-tune of Qwen's Qwen3.8-27B with a single objective: compress reasoning traces. Released as the second model in the ThinkingCap series, it offers one of the more granular published data sets on how much thinking-token overhead can be removed before accuracy meaningfully degrades. BottleCap AI reports 37.2% fewer thinking tokens on average across 12 benchmarks, at a macro-accuracy cost of 0.86 percentage points — from 86.65% to 85.79% — measured at reasoning_effort=xhigh. The weights are a drop-in replacement for Qwen3.8-27B; serving flags and sampling settings are unchanged. This fits the broader pattern of efficiency gains being extracted at the fine-tuning layer rather than through infrastructure rewrites.

Benchmark-by-Benchmark Breakdown

BottleCap AI reports both models were evaluated on identical hardware — a single NVIDIA H200 running vLLM 0.29.0 — with sampling fixed at temperature 1.0, top_p 0.95, top_k 20, and min_p 0.0. The 37.2% figure is the arithmetic mean of per-benchmark token reductions; pooled mean thinking tokens fall from 15,735 to 12,144.

Knowledge and multilingual tasks absorb the steepest cuts. BottleCap AI reports MMMLU shrinks from 1,656 to 571 thinking tokens per question — a 65.5% reduction — and MMLU-Pro drops 57.3%. RealWorldQA falls 50.4% and IFBench 46.4%, the latter with accuracy essentially flat at 79.75% versus 79.71%.

GPQA-Diamond drops from 12,772 to 7,267 tokens (43.1% cut) while accuracy falls 1.89 percentage points. The most expensive accuracy trade is AIME 2026: a 3.85pp drop — from 98.13% to 94.27% — for 30.2% fewer tokens. HMMT Feb 26 and HMMT Nov 25 each lose roughly 1.1–1.2 percentage points for reductions of 22.0% and 30.5%, respectively.

Two benchmarks improve outright. AA-LCR long-context retrieval accuracy rises 2.25pp (81.75% to 84.00%) while using 38.6% fewer tokens. LiveCodeBench v6 edges up 0.07pp with 20.3% fewer tokens. Agentic results hold close: τ²-bench gives up 1.01pp for 30.9% fewer tokens; Terminal-Bench 2.1 loses 0.56pp — inside its reported ±4.26pp confidence interval — for 10.7% fewer tokens, the smallest cut across the suite.

BottleCap AI also reports that under a 16K-token cap per response, ThinkingCap scores above the base. Truncated traces fall from 0.51% to 0.34%, and looping from 0.06% to 0.05%.

Effort-Level Interaction and Speculative Decoding

The compression stacks with Qwen3.8-27B's built-in reasoning-effort dial. BottleCap AI reports the following deltas against the base model at xhigh, averaged over 11 benchmarks (Terminal-Bench 2.1 ran only at xhigh): at medium, the base model alone cuts 52.1% of thinking for −9.16pp accuracy; ThinkingCap extends that to −60.2% thinking at −9.90pp. At low, the base reaches −55.4% thinking at −9.71pp; ThinkingCap delivers −62.3% at −10.79pp. With thinking disabled entirely, ThinkingCap trails the base by 5.7pp. BottleCap AI recommends xhigh for the optimal accuracy-to-token balance, noting individual effort modes will receive focused tuning in a future release.

On speculative decoding, BottleCap AI reports that MTP with 3 draft tokens is accuracy-neutral on AIME 2026, accepting 53% of drafted tokens — approximately 2.6 tokens per step — matching the base model's acceptance rate.

Deployment Builds and Licensing

The primary checkpoint is a 28-billion-parameter bf16 model accepting image and text input, evaluated by BottleCap AI on vLLM 0.29.0 with --reasoning-parser qwen3 and the qwen3_xml tool-call parser. Five quantized variants ship alongside it: FP8 (31 GB, Hopper and Blackwell), NVFP4 weight-only (21 GB, Hopper via Marlin kernel and Blackwell), NVFP4 W4A4 AWQ (23 GB, Blackwell only), GGUF (16–55 GB, IQ4_XS to f16, for llama.cpp, LM Studio, and Ollama across CUDA, Metal, Vulkan, or CPU), and MLX 4-bit DWQ (21 GB, Apple Silicon, 32 GB Mac).

The repo is gated on Hugging Face. Licensing is PolyForm Small Business 1.0.0 plus a BottleCap personal-use grant; commercial deployment beyond small-business scope requires a separate BottleCap agreement. Upstream Qwen materials remain under Apache-2.0. MarkTechPost reports no inference provider is hosting the model as of publication.

AI Mastery analysis

The benchmark architecture here reveals a skew practitioners should scrutinize before deploying: token reductions are largest where they cost least. Tasks with short, verifiable answers — MMMLU (65.5% cut, −1.29pp), MMLU-Pro (57.3%, −0.87pp), RealWorldQA (50.4%, −0.91pp) — are exactly where the efficiency argument is strongest. The hard-math regime tells a different story: AIME 2026 loses 3.85pp, non-trivial in competitive evaluation contexts even if 94.27% absolute accuracy remains high.

The AA-LCR result deserves attention. A 2.25pp accuracy gain alongside a 38.6% token reduction on long-context retrieval implies the base model's extended reasoning was generating noise, not signal — a pattern architectural specificity tends to surface.

One meaningful gap in the evaluation is prompt diversity. All benchmarks run under fixed sampling against standardised test sets; the single-seed MMLU-Pro and MMMLU results (versus 32 seeds on AIME 2026) mean confidence intervals on the most-compressed benchmarks are wider than they appear. And because all results are vendor-reported — BottleCap AI ran its own harness — independent replication on diverse prompt distributions remains the missing piece before high-stakes deployments treat the 0.86pp macro cost as a reliable bound.

The PolyForm Small Business license constrains organizational reach in ways fully permissive weights do not. Teams evaluating this against Apache-licensed alternatives should account for that ceiling, particularly where open-weight portability is already a deployment bottleneck. For teams already running Qwen3.8-27B at scale, the NVFP4 weight-only build at 21 GB and the GGUF family make this an accessible A/B test — and the per-task benchmark granularity sets a transparency bar worth noting for this class of release.

Sources

• Primary: bottlecapai/ThinkingCap-Qwen3.8-27B · Hugging Face
• Reported by: MarkTechPost: BottleCap AI Releases ThinkingCap-Qwen3.8-27B: 37.2% Fewer Thinking Tokens at a 0.86pp Accuracy Cost