Benchmarks
58 pieces on Benchmarks.
News & Analysis
NVIDIA Nemotron 3.5 Super VL: 120B MoE Runs on a Single GPU
NVIDIA's Nemotron 3.5 Super VL combines 120B total parameters with 12B active and runs the NVFP4 checkpoint on one DGX Spark, H200, or GB200.
Gemini 4 Argon Ships $2 Input, 1M Token Output
Google's Gemini 4 Argon launches with $2 per million input tokens, $10 per million output tokens, and a 1M-token output limit, undercutting frontier rivals on price.
OrcaCyber Zero 1.5 Claims 100% Cybench, 1M Context
OrcaRouter's gated cybersecurity model pairs a 1M-token context window with $3/$7.50 per 1M token pricing and self-reported 100% Cybench.
9B Drex 1.5 Scores 58.08, Ties Closed Jev
Nace AI's 8.95B Drex 1.5 scores 58.08 on Decision Index 0.3.1, matching closed Jev on a single GPU with zero output cost.
JetBrains Mellum2.1: 2.5B Active Params Reach 47 SWE-bench Verified
JetBrains' 12B MoE model activates 2.5B parameters per token and lifts SWE-bench Verified from 2.0 to 47.0 after reinforcement learning in real repositories.
Nemotron Scores 535.4 IOI and 30/42 IMO, Passing Gold in Both
NVIDIA reports one Nemotron family reached 535.4/600 on IOI 2026 and 30/42 on IMO 2026, above both gold thresholds.
pplx-embed-v2-late: 0.6B Queries 9B Index at 63.5% ViDoRe
Perplexity's MIT-licensed pplx-embed-v2-late pairs a 0.6B edge encoder with a 9B indexer in one embedding space; the 9B scores 92.4% on MADQA.
JEPA-Anything Cuts Interventional Pong Error 34.83% With One Recipe
Researchers report Orthogonal Predictive Factorization lets one JEPA recipe handle seven domains, cutting single-intervention Pong MSE by 34.83%.
Mistral Claims 82% Cyber Fix Score for 1T Le Chonk
Mistral's 1T-parameter Mistral Large 4 preview posts an 82% cyber fix score, with open weights promised by the end of October 2026.
MAI-Transcribe-2-Streaming Hits 2.5% WER in 0.13 Seconds
Microsoft’s streaming speech model ranks first of 38 systems for transcript accuracy, according to Artificial Analysis data reported by MarkTechPost.
Perplexity's 9B Contextual Embedder Beats voyage-context-4 by 14.4 Points
Perplexity releases pplx-embed-v2-context-9b-preview, an open-weights 9B RAG embedder that scores 45.5% Answer Recall@10 and cuts storage 8× vs float32.
Token Efficiency Is Repricing AI: Four Releases, One Signal
Four September releases show that cost-per-outcome, not per-token rate, is now the real unit of AI competition—driven by token compression, not price cuts.
GPT-6.1 Sol Matches Astra on Coding at One-Fifth the Price
OpenAI's GPT-6.1 Sol hits $2/$10 per million tokens—one-fifth of Astra's rate—while matching it on DeepSWE and closing to within 2.1 points on computer use.
Claude Sonnet 5.5 Beats Opus 5.5 on Agentic Coding at Half the Tier Price
Anthropic's Sonnet 5.5 scores 70.6% on Terminal-Bench 4.0, outpacing Opus 5.5's 66.4%, at the same $2/$10 per-million-token price as Sonnet 5.
Four AI Releases, One Pattern: The Benchmark Is Part of the Product
Qwen-Image-2.1, Voice of Reason, GPT-6 Sol/Luna, and ThinkingCap each pair a striking headline with a carefully bounded evaluation frame. Here is the decoding framework.
GPT-6 Sol and Luna Cut API Prices 50%—Benchmarks Included
OpenAI's GPT-6 Sol ($2/$10 per 1M tokens) and Luna ($0.10/$0.50) go live in the API with benchmark results against Claude Opus 5 and Fable 5.
Grok Voice Transcribe 2.0 Cuts Short-Phrase WER From 20.6% to 6.8%
SpaceXAI's Grok Voice Transcribe 2.0 claims 2x accuracy over 1.0 at unchanged pricing: $0.10/hr batch, $0.20/hr streaming.
AI Portability Is the Real Bottleneck, Not Capability
Four recent results show the same pattern: advances are real within their evaluation frame and collapse when one variable changes. Portability is the crisis.
ByteDance's HarnessDev: Only 34 of 64 LLM Harness Changes Generalize
HarnessDev benchmarks the harness LLMs build, not the answers they return. Execution feedback matched held-out results only 53.1% of the time.
Cohere's 218B MoE Scores 83.6 on WMT26, Beats DeepL and Google Translate
North Small Translate activates 25B of 218B parameters per token, scores 83.6 on WMT26 across 50 languages, and runs on a single B200 in 4-bit mode.
GPT-6 Astra: $10/M Tokens, 57.9% Terminal-Bench, Critical Cyber Flag
OpenAI's GPT-6 Astra scores 57.9% on Terminal-Bench 4.0, costs $10/$50 per million tokens, and is the first model to hit the Critical cybersecurity threshold.
150M-Parameter BDH-CQ Scores 29.2% on ARC-AGI-1 at $0.0007 per Task
Pathway's 150M-parameter BDH-CQ model scores 29.2% pass@2 on ARC-AGI-1 at $0.0007 per task, trained on SageMaker HyperPod with H200 GPUs.
G7 Beats G5 and G6 on MoE Inference With Half the GPUs
AWS benchmarks show G7 (Blackwell) delivers 60.8% more throughput than G6 and $0.90/1M tokens on 30B MoE models with only two GPUs.
IFM K2 Horizon: Six Apache 2.0 Models, 0.9B to 375B, With Self-Audit
IFM releases six open-weight models from 0.9B to 375B, plus training corpus, code, and a self-published reward-hacking audit that corrects 70.2% to 66.9%.
MiniCPM5-2B: 2.5B Model Beats Qwen3.5-4B With 53.9 Avg Across 34 Benchmarks
OpenBMB's MiniCPM5-2B averages 53.9 across 34 benchmarks, outscoring Qwen3.5-4B at 51.1 with roughly half the parameters and Apache 2.0 weights.
Meta FAIR's RPMs Cut H200 Time 1.6× by Ranking Experiments First
Meta FAIR's Research Preference Models rank unexecuted ML candidates via pairwise tournament, reaching equivalent quality 1.61× faster on AIRS-Bench.
CUA-Lite Cuts Desktop Memory 4.6× by Replacing KVM VMs with Docker
UC Berkeley's CUA-Lite unifies sandboxes, data, eval, and RL for computer-use agents in one Docker-native platform with 30k+ verifiable tasks.
Claude Fable 5.1 Hits 52.6% on Science Bench; Max Run Costs $3.30
Anthropic's Claude Fable 5.1 scores 52.6% on Terminal-Bench-Science 0.1 and ships five reasoning tiers ranging from ~$0.10 to $3.30 per prompt.
Gradium TTS: 81.0% Hard-Case Accuracy at 216 ms First Audio
Gradium AI's new default TTS model posts 81.0% on a 500-sentence hard-case eval and 216 ms P50 latency with a 30 ms interquartile spread.
207 WebGPU Kernels From Hugging Face Beat ORT by 2.57× on M4
Hugging Face releases 207 versioned WebGPU kernels with a JS loader and Fleet benchmarking tool, running 2.57× faster than ORT WebGPU on Apple M4.
NEEDLE Benchmark Rebuilds Search Queries Every Hour to Block Label Leakage
Keenable AI's MIT-licensed NEEDLE benchmark regenerates web search queries from live public sources, eliminating the fixed answer keys agents can fetch or memorize.
EnvHarness Wraps Static Benchmarks, Lifts ALFWorld OOD Score 9 Points
Google Cloud AI Research's EnvHarness reshapes frozen benchmarks via a plug-in layer, gaining 9.0 OOD points on ALFWorld and 9.8% fewer SWE-bench steps.
Gemini 3.5 Transcribe: 2.6% WER, 85+ Languages, Two API Surfaces
Google's Gemini 3.5 Transcribe posts 2.6% non-streaming WER across 85+ languages, with a dual-endpoint design that forces hard architectural choices.
Google DeepMind's Double-Blind AI Evals Use Cryptographic Isolation
Google DeepMind pilots the world's first double-blind frontier AI evaluation, using Confidential Space to cryptographically protect both model weights and test prompts.
OpenAI's Jalapeño ASIC Beats Nvidia GB200/GB300 on Latency and Throughput
OpenAI's Jalapeño chip delivers 1.5–1.9× more AI work per watt and 1.7–3.6× lower latency than Nvidia GB200/GB300 across three models.
Jalapeño Beats GB200/GB300 by 1.9× Efficiency, 3.6× Latency
OpenAI's custom Jalapeño chip outperforms NVIDIA GB200 and GB300 on InferenceX across three open-weight models up to 1T parameters.
OpenAI's Jalapeño Beats Blackwell on Inference Efficiency at Hot Chips
OpenAI's Jalapeño chip outperforms Nvidia Blackwell on tokens-per-user and throughput-per-kilowatt in SemiAnalysis' InferenceX benchmark.
Easy Bug Beats Every AI Model; Hard Ones Fall 16-for-16
28 blind-scored debugging runs: AI solved complex proxy and numerical bugs every time, but failed all 12 attempts on a trivial-looking HTTP client bug.
ASR Benchmarks Are Gameable: 6 of 11 Top Models Reproduce Audio Errors
Hume AI tested 11 open-source ASR models and found six reproduce VoxPopuli's transcript errors even when audio contradicts them — exposing WER as a gameable metric.
Nvidia's AVO Harness Takes Claude Opus 5 from 30% to 100% on ARC-AGI-3
Nvidia's custom AVO harness lifted Claude Opus 5 from 30% to 100% on ARC-AGI-3 — without changing the model at all.
ALTK-Evolve: Agent Memory Gains Depend on Model Tier, Not Just Size
IBM Research tests memory injection across 8 models on AppWorld: weaker models gain +16.1pp at +5% token cost via retrieval; strong models need the full set.
Harper 5.2 Beats Vercel Stack Up to 14× on Live Personalized Reads
Harper's benchmark across 474 load tests shows up to 14× latency advantage over a Vercel/Neon/Upstash/Ably stack on live, personalized-data paths.
Kimi K3's 1M-Token Window Costs 16× More Than RAG on 12 Questions
A controlled 12-question blind trial pits Kimi K3's full 127K-token prompt against a tuned RAG pipeline. Long-context wins on completeness, loses on cost and latency.
Sonic-3.6 Hits 1,283 Elo and Leads Both Artificial Analysis Speech Arenas
Cartesia's Sonic-3.6 tops both Artificial Analysis speech leaderboards with 1,283 Elo (Provider Voice) and 1,123 Elo (Controlled Voice), at $49/1M characters.
Flat Recovery Across All Densities: Edge Utilization Is What Moves
A 50-run benchmark shows information recovery stays between 0.924–0.976 across all densities. Edge utilization tells the real story.
Z.ai GLM-5.3: Benchmark Gains From Post-Training Alone
GLM-5.3 reuses the 743B GLM-5.2 base model unchanged. Every benchmark gain comes from scaled post-training environments and longer training runs.
Z.ai GLM-5.3: Big Benchmark Gains from Post-Training Alone
Z.ai's GLM-5.3 reuses the 743B GLM-5.2 base model unchanged, delivering major gains on long-horizon coding and cybersecurity benchmarks through post-training scale alone.
Z.ai GLM-5.3: Frontier Gains From Post-Training Alone
GLM-5.3 reuses GLM-5.2's 743B base model unchanged. Terminal-Bench 3.0 jumps from 4.6 to 28.3; CyberGym hits 84.5%, edging past GPT-5.6 Sol.
Z.ai GLM-5.3: Post-Training Gains on a Fixed 743B Base Model
GLM-5.3 reuses GLM-5.2's 743B base unchanged. Terminal-Bench 3.0 jumps from 4.6 to 28.3; CyberGym hits 84.5%, edging closed frontier models.
Z.ai GLM-5.3: Frozen 743B Base, All Gains from Post-Training
Z.ai's GLM-5.3 reuses the frozen GLM-5.2 743B base model, extracting every benchmark gain through scaled post-training alone.
Gemini 3.7 Flash: Coding and Agent Model at $0.75/1M Input Tokens
Google ships Gemini 3.7 Flash three weeks after 3.6 Flash — FrontierCode 43.6%, DeepSWE 65.3%, at $0.75/1M input until Dec 31 2026.
Grok 4.6: 500K-Context Post-Training Upgrade for Agents
SpaceXAI ships Grok 4.6 with a 500K-token context window, xhigh reasoning effort, and agentic RL — priced at $2/$6 per 1M with a 200K billing cliff.
Liquid AI LFM2.5-VL-3B: 3.1B On-Device Vision-Language Model
Liquid AI's 3.1B-parameter LFM2.5-VL-3B scores 69.4 across 28 vision benchmarks, matches 4.7B rivals, and adds tool calling for on-device agents.
PROVE: Xiaomi's Perception-Aligned Video Removal Metrics RC-S and RC-T
Xiaomi MiLM Plus releases PROVE, two reference-free metrics and a real-world benchmark that outperform PSNR, ReMOVE, and CFD on video object removal.
NOOA: NVIDIA's Object-Oriented Agent Framework Explained
NVIDIA open-sources NOOA, a Python framework that collapses prompt templates, tool schemas, and workflow graphs into one class—with 82.2% on SWE-bench Verified.
Harness-1 Shows Smaller Open Models Can Beat Frontier AI at Search
Harness-1 is a 20B open-source search agent that beats GPT-5.4 on recall by moving search memory out of the model and into a structured environment.
DeepSWE Reshuffles the AI Coding Leaderboard and Puts GPT-5.5 on Top
A tougher coding benchmark shows wider gaps between frontier AI models, with GPT-5.5 leading and verifier quality becoming the real story.
OpenAI Makes GPT-5.5 Instant the New Default for ChatGPT
OpenAI has rolled out GPT-5.5 Instant as ChatGPT's new default model, promising faster responses, fewer hallucinations in high-stakes domains, stronger reasoning scores, and broader memory-source visibility.