ToolGrad Hits 99.8% Pass Rate, Lifts Gemma-3-12B to Gemini 2.5 Pro Level

Google Research, in collaboration with the University of Tokyo, RIKEN AIP, and Tohoku University, has released ToolGrad, a framework that inverts the conventional pipeline for generating tool-use training data. Instead of composing a user query and then searching for a tool chain that satisfies it, ToolGrad executes real APIs first, constructs a verified chain, and derives a matching query in a single LLM call afterward. The result: a 99.8% pass rate on ToolBench data generation, against 63.8% for the DFS-based query-first baseline.

The practical stakes are significant. Agentic systems that rely on tool-calling are only as reliable as the training data used to teach that behaviour, and as discussed in our analysis of how production AI fails on architecture rather than model intelligence, the weakest link is often the data pipeline rather than the model itself. ToolGrad attacks that link directly.

The query-first failure mode and ToolGrad's inversion

Prior frameworks such as ToolBench and ToolACE follow a query-first recipe: sample APIs, instruct an LLM to generate a plausible user intent, then run a depth-first search agent to find a tool chain that satisfies that intent. The search carries no success guarantee. Dead-end explorations consume compute and are discarded — the paper characterises this as distilling valuable trajectories from a complex and often failing agent exploration.

ToolGrad eliminates the search step. Each sample is produced by a four-module loop, run for a configurable number of iterations. The repository default is 10 iterations over 50 sampled APIs per workflow:

1. API Proposer narrows the sampled API pool to a shortlist of candidates capable of extending the current workflow.
2. API Executors run those candidates in parallel and produce detailed execution reports.
3. API Selector reads the reports, selects the single best-performing call, and appends it to the workflow. Its directional reasoning constitutes the textual gradient.
4. LLM Updater rewrites the synthetic user query and the AI response so both remain consistent with the expanded API set.

One loop iteration appends one verified API call. After all iterations complete, the saved sample contains a user query, a verified API workflow, and a final response — all coherent by construction because the chain was executed before the query was written.

Generation efficiency against ToolBench DFS

The research team benchmarked ToolGrad against ToolBench's DFS approach on the ToolBench API database, which contains more than 16,000 real-world APIs. The 0.2% failure cases occurred when the agent failed to obtain a successful API response across all 10 iterations and saved an empty sample.

The chain length increase from 2.1 to 3.4 ground-truth tool uses per sample matters for training: longer chains expose models to multi-step coordination patterns that single-call examples cannot teach. The reduction in tool-use steps from 34.3 to 20.0 reflects the elimination of failed exploration branches, not a reduction in workflow complexity.

BFCL results with Gemma-3 fine-tuned on 500 samples

The team generated ToolGrad-500 using Gemini 2.5 Flash-Lite as the teacher model and post-trained Gemma-3 at 1B, 4B, and 12B parameter scales. Evaluation used the Berkeley Function Calling Leaderboard with a tool set distinct from ToolBench, making it an out-of-distribution test. Fine-tuning on ToolGrad-500 improved BFCL scores at every parameter size. The 12B variant reached 83.1, placing it alongside Gemini 2.5 Pro at 83.2 and above Claude 4.5 Opus at 82.8 and GPT-5 at 74.4, as measured at the time of publication. ToolGrad-12B also outperformed its Gemini 2.5 Flash-Lite teacher and led open tool-use specialists including ToolACE and Hammer-2.1-7B. Reproduction scripts target BFCL V1 and V2 through a customised fork and were verified on a single NVIDIA A100 40GB running inference via a vLLM Docker image.

AI Mastery analysis

The student-beats-teacher result deserves scrutiny. ToolGrad-12B surpassing Gemini 2.5 Flash-Lite on BFCL despite training on only 500 Flash-Lite-generated samples is consistent with a well-understood phenomenon: when the training signal is structurally correct — here, every sample is chain-verified — a smaller model can exceed the generalisation ceiling of the teacher on the specific capability being trained. This is distinct from classical capability distillation and is closer to architectural specificity outperforming raw scale.

Two caveats are worth flagging. First, the 99.8% pass rate is measured on ToolBench's API corpus; real-world tool environments with undocumented side effects, rate limits, or authentication flows will degrade that figure in ways the benchmark cannot capture. Second, the 500-sample ceiling raises a question the paper does not fully answer: does BFCL performance plateau there, or does it continue to climb with larger ToolGrad-generated sets? The answer would clarify whether the data-efficiency advantage holds at production dataset scales, where pipeline architecture rather than model choice drives AI gains.

The broader signal is that verified-execution data generation may be the more tractable path to reliable tool-calling than scaling search-based approaches. When 500 structurally sound samples can place a 12B open model within 0.1 points of Gemini 2.5 Pro on an out-of-distribution benchmark, the argument for expensive DFS-based pipelines weakens considerably. The code, ToolGrad-500, and the 1B, 4B, and 12B models are all available on Hugging Face under Apache-2.0.

Primary source

Google Research Releases ToolGrad: Answer-First Framework Hits 99.8% Pass Rate for Tool-Use Data Generation — MarkTechPost