Google ToolGrad Hits 99.8% Tool-Use Data Pass Rate

Google ToolGrad arrives with a number that is hard to ignore: a 99.8 percent pass rate for generating tool-use training data. The framework, built by researchers from Google alongside the University of Tokyo, RIKEN AIP, and Tohoku University, flips the standard recipe for teaching language models how to call tools. Instead of asking a model to invent a user request and then hunting for an API chain that satisfies it, Google ToolGrad builds a working chain first and writes the query afterward.

A Simple Inversion With Big Consequences

Prior pipelines like ToolBench and ToolACE follow what the researchers call a query-first approach. The system samples a pool of APIs, asks an LLM to dream up a plausible user instruction, then dispatches a depth-first search agent to find a tool-use path that satisfies the request. That search comes with no guarantee of success. When it dead-ends, the compute spent exploring is wasted and the sample gets thrown out. The paper describes this as distilling valuable trajectories from a complex and often failing agent exploration, which is inherently inefficient.

Google ToolGrad reverses the order entirely. It constructs a ground-truth tool-use chain by actually executing APIs, then annotates that chain with a matching user query. An explicit, working chain is far less ambiguous than a hypothetical prompt. That means the chain-to-query step needs just a single LLM call.

Four Modules, One Loop

Each iteration of Google ToolGrad runs four modules in sequence. The API Proposer narrows a sampled set of APIs down to a few candidates that could extend the current workflow. API Executors run those candidates in parallel and produce detailed execution reports. The API Selector reviews the reports, picks the single best-performing call, and appends it to the workflow. Its directional feedback acts as a textual gradient. Finally, the LLM Updater rewrites the synthetic user query and AI response so they match the new API set.

Repeat the loop and you get one sample: a user query, a verified API workflow, and the final response. The repository's default configuration runs 10 iterations over 50 sampled APIs per workflow.

The Numbers Behind the 99.8 Percent

The team evaluated data generation on the ToolBench API database, which holds more than 16,000 real-world APIs, and compared Google ToolGrad against ToolBench's DFS-based query-first approach. The results are stark.

• Pass rate rose from 63.8 percent with DFS to 99.8 percent with Google ToolGrad.
• Ground-truth tool uses per sample rose from 2.1 to 3.4, meaning longer chains.
• Tool-use steps per sample fell from 34.3 to 20.0.
• LLM invocations per sample dropped slightly, from 64.5 to 63.9.

That 0.2 percent failure case happened when the agent could not get a successful response from three selected APIs across all 10 iterations and saved an empty sample. Not a catastrophic breakdown. Just a quiet edge case.

Five Hundred Samples, Frontier-Level Scores

The researchers generated ToolGrad-500, a 500-sample dataset built with Gemini 2.5 Flash-Lite, and used it to post-train Gemma-3 at 1B, 4B, and 12B parameters. They evaluated on the Berkeley Function Calling Leaderboard, which uses a tool set that differs from ToolBench. That makes it an out-of-distribution test with unseen tools.

Fine-tuning on ToolGrad-500 improved tool-use scores at every parameter size. The headline result: ToolGrad-12B scored 83.1, compared with Gemini 2.5 Pro at 83.2, Claude 4.5 Opus at 82.8, and GPT-5 at 74.4, as measured at the time of publication.

Only 500 samples lift Gemma-3-12B to 83.1 on BFCL, next to Gemini 2.5 Pro at 83.2.

There is a twist worth pausing on. The 12B student outperformed Gemini 2.5 Flash-Lite, the very teacher model that generated its training data. Google ToolGrad-12B also led open tool-use specialists including ToolACE and Hammer-2.1-7B.

Why the Inversion Works

The pass rate improvement is not the whole story. The deeper shift is what happens to the data itself. Longer chains, fewer tool steps, and roughly the same number of LLM calls per sample means each generated example carries more verified complexity without extra generation cost. The system stops gambling on search and starts building on execution.

That distinction matters for anyone training agents on tool use. A 63.8 percent pass rate means roughly a third of attempted samples vanish into dead ends. At 99.8 percent, almost nothing is wasted. The compute goes into data you can actually use.

Available to Build On

Is it deployable? Yes. The code is Apache-2.0 licensed. The ToolGrad-500 dataset and the 1B, 4B, and 12B models are on Hugging Face, and there is a PyPI package. The repository's reproduction scripts target BFCL V1 and V2 through a customized fork, run inference in a vLLM Docker image, and were verified on a single NVIDIA A100 40GB.

For teams working on tool-calling models, Google ToolGrad offers something rare: a method that is both cheaper and more reliable than what came before. Verify the chain first, write the query second. The 99.8 percent figure is the proof, but the inversion itself is the idea worth stealing.