Thinking makes my model better at tool calls. So I made it think less.
A small model's own token probabilities tell you when a tool call is shaky. Think only then, and you keep most of the gain at half the cost.
Reasoning modes are great. Let a model think before it calls a tool and it gets the call right more often. On my tests with Qwen3 that was 3 to 5 points better on requests that actually needed a tool.
But it's not free. Thinking used six to seven times the tokens on every single request, most of which didn't need it. And then something I didn't expect: when no tool fits the request, a thinking model is worse at saying so. It reasons its way into calling something anyway. Correctly declining dropped by about 6 points at every model size.
So the question became: can the model tell me which requests need the extra thinking?
The model already knows when it's unsure
When the model makes a normal tool call, the server can return the probability of every token it wrote. Average those and you get a rough "how sure was it" number.
It turns out that number is a pretty good predictor of whether the call is right (AUROC around 0.82 to 0.84). And the probability of the very first token being a tool call is an even better signal for whether a tool was needed at all (0.95 to 0.97). I also tried just asking the model "does a tool fit here?" and that was worse. The call it was going to make anyway tells you more than asking it.
Both numbers come for free from the call you're already making.
Think only when it looks shaky
So the rule is simple: make the normal call. If it's a tool call and its confidence is low, redo it with thinking. If the model declined, leave it alone.
Thinking on the least confident 30% of requests:
- at 1.7B and 4B, it got back about two thirds of the gain from always thinking
- at 8B it actually beat always thinking (0.877 vs 0.874), because it skipped the thinking that was turning good declines into bad calls
- it used about half the tokens either way
- picking 30% at random instead did worse every time, so the confidence ranking is doing the work
To be clear about what this is: the 70% that don't get sent to thinking come out exactly the same as before. It doesn't make every answer better. It spends the expensive part where it matters.
Other stuff I found along the way
I tested a dozen cheap tweaks on the same data, and a few surprised me:
- parameter descriptions matter a lot (removing them cost 16 to 18 points), function names barely matter
- extra unrelated tools hurt small models the most
- keyword search to pick tools was sometimes worse than just sending all of them, while a small embedding model fixed that
- forcing the model to always call a tool helps a bit, then completely breaks the cases where it shouldn't
If you want to try it
Turn on logprobs for your tool calls, retry the shaky ones with thinking, and never send a decline to thinking. It's a few lines of code and works with any OpenAI-compatible server.
Paper: PDF · DOI 10.5281/zenodo.23580272
Code and data: github.com/alleyxd/SecondThought