I tried to make a small open model work like Jev
Reading answers straight from the model's probabilities instead of letting it write JSON. Way faster than I expected, and the confidence numbers needed some fixing.
When Jev came out there was a lot of hype around it. It's a decision model: you give it an input and a few typed questions (is this urgent, which team should handle it, yes or no) and it gives back fixed answers instead of text. Then OpenAI shipped their own Decisions API, and it was clear this was going to be a thing.
My first thought was: can a small open-weight model do this on my own GPU?
Where the idea came from
A language model doesn't really "write" an answer first. Before it picks each token, it scores every possible next token. So if the answer is a single token, like A or B, you don't need to let it write anything. You run one forward pass and read off how much probability it put on A versus B.
And if you have six questions about the same ticket, you can batch all six into that one pass.
That was the whole bet, and it worked better than I expected. On my RTX 5090 with a 4B model, six answers about a support ticket took 57 ms. Asking the same model to write those six answers as JSON took 1.1 to 2.5 seconds. That's 20 to 45 times slower, plus all the tokens it spends writing, plus parsing.
The catch: it's way too sure of itself
Speed was the easy part. The probabilities were the problem. Untrained chat models put near-certain probability on almost everything. In that ticket example the model gave 1.0 to all six answers, including ones it should have been unsure about. Across the tasks I tested it averaged 78% accuracy while acting like it was never wrong.
That matters if you want to use the confidence, like auto-approving anything above 0.9 and sending the rest to a person.
I tried the usual prompt tricks (option names instead of letters, few-shot examples, shuffling the order). Few-shot helped a little. Nothing fixed the overconfidence.
There's also a fix people suggest for yes/no questions: ask the question and its opposite and average them. I tested it. It barely helped, and when the opposite was phrased as a negation it made things worse. Small models are bad at "not".
What actually fixed it
Platt scaling. It's a two-number logistic fit from the 90s. Label a handful of examples, see what the model said versus what was right, fit a curve.
With just 25 labels per task, calibration error dropped from 0.197 to 0.089, and accuracy went up too, from 0.779 to 0.845, because it also corrects the model leaning towards one answer. With 100 labels every task I tried was under 0.075. Labelling 25 examples takes ten minutes.
Trying LoRA
The other thing I wanted to try was fine-tuning a model to work this way, like Jev does. I trained a small LoRA on a 4B model for 74 minutes, on 15 public datasets that don't overlap with the test tasks. Its average score went from 0.713 to 0.756 and calibration error from 0.241 to 0.108.
So training helps, but honestly less than I thought it would. Platt scaling with 25 labels did more.
What I'd tell someone trying this
If your model is making decisions, read the probabilities instead of generating text. It's much faster and you get confidence for free. Just don't trust that confidence until you've calibrated it, and 25 labels is enough to start.
Everything's open if you want to try it on your own data.
Paper: PDF · DOI 10.5281/zenodo.23555119
Code and data: github.com/alleyxd/AnyDecide