Introduction
When I’m building with AI, I want to understand when I can rely on an answer and when I should check it. A model can get most questions right and still be confidently wrong on the next one.
I built the Trust Table to explore whether models recognize that uncertainty, both in their own answers and in each other’s.
It’s a game where models answer questions, bet chips on whether they’re right, and judge each other’s answers. The idea came from Dwarkesh Patel’s Clocktower Bench, where models play a deception game. I was interested in a different question: which models know when to trust an answer?
The experiment includes 19 models, from small open models that can run on a laptop to Claude Opus. I also explored whether training on the game could improve how models handle uncertainty.
How it works
Each round has three parts:
- Answer and bet. A model answers a question and bets either 1 or 3 chips that it’s right. It can also say “I don’t know,” which costs a quarter chip.
- Judge the answer. The other models see its answer, but not the correct one. Each votes to trust it or call it wrong.
- Score the round. A correct answer wins the amount bet. A wrong answer loses twice that amount. Judges win chips for a correct vote and lose a chip for an incorrect one.
The cost of being wrong is what makes this interesting. A model can know a lot and still lose chips by betting too much on answers it isn’t sure about. Sometimes, passing is the better decision.
There are 600 questions covering trivia, multiple choice, true or false, and medical exam questions. Every model gets the same questions.
Play it yourself
Here are five rounds from the experiment. You’ll see the question, a model’s answer, and how much it bet.
Your job is to decide whether to trust the answer or call it wrong. Like the models, you won’t see the correct answer until after you vote.
The first (joint-)winner of the Nobel Peace Prize, Swiss Jean Henri Dunant, in 1901, founded which international organization?
Proposed answer: The Red Cross. True or false?
Qwen2.5-3B answers False and bets 3 chips.
Do you trust this answer?
What happened
Before training, Claude Opus 5.5 ranked first. It knew the most answers and was the hardest to fool. Inkling 2 Small, an open model from Thinking Machines, ranked third, tied with Claude Sonnet.
The smallest models had more trouble judging other models’ answers. Three of the four models with a billion parameters or fewer trusted at least 90% of the wrong answers they saw.
But size didn’t explain everything. Qwen2.5-3B was fooled less often than any other untrained open model in the experiment.
That distinction is part of what I wanted to explore. Answering a question correctly and recognizing when someone else is wrong are different abilities. The game measures both.
Can training help?
The next question was whether models could get better at this through training.
The game already provides a reward: the chips a model wins. I used that reward to train two open models on separate practice questions. The evaluation questions weren’t used in training.
| Model | Rank before → after | Training |
|---|---|---|
| Inkling 2 Small | 3rd → 1st | 20 steps, about $13 |
| Qwen3.5-4B | 10th → 6th | 16 minutes, about $11 |
Inkling 2 Small
After training, Inkling scored 1805 compared with Opus’s 1798. That put it first on the leaderboard, but the scores are close enough that I’d describe the overall result as a tie.
On judging, Inkling finished 22 points ahead of Opus. That lead held up in the follow-up checks.
Qwen3.5-4B
Qwen moved from tenth to sixth. Its wrong answers fell from 24% to 11%, while it answered just as many questions correctly. It learned to pass on more of the questions it would otherwise have guessed wrong.
That’s the result I find especially interesting from a product perspective. Knowing when to say “I don’t know” can make a model more useful, even if it doesn’t know more answers.
What I’m taking from this
In this experiment, training changed how the models handled uncertainty. Qwen became more willing to pass, and Inkling improved at judging other models’ answers.
These results are specific to this game and its questions. They don’t establish that one model is more reliable across every task, but they give me a starting point for exploring how training can shape the behavior we want in products.
I’m doing this because I think teams building with AI will need product people who understand these trade-offs, from which model to use down to how it’s trained. More on why.
This is an ongoing research project. I’m sharing the experiments, results, and questions as I go.
Want to compare notes?
If you’re exploring similar questions or want to collaborate, I’d love to hear from you.
Leave your email and I’ll get in touch.