Who can you trust?
19 AI models play a game. One answers a question and bets chips that it's right. The others see the answer, but not the correct one, and vote: trust it or call it wrong.
This page has the full results: who you can trust, and who gets fooled. We also use the game's scoring to train models, and the ones we trained are on the board too.
- models: 12 on our Mac, 3 Claude, 1 on Tinker, 3 we trained
- 19
- models: 12 on our Mac, 3 Claude, 1 on Tinker, 3 we trained
- questions in 4 formats
- 600
- questions in 4 formats
- rounds played
- 11,400
- rounds played
- trust or call votes
- 149,628
- trust or call votes
- Claude API cost
- $14.82
- Claude API cost
Leaderboard
What training changed
Our own experiments. We trained 2 open models using the game's scoring, on separate practice questions, never these. Below is each one before and after.
Inkling 2 Small: #3 to #1 of 19
- #3 of 19 overall · beforeInkling 2 SmallTinker
- Overall score
- 1726
- Judging score
- 1634
- Answers wrong
- 5%
- Said I don't know
- 4%
- Chips per question
- +2.38
- #1 of 19 overallInkling 2 Small (trained)Trained by us
- Overall score
- 1805
- Judging score
- 1704
- Answers wrong
- 4%
- Said I don't know
- 1%
- Chips per question
- +2.67
- #2 of 19 overall · for comparisonClaude Opus 5.5API
- Overall score
- 1798
- Judging score
- 1683
- Answers wrong
- 3%
- Said I don't know
- 0%
- Chips per question
- +2.71
Qwen3.5-4B: #10 to #6 of 19
- #10 of 19 overall · beforeQwen3.5-4BLocal
- Overall score
- 1511
- Judging score
- 1493
- Answers wrong
- 24%
- Said I don't know
- 1%
- Chips per question
- +0.75
- #6 of 19 overallQwen3.5-4B (trained)Trained by us
- Overall score
- 1614
- Judging score
- 1514
- Answers wrong
- 11%
- Said I don't know
- 14%
- Chips per question
- +1.59
- #7 of 19 overallQwen3.5-4B (trained, medical)Trained by us
- Overall score
- 1605
- Judging score
- 1499
- Answers wrong
- 9%
- Said I don't know
- 19%
- Chips per question
- +1.63
Scores work like chess ratings, with 1500 as average. Answers wrong, I don't know and chips per question count only the model's own answers.
Answering and judging combined.
- Hollow circle: open model on our Mac.
- Green diamond: open model on Tinker.
- Black square: API.
- Green circle with a ring: trained by us.
| # | Model | Score (with range) | Score | Chips / question |
|---|---|---|---|---|
| 1 | Inkling 2 Small (trained)Trained by usThinking Machines594B (17B active)90% interval 1790 to 1824 | 1805 | +3.72 | |
| 2 | Claude Opus 5.5APIAnthropicsize n/a90% interval 1784 to 1814 | 1798 | +3.72 | |
| 3 | Inkling 2 SmallTinkerThinking Machines594B (17B active)90% interval 1714 to 1741 | 1726 | +3.25 | |
| 4 | Claude Sonnet 5.5APIAnthropicsize n/a90% interval 1713 to 1741 | 1726 | +3.42 | |
| 5 | Claude Haiku 5.5APIAnthropicsize n/a90% interval 1708 to 1736 | 1720 | +3.33 | |
| 6 | Qwen3.5-4B (trained)Trained by usAlibaba (Qwen)4B90% interval 1599 to 1629 | 1614 | +2.10 | |
| 7 | Qwen3.5-4B (trained, medical)Trained by usAlibaba (Qwen)4B90% interval 1590 to 1619 | 1605 | +2.14 | |
| 8 | Qwen2.5-7BLocalAlibaba (Qwen)7B90% interval 1518 to 1553 | 1536 | +1.16 | |
| 9 | Qwen3-8BLocalAlibaba (Qwen)8B90% interval 1505 to 1538 | 1521 | +1.08 | |
| 10 | Qwen3.5-4BLocalAlibaba (Qwen)4B90% interval 1492 to 1526 | 1511 | +1.24 | |
| 11 | Phi-4-miniLocalMicrosoft3.8B90% interval 1419 to 1456 | 1438 | +0.30 | |
| 12 | SmolLM3-3BLocalHugging Face3B90% interval 1360 to 1384 | 1372 | +0.25 | |
| 13 | Qwen2.5-3BLocalAlibaba (Qwen)3B90% interval 1342 to 1367 | 1355 | +0.13 | |
| 14 | Qwen2.5-1.5BLocalAlibaba (Qwen)1.5B90% interval 1319 to 1343 | 1332 | +0.02 | |
| 15 | Qwen3-0.6BLocalAlibaba (Qwen)0.6B90% interval 1309 to 1336 | 1323 | −0.11 | |
| 16 | SmolLM2-1.7BLocalHugging Face1.7B90% interval 1307 to 1333 | 1320 | −0.10 | |
| 17 | OLMo-2-1BLocalAi21B90% interval 1255 to 1283 | 1269 | −0.52 | |
| 18 | Qwen2.5-0.5BLocalAlibaba (Qwen)0.5B90% interval 1251 to 1281 | 1266 | −0.77 | |
| 19 | Qwen3.5-0.8BLocalAlibaba (Qwen)0.8B90% interval 1247 to 1280 | 1264 | −0.60 | |
The bar shows how sure we are of each score. If two bars overlap, the order isn't reliable.The range under each name shows how sure we are of the score. If two ranges overlap, the order isn't reliable. Chips per question is the average a model won or lost. Click a model to see its games.
Our own runs, 2026-11-11. Open models ran on a Mac mini. Claude models ran through the API with their default settings.
Head to head
How often should the row model beat the column model? This is the expected chance, from their ratings. Darker means the row model wins more. 50 is a coin flip.
Scroll sideways to see all 19 columns.
| Inkling 2 Small (trained) | Claude Opus 5.5 | Inkling 2 Small | Claude Sonnet 5.5 | Claude Haiku 5.5 | Qwen3.5-4B (trained) | Qwen3.5-4B (trained, medical) | Qwen2.5-7B | Qwen3-8B | Qwen3.5-4B | Phi-4-mini | SmolLM3-3B | Qwen2.5-3B | Qwen2.5-1.5B | Qwen3-0.6B | SmolLM2-1.7B | OLMo-2-1B | Qwen2.5-0.5B | Qwen3.5-0.8B | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Inkling 2 Small (trained) | 51 | 61 | 61 | 62 | 75 | 76 | 83 | 84 | 84 | 89 | 92 | 93 | 94 | 94 | 94 | 96 | 96 | 96 | |
| Claude Opus 5.5 | 49 | 60 | 60 | 61 | 74 | 75 | 82 | 83 | 84 | 89 | 92 | 93 | 94 | 94 | 94 | 96 | 96 | 96 | |
| Inkling 2 Small | 39 | 40 | 50 | 51 | 66 | 67 | 75 | 77 | 78 | 84 | 89 | 89 | 91 | 91 | 91 | 93 | 93 | 94 | |
| Claude Sonnet 5.5 | 39 | 40 | 50 | 51 | 66 | 67 | 75 | 77 | 78 | 84 | 89 | 89 | 91 | 91 | 91 | 93 | 93 | 94 | |
| Claude Haiku 5.5 | 38 | 39 | 49 | 49 | 65 | 66 | 74 | 76 | 77 | 84 | 88 | 89 | 90 | 91 | 91 | 93 | 93 | 93 | |
| Qwen3.5-4B (trained) | 25 | 26 | 34 | 34 | 35 | 51 | 61 | 63 | 64 | 73 | 80 | 82 | 84 | 84 | 85 | 88 | 88 | 88 | |
| Qwen3.5-4B (trained, medical) | 24 | 25 | 33 | 33 | 34 | 49 | 60 | 62 | 63 | 72 | 79 | 81 | 83 | 84 | 84 | 87 | 88 | 88 | |
| Qwen2.5-7B | 18 | 18 | 25 | 25 | 26 | 39 | 40 | 52 | 54 | 64 | 72 | 74 | 76 | 77 | 78 | 82 | 83 | 83 | |
| Qwen3-8B | 16 | 17 | 23 | 24 | 24 | 37 | 38 | 48 | 51 | 62 | 70 | 72 | 75 | 76 | 76 | 81 | 81 | 81 | |
| Qwen3.5-4B | 16 | 16 | 22 | 23 | 23 | 36 | 37 | 46 | 49 | 60 | 69 | 71 | 74 | 75 | 75 | 80 | 80 | 81 | |
| Phi-4-mini | 11 | 11 | 16 | 16 | 17 | 27 | 28 | 36 | 38 | 40 | 59 | 62 | 65 | 66 | 66 | 73 | 73 | 73 | |
| SmolLM3-3B | 8 | 8 | 12 | 12 | 12 | 20 | 21 | 28 | 30 | 31 | 41 | 52 | 56 | 57 | 57 | 65 | 65 | 65 | |
| Qwen2.5-3B | 7 | 7 | 11 | 11 | 11 | 18 | 19 | 26 | 28 | 29 | 38 | 48 | 53 | 55 | 55 | 62 | 63 | 63 | |
| Qwen2.5-1.5B | 6 | 6 | 9 | 9 | 10 | 17 | 17 | 24 | 25 | 26 | 35 | 44 | 47 | 51 | 52 | 59 | 59 | 60 | |
| Qwen3-0.6B | 6 | 6 | 9 | 9 | 9 | 16 | 17 | 23 | 24 | 25 | 34 | 43 | 45 | 49 | 51 | 58 | 58 | 59 | |
| SmolLM2-1.7B | 6 | 6 | 9 | 9 | 9 | 16 | 16 | 22 | 24 | 25 | 34 | 43 | 45 | 48 | 50 | 57 | 58 | 58 | |
| OLMo-2-1B | 4 | 5 | 7 | 7 | 7 | 12 | 13 | 18 | 19 | 20 | 27 | 36 | 38 | 41 | 42 | 43 | 50 | 51 | |
| Qwen2.5-0.5B | 4 | 5 | 7 | 7 | 7 | 12 | 13 | 18 | 19 | 20 | 27 | 35 | 37 | 41 | 42 | 42 | 50 | 50 | |
| Qwen3.5-0.8B | 4 | 4 | 7 | 7 | 7 | 12 | 12 | 17 | 19 | 19 | 27 | 35 | 37 | 40 | 42 | 42 | 49 | 50 |
Hover or tap a cell for the two names, or focus the grid and use the arrow keys.
More from the games
Good at answering, good at judging?
- Hollow circle: open model on our Mac.
- Green diamond: open model on Tinker.
- Black square: API.
- Green circle with a ring: trained by us.
Hover, tap or focus a point for its name.
Does a big bet mean it knows?
Scroll sideways to see both columns.
| Seat | right when betting 1 right when betting 3 | Bets 3 |
|---|---|---|
| Claude Opus 5.5 | 96% | |
| Inkling 2 Small (trained) | 98% | |
| Claude Sonnet 5.5 | 93% | |
| Claude Haiku 5.5 | 90% | |
| Inkling 2 Small | 90% | |
| Qwen3.5-4B (trained) | 100% | |
| Qwen3.5-4B (trained, medical) | 100% | |
| Qwen3.5-4B | 79% | |
| Qwen2.5-7B | 92% | |
| Qwen3-8B | 89% | |
| Phi-4-mini | 67% | |
| SmolLM3-3B | 13% | |
| Qwen2.5-3B | 13% | |
| Qwen2.5-1.5B | 0% | |
| SmolLM2-1.7B | 0% | |
| Qwen3-0.6B | 17% | |
| Qwen2.5-0.5B | 0% | |
| OLMo-2-1B | 2% | |
| Qwen3.5-0.8B | 0% | |
Who fooled whom
Scroll sideways to see every model.
| judge ↓ / answered by → | Claude Opus 5.5 | Inkling 2 Small (trained) | Claude Sonnet 5.5 | Claude Haiku 5.5 | Inkling 2 Small | Qwen3.5-4B (trained) | Qwen3.5-4B (trained, medical) | Qwen3.5-4B | Qwen2.5-7B | Qwen3-8B | Phi-4-mini | SmolLM3-3B | Qwen2.5-3B | Qwen2.5-1.5B | SmolLM2-1.7B | Qwen3-0.6B | Qwen2.5-0.5B | OLMo-2-1B | Qwen3.5-0.8B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 50 | 100 | 67 | 27 | 17 | 17 | 7 | 3 | 11 | 5 | 17 | 10 | 3 | 10 | 16 | 1 | 13 | 16 | |
| Inkling 2 Small (trained) | 95 | 100 | 83 | 48 | 15 | 20 | 11 | 6 | 9 | 6 | 14 | 5 | 7 | 10 | 15 | 2 | 11 | 9 | |
| Claude Sonnet 5.5 | 100 | 20 | 67 | 33 | 19 | 14 | 11 | 31 | 6 | 39 | 29 | 32 | 33 | 8 | 57 | 18 | 10 | 26 | |
| Claude Haiku 5.5 | 75 | 75 | 100 | 43 | 21 | 25 | 21 | 11 | 16 | 22 | 32 | 12 | 35 | 10 | 46 | 21 | 16 | 20 | |
| Inkling 2 Small | 65 | 82 | 67 | 58 | 20 | 22 | 15 | 13 | 11 | 15 | 19 | 13 | 17 | 10 | 36 | 13 | 11 | 17 | |
| Qwen3.5-4B (trained) | 70 | 77 | 89 | 75 | 76 | 77 | 69 | 57 | 56 | 63 | 53 | 48 | 41 | 34 | 58 | 23 | 32 | 36 | |
| Qwen3.5-4B (trained, medical) | 60 | 68 | 78 | 71 | 72 | 74 | 68 | 58 | 59 | 69 | 57 | 55 | 49 | 36 | 61 | 26 | 36 | 39 | |
| Qwen3.5-4B | 65 | 68 | 78 | 63 | 76 | 75 | 78 | 55 | 54 | 66 | 53 | 58 | 43 | 38 | 53 | 24 | 34 | 39 | |
| Qwen2.5-7B | 40 | 32 | 44 | 33 | 28 | 28 | 26 | 28 | 21 | 23 | 28 | 32 | 19 | 17 | 18 | 6 | 13 | 8 | |
| Qwen3-8B | 45 | 32 | 33 | 29 | 38 | 39 | 33 | 32 | 34 | 40 | 35 | 45 | 29 | 20 | 27 | 15 | 19 | 21 | |
| Phi-4-mini | 25 | 14 | 28 | 17 | 21 | 22 | 16 | 26 | 17 | 23 | 29 | 27 | 20 | 19 | 14 | 3 | 20 | 12 | |
| SmolLM3-3B | 60 | 46 | 56 | 42 | 48 | 45 | 45 | 51 | 49 | 52 | 55 | 47 | 38 | 37 | 49 | 15 | 35 | 35 | |
| Qwen2.5-3B | 30 | 23 | 33 | 13 | 21 | 17 | 14 | 17 | 21 | 16 | 20 | 22 | 17 | 9 | 25 | 7 | 11 | 13 | |
| Qwen2.5-1.5B | 45 | 36 | 39 | 42 | 48 | 42 | 49 | 31 | 52 | 45 | 39 | 47 | 53 | 49 | 52 | 29 | 32 | 35 | |
| SmolLM2-1.7B | 65 | 55 | 67 | 54 | 72 | 77 | 69 | 76 | 63 | 76 | 68 | 64 | 60 | 72 | 69 | 56 | 74 | 66 | |
| Qwen3-0.6B | 100 | 100 | 100 | 100 | 100 | 99 | 98 | 99 | 99 | 100 | 100 | 99 | 100 | 100 | 100 | 100 | 100 | 100 | |
| Qwen2.5-0.5B | 40 | 46 | 50 | 54 | 59 | 51 | 63 | 42 | 52 | 44 | 41 | 55 | 57 | 64 | 49 | 60 | 55 | 47 | |
| OLMo-2-1B | 100 | 100 | 100 | 100 | 100 | 95 | 94 | 98 | 94 | 96 | 98 | 94 | 97 | 98 | 100 | 100 | 98 | 100 | |
| Qwen3.5-0.8B | 100 | 96 | 100 | 100 | 100 | 99 | 98 | 98 | 94 | 93 | 99 | 94 | 87 | 94 | 89 | 95 | 73 | 88 |
Numbers are percent. Hover or tap a cell for names, or focus the map and use the arrow keys.
Right, wrong or pass, by format
| Seat | Open trivia | Multiple choice | True or false | Medical exam |
|---|---|---|---|---|
| Claude Opus 5.5 | ||||
| Inkling 2 Small (trained) | ||||
| Claude Sonnet 5.5 | ||||
| Claude Haiku 5.5 | ||||
| Inkling 2 Small | ||||
| Qwen3.5-4B (trained) | ||||
| Qwen3.5-4B (trained, medical) | ||||
| Qwen3.5-4B | ||||
| Qwen2.5-7B | ||||
| Qwen3-8B | ||||
| Phi-4-mini | ||||
| SmolLM3-3B | ||||
| Qwen2.5-3B | ||||
| Qwen2.5-1.5B | ||||
| SmolLM2-1.7B | ||||
| Qwen3-0.6B | ||||
| Qwen2.5-0.5B | ||||
| OLMo-2-1B | ||||
| Qwen3.5-0.8B |
How the ratings work
- On every question, we compare each pair of models by the chips they won. More chips counts as a win.
- Those wins become one score per model, the way chess ratings work. 1500 is average. A model 100 points higher wins about 64% of the time.
- The bar around each score shows how sure we are. If two bars overlap, we wouldn't bet on which model is better.
- To top Overall, a model has to do both jobs well: answer honestly and judge well.
Want to collaborate?
Want your model at the table, or a model trained this way for your own task? Leave your email and we'll reach out.