Who can you trust?

19 AI models play a game. One answers a question and bets chips that it's right. The others see the answer, but not the correct one, and vote: trust it or call it wrong.

This page has the full results: who you can trust, and who gets fooled. We also use the game's scoring to train models, and the ones we trained are on the board too.

models: 12 on our Mac, 3 Claude, 1 on Tinker, 3 we trained
19
models: 12 on our Mac, 3 Claude, 1 on Tinker, 3 we trained
questions in 4 formats
600
questions in 4 formats
rounds played
11,400
rounds played
trust or call votes
149,628
trust or call votes
Claude API cost
$14.82
Claude API cost

Leaderboard

What training changed

Our own experiments. We trained 2 open models using the game's scoring, on separate practice questions, never these. Below is each one before and after.

Inkling 2 Small: #3 to #1 of 19

  1. #3 of 19 overall · beforeInkling 2 SmallTinker
    Overall score
    1726
    Judging score
    1634
    Answers wrong
    5%
    Said I don't know
    4%
    Chips per question
    +2.38
  2. #1 of 19 overallInkling 2 Small (trained)Trained by us
    Overall score
    1805
    Judging score
    1704
    Answers wrong
    4%
    Said I don't know
    1%
    Chips per question
    +2.67
  3. #2 of 19 overall · for comparisonClaude Opus 5.5API
    Overall score
    1798
    Judging score
    1683
    Answers wrong
    3%
    Said I don't know
    0%
    Chips per question
    +2.71

Qwen3.5-4B: #10 to #6 of 19

  1. #10 of 19 overall · beforeQwen3.5-4BLocal
    Overall score
    1511
    Judging score
    1493
    Answers wrong
    24%
    Said I don't know
    1%
    Chips per question
    +0.75
  2. #6 of 19 overallQwen3.5-4B (trained)Trained by us
    Overall score
    1614
    Judging score
    1514
    Answers wrong
    11%
    Said I don't know
    14%
    Chips per question
    +1.59
  3. #7 of 19 overallQwen3.5-4B (trained, medical)Trained by us
    Overall score
    1605
    Judging score
    1499
    Answers wrong
    9%
    Said I don't know
    19%
    Chips per question
    +1.63

Scores work like chess ratings, with 1500 as average. Answers wrong, I don't know and chips per question count only the model's own answers.

Answering and judging combined.

  • Hollow circle: open model on our Mac.
  • Green diamond: open model on Tinker.
  • Black square: API.
  • Green circle with a ring: trained by us.
Overall leaderboard, ranked by score, with the range and chips won per question.
#ModelScoreChips
1Inkling 2 Small (trained)Trained by usThinking Machines594B (17B active)90% interval 1790 to 18241805+3.72
2Claude Opus 5.5APIAnthropicsize n/a90% interval 1784 to 18141798+3.72
3Inkling 2 SmallTinkerThinking Machines594B (17B active)90% interval 1714 to 17411726+3.25
4Claude Sonnet 5.5APIAnthropicsize n/a90% interval 1713 to 17411726+3.42
5Claude Haiku 5.5APIAnthropicsize n/a90% interval 1708 to 17361720+3.33
6Qwen3.5-4B (trained)Trained by usAlibaba (Qwen)4B90% interval 1599 to 16291614+2.10
7Qwen3.5-4B (trained, medical)Trained by usAlibaba (Qwen)4B90% interval 1590 to 16191605+2.14
8Qwen2.5-7BLocalAlibaba (Qwen)7B90% interval 1518 to 15531536+1.16
9Qwen3-8BLocalAlibaba (Qwen)8B90% interval 1505 to 15381521+1.08
10Qwen3.5-4BLocalAlibaba (Qwen)4B90% interval 1492 to 15261511+1.24
11Phi-4-miniLocalMicrosoft3.8B90% interval 1419 to 14561438+0.30
12SmolLM3-3BLocalHugging Face3B90% interval 1360 to 13841372+0.25
13Qwen2.5-3BLocalAlibaba (Qwen)3B90% interval 1342 to 13671355+0.13
14Qwen2.5-1.5BLocalAlibaba (Qwen)1.5B90% interval 1319 to 13431332+0.02
15Qwen3-0.6BLocalAlibaba (Qwen)0.6B90% interval 1309 to 13361323−0.11
16SmolLM2-1.7BLocalHugging Face1.7B90% interval 1307 to 13331320−0.10
17OLMo-2-1BLocalAi21B90% interval 1255 to 12831269−0.52
18Qwen2.5-0.5BLocalAlibaba (Qwen)0.5B90% interval 1251 to 12811266−0.77
19Qwen3.5-0.8BLocalAlibaba (Qwen)0.8B90% interval 1247 to 12801264−0.60

The range under each name shows how sure we are of the score. If two ranges overlap, the order isn't reliable. Chips per question is the average a model won or lost. Click a model to see its games.

Our own runs, 2026-11-11. Open models ran on a Mac mini. Claude models ran through the API with their default settings.

Head to head

How often should the row model beat the column model? This is the expected chance, from their ratings. Darker means the row model wins more. 50 is a coin flip.

Darker: the row model wins more often. 50 is a coin flip.

Scroll sideways to see all 19 columns.

Expected chance that the row model beats the column model on the overall board, in percent. Rows and columns are ordered by rating.
Inkling 2 Small (trained)Claude Opus 5.5Inkling 2 SmallClaude Sonnet 5.5Claude Haiku 5.5Qwen3.5-4B (trained)Qwen3.5-4B (trained, medical)Qwen2.5-7BQwen3-8BQwen3.5-4BPhi-4-miniSmolLM3-3BQwen2.5-3BQwen2.5-1.5BQwen3-0.6BSmolLM2-1.7BOLMo-2-1BQwen2.5-0.5BQwen3.5-0.8B
Inkling 2 Small (trained)516161627576838484899293949494969696
Claude Opus 5.5496060617475828384899293949494969696
Inkling 2 Small394050516667757778848989919191939394
Claude Sonnet 5.5394050516667757778848989919191939394
Claude Haiku 5.5383949496566747677848889909191939393
Qwen3.5-4B (trained)252634343551616364738082848485888888
Qwen3.5-4B (trained, medical)242533333449606263727981838484878888
Qwen2.5-7B181825252639405254647274767778828383
Qwen3-8B161723242437384851627072757676818181
Qwen3.5-4B161622232336374649606971747575808081
Phi-4-mini111116161727283638405962656666737373
SmolLM3-3B8812121220212830314152565757656565
Qwen2.5-3B7711111118192628293848535555626363
Qwen2.5-1.5B66991017172425263544475152595960
Qwen3-0.6B6699916172324253443454951585859
SmolLM2-1.7B6699916162224253443454850575858
OLMo-2-1B4577712131819202736384142435051
Qwen2.5-0.5B4577712131819202735374142425050
Qwen3.5-0.8B4477712121719192735374042424950

Hover or tap a cell for the two names, or focus the grid and use the arrow keys.

More from the games

Good at answering, good at judging?

  • Hollow circle: open model on our Mac.
  • Green diamond: open model on Tinker.
  • Black square: API.
  • Green circle with a ring: trained by us.
12001200130013001400140015001500160016001700170018001800Answering scoreJudging score

Hover, tap or focus a point for its name.

How to read this. Each dot is a model. Further right means better at answering. Higher up means better at judging. Models above the dashed line are better judges than answerers.

Does a big bet mean it knows?

Scroll sideways to see both columns.

Accuracy when betting 1 chip and when betting 3 chips, per seat, and how often each seat bet 3.
Seat right when betting 1 right when betting 3Bets 3
Claude Opus 5.596%
Inkling 2 Small (trained)98%
Claude Sonnet 5.593%
Claude Haiku 5.590%
Inkling 2 Small90%
Qwen3.5-4B (trained)100%
Qwen3.5-4B (trained, medical)100%
Qwen3.5-4B79%
Qwen2.5-7B92%
Qwen3-8B89%
Phi-4-mini67%
SmolLM3-3B13%
Qwen2.5-3B13%
Qwen2.5-1.5B0%
SmolLM2-1.7B0%
Qwen3-0.6B17%
Qwen2.5-0.5B0%
OLMo-2-1B2%
Qwen3.5-0.8B0%
How to read this. How often each model was right when it bet 1 chip (circle) and when it bet 3 (square). A model that knows when it's sure has its square well to the right. If the two marks sit on top of each other, its bet size means nothing.

Who fooled whom

Scroll sideways to see every model.

Share of each model's wrong answers that each judge trusted, in percent. Rows are judges, columns are the models that answered.
judge ↓ / answered by →Claude Opus 5.5Inkling 2 Small (trained)Claude Sonnet 5.5Claude Haiku 5.5Inkling 2 SmallQwen3.5-4B (trained)Qwen3.5-4B (trained, medical)Qwen3.5-4BQwen2.5-7BQwen3-8BPhi-4-miniSmolLM3-3BQwen2.5-3BQwen2.5-1.5BSmolLM2-1.7BQwen3-0.6BQwen2.5-0.5BOLMo-2-1BQwen3.5-0.8B
Claude Opus 5.550100672717177311517103101611316
Inkling 2 Small (trained)951008348152011696145710152119
Claude Sonnet 5.510020673319141131639293233857181026
Claude Haiku 5.57575100432125211116223212351046211620
Inkling 2 Small658267582022151311151913171036131117
Qwen3.5-4B (trained)707789757677695756635348413458233236
Qwen3.5-4B (trained, medical)606878717274685859695755493661263639
Qwen3.5-4B656878637675785554665358433853243439
Qwen2.5-7B4032443328282628212328321917186138
Qwen3-8B453233293839333234403545292027151921
Phi-4-mini25142817212216261723292720191432012
SmolLM3-3B604656424845455149525547383749153535
Qwen2.5-3B3023331321171417211620221792571113
Qwen2.5-1.5B453639424842493152453947534952293235
SmolLM2-1.7B655567547277697663766864607269567466
Qwen3-0.6B1001001001001009998999910010099100100100100100100
Qwen2.5-0.5B404650545951634252444155576449605547
OLMo-2-1B10010010010010095949894969894979810010098100
Qwen3.5-0.8B1009610010010099989894939994879489957388

Numbers are percent. Hover or tap a cell for names, or focus the map and use the arrow keys.

How to read this. Each row is a judge and each column is the model that answered. A cell shows how often the judge trusted that model's wrong answers. Darker means fooled more often.

Right, wrong or pass, by format

Share of right, wrong and passed answers per seat and question format.
SeatOpen triviaMultiple choiceTrue or falseMedical exam
Claude Opus 5.5
Inkling 2 Small (trained)
Claude Sonnet 5.5
Claude Haiku 5.5
Inkling 2 Small
Qwen3.5-4B (trained)
Qwen3.5-4B (trained, medical)
Qwen3.5-4B
Qwen2.5-7B
Qwen3-8B
Phi-4-mini
SmolLM3-3B
Qwen2.5-3B
Qwen2.5-1.5B
SmolLM2-1.7B
Qwen3-0.6B
Qwen2.5-0.5B
OLMo-2-1B
Qwen3.5-0.8B
How to read this. Each bar is one model on 150 questions of one type: right, wrong, or “I don't know”. A long red segment means a model that guesses and misses.

How the ratings work

  • On every question, we compare each pair of models by the chips they won. More chips counts as a win.
  • Those wins become one score per model, the way chess ratings work. 1500 is average. A model 100 points higher wins about 64% of the time.
  • The bar around each score shows how sure we are. If two bars overlap, we wouldn't bet on which model is better.
  • To top Overall, a model has to do both jobs well: answer honestly and judge well.
Full rules and harness

Want to collaborate?

Want your model at the table, or a model trained this way for your own task? Leave your email and we'll reach out.