Benchmark Almanac

SynthBench

Qwen3.8 Flash + Claude Sonnet 4 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite 45.6 tops its leaderboard Qwen3.8 Flash 32.2 leads the models we track
Not in model cards

Models answer survey questions. The index compares their answer distributions with human responses across four datasets, using random answers and human agreement as reference points.

The leaderboard lists 16 distinct base models. Its 48 system configurations include different prompts, access methods, and ensembles.

Made by
DataViking
Released
April 2026
Kind
Social simulation
Level
Everyday
Scored
Checked automatically
Answers
Multiple choice
Tasks
Public, plus a private test set
Measures
Capability
Subject
Sociology
Language
English
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
Strongest model tested
Claude Sonnet 5.5 #9 on the capability index
Last model added
October 2026
Scores refreshed
November 20th, 2026
21
Entries on its own leaderboard, 16 models we track

Qwen3.8 Flash + Claude Sonnet 4 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite tops the leaderboard with 45.6 Qwen3.8 Flash leads with 32.2

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen3.8 Flash, at 32.2. This view shows only models we track, each at its best.

  1. 1 Qwen3.8 Flash + Claude Sonnet 4 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite ensemble 45.6
  2. 2 Claude Haiku 5.5 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite ensemble 42.2
  3. 3 Qwen3.8 Flash + DeepSeek V4.1 Flash + Gemini 3.5 Flash-Lite ensemble 39.4
  4. 4 Claude Haiku 5.5 + Llama 3.3 70B Instruct + Gemini 2.5 Flash-Lite ensemble 39.1
  5. 5 Qwen3.8 Flash althing 32.2
  6. 6 DeepSeek V4.1 Flash raw 31.7
  7. 7 Claude Haiku 4.5 + Gemini 2.5 Flash-Lite + GPT-4o mini ensemble 30.1
  8. 8 Claude Haiku 4.5 althing, temperature 0.3 21.8
  9. 9 Gemini 3.5 Flash-Lite althing 21.7
  10. 10 Claude Sonnet 4 althing 19.8
  11. 11 Muse Glimmer althing 18.0
  12. 12 Claude Haiku 5.5 althing, structured template, global-respondents personas 16.4
  13. 13 GPT-6 Luna althing 16.1
  14. 14 Gemini 2.5 Flash-Lite Sep 2025, raw 14.9
  15. 15 Claude Sonnet 4.6 althing 13.0
  16. 16 Llama 3.3 70B althing 12.5
  17. 17 GPT-4o Mar 2025, raw 12.5
  18. 18 GPT-4o mini althing, temperature 0.3 12.3
  19. 19 Claude Sonnet 5.5 althing 3.4
  20. 20 Gemini 2.5 Flash Sep 2025, raw 2.0
  21. 21 Gemini 3.8 Flash raw -51.4

Source: Official SynthBench leaderboard

#EntryScore ±SettingDateSource
1 Qwen3.8 Flash + Claude Sonnet 4 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite 45.6 ±4.5 ensemble Official leaderboard
2 Claude Haiku 5.5 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite 42.2 ±4.4 ensemble Official leaderboard
3 Qwen3.8 Flash + DeepSeek V4.1 Flash + Gemini 3.5 Flash-Lite 39.4 ±4.2 ensemble Official leaderboard
4 Claude Haiku 5.5 + Llama 3.3 70B Instruct + Gemini 2.5 Flash-Lite 39.1 ±4.6 ensemble Official leaderboard
5 Qwen3.8 Flash 32.2 ±4.7 althing Aug 2026 Official leaderboard
6 DeepSeek V4.1 Flash 31.7 ±4.8 raw Sep 2026 Official leaderboard
7 Claude Haiku 4.5 + Gemini 2.5 Flash-Lite + GPT-4o mini 30.1 ±4.4 ensemble Official leaderboard
8 Claude Haiku 4.5 21.8 ±7.8 althing, temperature 0.3 Oct 2025 Official leaderboard
9 Gemini 3.5 Flash-Lite 21.7 ±4.9 althing Jul 2026 Official leaderboard
10 Claude Sonnet 4 19.8 ±5.2 althing May 2025 Official leaderboard
11 Muse Glimmer 18.0 ±5.2 althing Aug 2026 Official leaderboard
12 Claude Haiku 5.5 16.4 ±11.6 althing, structured template, global-respondents personas Oct 2026 Official leaderboard
13 GPT-6 Luna 16.1 ±5.4 althing Sep 2026 Official leaderboard
14 Gemini 2.5 Flash-Lite Sep 2025 14.9 ±5.7 raw Sep 2025 Official leaderboard
15 Claude Sonnet 4.6 13.0 ±5.2 althing Feb 2026 Official leaderboard
16 Llama 3.3 70B 12.5 ±4.8 althing Dec 2024 Official leaderboard
17 GPT-4o Mar 2025 12.5 ±5.8 raw Mar 2025 Official leaderboard
18 GPT-4o mini 12.3 ±8.9 althing, temperature 0.3 Jul 2024 Official leaderboard
19 Claude Sonnet 5.5 3.4 ±6.3 althing Sep 2026 Official leaderboard
20 Gemini 2.5 Flash Sep 2025 2.0 ±5 raw Sep 2025 Official leaderboard
21 Gemini 3.8 Flash -51.4 ±8.1 raw Sep 2026 Official leaderboard
#ModelScore ±SettingDateSource
1 Qwen3.8 Flash 32.2 ±4.7 althing Aug 2026 Official leaderboard
2 DeepSeek V4.1 Flash 31.7 ±4.8 raw Sep 2026 Official leaderboard
3 Claude Haiku 4.5 21.8 ±7.8 althing, temperature 0.3 Oct 2025 Official leaderboard
4 Gemini 3.5 Flash-Lite 21.7 ±4.9 althing Jul 2026 Official leaderboard
5 Claude Sonnet 4 19.8 ±5.2 althing May 2025 Official leaderboard
6 Muse Glimmer 18.0 ±5.2 althing Aug 2026 Official leaderboard
7 Claude Haiku 5.5 16.4 ±11.6 althing, structured template, global-respondents personas Oct 2026 Official leaderboard
8 GPT-6 Luna 16.1 ±5.4 althing Sep 2026 Official leaderboard
9 Gemini 2.5 Flash-Lite Sep 2025 14.9 ±5.7 raw Sep 2025 Official leaderboard
10 Claude Sonnet 4.6 13.0 ±5.2 althing Feb 2026 Official leaderboard
11 Llama 3.3 70B 12.5 ±4.8 althing Dec 2024 Official leaderboard
12 GPT-4o Mar 2025 12.5 ±5.8 raw Mar 2025 Official leaderboard
13 GPT-4o mini 12.3 ±8.9 althing, temperature 0.3 Jul 2024 Official leaderboard
14 Claude Sonnet 5.5 3.4 ±6.3 althing Sep 2026 Official leaderboard
15 Gemini 2.5 Flash Sep 2025 2.0 ±5 raw Sep 2025 Official leaderboard
16 Gemini 3.8 Flash -51.4 ±8.1 raw Sep 2026 Official leaderboard

Of runs with a cost, the best scores 21.8 for $0.02 a task

Runs on the cost frontier score higher than any run that's cheaper. Gemini 3.5 Flash-Lite at althing comes within 0.1 of the best for $0.0005 a task, 44 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind Anthropic Cost frontier One model at each effort level
0.0 5.0 10.0 15.0 20.0 25.0 $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Claude Haiku 4.5, althing, temperature 0.3 Gemini 3.5 Flash-Lite, raw Gemini 2.5 Flash-Lite, raw 0.0 5.0 10.0 15.0 20.0 25.0 $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Claude Haiku 4.5, althing, temperature 0.3 Gemini 3.5 Flash-Lite, raw

Source: Official SynthBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Qwen3.8 Flash althing 32.2
DeepSeek V4.1 Flash raw 31.7
Qwen3.8 Flash raw 28.8
DeepSeek V4.1 Flash althing 22.2
Claude Haiku 4.5 althing, temperature 0.3 21.8 $0.02
Gemini 3.5 Flash-Lite althing 21.7 $0.0005
Claude Haiku 4.5 althing, structured template 21.3
Claude Sonnet 4 althing 19.8 $0.06
Gemini 3.5 Flash-Lite raw 18.7 $0.0003
Muse Glimmer althing 18.0
Claude Haiku 5.5 althing, structured template, global-respondents personas 16.4 $0.006
GPT-6 Luna althing 16.1
GPT-6 Luna althing, structured template, global-respondents personas codex 15.6
Claude Haiku 5.5 althing 15.5 $0.03
GPT-6 Luna althing codex 15.2
Gemini 2.5 Flash-Lite Sep 2025 raw 14.9 $0.0003
Gemini 2.5 Flash-Lite Sep 2025 althing, temperature 0.3 13.9 $0.0004
Claude Sonnet 4.6 althing 13.0 $0.06
Llama 3.3 70B althing 12.5 $0.0009
GPT-4o Mar 2025 raw 12.5 $0.009
GPT-4o mini althing, temperature 0.3 12.3 $0.001
Claude Haiku 5.5 althing, structured template 11.6 $0.006
GPT-6 Luna althing, structured template codex 10.7
GPT-4o mini althing 9.3 $0.001
Claude Haiku 4.5 althing 5.9 $0.02
Llama 3.3 70B raw 4.2 $0.0004
Claude Sonnet 5.5 althing 3.4 $0.10
Gemini 2.5 Flash Sep 2025 raw 2.0 $0.0009
Claude Sonnet 4.6 raw 1.8 $0.01
Gemini 2.5 Flash-Lite Sep 2025 althing -0.4 $0.0004
GPT-4o mini raw -0.9 $0.0004
Claude Sonnet 5.5 raw -1.5 $0.02
GPT-6 Luna raw codex -9.2
Claude Haiku 4.5 althing, temperature 0.85 -13.4 $0.004
GPT-6 Luna raw -13.4
Claude Haiku 4.5 raw -16.9 $0.005
Claude Haiku 5.5 raw -25.5 $0.006
Claude Sonnet 4 raw -31.7 $0.02
Claude Haiku 4.5 althing, temperature 0.85, demo template -37.8 $0.004
Claude Haiku 4.5 althing, temperature 0.85, minimal template -45.9 $0.004
Claude Haiku 4.5 althing, temperature 0.85, values template -50.4 $0.004
Gemini 3.8 Flash raw -51.4 $0.02
Muse Glimmer raw -52.4

Of runs with a cost, the best scores 21.8 for $0.02 a task

Runs on the cost frontier score higher than any run that's cheaper. Gemini 3.5 Flash-Lite at althing comes within 0.1 of the best for $0.0005 a task, 44 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind Anthropic Cost frontier One model at each effort level
0.0 5.0 10.0 15.0 20.0 25.0 $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Claude Haiku 4.5, althing, temperature 0.3 Gemini 3.5 Flash-Lite, raw Gemini 2.5 Flash-Lite, raw 0.0 5.0 10.0 15.0 20.0 25.0 $0.0001 $0.001 $0.01 $0.10 $1.00 Cost per task, log scale Claude Haiku 4.5, althing, temperature 0.3 Gemini 3.5 Flash-Lite, raw

Source: Official SynthBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Qwen3.8 Flash althing 32.2
DeepSeek V4.1 Flash raw 31.7
Qwen3.8 Flash raw 28.8
DeepSeek V4.1 Flash althing 22.2
Claude Haiku 4.5 althing, temperature 0.3 21.8 $0.02
Gemini 3.5 Flash-Lite althing 21.7 $0.0005
Claude Haiku 4.5 althing, structured template 21.3
Claude Sonnet 4 althing 19.8 $0.06
Gemini 3.5 Flash-Lite raw 18.7 $0.0003
Muse Glimmer althing 18.0
Claude Haiku 5.5 althing, structured template, global-respondents personas 16.4 $0.006
GPT-6 Luna althing 16.1
GPT-6 Luna althing, structured template, global-respondents personas codex 15.6
Claude Haiku 5.5 althing 15.5 $0.03
GPT-6 Luna althing codex 15.2
Gemini 2.5 Flash-Lite Sep 2025 raw 14.9 $0.0003
Gemini 2.5 Flash-Lite Sep 2025 althing, temperature 0.3 13.9 $0.0004
Claude Sonnet 4.6 althing 13.0 $0.06
Llama 3.3 70B althing 12.5 $0.0009
GPT-4o Mar 2025 raw 12.5 $0.009
GPT-4o mini althing, temperature 0.3 12.3 $0.001
Claude Haiku 5.5 althing, structured template 11.6 $0.006
GPT-6 Luna althing, structured template codex 10.7
GPT-4o mini althing 9.3 $0.001
Claude Haiku 4.5 althing 5.9 $0.02
Llama 3.3 70B raw 4.2 $0.0004
Claude Sonnet 5.5 althing 3.4 $0.10
Gemini 2.5 Flash Sep 2025 raw 2.0 $0.0009
Claude Sonnet 4.6 raw 1.8 $0.01
Gemini 2.5 Flash-Lite Sep 2025 althing -0.4 $0.0004
GPT-4o mini raw -0.9 $0.0004
Claude Sonnet 5.5 raw -1.5 $0.02
GPT-6 Luna raw codex -9.2
Claude Haiku 4.5 althing, temperature 0.85 -13.4 $0.004
GPT-6 Luna raw -13.4
Claude Haiku 4.5 raw -16.9 $0.005
Claude Haiku 5.5 raw -25.5 $0.006
Claude Sonnet 4 raw -31.7 $0.02
Claude Haiku 4.5 althing, temperature 0.85, demo template -37.8 $0.004
Claude Haiku 4.5 althing, temperature 0.85, minimal template -45.9 $0.004
Claude Haiku 4.5 althing, temperature 0.85, values template -50.4 $0.004
Gemini 3.8 Flash raw -51.4 $0.02
Muse Glimmer raw -52.4

2 models have held number one on the official leaderboard since its release in April 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic Qwen
Own leaderboard 2 leaders

Source: Official SynthBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
-60 -40 -20 0 20 40 2025 2026 -60 -40 -20 0 20 40 2025 2026

Source: Official SynthBench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Forecasting

All 47 in Forecasting
  • ForecastBench In use Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. GPT-5.5 0.11
  • Prophet Arena In use Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. Claude Opus 5.5 86.2
  • Guesswork (The Aggregate) Not in model cards Models predict benchmark scores before publication. The maker ranks forecasts by mean absolute error after scaling each error by the variation in that benchmark’s scores. GPT-6 Astra 0.56
  • AGI-Pilled Bench Not in model cards Models forecast when AI will reach milestones that already happened in 2026. The score is the median years after 2026 across the forecast tasks. Claude Fable 5.2 6.50
  • Dutch Book Bench Not in model cards Models assign probabilities to related events in separate conversations. The grader checks whether inconsistent probabilities let a bettor guarantee a profit. Gemini 3.8 Flash 89.2%
  • Echo General AI Prediction Leaderboard Not in model cards Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. DeepSeek-V4-Pro 1,018

Data audit

All 9 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access Public tasks, plus a private test set.
  • Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (21 entries).
  • Pass Models scored 16 models we track, plus 5 other entries on its leaderboard.
  • Pass Newest score The newest is from October 2026, for Claude Haiku 5.5.
  • Note Frontier models Only 1 of the top 10 has a score. The strongest tested is Claude Sonnet 5.5 (#9).
  • Note Headroom Scored as an index, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on synthbench.org: 43 rows for models we track and 5 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check