SynthBench
Models answer survey questions. The index compares their answer distributions with human responses across four datasets, using random answers and human agreement as reference points.
The leaderboard lists 16 distinct base models. Its 48 system configurations include different prompts, access methods, and ensembles.
- Made by
- DataViking
- Released
- April 2026
- Kind
- Social simulation
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- Public, plus a private test set
- Measures
- Capability
- Subject
- Sociology
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use, Another benchmark
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
- Claude Sonnet 5.5 #9 on the capability index
- Last model added
- October 2026
- Scores refreshed
- November 20th, 2026
Qwen3.8 Flash + Claude Sonnet 4 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite tops the leaderboard with 45.6 Qwen3.8 Flash leads with 32.2
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen3.8 Flash, at 32.2. This view shows only models we track, each at its best.
- 45.6 Ensemble of Qwen 3.8 Flash (persona), Sonnet 4 (persona), DeepSeek V4.1 Flash (raw), Gemini 2.5 Flash-Lite (raw)
- 42.2 Ensemble of Haiku 5.5 (persona), DeepSeek V4.1 Flash (raw), Gemini 2.5 Flash-Lite (raw)
- 39.4 Ensemble of Qwen 3.8 Flash, DeepSeek V4.1 Flash, Gemini 3.5 Flash-Lite (all raw)
- 39.1 Ensemble of Haiku 5.5 (persona), Llama 3.3 70B (raw), Gemini 2.5 Flash-Lite (raw)
- 32.2
Qwen3.8 Flash
- 31.7
DeepSeek V4.1 Flash
- 30.1 Ensemble of Haiku 4.5, Gemini 2.5 Flash-Lite, GPT-4o mini (all persona)
- 21.8 Claude Haiku 4.5
- 21.7
Gemini 3.5 Flash-Lite
- 19.8 Claude Sonnet 4
- 18.0
Muse Glimmer
- 16.4 Claude Haiku 5.5
- 16.1 GPT-6 Luna
- 14.9
Gemini 2.5 Flash-Lite
- 13.0 Claude Sonnet 4.6
- 12.5
Llama 3.3 70B
- 12.5 GPT-4o
- 12.3 GPT-4o mini
- 3.4 Claude Sonnet 5.5
- 2.0
Gemini 2.5 Flash
- -51.4
Gemini 3.8 Flash
- 1 Qwen3.8 Flash + Claude Sonnet 4 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite ensemble 45.6
- 2 Claude Haiku 5.5 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite ensemble 42.2
- 3 Qwen3.8 Flash + DeepSeek V4.1 Flash + Gemini 3.5 Flash-Lite ensemble 39.4
- 4 Claude Haiku 5.5 + Llama 3.3 70B Instruct + Gemini 2.5 Flash-Lite ensemble 39.1
- 5
Qwen3.8 Flash althing 32.2
- 6
DeepSeek V4.1 Flash raw 31.7
- 7 Claude Haiku 4.5 + Gemini 2.5 Flash-Lite + GPT-4o mini ensemble 30.1
- 8 Claude Haiku 4.5 althing, temperature 0.3 21.8
- 9
Gemini 3.5 Flash-Lite althing 21.7
- 10 Claude Sonnet 4 althing 19.8
- 11
Muse Glimmer althing 18.0
- 12 Claude Haiku 5.5 althing, structured template, global-respondents personas 16.4
- 13 GPT-6 Luna althing 16.1
- 14
Gemini 2.5 Flash-Lite Sep 2025, raw 14.9
- 15 Claude Sonnet 4.6 althing 13.0
- 16
Llama 3.3 70B althing 12.5
- 17 GPT-4o Mar 2025, raw 12.5
- 18 GPT-4o mini althing, temperature 0.3 12.3
- 19 Claude Sonnet 5.5 althing 3.4
- 20
Gemini 2.5 Flash Sep 2025, raw 2.0
- 21
Gemini 3.8 Flash raw -51.4
Source: Official SynthBench leaderboard
- 32.2
Qwen3.8 Flash
- 31.7
DeepSeek V4.1 Flash
- 21.8 Claude Haiku 4.5
- 21.7
Gemini 3.5 Flash-Lite
- 19.8 Claude Sonnet 4
- 18.0
Muse Glimmer
- 16.4 Claude Haiku 5.5
- 16.1 GPT-6 Luna
- 14.9
Gemini 2.5 Flash-Lite
- 13.0 Claude Sonnet 4.6
- 12.5
Llama 3.3 70B
- 12.5 GPT-4o
- 12.3 GPT-4o mini
- 3.4 Claude Sonnet 5.5
- 2.0
Gemini 2.5 Flash
- -51.4
Gemini 3.8 Flash
- 1
Qwen3.8 Flash althing 32.2
- 2
DeepSeek V4.1 Flash raw 31.7
- 3 Claude Haiku 4.5 althing, temperature 0.3 21.8
- 4
Gemini 3.5 Flash-Lite althing 21.7
- 5 Claude Sonnet 4 althing 19.8
- 6
Muse Glimmer althing 18.0
- 7 Claude Haiku 5.5 althing, structured template, global-respondents personas 16.4
- 8 GPT-6 Luna althing 16.1
- 9
Gemini 2.5 Flash-Lite Sep 2025, raw 14.9
- 10 Claude Sonnet 4.6 althing 13.0
- 11
Llama 3.3 70B althing 12.5
- 12 GPT-4o Mar 2025, raw 12.5
- 13 GPT-4o mini althing, temperature 0.3 12.3
- 14 Claude Sonnet 5.5 althing 3.4
- 15
Gemini 2.5 Flash Sep 2025, raw 2.0
- 16
Gemini 3.8 Flash raw -51.4
Source: Official SynthBench leaderboard
| # | Entry | Score ± | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Qwen3.8 Flash + Claude Sonnet 4 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite | 45.6 ±4.5 | ensemble | Official leaderboard | |
| 2 | Claude Haiku 5.5 + DeepSeek V4.1 Flash + Gemini 2.5 Flash-Lite | 42.2 ±4.4 | ensemble | Official leaderboard | |
| 3 | Qwen3.8 Flash + DeepSeek V4.1 Flash + Gemini 3.5 Flash-Lite | 39.4 ±4.2 | ensemble | Official leaderboard | |
| 4 | Claude Haiku 5.5 + Llama 3.3 70B Instruct + Gemini 2.5 Flash-Lite | 39.1 ±4.6 | ensemble | Official leaderboard | |
| 5 | | 32.2 ±4.7 | althing | Aug 2026 | Official leaderboard |
| 6 | | 31.7 ±4.8 | raw | Sep 2026 | Official leaderboard |
| 7 | Claude Haiku 4.5 + Gemini 2.5 Flash-Lite + GPT-4o mini | 30.1 ±4.4 | ensemble | Official leaderboard | |
| 8 | Claude Haiku 4.5 | 21.8 ±7.8 | althing, temperature 0.3 | Oct 2025 | Official leaderboard |
| 9 | | 21.7 ±4.9 | althing | Jul 2026 | Official leaderboard |
| 10 | Claude Sonnet 4 | 19.8 ±5.2 | althing | May 2025 | Official leaderboard |
| 11 | | 18.0 ±5.2 | althing | Aug 2026 | Official leaderboard |
| 12 | Claude Haiku 5.5 | 16.4 ±11.6 | althing, structured template, global-respondents personas | Oct 2026 | Official leaderboard |
| 13 | GPT-6 Luna | 16.1 ±5.4 | althing | Sep 2026 | Official leaderboard |
| 14 | | 14.9 ±5.7 | raw | Sep 2025 | Official leaderboard |
| 15 | Claude Sonnet 4.6 | 13.0 ±5.2 | althing | Feb 2026 | Official leaderboard |
| 16 | | 12.5 ±4.8 | althing | Dec 2024 | Official leaderboard |
| 17 | GPT-4o Mar 2025 | 12.5 ±5.8 | raw | Mar 2025 | Official leaderboard |
| 18 | GPT-4o mini | 12.3 ±8.9 | althing, temperature 0.3 | Jul 2024 | Official leaderboard |
| 19 | Claude Sonnet 5.5 | 3.4 ±6.3 | althing | Sep 2026 | Official leaderboard |
| 20 | | 2.0 ±5 | raw | Sep 2025 | Official leaderboard |
| 21 | | -51.4 ±8.1 | raw | Sep 2026 | Official leaderboard |
| # | Model | Score ± | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 32.2 ±4.7 | althing | Aug 2026 | Official leaderboard |
| 2 | | 31.7 ±4.8 | raw | Sep 2026 | Official leaderboard |
| 3 | Claude Haiku 4.5 | 21.8 ±7.8 | althing, temperature 0.3 | Oct 2025 | Official leaderboard |
| 4 | | 21.7 ±4.9 | althing | Jul 2026 | Official leaderboard |
| 5 | Claude Sonnet 4 | 19.8 ±5.2 | althing | May 2025 | Official leaderboard |
| 6 | | 18.0 ±5.2 | althing | Aug 2026 | Official leaderboard |
| 7 | Claude Haiku 5.5 | 16.4 ±11.6 | althing, structured template, global-respondents personas | Oct 2026 | Official leaderboard |
| 8 | GPT-6 Luna | 16.1 ±5.4 | althing | Sep 2026 | Official leaderboard |
| 9 | | 14.9 ±5.7 | raw | Sep 2025 | Official leaderboard |
| 10 | Claude Sonnet 4.6 | 13.0 ±5.2 | althing | Feb 2026 | Official leaderboard |
| 11 | | 12.5 ±4.8 | althing | Dec 2024 | Official leaderboard |
| 12 | GPT-4o Mar 2025 | 12.5 ±5.8 | raw | Mar 2025 | Official leaderboard |
| 13 | GPT-4o mini | 12.3 ±8.9 | althing, temperature 0.3 | Jul 2024 | Official leaderboard |
| 14 | Claude Sonnet 5.5 | 3.4 ±6.3 | althing | Sep 2026 | Official leaderboard |
| 15 | | 2.0 ±5 | raw | Sep 2025 | Official leaderboard |
| 16 | | -51.4 ±8.1 | raw | Sep 2026 | Official leaderboard |
Of runs with a cost, the best scores 21.8 for $0.02 a task
Runs on the cost frontier score higher than any run that's cheaper. Gemini 3.5 Flash-Lite at althing comes within 0.1 of the best for $0.0005 a task, 44 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official SynthBench leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| althing | 32.2 | ||||
| raw | 31.7 | ||||
| raw | 28.8 | ||||
| althing | 22.2 | ||||
| Claude Haiku 4.5 | althing, temperature 0.3 | 21.8 | $0.02 | ||
| althing | 21.7 | $0.0005 | |||
| Claude Haiku 4.5 | althing, structured template | 21.3 | |||
| Claude Sonnet 4 | althing | 19.8 | $0.06 | ||
| raw | 18.7 | $0.0003 | |||
| althing | 18.0 | ||||
| Claude Haiku 5.5 | althing, structured template, global-respondents personas | 16.4 | $0.006 | ||
| GPT-6 Luna | althing | 16.1 | |||
| GPT-6 Luna | althing, structured template, global-respondents personas | codex | 15.6 | ||
| Claude Haiku 5.5 | althing | 15.5 | $0.03 | ||
| GPT-6 Luna | althing | codex | 15.2 | ||
| raw | 14.9 | $0.0003 | |||
| althing, temperature 0.3 | 13.9 | $0.0004 | |||
| Claude Sonnet 4.6 | althing | 13.0 | $0.06 | ||
| althing | 12.5 | $0.0009 | |||
| GPT-4o Mar 2025 | raw | 12.5 | $0.009 | ||
| GPT-4o mini | althing, temperature 0.3 | 12.3 | $0.001 | ||
| Claude Haiku 5.5 | althing, structured template | 11.6 | $0.006 | ||
| GPT-6 Luna | althing, structured template | codex | 10.7 | ||
| GPT-4o mini | althing | 9.3 | $0.001 | ||
| Claude Haiku 4.5 | althing | 5.9 | $0.02 | ||
| raw | 4.2 | $0.0004 | |||
| Claude Sonnet 5.5 | althing | 3.4 | $0.10 | ||
| raw | 2.0 | $0.0009 | |||
| Claude Sonnet 4.6 | raw | 1.8 | $0.01 | ||
| althing | -0.4 | $0.0004 | |||
| GPT-4o mini | raw | -0.9 | $0.0004 | ||
| Claude Sonnet 5.5 | raw | -1.5 | $0.02 | ||
| GPT-6 Luna | raw | codex | -9.2 | ||
| Claude Haiku 4.5 | althing, temperature 0.85 | -13.4 | $0.004 | ||
| GPT-6 Luna | raw | -13.4 | |||
| Claude Haiku 4.5 | raw | -16.9 | $0.005 | ||
| Claude Haiku 5.5 | raw | -25.5 | $0.006 | ||
| Claude Sonnet 4 | raw | -31.7 | $0.02 | ||
| Claude Haiku 4.5 | althing, temperature 0.85, demo template | -37.8 | $0.004 | ||
| Claude Haiku 4.5 | althing, temperature 0.85, minimal template | -45.9 | $0.004 | ||
| Claude Haiku 4.5 | althing, temperature 0.85, values template | -50.4 | $0.004 | ||
| raw | -51.4 | $0.02 | |||
| raw | -52.4 |
Of runs with a cost, the best scores 21.8 for $0.02 a task
Runs on the cost frontier score higher than any run that's cheaper. Gemini 3.5 Flash-Lite at althing comes within 0.1 of the best for $0.0005 a task, 44 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official SynthBench leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| althing | 32.2 | ||||
| raw | 31.7 | ||||
| raw | 28.8 | ||||
| althing | 22.2 | ||||
| Claude Haiku 4.5 | althing, temperature 0.3 | 21.8 | $0.02 | ||
| althing | 21.7 | $0.0005 | |||
| Claude Haiku 4.5 | althing, structured template | 21.3 | |||
| Claude Sonnet 4 | althing | 19.8 | $0.06 | ||
| raw | 18.7 | $0.0003 | |||
| althing | 18.0 | ||||
| Claude Haiku 5.5 | althing, structured template, global-respondents personas | 16.4 | $0.006 | ||
| GPT-6 Luna | althing | 16.1 | |||
| GPT-6 Luna | althing, structured template, global-respondents personas | codex | 15.6 | ||
| Claude Haiku 5.5 | althing | 15.5 | $0.03 | ||
| GPT-6 Luna | althing | codex | 15.2 | ||
| raw | 14.9 | $0.0003 | |||
| althing, temperature 0.3 | 13.9 | $0.0004 | |||
| Claude Sonnet 4.6 | althing | 13.0 | $0.06 | ||
| althing | 12.5 | $0.0009 | |||
| GPT-4o Mar 2025 | raw | 12.5 | $0.009 | ||
| GPT-4o mini | althing, temperature 0.3 | 12.3 | $0.001 | ||
| Claude Haiku 5.5 | althing, structured template | 11.6 | $0.006 | ||
| GPT-6 Luna | althing, structured template | codex | 10.7 | ||
| GPT-4o mini | althing | 9.3 | $0.001 | ||
| Claude Haiku 4.5 | althing | 5.9 | $0.02 | ||
| raw | 4.2 | $0.0004 | |||
| Claude Sonnet 5.5 | althing | 3.4 | $0.10 | ||
| raw | 2.0 | $0.0009 | |||
| Claude Sonnet 4.6 | raw | 1.8 | $0.01 | ||
| althing | -0.4 | $0.0004 | |||
| GPT-4o mini | raw | -0.9 | $0.0004 | ||
| Claude Sonnet 5.5 | raw | -1.5 | $0.02 | ||
| GPT-6 Luna | raw | codex | -9.2 | ||
| Claude Haiku 4.5 | althing, temperature 0.85 | -13.4 | $0.004 | ||
| GPT-6 Luna | raw | -13.4 | |||
| Claude Haiku 4.5 | raw | -16.9 | $0.005 | ||
| Claude Haiku 5.5 | raw | -25.5 | $0.006 | ||
| Claude Sonnet 4 | raw | -31.7 | $0.02 | ||
| Claude Haiku 4.5 | althing, temperature 0.85, demo template | -37.8 | $0.004 | ||
| Claude Haiku 4.5 | althing, temperature 0.85, minimal template | -45.9 | $0.004 | ||
| Claude Haiku 4.5 | althing, temperature 0.85, values template | -50.4 | $0.004 | ||
| raw | -51.4 | $0.02 | |||
| raw | -52.4 |
2 models have held number one on the official leaderboard since its release in April 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official SynthBench leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official SynthBench leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Forecasting
- ForecastBench In use Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. GPT-5.5 0.11
- Prophet Arena In use Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. Claude Opus 5.5 86.2
- Guesswork (The Aggregate) Not in model cards Models predict benchmark scores before publication. The maker ranks forecasts by mean absolute error after scaling each error by the variation in that benchmark’s scores. GPT-6 Astra 0.56
- AGI-Pilled Bench Not in model cards Models forecast when AI will reach milestones that already happened in 2026. The score is the median years after 2026 across the forecast tasks. Claude Fable 5.2 6.50
- Dutch Book Bench Not in model cards Models assign probabilities to related events in separate conversations. The grader checks whether inconsistent probabilities let a bettor guarantee a profit.
Gemini 3.8 Flash 89.2%
- Echo General AI Prediction Leaderboard Not in model cards Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models.
DeepSeek-V4-Pro 1,018
Data audit
All 9 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access Public tasks, plus a private test set.
- Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
- Pass Independent runs Scores from the official leaderboard (21 entries).
- Pass Models scored 16 models we track, plus 5 other entries on its leaderboard.
- Pass Newest score The newest is from October 2026, for Claude Haiku 5.5.
- Note Frontier models Only 1 of the top 10 has a score. The strongest tested is Claude Sonnet 5.5 (#9).
- Note Headroom Scored as an index, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on synthbench.org: 43 rows for models we track and 5 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.