Benchmark Almanac

WeirdML v3

Claude Fable 5.2 43% leads since November 2026
Not in model cards

Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results.

Scores are not raw accuracy and are not comparable with WeirdML v2.

Released
September 2026
Kind
Machine learning engineering
Level
Expert
Scored
Checked automatically
Answers
Agent task
Strongest model tested
Claude Fable 5.2 #1 on the capability index
Last model added
November 2026
Scores refreshed
November 12th, 2026
27
Models scored on its own leaderboard

Claude Fable 5.2 leads with 43%

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

27 models. Scroll sideways for the rest.

  1. 43.0 Claude Fable 5.2
  2. 42.2 GPT-6 Astra
  3. 38.3 GPT-6.1 Sol
  4. 31.2 Claude Opus 5.5
  5. 28.1 GPT-6.1 Terra
  6. 26.0 Claude Fable 5.1
  7. 20.3 Gemini 4 Argon
  8. 19.9 Claude Sonnet 5.5
  9. 19.7 GPT-6 Sol
  10. 19.0 Claude Opus 5
  11. 15.6 Claude Fable 5
  12. 15.1 Muse Wonder
  13. 15.0 GPT-5.6 Sol
  14. 11.7 Claude Haiku 5.5
  15. 9.89 Kimi K3.1 Pro
  16. 9.10 DeepSeek V4.1 Flash 1017
  17. 8.45 Grok 4.7
  18. 7.87 GPT-6 Luna
  19. 7.54 Kimi K3
  20. 7.38 Gemini 3.8 Flash
  21. 6.66 Muse Spark 1.3
  22. 6.51 Claude Opus 4.5
  23. 5.95 DeepSeek V4.1 Flash
  24. 5.16 GPT-5.6 Luna
  25. 5.01 GPT-5.2
  26. 4.20 Mistral Large 4
  27. 3.90 Inkling
  1. 1 Claude Fable 5.2 xhigh 43%
  2. 2 GPT-6 Astra xhigh 42.2%
  3. 3 GPT-6.1 Sol xhigh 38.3%
  4. 4 Claude Opus 5.5 xhigh 31.2%
  5. 5 GPT-6.1 Terra xhigh 28.1%
  6. 6 Claude Fable 5.1 xhigh 26%
  7. 7 Gemini 4 Argon high 20.3%
  8. 8 Claude Sonnet 5.5 xhigh 19.9%
  9. 9 GPT-6 Sol xhigh 19.7%
  10. 10 Claude Opus 5 xhigh 19%
  11. 11 Claude Fable 5 xhigh 15.6%
  12. 12 Muse Wonder xhigh 15.1%
  13. 13 GPT-5.6 Sol xhigh 15%
  14. 14 Claude Haiku 5.5 xhigh 11.7%
  15. 15 Kimi K3.1 Pro xhigh 9.89%
  16. 16 DeepSeek V4.1 Flash 1017 high 9.10%
  17. 17 Grok 4.7 xhigh 8.45%
  18. 18 GPT-6 Luna xhigh 7.87%
  19. 19 Kimi K3 high 7.54%
  20. 20 Gemini 3.8 Flash high 7.38%
  21. 21 Muse Spark 1.3 xhigh 6.66%
  22. 22 Claude Opus 4.5 high 6.51%
  23. 23 DeepSeek V4.1 Flash high 5.95%
  24. 24 GPT-5.6 Luna xhigh 5.16%
  25. 25 GPT-5.2 xhigh 5.01%
  26. 26 Mistral Large 4 high 4.20%
  27. 27 Inkling xhigh 3.90%

Source: Official WeirdML v3 leaderboard

#ModelScoreSettingDateSource
1 Claude Fable 5.2 43% xhigh, claude_code Nov 2026 Official leaderboard
2 GPT-6 Astra 42.2% xhigh, codex_cli Sep 2026 Official leaderboard
3 GPT-6.1 Sol 38.3% xhigh, codex_cli Sep 2026 Official leaderboard
4 Claude Opus 5.5 31.2% xhigh, claude_code Sep 2026 Official leaderboard
5 GPT-6.1 Terra 28.1% xhigh, codex_cli Oct 2026 Official leaderboard
6 Claude Fable 5.1 26% xhigh, claude_code Sep 2026 Official leaderboard
7 Gemini 4 Argon 20.3% high, gemini_cli Sep 2026 Official leaderboard
8 Claude Sonnet 5.5 19.9% xhigh, claude_code Sep 2026 Official leaderboard
9 GPT-6 Sol 19.7% xhigh, codex_cli Sep 2026 Official leaderboard
10 Claude Opus 5 19% xhigh, claude_code Jul 2026 Official leaderboard
11 Claude Fable 5 15.6% xhigh, claude_code Jun 2026 Official leaderboard
12 Muse Wonder 15.1% xhigh, codex_cli Oct 2026 Official leaderboard
13 GPT-5.6 Sol 15% xhigh, codex_cli Jul 2026 Official leaderboard
14 Claude Haiku 5.5 11.7% xhigh, claude_code Oct 2026 Official leaderboard
15 Kimi K3.1 Pro 9.89% xhigh, kimi_code via Fireworks Oct 2026 Official leaderboard
16 DeepSeek V4.1 Flash 1017 9.10% high, codex_cli via InferenceNet Oct 2026 Official leaderboard
17 Grok 4.7 8.45% xhigh, codex_cli Sep 2026 Official leaderboard
18 GPT-6 Luna 7.87% xhigh, codex_cli Sep 2026 Official leaderboard
19 Kimi K3 7.54% high, kimi_code via Fireworks Jul 2026 Official leaderboard
20 Gemini 3.8 Flash 7.38% high, gemini_cli Sep 2026 Official leaderboard
21 Muse Spark 1.3 6.66% xhigh, codex_cli Sep 2026 Official leaderboard
22 Claude Opus 4.5 6.51% high, claude_code Nov 2025 Official leaderboard
23 DeepSeek V4.1 Flash 5.95% high, opencode via Novita Sep 2026 Official leaderboard
24 GPT-5.6 Luna 5.16% xhigh, codex_cli Jul 2026 Official leaderboard
25 GPT-5.2 5.01% xhigh, codex_cli Dec 2025 Official leaderboard
26 Mistral Large 4 4.20% high, opencode via Mistral Oct 2026 Official leaderboard
27 Inkling 3.90% xhigh, opencode via Together Jul 2026 Official leaderboard

The best score, 43%, costs $29 a task

Runs on the cost frontier score higher than any run that's cheaper. GPT-6.1 Sol at xhigh comes within 4.8 points of the best for $5.29 a task, 6 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Anthropic DeepSeek Cost frontier One model at each effort level
0% 10% 20% 30% 40% 50% $0.10 $1.00 $10 $100 Cost per task, log scale Claude Fable 5.2, xhigh GPT-6.1 Sol, xhigh GPT-6.1 Terra, xhigh Claude Haiku 5.5, xhigh DeepSeek V4.1 Flash 1017, high 0% 10% 20% 30% 40% 50% $0.10 $1.00 $10 $100 Cost per task, log scale Claude Fable 5.2, xhigh GPT-6.1 Sol, xhigh

Source: Official WeirdML v3 leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude Fable 5.2 xhigh claude_code 43% $29 352,618
GPT-6 Astra xhigh codex_cli 42.2% $28 153,241
GPT-6.1 Sol xhigh codex_cli 38.3% $5.29 215,529
Claude Opus 5.5 xhigh claude_code 31.2% $10 180,952
GPT-6.1 Terra xhigh codex_cli 28.1% $2.86 147,165
Claude Fable 5.1 xhigh claude_code 26% $32 251,090
Gemini 4 Argon high gemini_cli 20.3% $16 310,099
Claude Sonnet 5.5 xhigh claude_code 19.9% $9.28 278,720
GPT-6 Sol xhigh codex_cli 19.7% $6.91 143,614
Claude Opus 5 xhigh claude_code 19% $25 288,399
Claude Fable 5 xhigh claude_code 15.6% $53 300,212
Muse Wonder xhigh codex_cli 15.1% $11 169,458
GPT-5.6 Sol xhigh codex_cli 15% $6.28 116,059
Claude Haiku 5.5 xhigh claude_code 11.7% $2.30 315,447
Kimi K3.1 Pro xhigh kimi_code via Fireworks 9.89% $10 116,936
DeepSeek V4.1 Flash 1017 high codex_cli via InferenceNet 9.10% $0.88 827,719
Grok 4.7 xhigh codex_cli 8.45% $27 275,430
GPT-6 Luna xhigh codex_cli 7.87% $0.32 180,561
Kimi K3 high kimi_code via Fireworks 7.54% $20 209,415
Gemini 3.8 Flash high gemini_cli 7.38% $3.94 156,103
Muse Spark 1.3 xhigh codex_cli 6.66% $5.56 280,208
Claude Opus 4.5 high claude_code 6.51% $8.70 82,595
DeepSeek V4.1 Flash high opencode via Novita 5.95% $0.74 323,336
DeepSeek V4.1 Flash high codex_cli via Novita 5.83% $0.51 385,683
GPT-5.6 Luna xhigh codex_cli 5.16% $0.74 99,311
GPT-5.2 xhigh codex_cli 5.01% $9.29 264,947
Mistral Large 4 high opencode via Mistral 4.20% $28 1,109,580
Inkling xhigh opencode via Together 3.90% $1.33 19,902
Inkling xhigh opencode via DeepInfra 3.75% $0.45 18,515

3 models have held number one on the official leaderboard since its release in September 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI
Own leaderboard 3 leaders
Claude Fable 5.2 43%

Source: Official WeirdML v3 leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Highest possible score Claude Fable 5.2 Claude Fable 5.1 Claude Opus 5 Claude Opus 4.5 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Highest possible score Claude Fable 5.2 Claude Fable 5.1

Source: Official WeirdML v3 leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI Google DeepMind Anthropic
WeirdML v3 15 leaders
Claude Fable 5.2 43%
WeirdML Passed 85% in Jun 2026, by Claude Fable 5
GPT-6 Astra Pro 93.6%
WeirdML v3 3 leaders
Claude Fable 5.2 43%

Source: Official leaderboards

Replaces WeirdML

More in AI research

All 104 in AI research
  • PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. Qwen 3.8 Max 93%
  • MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
  • PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
  • MLE-bench In use Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. GPT-6 Astra 93.8%
  • KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. MiniMax-M3 28.8%
  • PMPP-Hard In use Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. Inkling 2 Small 494.3