Benchmark Almanac

ZeroBench

GPT-5.6 Sol 54.6% leads since July 2026
GPT-5.6 Sol 54.6% leads since July 2026
In use

100 hand-made visual questions that all frontier models originally failed. Scored by exact answer, pass@1 and pass@5.

Point releases keep scores comparable: v2 (2025-03) revised questions, v3 (2025-12) clarified two questions and three subquestions. A 2026-08 grading fix raised mean pass@5 by 0.79 points.

Released
February 2025
Kind
General visual questions
Level
University
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Strongest model tested
GPT-6 Astra #3 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026

In the authors’ words

Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench - a lightweight visual reasoning benchmark curated using adversarial filtering to be "impossible" for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass^5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass^5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io.

Abstract of ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (arXiv:2502.09696, CC0)
80
Entries on its own leaderboard, 76 models we track
26
Launch posts list it, from 7 labs
16
System cards report it, from 8 labs

GPT-5.6 Sol leads with 54.6% among models released by September 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

GPT-5.6 Sol leads with 54.6%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in September 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

80 entries. Scroll sideways for the rest.

  1. 54.6 GPT-5.6 Sol
  2. 54.0 Muse Spark 1.2
  3. 52.0 Claude Opus 5
  4. 52.0 GPT-6 Astra
  5. 49.0 Qwen 3.8 Max
  6. 49.0 DeepSeek V4.1 Flash
  7. 46.0 Claude Fable 5
  8. 46.0 Muse Spark 1.1
  9. 43.0 Claude Opus 5.5
  10. 41.0 GPT-5.4
  11. 41.0 GPT-5.5
  12. 41.0 Kimi K3
  13. 41.0 GPT-6 Sol
  14. 35.0 DeepSeek-V4-Flash-Vision-Exp
  15. 34.0 Claude Opus 4.8
  16. 33.0 Muse Spark
  17. 30.0 Gemini 3.7 Flash
  18. 29.0 Gemini 3.1 Pro
  19. 22.0 Gemini 3.5 Flash
  20. 21.0 GPT-5.6 Luna
  21. 20.0 Gemini 3 Pro
  22. 20.0 GPT-6 Luna
  23. 19.0 dots3-note Preview
  24. 19.0 GPT-5.6 Terra
  25. 18.0 Seed2.1 Turbo
  26. 18.0 GPT-5.2
  27. 18.0 Gemini 3.6 Flash
  28. 17.0 Grok 4.6
  29. 15.0 Gemini 3 Flash
  30. 15.0 Claude Opus 4.7
  31. 15.0 Grok 4.5
  32. 14.0 Gemini 3.5 Flash-Lite
  33. 13.0 Grok 4.3
  34. 13.0 Qwen3.7-Plus
  35. 13.0 Claude Sonnet 5
  36. 12.0 Claude Opus 4.6
  37. 12.0 Claude Sonnet 4.6
  38. 11.0 o4-mini
  39. 11.0 GPT-5.4 Mini
  40. 11.0 Inkling-Small
  41. 10.0 Grok 4.20
  42. 9.00 GPT-5 mini
  43. 9.00 Claude Opus 4.5
  44. 8.00 Claude Opus 4.1
  45. 7.00 GPT-5
  46. 6.00 Claude 3.7 Sonnet
  47. 6.00 GPT-4.5
  48. 6.00 o3
  49. 6.00 Claude Opus 4
  50. 6.00 Claude Sonnet 4
  51. 6.00 Grok 4
  52. 6.00 GPT-5.1
  53. 5.00 Gemini 2.0 Flash Thinking
  54. 5.00 Gemini 2.5 Pro
  55. 5.00 Claude Sonnet 4.5
  56. 5.00 Gemini 3.1 Flash-Lite
  57. 4.00 QVQ
  58. 3.00 Claude 3.5 Sonnet (October 2024)
  59. 3.00 Pixtral Large
  60. 3.00 Gemini 2.0 Flash
  61. 3.00 Llama 4 Scout
  62. 3.00 GPT-4.1
  63. 3.00 GPT-5 nano
  64. 3.00 Gemini 2.5 Flash
  65. 2.00 Gemini 1.0 Pro Vision
  66. 2.00 Claude 3.5 Sonnet
  67. 2.00 GPT-4o mini
  68. 2.00 Gemini 1.5 Flash
  69. 2.00 Qwen2-VL-72B-Instruct
  70. 2.00 Gemini 1.5 Pro
  71. 2.00 Llama 4 Maverick
  72. 1.00 Claude 3 Sonnet
  73. 1.00 NVLM-D-72B
  74. 1.00 GPT-4o
  75. 0.0 OpenAI o1-mini
  76. 0.0 Claude 3 Opus
  77. 0.0 Claude 3 Haiku
  78. 0.0 Llama 3.2 90B
  79. 0.0 o1
  80. 0.0 Reka Edge
  1. 1 GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
  2. 2 Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
  3. 3 Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
  4. 4 GPT-6 Astra max 52%
  5. 5 Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
  6. 6 DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
  7. 7 Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
  8. 8 Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
  9. 9 Claude Opus 5.5 max 43%
  10. 10 GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
  11. 11 GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  12. 12 Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  13. 13 GPT-6 Sol max 41%
  14. 14 DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
  15. 15 Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
  16. 16 Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
  17. 17 Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
  18. 18 Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
  19. 19 Gemini 3.5 Flash 22%
  20. 20 GPT-5.6 Luna max 21%
  21. 21 Gemini 3 Pro 20%
  22. 22 GPT-6 Luna max 20%
  23. 23 dots3-note Preview externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge; values from card image. 19%
  24. 24 GPT-5.6 Terra max 19%
  25. 25 Seed2.1 Turbo externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 18%
  26. 26 GPT-5.2 medium 18%
  27. 27 Gemini 3.6 Flash high 18%
  28. 28 Grok 4.6 xhigh 17%
  29. 29 Gemini 3 Flash 15%
  30. 30 Claude Opus 4.7 xhigh 15%
  31. 31 Grok 4.5 high 15%
  32. 32 Gemini 3.5 Flash-Lite high 14%
  33. 33 Grok 4.3 high 13%
  34. 34 Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
  35. 35 Claude Sonnet 5 max 13%
  36. 36 Claude Opus 4.6 12%
  37. 37 Claude Sonnet 4.6 12%
  38. 38 o4-mini 11%
  39. 39 GPT-5.4 Mini 11%
  40. 40 Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
  41. 41 Grok 4.20 reasoning 10%
  42. 42 GPT-5 mini high 9.00%
  43. 43 Claude Opus 4.5 9.00%
  44. 44 Claude Opus 4.1 thinking 8.00%
  45. 45 GPT-5 medium 7.00%
  46. 46 Claude 3.7 Sonnet thinking 6.00%
  47. 47 GPT-4.5 6.00%
  48. 48 o3 6.00%
  49. 49 Claude Opus 4 thinking 6.00%
  50. 50 Claude Sonnet 4 6.00%
  51. 51 Grok 4 6.00%
  52. 52 GPT-5.1 medium 6.00%
  53. 53 Gemini 2.0 Flash Thinking Jan 2025 5.00%
  54. 54 Gemini 2.5 Pro Jun 2025 5.00%
  55. 55 Claude Sonnet 4.5 5.00%
  56. 56 Gemini 3.1 Flash-Lite 5.00%
  57. 57 QVQ original release evaluation 4.00%
  58. 58 Claude 3.5 Sonnet (October 2024) 3.00%
  59. 59 Pixtral Large 3.00%
  60. 60 Gemini 2.0 Flash Feb 2025 3.00%
  61. 61 Llama 4 Scout 3.00%
  62. 62 GPT-4.1 3.00%
  63. 63 GPT-5 nano high 3.00%
  64. 64 Gemini 2.5 Flash Sep 2025 3.00%
  65. 65 Gemini 1.0 Pro Vision 2.00%
  66. 66 Claude 3.5 Sonnet 2.00%
  67. 67 GPT-4o mini 2.00%
  68. 68 Gemini 1.5 Flash Sep 2024 2.00%
  69. 69 Qwen2-VL-72B-Instruct original release evaluation 2.00%
  70. 70 Gemini 1.5 Pro Sep 2024 2.00%
  71. 71 Llama 4 Maverick 2.00%
  72. 72 Claude 3 Sonnet 1.00%
  73. 73 NVLM-D-72B 1.00%
  74. 74 GPT-4o Mar 2025, externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
  75. 75 OpenAI o1-mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 0%
  76. 76 Claude 3 Opus 0%
  77. 77 Claude 3 Haiku 0%
  78. 78 Llama 3.2 90B 0%
  79. 79 o1 0%
  80. 80 Reka Edge 0%

Source: Official ZeroBench leaderboard

101 models. Scroll sideways for the rest.

  1. 54.6 GPT-5.6 Sol
  2. 54.0 Muse Spark 1.2
  3. 52.0 Claude Opus 5
  4. 52.0 GPT-6 Astra
  5. 51.0 DeepSeek V4.1 Flash 1017
  6. 50.0 Muse Spark Open
  7. 49.0 Qwen 3.8 Max
  8. 49.0 DeepSeek V4.1 Flash
  9. 46.0 Claude Fable 5
  10. 46.0 Muse Spark 1.1
  11. 43.0 Claude Opus 5.5
  12. 41.0 GPT-5.4
  13. 41.0 GPT-5.5
  14. 41.0 Kimi K3
  15. 41.0 GPT-6 Sol
  16. 39.0 Kimi K3.1 Pro
  17. 36.2 Qwen3.5-122B-A10B
  18. 36.2 Qwen3.5-27B
  19. 35.0 DeepSeek-V4-Flash-Vision-Exp
  20. 34.4 Qwen3.5-Omni-Plus
  21. 34.4 Qwen 3.6 35B-A3B
  22. 34.1 Qwen3.5-35B-A3B
  23. 34.0 Claude Opus 4.8
  24. 33.0 Muse Spark
  25. 31.1 Qwen3.5-9B
  26. 30.0 Gemini 3.7 Flash
  27. 29.0 Gemini 3.1 Pro
  28. 27.7 Qwen3-VL-235B-A22B
  29. 26.3 Qwen3.5-4B
  30. 25.8 GLM-4.6V
  31. 23.4 GLM-4.5V
  32. 23.0 Step 3
  33. 22.0 Gemini 3.5 Flash
  34. 22.0 Seed2.1 Pro
  35. 21.0 GPT-5.6 Luna
  36. 20.0 Gemini 3 Pro
  37. 20.0 GPT-6 Luna
  38. 19.2 GLM-4.1V-9B-Thinking
  39. 19.0 GPT-5.6 Terra
  40. 18.3 Nemotron Nano 12B v2 VL
  41. 18.0 GPT-5.2
  42. 18.0 Gemini 3.6 Flash
  43. 17.1 Qwen3.5-2B
  44. 17.0 Grok 4.6
  45. 15.0 Gemini 3 Flash
  46. 15.0 Claude Opus 4.7
  47. 15.0 Grok 4.5
  48. 14.0 Gemini 3.5 Flash-Lite
  49. 13.0 Grok 4.3
  50. 13.0 Qwen3.7-Plus
  51. 13.0 Claude Sonnet 5
  52. 12.9 Qwen3.5-0.8B
  53. 12.0 Claude Opus 4.6
  54. 12.0 Seed2.0 Pro
  55. 12.0 Claude Sonnet 4.6
  56. 11.0 o4-mini
  57. 11.0 Seed1.8
  58. 11.0 GPT-5.4 Mini
  59. 11.0 Inkling-Small
  60. 10.0 Grok 4.20
  61. 9.00 GPT-5 mini
  62. 9.00 Claude Opus 4.5
  63. 9.00 Kimi K2.5
  64. 8.00 Claude Opus 4.1
  65. 7.00 GPT-5
  66. 6.00 Claude 3.7 Sonnet
  67. 6.00 GPT-4.5
  68. 6.00 o3
  69. 6.00 Claude Opus 4
  70. 6.00 Claude Sonnet 4
  71. 6.00 Grok 4
  72. 6.00 GPT-5.1
  73. 5.00 Gemini 2.0 Flash Thinking
  74. 5.00 Gemini 2.5 Pro
  75. 5.00 Claude Sonnet 4.5
  76. 5.00 Gemini 3.1 Flash-Lite
  77. 3.00 Claude 3.5 Sonnet (October 2024)
  78. 3.00 Pixtral Large
  79. 3.00 Gemini 2.0 Flash
  80. 3.00 Llama 4 Scout
  81. 3.00 GPT-4.1
  82. 3.00 GPT-5 nano
  83. 3.00 Gemini 2.5 Flash
  84. 2.00 Gemini 1.0 Pro Vision
  85. 2.00 Claude 3.5 Sonnet
  86. 2.00 GPT-4o mini
  87. 2.00 Gemini 1.5 Flash
  88. 2.00 Qwen2-VL-72B-Instruct
  89. 2.00 Gemini 1.5 Pro
  90. 2.00 Llama 4 Maverick
  91. 2.00 Seed1.5-VL
  92. 2.00 Gemini 2.5 Flash
  93. 1.00 Claude 3 Sonnet
  94. 1.00 NVLM-D-72B
  95. 1.00 GPT-4o
  96. 1.00 STEP3-VL-10B
  97. 0.0 Claude 3 Opus
  98. 0.0 Claude 3 Haiku
  99. 0.0 Llama 3.2 90B
  100. 0.0 o1
  101. 0.0 Reka Edge
  1. 1 GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
  2. 2 Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
  3. 3 Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
  4. 4 GPT-6 Astra max 52%
  5. 5 DeepSeek V4.1 Flash 1017 51%
  6. 6 Muse Spark Open 50%
  7. 7 Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
  8. 8 DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
  9. 9 Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
  10. 10 Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
  11. 11 Claude Opus 5.5 max 43%
  12. 12 GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
  13. 13 GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  14. 14 Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  15. 15 GPT-6 Sol max 41%
  16. 16 Kimi K3.1 Pro 39%
  17. 17 Qwen3.5-122B-A10B 36.2%
  18. 18 Qwen3.5-27B 36.2%
  19. 19 DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
  20. 20 Qwen3.5-Omni-Plus 34.4%
  21. 21 Qwen 3.6 35B-A3B 34.4%
  22. 22 Qwen3.5-35B-A3B 34.1%
  23. 23 Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
  24. 24 Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
  25. 25 Qwen3.5-9B 31.1%
  26. 26 Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
  27. 27 Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
  28. 28 Qwen3-VL-235B-A22B 27.7%
  29. 29 Qwen3.5-4B 26.3%
  30. 30 GLM-4.6V 25.8%
  31. 31 GLM-4.5V 23.4%
  32. 32 Step 3 23%
  33. 33 Gemini 3.5 Flash 22%
  34. 34 Seed2.1 Pro 22%
  35. 35 GPT-5.6 Luna max 21%
  36. 36 Gemini 3 Pro 20%
  37. 37 GPT-6 Luna max 20%
  38. 38 GLM-4.1V-9B-Thinking 19.2%
  39. 39 GPT-5.6 Terra max 19%
  40. 40 Nemotron Nano 12B v2 VL 18.3%
  41. 41 GPT-5.2 medium 18%
  42. 42 Gemini 3.6 Flash high 18%
  43. 43 Qwen3.5-2B 17.1%
  44. 44 Grok 4.6 xhigh 17%
  45. 45 Gemini 3 Flash 15%
  46. 46 Claude Opus 4.7 xhigh 15%
  47. 47 Grok 4.5 high 15%
  48. 48 Gemini 3.5 Flash-Lite high 14%
  49. 49 Grok 4.3 high 13%
  50. 50 Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
  51. 51 Claude Sonnet 5 max 13%
  52. 52 Qwen3.5-0.8B 12.9%
  53. 53 Claude Opus 4.6 12%
  54. 54 Seed2.0 Pro 12%
  55. 55 Claude Sonnet 4.6 12%
  56. 56 o4-mini 11%
  57. 57 Seed1.8 11%
  58. 58 GPT-5.4 Mini 11%
  59. 59 Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
  60. 60 Grok 4.20 reasoning 10%
  61. 61 GPT-5 mini high 9.00%
  62. 62 Claude Opus 4.5 9.00%
  63. 63 Kimi K2.5 9.00%
  64. 64 Claude Opus 4.1 thinking 8.00%
  65. 65 GPT-5 medium 7.00%
  66. 66 Claude 3.7 Sonnet thinking 6.00%
  67. 67 GPT-4.5 6.00%
  68. 68 o3 6.00%
  69. 69 Claude Opus 4 thinking 6.00%
  70. 70 Claude Sonnet 4 6.00%
  71. 71 Grok 4 6.00%
  72. 72 GPT-5.1 medium 6.00%
  73. 73 Gemini 2.0 Flash Thinking Jan 2025 5.00%
  74. 74 Gemini 2.5 Pro Jun 2025 5.00%
  75. 75 Claude Sonnet 4.5 5.00%
  76. 76 Gemini 3.1 Flash-Lite 5.00%
  77. 77 Claude 3.5 Sonnet (October 2024) 3.00%
  78. 78 Pixtral Large 3.00%
  79. 79 Gemini 2.0 Flash Feb 2025 3.00%
  80. 80 Llama 4 Scout 3.00%
  81. 81 GPT-4.1 3.00%
  82. 82 GPT-5 nano high 3.00%
  83. 83 Gemini 2.5 Flash Sep 2025 3.00%
  84. 84 Gemini 1.0 Pro Vision 2.00%
  85. 85 Claude 3.5 Sonnet 2.00%
  86. 86 GPT-4o mini 2.00%
  87. 87 Gemini 1.5 Flash Sep 2024 2.00%
  88. 88 Qwen2-VL-72B-Instruct original release evaluation 2.00%
  89. 89 Gemini 1.5 Pro Sep 2024 2.00%
  90. 90 Llama 4 Maverick 2.00%
  91. 91 Seed1.5-VL 2.00%
  92. 92 Gemini 2.5 Flash Jun 2025 2.00%
  93. 93 Claude 3 Sonnet 1.00%
  94. 94 NVLM-D-72B 1.00%
  95. 95 GPT-4o Mar 2025, externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
  96. 96 STEP3-VL-10B 1.00%
  97. 97 Claude 3 Opus 0%
  98. 98 Claude 3 Haiku 0%
  99. 99 Llama 3.2 90B 0%
  100. 100 o1 0%
  101. 101 Reka Edge 0%

Sources: Official ZeroBench leaderboard and labs’ model cards

#EntryScoreSettingDateSource
1 GPT-5.6 Sol 54.6% max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
2 Muse Spark 1.2 54% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Aug 2026 Official leaderboard
3 Claude Opus 5 52% max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. Jul 2026 Official leaderboard
4 GPT-6 Astra 52% max Sep 2026 Official leaderboard
5 Qwen 3.8 Max 49% w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. Aug 2026 Official leaderboard
6 DeepSeek V4.1 Flash 49% w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. Sep 2026 Official leaderboard
7 Claude Fable 5 46% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jun 2026 Official leaderboard
8 Muse Spark 1.1 46% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
9 Claude Opus 5.5 43% max Sep 2026 Official leaderboard
10 GPT-5.4 41% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Mar 2026 Official leaderboard
11 GPT-5.5 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Apr 2026 Official leaderboard
12 Kimi K3 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jul 2026 Official leaderboard
13 GPT-6 Sol 41% max Sep 2026 Official leaderboard
14 DeepSeek-V4-Flash-Vision-Exp 35% externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. Aug 2026 Official leaderboard
15 Claude Opus 4.8 34% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. May 2026 Official leaderboard
16 Muse Spark 33% w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. Apr 2026 Official leaderboard
17 Gemini 3.7 Flash 30% high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. Aug 2026 Official leaderboard
18 Gemini 3.1 Pro 29% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Feb 2026 Official leaderboard
19 Gemini 3.5 Flash 22% May 2026 Official leaderboard
20 GPT-5.6 Luna 21% max Jul 2026 Official leaderboard
21 Gemini 3 Pro 20% Nov 2025 Official leaderboard
22 GPT-6 Luna 20% max Sep 2026 Official leaderboard
23 dots3-note Preview 19% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge; values from card image. Official leaderboard
24 GPT-5.6 Terra 19% max Jul 2026 Official leaderboard
25 Seed2.1 Turbo 18% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Official leaderboard
26 GPT-5.2 18% medium Dec 2025 Official leaderboard
27 Gemini 3.6 Flash 18% high Jul 2026 Official leaderboard
28 Grok 4.6 17% xhigh Aug 2026 Official leaderboard
29 Gemini 3 Flash 15% Dec 2025 Official leaderboard
30 Claude Opus 4.7 15% xhigh Apr 2026 Official leaderboard
31 Grok 4.5 15% high Jul 2026 Official leaderboard
32 Gemini 3.5 Flash-Lite 14% high Jul 2026 Official leaderboard
33 Grok 4.3 13% high Apr 2026 Official leaderboard
34 Qwen3.7-Plus 13% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jun 2026 Official leaderboard
35 Claude Sonnet 5 13% max Jun 2026 Official leaderboard
36 Claude Opus 4.6 12% Feb 2026 Official leaderboard
37 Claude Sonnet 4.6 12% Feb 2026 Official leaderboard
38 o4-mini 11% Apr 2025 Official leaderboard
39 GPT-5.4 Mini 11% Mar 2026 Official leaderboard
40 Inkling-Small 11% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jul 2026 Official leaderboard
41 Grok 4.20 10% reasoning Feb 2026 Official leaderboard
42 GPT-5 mini 9.00% high Aug 2025 Official leaderboard
43 Claude Opus 4.5 9.00% Nov 2025 Official leaderboard
44 Claude Opus 4.1 8.00% thinking Aug 2025 Official leaderboard
45 GPT-5 7.00% medium Aug 2025 Official leaderboard
46 Claude 3.7 Sonnet 6.00% thinking Feb 2025 Official leaderboard
47 GPT-4.5 6.00% Feb 2025 Official leaderboard
48 o3 6.00% Apr 2025 Official leaderboard
49 Claude Opus 4 6.00% thinking May 2025 Official leaderboard
50 Claude Sonnet 4 6.00% May 2025 Official leaderboard
51 Grok 4 6.00% Jul 2025 Official leaderboard
52 GPT-5.1 6.00% medium Nov 2025 Official leaderboard
53 Gemini 2.0 Flash Thinking Jan 2025 5.00% Jan 2025 Official leaderboard
54 Gemini 2.5 Pro Jun 2025 5.00% Jun 2025 Official leaderboard
55 Claude Sonnet 4.5 5.00% Sep 2025 Official leaderboard
56 Gemini 3.1 Flash-Lite 5.00% Mar 2026 Official leaderboard
57 QVQ 4.00% original release evaluation Official leaderboard
58 Claude 3.5 Sonnet (October 2024) 3.00% Oct 2024 Official leaderboard
59 Pixtral Large 3.00% Nov 2024 Official leaderboard
60 Gemini 2.0 Flash Feb 2025 3.00% Feb 2025 Official leaderboard
61 Llama 4 Scout 3.00% Apr 2025 Official leaderboard
62 GPT-4.1 3.00% Apr 2025 Official leaderboard
63 GPT-5 nano 3.00% high Aug 2025 Official leaderboard
64 Gemini 2.5 Flash Sep 2025 3.00% Sep 2025 Official leaderboard
65 Gemini 1.0 Pro Vision 2.00% Jan 2024 Official leaderboard
66 Claude 3.5 Sonnet 2.00% Jun 2024 Official leaderboard
67 GPT-4o mini 2.00% Jul 2024 Official leaderboard
68 Gemini 1.5 Flash Sep 2024 2.00% Aug 2024 Official leaderboard
69 Qwen2-VL-72B-Instruct 2.00% original release evaluation Aug 2024 Official leaderboard
70 Gemini 1.5 Pro Sep 2024 2.00% Sep 2024 Official leaderboard
71 Llama 4 Maverick 2.00% Apr 2025 Official leaderboard
72 Claude 3 Sonnet 1.00% Feb 2024 Official leaderboard
73 NVLM-D-72B 1.00% Sep 2024 Official leaderboard
74 GPT-4o Mar 2025 1.00% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Mar 2025 Official leaderboard
75 OpenAI o1-mini 0% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Official leaderboard
76 Claude 3 Opus 0% Feb 2024 Official leaderboard
77 Claude 3 Haiku 0% Mar 2024 Official leaderboard
78 Llama 3.2 90B 0% Sep 2024 Official leaderboard
79 o1 0% Dec 2024 Official leaderboard
80 Reka Edge 0% Mar 2026 Official leaderboard
#ModelScoreSettingDateSource
1 GPT-5.6 Sol 54.6% max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
2 Muse Spark 1.2 54% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Aug 2026 Official leaderboard
3 Claude Opus 5 52% max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. Jul 2026 Official leaderboard
4 GPT-6 Astra 52% max Sep 2026 Official leaderboard
5 DeepSeek V4.1 Flash 1017 51% Oct 2026 Lab model card
6 Muse Spark Open 50% Oct 2026 Lab model card
7 Qwen 3.8 Max 49% w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. Aug 2026 Official leaderboard
8 DeepSeek V4.1 Flash 49% w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. Sep 2026 Official leaderboard
9 Claude Fable 5 46% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jun 2026 Official leaderboard
10 Muse Spark 1.1 46% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
11 Claude Opus 5.5 43% max Sep 2026 Official leaderboard
12 GPT-5.4 41% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Mar 2026 Official leaderboard
13 GPT-5.5 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Apr 2026 Official leaderboard
14 Kimi K3 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jul 2026 Official leaderboard
15 GPT-6 Sol 41% max Sep 2026 Official leaderboard
16 Kimi K3.1 Pro 39% Oct 2026 Lab model card
17 Qwen3.5-122B-A10B 36.2% Feb 2026 Lab model card
18 Qwen3.5-27B 36.2% Feb 2026 Lab model card
19 DeepSeek-V4-Flash-Vision-Exp 35% externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. Aug 2026 Official leaderboard
20 Qwen3.5-Omni-Plus 34.4% Mar 2026 Lab model card
21 Qwen 3.6 35B-A3B 34.4% Apr 2026 Lab model card
22 Qwen3.5-35B-A3B 34.1% Feb 2026 Lab model card
23 Claude Opus 4.8 34% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. May 2026 Official leaderboard
24 Muse Spark 33% w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. Apr 2026 Official leaderboard
25 Qwen3.5-9B 31.1% Mar 2026 Lab model card
26 Gemini 3.7 Flash 30% high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. Aug 2026 Official leaderboard
27 Gemini 3.1 Pro 29% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Feb 2026 Official leaderboard
28 Qwen3-VL-235B-A22B 27.7% Sep 2025 Lab model card
29 Qwen3.5-4B 26.3% Mar 2026 Lab model card
30 GLM-4.6V 25.8% Dec 2025 Lab model card
31 GLM-4.5V 23.4% Aug 2025 Lab model card
32 Step 3 23% Jul 2025 Lab model card
33 Gemini 3.5 Flash 22% May 2026 Official leaderboard
34 Seed2.1 Pro 22% Jun 2026 Lab model card
35 GPT-5.6 Luna 21% max Jul 2026 Official leaderboard
36 Gemini 3 Pro 20% Nov 2025 Official leaderboard
37 GPT-6 Luna 20% max Sep 2026 Official leaderboard
38 GLM-4.1V-9B-Thinking 19.2% Jul 2025 Lab model card
39 GPT-5.6 Terra 19% max Jul 2026 Official leaderboard
40 Nemotron Nano 12B v2 VL 18.3% Nov 2025 Lab model card
41 GPT-5.2 18% medium Dec 2025 Official leaderboard
42 Gemini 3.6 Flash 18% high Jul 2026 Official leaderboard
43 Qwen3.5-2B 17.1% Mar 2026 Lab model card
44 Grok 4.6 17% xhigh Aug 2026 Official leaderboard
45 Gemini 3 Flash 15% Dec 2025 Official leaderboard
46 Claude Opus 4.7 15% xhigh Apr 2026 Official leaderboard
47 Grok 4.5 15% high Jul 2026 Official leaderboard
48 Gemini 3.5 Flash-Lite 14% high Jul 2026 Official leaderboard
49 Grok 4.3 13% high Apr 2026 Official leaderboard
50 Qwen3.7-Plus 13% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jun 2026 Official leaderboard
51 Claude Sonnet 5 13% max Jun 2026 Official leaderboard
52 Qwen3.5-0.8B 12.9% Mar 2026 Lab model card
53 Claude Opus 4.6 12% Feb 2026 Official leaderboard
54 Seed2.0 Pro 12% Feb 2026 Lab model card
55 Claude Sonnet 4.6 12% Feb 2026 Official leaderboard
56 o4-mini 11% Apr 2025 Official leaderboard
57 Seed1.8 11% Dec 2025 Lab model card
58 GPT-5.4 Mini 11% Mar 2026 Official leaderboard
59 Inkling-Small 11% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jul 2026 Official leaderboard
60 Grok 4.20 10% reasoning Feb 2026 Official leaderboard
61 GPT-5 mini 9.00% high Aug 2025 Official leaderboard
62 Claude Opus 4.5 9.00% Nov 2025 Official leaderboard
63 Kimi K2.5 9.00% Jan 2026 Lab model card
64 Claude Opus 4.1 8.00% thinking Aug 2025 Official leaderboard
65 GPT-5 7.00% medium Aug 2025 Official leaderboard
66 Claude 3.7 Sonnet 6.00% thinking Feb 2025 Official leaderboard
67 GPT-4.5 6.00% Feb 2025 Official leaderboard
68 o3 6.00% Apr 2025 Official leaderboard
69 Claude Opus 4 6.00% thinking May 2025 Official leaderboard
70 Claude Sonnet 4 6.00% May 2025 Official leaderboard
71 Grok 4 6.00% Jul 2025 Official leaderboard
72 GPT-5.1 6.00% medium Nov 2025 Official leaderboard
73 Gemini 2.0 Flash Thinking Jan 2025 5.00% Jan 2025 Official leaderboard
74 Gemini 2.5 Pro Jun 2025 5.00% Jun 2025 Official leaderboard
75 Claude Sonnet 4.5 5.00% Sep 2025 Official leaderboard
76 Gemini 3.1 Flash-Lite 5.00% Mar 2026 Official leaderboard
77 Claude 3.5 Sonnet (October 2024) 3.00% Oct 2024 Official leaderboard
78 Pixtral Large 3.00% Nov 2024 Official leaderboard
79 Gemini 2.0 Flash Feb 2025 3.00% Feb 2025 Official leaderboard
80 Llama 4 Scout 3.00% Apr 2025 Official leaderboard
81 GPT-4.1 3.00% Apr 2025 Official leaderboard
82 GPT-5 nano 3.00% high Aug 2025 Official leaderboard
83 Gemini 2.5 Flash Sep 2025 3.00% Sep 2025 Official leaderboard
84 Gemini 1.0 Pro Vision 2.00% Jan 2024 Official leaderboard
85 Claude 3.5 Sonnet 2.00% Jun 2024 Official leaderboard
86 GPT-4o mini 2.00% Jul 2024 Official leaderboard
87 Gemini 1.5 Flash Sep 2024 2.00% Aug 2024 Official leaderboard
88 Qwen2-VL-72B-Instruct 2.00% original release evaluation Aug 2024 Official leaderboard
89 Gemini 1.5 Pro Sep 2024 2.00% Sep 2024 Official leaderboard
90 Llama 4 Maverick 2.00% Apr 2025 Official leaderboard
91 Seed1.5-VL 2.00% May 2025 Lab model card
92 Gemini 2.5 Flash Jun 2025 2.00% Jul 2025 Lab model card
93 Claude 3 Sonnet 1.00% Feb 2024 Official leaderboard
94 NVLM-D-72B 1.00% Sep 2024 Official leaderboard
95 GPT-4o Mar 2025 1.00% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Mar 2025 Official leaderboard
96 STEP3-VL-10B 1.00% Jan 2026 Lab model card
97 Claude 3 Opus 0% Feb 2024 Official leaderboard
98 Claude 3 Haiku 0% Mar 2024 Official leaderboard
99 Llama 3.2 90B 0% Sep 2024 Official leaderboard
100 o1 0% Dec 2024 Official leaderboard
101 Reka Edge 0% Mar 2026 Official leaderboard

Labs report scores for 31 models in their cards. Switch the scores above to rank them together.

Results plot: tokens per question

Official runs only: pass@5 on the 100 main questions against the mean total (input and output) tokens per question, with the Pareto frontier. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 1001,00010K100K Mean total tokens per question (tokens), log scale, lower is better pass@5 Gemini 1.0 Pro Vision (original release evaluation): pass@5 2%, mean total tokens per question 610 tokens Claude 3 Opus: pass@5 0%, mean total tokens per question 1,915 tokens Claude 3 Opus (original release evaluation): pass@5 0%, mean total tokens per question 1,915 tokens Claude 3 Sonnet: pass@5 1%, mean total tokens per question 1,920 tokens Claude 3 Sonnet (original release evaluation): pass@5 1%, mean total tokens per question 1,920 tokens Claude 3 Haiku: pass@5 0%, mean total tokens per question 1,919 tokens Claude 3 Haiku (original release evaluation): pass@5 0%, mean total tokens per question 1,919 tokens Claude 3.5 Sonnet: pass@5 2%, mean total tokens per question 1,940 tokens Claude 3.5 Sonnet (original release evaluation): pass@5 1%, mean total tokens per question 1,940 tokens GPT-4o mini: pass@5 2%, mean total tokens per question 1,546 tokens GPT-4o mini (original release evaluation): pass@5 2%, mean total tokens per question 1,546 tokens Gemini 1.5 Flash: pass@5 2%, mean total tokens per question 728 tokens Gemini 1.5 Flash (original release evaluation): pass@5 2%, mean total tokens per question 728 tokens Qwen2-VL-72B-Instruct: pass@5 1%, mean total tokens per question 3,313 tokens Qwen2-VL-72B-Instruct (original release evaluation): pass@5 2%, mean total tokens per question 3,313 tokens Gemini 1.5 Pro: pass@5 2%, mean total tokens per question 707 tokens Gemini 1.5 Pro (original release evaluation): pass@5 2%, mean total tokens per question 707 tokens Llama 3.2 90B: pass@5 0%, mean total tokens per question 989 tokens Llama 3.2 90B (original release evaluation): pass@5 0%, mean total tokens per question 989 tokens NVLM-D-72B: pass@5 1%, mean total tokens per question 3,728 tokens NVLM-D-72B (original release evaluation): pass@5 1%, mean total tokens per question 3,728 tokens Claude 3.5 Sonnet (October 2024): pass@5 3%, mean total tokens per question 1,905 tokens Claude 3.5 Sonnet (October 2024) (original release evaluation): pass@5 2%, mean total tokens per question 1,905 tokens Pixtral Large: pass@5 3%, mean total tokens per question 4,107 tokens Pixtral Large (original release evaluation): pass@5 3%, mean total tokens per question 4,107 tokens o1: pass@5 0%, mean total tokens per question 8,206 tokens o1 (original release evaluation): pass@5 0%, mean total tokens per question 8,206 tokens Gemini 2.0 Flash Thinking: pass@5 5%, mean total tokens per question 2,919 tokens Gemini 2.0 Flash Thinking (original release evaluation): pass@5 5%, mean total tokens per question 2,919 tokens Gemini 2.0 Flash: pass@5 3%, mean total tokens per question 2,798 tokens Gemini 2.0 Flash (original release evaluation): pass@5 3%, mean total tokens per question 2,798 tokens Claude 3.7 Sonnet: pass@5 2%, mean total tokens per question 1,981 tokens Claude 3.7 Sonnet (thinking): pass@5 6%, mean total tokens per question 14K tokens GPT-4o: pass@5 0%, mean total tokens per question 1,582 tokens GPT-4o (original release evaluation): pass@5 0%, mean total tokens per question 1,582 tokens Llama 4 Maverick: pass@5 2%, mean total tokens per question 985 tokens Llama 4 Scout: pass@5 3%, mean total tokens per question 3,150 tokens GPT-4.1: pass@5 3%, mean total tokens per question 1,650 tokens o3: pass@5 6%, mean total tokens per question 6,454 tokens o4-mini: pass@5 11%, mean total tokens per question 18K tokens Claude Opus 4: pass@5 4%, mean total tokens per question 2,181 tokens Claude Opus 4 (thinking): pass@5 6%, mean total tokens per question 7,486 tokens Claude Sonnet 4: pass@5 6%, mean total tokens per question 2,179 tokens Claude Sonnet 4 (thinking): pass@5 6%, mean total tokens per question 13K tokens Grok 4: pass@5 6%, mean total tokens per question 22K tokens Claude Opus 4.1: pass@5 5%, mean total tokens per question 2,222 tokens Claude Opus 4.1 (thinking): pass@5 8%, mean total tokens per question 8,001 tokens GPT-5 (medium): pass@5 7%, mean total tokens per question 6,040 tokens GPT-5 mini (high): pass@5 9%, mean total tokens per question 12K tokens GPT-5 nano (high): pass@5 3%, mean total tokens per question 10K tokens Gemini 2.5 Flash: pass@5 3%, mean total tokens per question 3,226 tokens Claude Sonnet 4.5 (thinking): pass@5 3%, mean total tokens per question 15K tokens Claude Sonnet 4.5: pass@5 5%, mean total tokens per question 2,321 tokens GPT-5.1 (medium): pass@5 6%, mean total tokens per question 6,206 tokens Gemini 3 Pro: pass@5 20%, mean total tokens per question 20K tokens Claude Opus 4.5: pass@5 9%, mean total tokens per question 20K tokens Gemini 3 Flash: pass@5 15%, mean total tokens per question 24K tokens Claude Opus 4.6: pass@5 12%, mean total tokens per question 30K tokens Claude Sonnet 4.6: pass@5 12%, mean total tokens per question 25K tokens Grok 4.20 (reasoning): pass@5 10%, mean total tokens per question 20K tokens Grok 4.20 (non-reasoning): pass@5 10%, mean total tokens per question 13K tokens Gemini 3.1 Pro: pass@5 19%, mean total tokens per question 22K tokens Gemini 3.1 Flash-Lite: pass@5 5%, mean total tokens per question 14K tokens GPT-5.4 (xhigh): pass@5 23%, mean total tokens per question 22K tokens Reka Edge (original release evaluation): pass@5 0%, mean total tokens per question 530 tokens GPT-5.4 Mini: pass@5 11%, mean total tokens per question 45K tokens Claude Opus 4.7 (xhigh): pass@5 15%, mean total tokens per question 16K tokens GPT-5.5 (xhigh): pass@5 23%, mean total tokens per question 20K tokens Grok 4.3 (high): pass@5 13%, mean total tokens per question 19K tokens Gemini 3.5 Flash: pass@5 22%, mean total tokens per question 22K tokens Claude Opus 4.8: pass@5 18%, mean total tokens per question 36K tokens Claude Fable 5 (max): pass@5 24%, mean total tokens per question 23K tokens Claude Sonnet 5 (max): pass@5 13%, mean total tokens per question 72K tokens Grok 4.5 (high): pass@5 15%, mean total tokens per question 26K tokens GPT-5.6 Luna (max): pass@5 21%, mean total tokens per question 52K tokens GPT-5.6 Sol (max): pass@5 30%, mean total tokens per question 31K tokens GPT-5.6 Terra (max): pass@5 19%, mean total tokens per question 42K tokens Gemini 3.5 Flash-Lite (high): pass@5 14%, mean total tokens per question 13K tokens Gemini 3.6 Flash (high): pass@5 18%, mean total tokens per question 21K tokens Claude Opus 5 (max): pass@5 26%, mean total tokens per question 30K tokens Grok 4.6 (xhigh): pass@5 17%, mean total tokens per question 26K tokens Claude Opus 5.5 (max): pass@5 43%, mean total tokens per question 59K tokens GPT-6 Luna (max): pass@5 20%, mean total tokens per question 41K tokens GPT-6 Sol (max): pass@5 41%, mean total tokens per question 23K tokens QVQ (original release evaluation): pass@5 4%, mean total tokens per question 4,138 tokens. Not in our model list QVQ: pass@5 2%, mean total tokens per question 4,138 tokens. Not in our model list Gemini 1.0 Pro Vision: pass@5 2%, mean total tokens per question 610 tokens GPT-4.5: pass@5 6%, mean total tokens per question 1,509 tokens Gemini 2.5 Pro: pass@5 5%, mean total tokens per question 1,480 tokens GPT-5.2 (medium): pass@5 18%, mean total tokens per question 5,804 tokens Reka Edge: pass@5 0%, mean total tokens per question 530 tokens GPT-6 Astra (max): pass@5 52%, mean total tokens per question 17K tokens GPT-6 Astra (max)GPT-5.2 (medium)GPT-4.5Gemini 2.5 ProGemini 1.0 Pro VisionReka Edge 0% 20% 40% 60% 1001,00010K100K Mean total tokens per question (tokens), log scale, lower is better pass@5 Gemini 1.0 Pro Vision (original release evaluation): pass@5 2%, mean total tokens per question 610 tokens Claude 3 Opus: pass@5 0%, mean total tokens per question 1,915 tokens Claude 3 Opus (original release evaluation): pass@5 0%, mean total tokens per question 1,915 tokens Claude 3 Sonnet: pass@5 1%, mean total tokens per question 1,920 tokens Claude 3 Sonnet (original release evaluation): pass@5 1%, mean total tokens per question 1,920 tokens Claude 3 Haiku: pass@5 0%, mean total tokens per question 1,919 tokens Claude 3 Haiku (original release evaluation): pass@5 0%, mean total tokens per question 1,919 tokens Claude 3.5 Sonnet: pass@5 2%, mean total tokens per question 1,940 tokens Claude 3.5 Sonnet (original release evaluation): pass@5 1%, mean total tokens per question 1,940 tokens GPT-4o mini: pass@5 2%, mean total tokens per question 1,546 tokens GPT-4o mini (original release evaluation): pass@5 2%, mean total tokens per question 1,546 tokens Gemini 1.5 Flash: pass@5 2%, mean total tokens per question 728 tokens Gemini 1.5 Flash (original release evaluation): pass@5 2%, mean total tokens per question 728 tokens Qwen2-VL-72B-Instruct: pass@5 1%, mean total tokens per question 3,313 tokens Qwen2-VL-72B-Instruct (original release evaluation): pass@5 2%, mean total tokens per question 3,313 tokens Gemini 1.5 Pro: pass@5 2%, mean total tokens per question 707 tokens Gemini 1.5 Pro (original release evaluation): pass@5 2%, mean total tokens per question 707 tokens Llama 3.2 90B: pass@5 0%, mean total tokens per question 989 tokens Llama 3.2 90B (original release evaluation): pass@5 0%, mean total tokens per question 989 tokens NVLM-D-72B: pass@5 1%, mean total tokens per question 3,728 tokens NVLM-D-72B (original release evaluation): pass@5 1%, mean total tokens per question 3,728 tokens Claude 3.5 Sonnet (October 2024): pass@5 3%, mean total tokens per question 1,905 tokens Claude 3.5 Sonnet (October 2024) (original release evaluation): pass@5 2%, mean total tokens per question 1,905 tokens Pixtral Large: pass@5 3%, mean total tokens per question 4,107 tokens Pixtral Large (original release evaluation): pass@5 3%, mean total tokens per question 4,107 tokens o1: pass@5 0%, mean total tokens per question 8,206 tokens o1 (original release evaluation): pass@5 0%, mean total tokens per question 8,206 tokens Gemini 2.0 Flash Thinking: pass@5 5%, mean total tokens per question 2,919 tokens Gemini 2.0 Flash Thinking (original release evaluation): pass@5 5%, mean total tokens per question 2,919 tokens Gemini 2.0 Flash: pass@5 3%, mean total tokens per question 2,798 tokens Gemini 2.0 Flash (original release evaluation): pass@5 3%, mean total tokens per question 2,798 tokens Claude 3.7 Sonnet: pass@5 2%, mean total tokens per question 1,981 tokens Claude 3.7 Sonnet (thinking): pass@5 6%, mean total tokens per question 14K tokens GPT-4o: pass@5 0%, mean total tokens per question 1,582 tokens GPT-4o (original release evaluation): pass@5 0%, mean total tokens per question 1,582 tokens Llama 4 Maverick: pass@5 2%, mean total tokens per question 985 tokens Llama 4 Scout: pass@5 3%, mean total tokens per question 3,150 tokens GPT-4.1: pass@5 3%, mean total tokens per question 1,650 tokens o3: pass@5 6%, mean total tokens per question 6,454 tokens o4-mini: pass@5 11%, mean total tokens per question 18K tokens Claude Opus 4: pass@5 4%, mean total tokens per question 2,181 tokens Claude Opus 4 (thinking): pass@5 6%, mean total tokens per question 7,486 tokens Claude Sonnet 4: pass@5 6%, mean total tokens per question 2,179 tokens Claude Sonnet 4 (thinking): pass@5 6%, mean total tokens per question 13K tokens Grok 4: pass@5 6%, mean total tokens per question 22K tokens Claude Opus 4.1: pass@5 5%, mean total tokens per question 2,222 tokens Claude Opus 4.1 (thinking): pass@5 8%, mean total tokens per question 8,001 tokens GPT-5 (medium): pass@5 7%, mean total tokens per question 6,040 tokens GPT-5 mini (high): pass@5 9%, mean total tokens per question 12K tokens GPT-5 nano (high): pass@5 3%, mean total tokens per question 10K tokens Gemini 2.5 Flash: pass@5 3%, mean total tokens per question 3,226 tokens Claude Sonnet 4.5 (thinking): pass@5 3%, mean total tokens per question 15K tokens Claude Sonnet 4.5: pass@5 5%, mean total tokens per question 2,321 tokens GPT-5.1 (medium): pass@5 6%, mean total tokens per question 6,206 tokens Gemini 3 Pro: pass@5 20%, mean total tokens per question 20K tokens Claude Opus 4.5: pass@5 9%, mean total tokens per question 20K tokens Gemini 3 Flash: pass@5 15%, mean total tokens per question 24K tokens Claude Opus 4.6: pass@5 12%, mean total tokens per question 30K tokens Claude Sonnet 4.6: pass@5 12%, mean total tokens per question 25K tokens Grok 4.20 (reasoning): pass@5 10%, mean total tokens per question 20K tokens Grok 4.20 (non-reasoning): pass@5 10%, mean total tokens per question 13K tokens Gemini 3.1 Pro: pass@5 19%, mean total tokens per question 22K tokens Gemini 3.1 Flash-Lite: pass@5 5%, mean total tokens per question 14K tokens GPT-5.4 (xhigh): pass@5 23%, mean total tokens per question 22K tokens Reka Edge (original release evaluation): pass@5 0%, mean total tokens per question 530 tokens GPT-5.4 Mini: pass@5 11%, mean total tokens per question 45K tokens Claude Opus 4.7 (xhigh): pass@5 15%, mean total tokens per question 16K tokens GPT-5.5 (xhigh): pass@5 23%, mean total tokens per question 20K tokens Grok 4.3 (high): pass@5 13%, mean total tokens per question 19K tokens Gemini 3.5 Flash: pass@5 22%, mean total tokens per question 22K tokens Claude Opus 4.8: pass@5 18%, mean total tokens per question 36K tokens Claude Fable 5 (max): pass@5 24%, mean total tokens per question 23K tokens Claude Sonnet 5 (max): pass@5 13%, mean total tokens per question 72K tokens Grok 4.5 (high): pass@5 15%, mean total tokens per question 26K tokens GPT-5.6 Luna (max): pass@5 21%, mean total tokens per question 52K tokens GPT-5.6 Sol (max): pass@5 30%, mean total tokens per question 31K tokens GPT-5.6 Terra (max): pass@5 19%, mean total tokens per question 42K tokens Gemini 3.5 Flash-Lite (high): pass@5 14%, mean total tokens per question 13K tokens Gemini 3.6 Flash (high): pass@5 18%, mean total tokens per question 21K tokens Claude Opus 5 (max): pass@5 26%, mean total tokens per question 30K tokens Grok 4.6 (xhigh): pass@5 17%, mean total tokens per question 26K tokens Claude Opus 5.5 (max): pass@5 43%, mean total tokens per question 59K tokens GPT-6 Luna (max): pass@5 20%, mean total tokens per question 41K tokens GPT-6 Sol (max): pass@5 41%, mean total tokens per question 23K tokens QVQ (original release evaluation): pass@5 4%, mean total tokens per question 4,138 tokens. Not in our model list QVQ: pass@5 2%, mean total tokens per question 4,138 tokens. Not in our model list Gemini 1.0 Pro Vision: pass@5 2%, mean total tokens per question 610 tokens GPT-4.5: pass@5 6%, mean total tokens per question 1,509 tokens Gemini 2.5 Pro: pass@5 5%, mean total tokens per question 1,480 tokens GPT-5.2 (medium): pass@5 18%, mean total tokens per question 5,804 tokens Reka Edge: pass@5 0%, mean total tokens per question 530 tokens GPT-6 Astra (max): pass@5 52%, mean total tokens per question 17K tokens GPT-6 Astra (max)GPT-5.2 (medium)GPT-4.5Gemini 2.5 ProGemini 1.0 Pro VisionReka Edge
ModelSettingpass@5Mean total tokens per question
GPT-6 Astra max 52% 16,763 tokens
Claude Opus 5.5 max 43% 58,573 tokens
GPT-6 Sol max 41% 22,672 tokens
GPT-5.6 Sol max 30% 31,065 tokens
Claude Opus 5 max 26% 29,995 tokens
Claude Fable 5 max 24% 23,151 tokens
GPT-5.5 xhigh 23% 20,174 tokens
GPT-5.4 xhigh 23% 22,315 tokens
Gemini 3.5 Flash 22% 21,955 tokens
GPT-5.6 Luna max 21% 52,301 tokens
Gemini 3 Pro 20% 19,825 tokens
GPT-6 Luna max 20% 41,067 tokens
Gemini 3.1 Pro 19% 21,764 tokens
GPT-5.6 Terra max 19% 42,219 tokens
GPT-5.2 medium 18% 5,804 tokens
Gemini 3.6 Flash high 18% 20,539 tokens
Claude Opus 4.8 18% 35,501 tokens
Grok 4.6 xhigh 17% 26,165 tokens
Claude Opus 4.7 xhigh 15% 15,895 tokens
Gemini 3 Flash 15% 23,650 tokens
Grok 4.5 high 15% 25,758 tokens
Gemini 3.5 Flash-Lite high 14% 12,939 tokens
Grok 4.3 high 13% 19,309 tokens
Claude Sonnet 5 max 13% 72,104 tokens
Claude Sonnet 4.6 12% 24,824 tokens
Claude Opus 4.6 12% 30,000 tokens
o4-mini 11% 18,025 tokens
GPT-5.4 Mini 11% 44,650 tokens
Grok 4.20 non-reasoning 10% 12,893 tokens
Grok 4.20 reasoning 10% 20,485 tokens
GPT-5 mini high 9% 12,067 tokens
Claude Opus 4.5 9% 20,194 tokens
Claude Opus 4.1 thinking 8% 8,001 tokens
GPT-5 medium 7% 6,040 tokens
GPT-4.5 6% 1,509 tokens
Claude Sonnet 4 6% 2,179 tokens
GPT-5.1 medium 6% 6,206 tokens
o3 6% 6,454 tokens
Claude Opus 4 thinking 6% 7,486 tokens
Claude Sonnet 4 thinking 6% 12,654 tokens
Claude 3.7 Sonnet thinking 6% 13,853 tokens
Grok 4 6% 22,343 tokens
Gemini 2.5 Pro Jun 2025 5% 1,480 tokens
Claude Opus 4.1 5% 2,222 tokens
Claude Sonnet 4.5 5% 2,321 tokens
Gemini 2.0 Flash Thinking Jan 2025 5% 2,919 tokens
Gemini 2.0 Flash Thinking Jan 2025 original release evaluation 5% 2,919 tokens
Gemini 3.1 Flash-Lite 5% 14,139 tokens
Claude Opus 4 4% 2,181 tokens
QVQ original release evaluation 4% 4,138 tokens
GPT-4.1 3% 1,650 tokens
Claude 3.5 Sonnet (October 2024) 3% 1,905 tokens
Gemini 2.0 Flash Feb 2025 3% 2,798 tokens
Gemini 2.0 Flash Feb 2025 original release evaluation 3% 2,798 tokens
Llama 4 Scout 3% 3,150 tokens
Gemini 2.5 Flash Sep 2025 3% 3,226 tokens
Pixtral Large 3% 4,107 tokens
Pixtral Large original release evaluation 3% 4,107 tokens
GPT-5 nano high 3% 10,351 tokens
Claude Sonnet 4.5 thinking 3% 14,680 tokens
Gemini 1.0 Pro Vision 2% 610 tokens
Gemini 1.0 Pro Vision original release evaluation 2% 610 tokens
Gemini 1.5 Pro Sep 2024 2% 707 tokens
Gemini 1.5 Pro Sep 2024 original release evaluation 2% 707 tokens
Gemini 1.5 Flash Sep 2024 2% 728 tokens
Gemini 1.5 Flash Sep 2024 original release evaluation 2% 728 tokens
Llama 4 Maverick 2% 985 tokens
GPT-4o mini 2% 1,546 tokens
GPT-4o mini original release evaluation 2% 1,546 tokens
Claude 3.5 Sonnet (October 2024) original release evaluation 2% 1,905 tokens
Claude 3.5 Sonnet 2% 1,940 tokens
Claude 3.7 Sonnet 2% 1,981 tokens
Qwen2-VL-72B-Instruct original release evaluation 2% 3,313 tokens
QVQ 2% 4,138 tokens
Claude 3 Sonnet 1% 1,920 tokens
Claude 3 Sonnet original release evaluation 1% 1,920 tokens
Claude 3.5 Sonnet original release evaluation 1% 1,940 tokens
Qwen2-VL-72B-Instruct 1% 3,313 tokens
NVLM-D-72B 1% 3,728 tokens
NVLM-D-72B original release evaluation 1% 3,728 tokens
Reka Edge 0% 530 tokens
Reka Edge original release evaluation 0% 530 tokens
Llama 3.2 90B 0% 989 tokens
Llama 3.2 90B original release evaluation 0% 989 tokens
GPT-4o Mar 2025 0% 1,582 tokens
GPT-4o Mar 2025 original release evaluation 0% 1,582 tokens
Claude 3 Opus 0% 1,915 tokens
Claude 3 Opus original release evaluation 0% 1,915 tokens
Claude 3 Haiku 0% 1,919 tokens
Claude 3 Haiku original release evaluation 0% 1,919 tokens
o1 0% 8,206 tokens
o1 original release evaluation 0% 8,206 tokens

Of runs with a cost, the best scores 52% for $0.56 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind Anthropic OpenAI Cost frontier One model at each effort level
0% 20% 40% 60% $0.0001 $0.001 $0.01 $0.10 $1.00 $10 Cost per task, log scale GPT-6 Astra, max GPT-6 Sol, max GPT-6 Luna, max Claude Sonnet 4 Gemini 2.0 Flash 0% 20% 40% 60% $0.0001 $0.001 $0.01 $0.10 $1.00 $10 Cost per task, log scale GPT-6 Astra, max GPT-6 Sol, max

Source: Official ZeroBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
GPT-5.6 Sol max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 53%
Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
GPT-6 Astra max 52% $0.56
Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
Claude Opus 5 max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Opus 5 number; values from chart image. 47.5%
Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
Claude Opus 5.5 max 43% $1.11
GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
Kimi K3 w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; matches Moonshot’s own with-tools value; possibly relayed. 41%
GPT-6 Sol max 41% $0.17
GPT-5.6 Sol w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 35%
DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
Claude Opus 4.8 externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 34%
Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
GPT-5.6 Sol max 30% $0.76
Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
Claude Opus 5 max 26% $0.68
Claude Fable 5 max 24% $1.01
Kimi K3 externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge; independent re-run by dots-studio. 24%
Qwen 3.8 Max externally reported: Qwen3.8 release; 5 samplings. 24%
GPT-5.4 xhigh 23% $0.30
GPT-5.5 xhigh 23% $0.55
Claude Fable 5 externally reported: Kimi K3 report; 5 samplings. 23%
Kimi K3 externally reported: Kimi K3 report; 5 samplings. 23%
GPT-5.5 externally reported: Kimi K3 report; 5 samplings. 22%
Gemini 3.5 Flash 22% $0.18
Gemini 3.5 Flash externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 21%
GPT-5.6 Luna max 21% $0.28
Gemini 3 Pro 20% $0.22
GPT-6 Luna max 20% $0.02
Gemini 3.1 Pro 19% $0.25
GPT-5.6 Terra max 19% $0.55
GPT-5.2 medium 18% $0.06
Claude Opus 4.8 18% $0.81
Gemini 3.6 Flash high 18% $0.14
Claude Opus 4.8 externally reported: Kimi K3 report; 5 samplings. 17%
GPT-5.6 Sol externally reported: Kimi K3 report; 5 samplings. 17%
Grok 4.6 xhigh 17% $0.15
Gemini 3.1 Pro externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 16%
Gemini 3 Flash 15% $0.07
Claude Opus 4.7 xhigh 15% $0.32
Grok 4.5 high 15% $0.15
Gemini 3.5 Flash-Lite high 14% $0.03
Grok 4.3 high 13% $0.05
Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
Claude Sonnet 5 max 13% $0.69
Claude Opus 4.6 12% $0.72
Claude Sonnet 4.6 12% $0.35
o4-mini 11% $0.07
GPT-5.4 Mini 11% $0.19
Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
Grok 4.20 reasoning 10% $0.11
Grok 4.20 non-reasoning 10% $0.07
GPT-5 mini high 9.00% $0.02
Claude Opus 4.5 9.00% $0.47
Claude Opus 4.1 thinking 8.00% $0.50
GPT-5 medium 7.00% $0.05
Claude 3.7 Sonnet thinking 6.00% $0.19
GPT-4.5 6.00% $0.15
o3 6.00% $0.23
Claude Opus 4 thinking 6.00% $0.46
Claude Sonnet 4 6.00% $0.01
Claude Sonnet 4 thinking 6.00% $0.17
Grok 4 6.00% $0.31
GPT-5.1 medium 6.00% $0.05
Gemini 2.0 Flash Thinking Jan 2025 5.00%
Gemini 2.0 Flash Thinking Jan 2025 original release evaluation 5.00%
Gemini 2.5 Pro Jun 2025 5.00% $0.01
Claude Opus 4.1 5.00% $0.07
Claude Sonnet 4.5 5.00% $0.02
Gemini 3.1 Flash-Lite 5.00% $0.02
Claude Opus 4 4.00% $0.07
Claude 3.5 Sonnet (October 2024) 3.00% $0.009
Pixtral Large 3.00% $0.01
Pixtral Large original release evaluation 3.00% $0.01
Gemini 2.0 Flash Feb 2025 3.00% $0.0004
Gemini 2.0 Flash Feb 2025 original release evaluation 3.00% $0.0004
Llama 4 Scout 3.00% $0.0005
GPT-4.1 3.00% $0.007
GPT-5 nano high 3.00% $0.003
Gemini 2.5 Flash Sep 2025 3.00% $0.003
Claude Sonnet 4.5 thinking 3.00% $0.20
Gemini 1.0 Pro Vision 2.00% $0.0002
Gemini 1.0 Pro Vision original release evaluation 2.00% $0.0002
Claude 3.5 Sonnet 2.00% $0.009
GPT-4o mini 2.00% $0.0004
GPT-4o mini original release evaluation 2.00% $0.0004
Gemini 1.5 Flash Sep 2024 2.00% $6.05
Gemini 1.5 Flash Sep 2024 original release evaluation 2.00% $6.05
Qwen2-VL-72B-Instruct original release evaluation 2.00% $0.001
Gemini 1.5 Pro Sep 2024 2.00% $0.003
Gemini 1.5 Pro Sep 2024 original release evaluation 2.00% $0.003
Claude 3.5 Sonnet (October 2024) original release evaluation 2.00% $0.009
Claude 3.7 Sonnet 2.00% $0.01
Llama 4 Maverick 2.00% $0.0006
Claude 3 Sonnet 1.00% $0.009
Claude 3 Sonnet original release evaluation 1.00% $0.009
Claude 3.5 Sonnet original release evaluation 1.00% $0.009
GPT-4o mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
Qwen2-VL-72B-Instruct 1.00% $0.001
NVLM-D-72B 1.00%
NVLM-D-72B original release evaluation 1.00%
GPT-4o Mar 2025 externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
Claude 3 Opus 0% $0.05
Claude 3 Opus original release evaluation 0% $0.05
Claude 3 Haiku 0% $0.0008
Claude 3 Haiku original release evaluation 0% $0.0008
Llama 3.2 90B 0% $0.0004
Llama 3.2 90B original release evaluation 0% $0.0004
o1 0% $0.45
o1 original release evaluation 0% $0.45
GPT-4o Mar 2025 0% $0.007
GPT-4o Mar 2025 original release evaluation 0% $0.007
Reka Edge 0% $5.30
Reka Edge original release evaluation 0% $5.30

8 models have held number one on the official leaderboard since its release in February 2025 12 models have held number one with claims included since its release in February 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Google DeepMind Anthropic OpenAI ByteDance Seed Z.ai StepFun Qwen Meta DeepSeek
Own leaderboard Not tested since Sep 2026
GPT-5.6 Sol 54.6%
Lab claims 9 leaders
DeepSeek V4.1 Flash 1017 51%

Sources: Official ZeroBench leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) GPT-5.6 Sol Claude Fable 5 GPT-5.4 Qwen3.5-122B-A10B Gemini 3.1 Pro GLM-4.5V GLM-4.1V-9B-Thinking 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) GPT-5.6 Sol Claude Fable 5

Sources: Official ZeroBench leaderboard and labs’ model cards

5 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

ByteDance Seed
StepFun
Z.ai
Qwen
Moonshot AI
Meta
DeepSeek
DateLaunch postScoreAs the post writes it
Oct 24, 2026 Kimi K3.1 Pro 39 ZeroBench_main (avg@5 / pass@5) 29.8 / 39.0
Oct 17, 2026 DeepSeek V4.1 Flash 1017 51 ZeroBench-main (w/tools)
Oct 15, 2026 Muse Spark Open 50 ZeroBench (w/ tools)
Sep 10, 2026 DeepSeek V4.1 Flash 49 ZeroBench-main w/ tools (Pass@5)
Aug 21, 2026 DeepSeek-V4-Flash-Vision-Exp 35 ZeroBench (Pass@5)
Aug 3, 2026 Qwen 3.8 Max 49 Best of ZeroBench (Pass@5) (without CI / with CI) 24.0 / 49.0 / ZeroBench-Sub
Jul 16, 2026 Kimi K3 23 ZeroBench (pass@5) (without python)
Jun 23, 2026 Seed2.1 Pro 22 ZEROBench (w. Tool) 18.0 (22.0), main set
Apr 15, 2026 Qwen 3.6 35B-A3B 34.4 ZEROBench_sub
Apr 8, 2026 Muse Spark 33 ZeroBench (Multi-Step Visual Reasoning (pass@5) - With Python)
Mar 30, 2026 Qwen3.5-Omni-Plus 34.4 Best of ZEROBench 5 / ZEROBench_sub 34.4
Mar 2, 2026 Qwen3.5-0.8B 12.9 Best of ZEROBench / ZEROBench_sub 12.9/11.4
Mar 2, 2026 Qwen3.5-2B 17.1 Best of ZEROBench / ZEROBench_sub 17.1/18.6
Mar 2, 2026 Qwen3.5-4B 26.3 Best of ZEROBench / ZEROBench_sub 26.3
Mar 2, 2026 Qwen3.5-9B 31.1 Best of ZEROBench / ZEROBench_sub 31.1
Feb 24, 2026 Qwen3.5-35B-A3B 34.1 Best of ZEROBench / ZEROBench_sub 34.1
Feb 24, 2026 Qwen3.5-27B 36.2 Best of ZEROBench / ZEROBench_sub 36.2
Feb 24, 2026 Qwen3.5-122B-A10B 36.2 Best of ZEROBench 9 / ZEROBench_sub 36.2
Feb 16, 2026 Qwen3.5 397B-A17B 12 Best of ZEROBench / ZEROBench_sub
Jan 27, 2026 Kimi K2.5 9 ZeroBench
Dec 18, 2025 Seed1.8 11 ZeroBench (main)
Dec 8, 2025 GLM-4.6V 25.8 ZeroBench (sub)
Sep 23, 2025 Qwen3-VL-235B-A22B 27.7 Best of ZEROBench 4.0 / ZEROBench_Sub 27.7
Aug 11, 2025 GLM-4.5V 23.4 ZeroBench (sub)
Jul 31, 2025 Step 3 23 ZeroBench (sub)
May 12, 2025 Seed1.5-VL 2 ZeroBench (main) 2 (sub 30.8)

16 system cards report it, from 8 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Sep 10, 2026 DeepSeek V4.1 Flash technical report 49 ZeroBench-main w/ tools (Pass@5) (Table 3) Capabilities
Jul 27, 2026 Kimi K3 technical report 23 ZeroBench-main (pass@5) without Python, cell 23.0 / 41.0, 5 runs (Table 2) Capabilities
Jun 23, 2026 Seed2.1 Pro model card 18 ZeroBench (main), no tools, 22.0 with tools. ZeroBench (sub) 49.4, 56.3 with tools (Table 9) Capabilities
Apr 17, 2026 Qwen3.5-Omni-Plus technical report 5 ZEROBench 5, ZEROBench_sub 34.4 (Table 6) Capabilities
Mar 21, 2026 Seed1.8 model card 11 ZeroBench (main) (Table 2) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 12 ZeroBench (main) 12.0, ZeroBench (sub) 47.6 (Table 8) Capabilities
Jan 30, 2026 Kimi K2.5 technical report 9 ZeroBench without tools, 11 with tools (Table 4) Capabilities
Jan 14, 2026 STEP3-VL-10B technical report 1 ZeroBench (main), SeRe, sub 27.54 (Tables 1 and 3). PaCoRe main 5.00, sub 29.94 Capabilities
Jan 1, 2026 GLM-4.6V technical report 25.8 ZeroBench (sub), GLM-4.6V thinking (Table 2) Capabilities
Nov 26, 2025 Qwen3-VL-235B-A22B technical report 4 ZeroBench (Table 2) Capabilities
Nov 6, 2025 Nemotron Nano 12B v2 VL technical report 18.3 ZeroBench (sub), reasoning on (Table 3). Reasoning off 15.0 Capabilities
Aug 13, 2025 GLM-4.5V technical report 23.4 ZeroBench (sub), GLM-4.5V thinking (Table 2) Capabilities
Jul 7, 2025 Gemini 2.5 Pro technical report 4.5 ZeroBench, Gemini as judge, average of 4 runs (Table 3) Capabilities
Jul 7, 2025 Gemini 2.5 Flash technical report 2 ZeroBench, Gemini as judge, average of 4 runs (Table 3) Capabilities
Jul 1, 2025 GLM-4.1V-9B-Thinking technical report 19.2 ZeroBench (sub), GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities
May 11, 2025 Seed1.5-VL technical report 2 ZeroBench (main) 2, ZeroBench (sub) 30.8, thinking (Table 6). Non-thinking main 0, sub 29.0 Capabilities

The lab's number comes out ahead in 0 of 2 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 4 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap
Kimi K3 23% ZeroBench (pass@5) (without python) 41% Leaderboard, w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. −18 pts
Gemini 2.5 Pro Jun 2025 4.50% ZeroBench, Gemini as judge, average of 4 runs (Table 3) 5.00% Leaderboard −0.5 pts
Muse Spark 33% ZeroBench (Multi-Step Visual Reasoning (pass@5) - With Python) 33% Leaderboard, w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. Same
Qwen 3.8 Max 49% Best of ZeroBench (Pass@5) (without CI / with CI) 24.0 / 49.0 / ZeroBench-Sub 49% Leaderboard, w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. Same
DeepSeek-V4-Flash-Vision-Exp 35% ZeroBench (Pass@5) 35% Leaderboard, externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. Same
DeepSeek V4.1 Flash 49% ZeroBench-main w/ tools (Pass@5) 49% Leaderboard, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. Same

2 logged changes that move scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Scores recomputed August 2026

    The maintainers corrected 0.71% of grading decisions and updated official scores. Mean pass@5 increased by 0.79 percentage points without question changes. Source

  2. Questions corrected March 2025

    The maintainers revised questions after post-release review. Results should identify the question revision used for evaluation. Source

More in Vision

All 281 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%
  • ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Command A Vision 90.9%