Benchmark Almanac

ZeroBench

GPT-5.6 Sol 54.6% leads since July 2026
GPT-5.6 Sol 54.6% leads since July 2026
In use

The official leaderboard hasn’t scored a model released after Claude Opus 5.5 (September 2026). Labs report their own numbers for 4 newer models, such as DeepSeek V4.1 Flash 1017 at 51%.

100 hand-made visual questions that all frontier models originally failed. Scored by exact answer, pass@1 and pass@5.

Point releases keep scores comparable: v2 (2025-03) revised questions, v3 (2025-12) clarified two questions and three subquestions. A 2026-08 grading fix raised mean pass@5 by 0.79 points.

Released
February 2025
Kind
General visual questions
Level
University
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Images → Text
Interaction
Single turn
Run by
Anyone
In views
Vision
Strongest model tested
GPT-6 Astra #4 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 18th, 2026

In the authors’ words

Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench - a lightweight visual reasoning benchmark curated using adversarial filtering to be "impossible" for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass^5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass^5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io.

Abstract of ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (arXiv:2502.09696, CC0)
80
Entries on its own leaderboard, 77 models we track
27
Launch posts list it, from 8 labs
17
System cards report it, from 9 labs

GPT-5.6 Sol leads with 54.6% among models released by September 2026 GPT-5.6 Sol leads with 54.6% among models released by September 2026

Best score per entry from the benchmark's own leaderboard, read on November 18th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

GPT-5.6 Sol leads with 54.6%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

The newest model with a score here came out in September 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

80 entries. Scroll sideways for the rest.

  1. 54.6 GPT-5.6 Sol
  2. 54.0 Muse Spark 1.2
  3. 52.0 Claude Opus 5
  4. 52.0 GPT-6 Astra
  5. 49.0 Qwen 3.8 Max
  6. 49.0 DeepSeek V4.1 Flash
  7. 46.0 Claude Fable 5
  8. 46.0 Muse Spark 1.1
  9. 43.0 Claude Opus 5.5
  10. 41.0 GPT-5.4
  11. 41.0 GPT-5.5
  12. 41.0 Kimi K3
  13. 41.0 GPT-6 Sol
  14. 35.0 DeepSeek-V4-Flash-Vision-Exp
  15. 34.0 Claude Opus 4.8
  16. 33.0 Muse Spark
  17. 30.0 Gemini 3.7 Flash
  18. 29.0 Gemini 3.1 Pro
  19. 22.0 Gemini 3.5 Flash
  20. 21.0 GPT-5.6 Luna
  21. 20.0 Gemini 3 Pro
  22. 20.0 GPT-6 Luna
  23. 19.0 dots3-note Preview
  24. 19.0 GPT-5.6 Terra
  25. 18.0 Seed2.1 Turbo
  26. 18.0 GPT-5.2
  27. 18.0 Gemini 3.6 Flash
  28. 17.0 Grok 4.6
  29. 15.0 Gemini 3 Flash
  30. 15.0 Claude Opus 4.7
  31. 15.0 Grok 4.5
  32. 14.0 Gemini 3.5 Flash-Lite
  33. 13.0 Grok 4.3
  34. 13.0 Qwen3.7-Plus
  35. 13.0 Claude Sonnet 5
  36. 12.0 Claude Opus 4.6
  37. 12.0 Claude Sonnet 4.6
  38. 11.0 o4-mini
  39. 11.0 GPT-5.4 Mini
  40. 11.0 Inkling-Small
  41. 10.0 Grok 4.20
  42. 9.00 GPT-5 mini
  43. 9.00 Claude Opus 4.5
  44. 8.00 Claude Opus 4.1
  45. 7.00 GPT-5
  46. 6.00 Claude 3.7 Sonnet
  47. 6.00 GPT-4.5
  48. 6.00 o3
  49. 6.00 Claude Opus 4
  50. 6.00 Claude Sonnet 4
  51. 6.00 Grok 4
  52. 6.00 GPT-5.1
  53. 5.00 Gemini 2.0 Flash Thinking
  54. 5.00 Gemini 2.5 Pro
  55. 5.00 Claude Sonnet 4.5
  56. 5.00 Gemini 3.1 Flash-Lite
  57. 4.00 QVQ
  58. 3.00 Claude 3.5 Sonnet (October 2024)
  59. 3.00 Pixtral Large
  60. 3.00 Gemini 2.0 Flash
  61. 3.00 Llama 4 Scout
  62. 3.00 GPT-4.1
  63. 3.00 GPT-5 nano
  64. 3.00 Gemini 2.5 Flash
  65. 2.00 Gemini 1.0 Pro Vision
  66. 2.00 Claude 3.5 Sonnet
  67. 2.00 GPT-4o mini
  68. 2.00 Gemini 1.5 Flash
  69. 2.00 Qwen2-VL-72B-Instruct
  70. 2.00 Gemini 1.5 Pro
  71. 2.00 Llama 4 Maverick
  72. 1.00 Claude 3 Sonnet
  73. 1.00 NVLM-D-72B
  74. 1.00 GPT-4o
  75. 0.0 Claude 3 Opus
  76. 0.0 Claude 3 Haiku
  77. 0.0 o1-mini
  78. 0.0 Llama 3.2 90B
  79. 0.0 o1
  80. 0.0 Reka Edge
  1. 1 GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
  2. 2 Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
  3. 3 Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
  4. 4 GPT-6 Astra max 52%
  5. 5 Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
  6. 6 DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
  7. 7 Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
  8. 8 Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
  9. 9 Claude Opus 5.5 max 43%
  10. 10 GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
  11. 11 GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  12. 12 Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  13. 13 GPT-6 Sol max 41%
  14. 14 DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
  15. 15 Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
  16. 16 Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
  17. 17 Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
  18. 18 Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
  19. 19 Gemini 3.5 Flash 22%
  20. 20 GPT-5.6 Luna max 21%
  21. 21 Gemini 3 Pro 20%
  22. 22 GPT-6 Luna max 20%
  23. 23 dots3-note Preview externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge; values from card image. 19%
  24. 24 GPT-5.6 Terra max 19%
  25. 25 Seed2.1 Turbo externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 18%
  26. 26 GPT-5.2 medium 18%
  27. 27 Gemini 3.6 Flash high 18%
  28. 28 Grok 4.6 xhigh 17%
  29. 29 Gemini 3 Flash 15%
  30. 30 Claude Opus 4.7 xhigh 15%
  31. 31 Grok 4.5 high 15%
  32. 32 Gemini 3.5 Flash-Lite high 14%
  33. 33 Grok 4.3 high 13%
  34. 34 Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
  35. 35 Claude Sonnet 5 max 13%
  36. 36 Claude Opus 4.6 12%
  37. 37 Claude Sonnet 4.6 12%
  38. 38 o4-mini 11%
  39. 39 GPT-5.4 Mini 11%
  40. 40 Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
  41. 41 Grok 4.20 reasoning 10%
  42. 42 GPT-5 mini high 9.00%
  43. 43 Claude Opus 4.5 9.00%
  44. 44 Claude Opus 4.1 thinking 8.00%
  45. 45 GPT-5 medium 7.00%
  46. 46 Claude 3.7 Sonnet thinking 6.00%
  47. 47 GPT-4.5 6.00%
  48. 48 o3 6.00%
  49. 49 Claude Opus 4 thinking 6.00%
  50. 50 Claude Sonnet 4 6.00%
  51. 51 Grok 4 6.00%
  52. 52 GPT-5.1 medium 6.00%
  53. 53 Gemini 2.0 Flash Thinking Jan 2025 5.00%
  54. 54 Gemini 2.5 Pro Jun 2025 5.00%
  55. 55 Claude Sonnet 4.5 5.00%
  56. 56 Gemini 3.1 Flash-Lite 5.00%
  57. 57 QVQ original release evaluation 4.00%
  58. 58 Claude 3.5 Sonnet (October 2024) 3.00%
  59. 59 Pixtral Large 3.00%
  60. 60 Gemini 2.0 Flash Feb 2025 3.00%
  61. 61 Llama 4 Scout 3.00%
  62. 62 GPT-4.1 3.00%
  63. 63 GPT-5 nano high 3.00%
  64. 64 Gemini 2.5 Flash Sep 2025 3.00%
  65. 65 Gemini 1.0 Pro Vision 2.00%
  66. 66 Claude 3.5 Sonnet 2.00%
  67. 67 GPT-4o mini 2.00%
  68. 68 Gemini 1.5 Flash Sep 2024 2.00%
  69. 69 Qwen2-VL-72B-Instruct original release evaluation 2.00%
  70. 70 Gemini 1.5 Pro Sep 2024 2.00%
  71. 71 Llama 4 Maverick 2.00%
  72. 72 Claude 3 Sonnet 1.00%
  73. 73 NVLM-D-72B 1.00%
  74. 74 GPT-4o Mar 2025, externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
  75. 75 Claude 3 Opus 0%
  76. 76 Claude 3 Haiku 0%
  77. 77 o1-mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 0%
  78. 78 Llama 3.2 90B 0%
  79. 79 o1 0%
  80. 80 Reka Edge 0%

Source: Official ZeroBench leaderboard

  1. 1 GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
  2. 2 Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
  3. 3 Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
  4. 4 GPT-6 Astra max 52%
  5. 5 Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
  6. 6 DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
  7. 7 Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
  8. 8 Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
  9. 9 Claude Opus 5.5 max 43%
  10. 10 GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
  11. 11 GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  12. 12 Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  13. 13 GPT-6 Sol max 41%
  14. 14 DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
  15. 15 Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
  16. 16 Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
  17. 17 Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
  18. 18 Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
  19. 19 Gemini 3.5 Flash 22%
  20. 20 GPT-5.6 Luna max 21%
  21. 21 Gemini 3 Pro 20%
  22. 22 GPT-6 Luna max 20%
  23. 23 GPT-5.6 Terra max 19%
  24. 24 GPT-5.2 medium 18%
  25. 25 Gemini 3.6 Flash high 18%
  26. 26 Grok 4.6 xhigh 17%
  27. 27 Gemini 3 Flash 15%
  28. 28 Claude Opus 4.7 xhigh 15%
  29. 29 Grok 4.5 high 15%
  30. 30 Gemini 3.5 Flash-Lite high 14%
  31. 31 Grok 4.3 high 13%
  32. 32 Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
  33. 33 Claude Sonnet 5 max 13%
  34. 34 Claude Opus 4.6 12%
  35. 35 Claude Sonnet 4.6 12%
  36. 36 o4-mini 11%
  37. 37 GPT-5.4 Mini 11%
  38. 38 Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
  39. 39 Grok 4.20 reasoning 10%
  40. 40 GPT-5 mini high 9.00%
  41. 41 Claude Opus 4.5 9.00%
  42. 42 Claude Opus 4.1 thinking 8.00%
  43. 43 GPT-5 medium 7.00%
  44. 44 Claude 3.7 Sonnet thinking 6.00%
  45. 45 GPT-4.5 6.00%
  46. 46 o3 6.00%
  47. 47 Claude Opus 4 thinking 6.00%
  48. 48 Claude Sonnet 4 6.00%
  49. 49 Grok 4 6.00%
  50. 50 GPT-5.1 medium 6.00%
  51. 51 Gemini 2.0 Flash Thinking Jan 2025 5.00%
  52. 52 Gemini 2.5 Pro Jun 2025 5.00%
  53. 53 Claude Sonnet 4.5 5.00%
  54. 54 Gemini 3.1 Flash-Lite 5.00%
  55. 55 Claude 3.5 Sonnet (October 2024) 3.00%
  56. 56 Pixtral Large 3.00%
  57. 57 Gemini 2.0 Flash Feb 2025 3.00%
  58. 58 Llama 4 Scout 3.00%
  59. 59 GPT-4.1 3.00%
  60. 60 GPT-5 nano high 3.00%
  61. 61 Gemini 2.5 Flash Sep 2025 3.00%
  62. 62 Gemini 1.0 Pro Vision 2.00%
  63. 63 Claude 3.5 Sonnet 2.00%
  64. 64 GPT-4o mini 2.00%
  65. 65 Gemini 1.5 Flash Sep 2024 2.00%
  66. 66 Qwen2-VL-72B-Instruct original release evaluation 2.00%
  67. 67 Gemini 1.5 Pro Sep 2024 2.00%
  68. 68 Llama 4 Maverick 2.00%
  69. 69 Claude 3 Sonnet 1.00%
  70. 70 NVLM-D-72B 1.00%
  71. 71 GPT-4o Mar 2025, externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
  72. 72 Claude 3 Opus 0%
  73. 73 Claude 3 Haiku 0%
  74. 74 o1-mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 0%
  75. 75 Llama 3.2 90B 0%
  76. 76 o1 0%
  77. 77 Reka Edge 0%

Source: Official ZeroBench leaderboard

103 models. Scroll sideways for the rest.

  1. 54.6 GPT-5.6 Sol
  2. 54.0 Muse Spark 1.2
  3. 52.0 Claude Opus 5
  4. 52.0 GPT-6 Astra
  5. 51.0 DeepSeek V4.1 Flash 1017
  6. 50.0 Muse Spark Open
  7. 49.0 Qwen 3.8 Max
  8. 49.0 DeepSeek V4.1 Flash
  9. 46.0 Claude Fable 5
  10. 46.0 Muse Spark 1.1
  11. 43.0 Claude Opus 5.5
  12. 41.0 GPT-5.4
  13. 41.0 GPT-5.5
  14. 41.0 Kimi K3
  15. 41.0 GPT-6 Sol
  16. 39.0 Kimi K3.1 Pro
  17. 36.2 Qwen3.5-122B-A10B
  18. 36.2 Qwen3.5-27B
  19. 35.0 DeepSeek-V4-Flash-Vision-Exp
  20. 34.4 Qwen3.5-Omni-Plus
  21. 34.4 Qwen 3.6 35B-A3B
  22. 34.1 Qwen3.5-35B-A3B
  23. 34.0 Claude Opus 4.8
  24. 33.0 Muse Spark
  25. 31.1 Qwen3.5-9B
  26. 30.0 Gemini 3.7 Flash
  27. 29.0 Gemini 3.1 Pro
  28. 27.7 Qwen3-VL-235B-A22B
  29. 26.3 Qwen3.5-4B
  30. 25.8 GLM-4.6V
  31. 23.4 GLM-4.5V
  32. 23.0 Step 3
  33. 22.0 Gemini 3.5 Flash
  34. 22.0 Seed2.1 Pro
  35. 21.0 GPT-5.6 Luna
  36. 20.0 Gemini 3 Pro
  37. 20.0 GPT-6 Luna
  38. 19.2 GLM-4.1V-9B-Thinking
  39. 19.0 GPT-5.6 Terra
  40. 18.3 Nemotron Nano 12B v2 VL
  41. 18.0 GPT-5.2
  42. 18.0 Gemini 3.6 Flash
  43. 17.1 Qwen3.5-2B
  44. 17.0 Grok 4.6
  45. 17.0 Inkling 2 Mini
  46. 15.0 Gemini 3 Flash
  47. 15.0 Claude Opus 4.7
  48. 15.0 Grok 4.5
  49. 14.0 Gemini 3.5 Flash-Lite
  50. 13.0 Grok 4.3
  51. 13.0 Qwen3.7-Plus
  52. 13.0 Claude Sonnet 5
  53. 12.9 Qwen3.5-0.8B
  54. 12.0 Claude Opus 4.6
  55. 12.0 Seed2.0 Pro
  56. 12.0 Claude Sonnet 4.6
  57. 11.0 o4-mini
  58. 11.0 Seed1.8
  59. 11.0 GPT-5.4 Mini
  60. 11.0 Inkling-Small
  61. 10.0 Grok 4.20
  62. 9.00 GPT-5 mini
  63. 9.00 Claude Opus 4.5
  64. 9.00 Kimi K2.5
  65. 8.00 Claude Opus 4.1
  66. 7.00 GPT-5
  67. 6.00 Claude 3.7 Sonnet
  68. 6.00 GPT-4.5
  69. 6.00 o3
  70. 6.00 Claude Opus 4
  71. 6.00 Claude Sonnet 4
  72. 6.00 Grok 4
  73. 6.00 GPT-5.1
  74. 5.00 Gemini 2.0 Flash Thinking
  75. 5.00 Gemini 2.5 Pro
  76. 5.00 Claude Sonnet 4.5
  77. 5.00 Gemini 3.1 Flash-Lite
  78. 3.00 Claude 3.5 Sonnet (October 2024)
  79. 3.00 Pixtral Large
  80. 3.00 Gemini 2.0 Flash
  81. 3.00 Llama 4 Scout
  82. 3.00 GPT-4.1
  83. 3.00 GPT-5 nano
  84. 3.00 Gemini 2.5 Flash
  85. 2.00 Gemini 1.0 Pro Vision
  86. 2.00 Claude 3.5 Sonnet
  87. 2.00 GPT-4o mini
  88. 2.00 Gemini 1.5 Flash
  89. 2.00 Qwen2-VL-72B-Instruct
  90. 2.00 Gemini 1.5 Pro
  91. 2.00 Llama 4 Maverick
  92. 2.00 Seed1.5-VL
  93. 2.00 Gemini 2.5 Flash
  94. 1.00 Claude 3 Sonnet
  95. 1.00 NVLM-D-72B
  96. 1.00 GPT-4o
  97. 1.00 STEP3-VL-10B
  98. 0.0 Claude 3 Opus
  99. 0.0 Claude 3 Haiku
  100. 0.0 o1-mini
  101. 0.0 Llama 3.2 90B
  102. 0.0 o1
  103. 0.0 Reka Edge
  1. 1 GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
  2. 2 Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
  3. 3 Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
  4. 4 GPT-6 Astra max 52%
  5. 5 DeepSeek V4.1 Flash 1017 51%
  6. 6 Muse Spark Open 50%
  7. 7 Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
  8. 8 DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
  9. 9 Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
  10. 10 Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
  11. 11 Claude Opus 5.5 max 43%
  12. 12 GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
  13. 13 GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  14. 14 Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
  15. 15 GPT-6 Sol max 41%
  16. 16 Kimi K3.1 Pro 39%
  17. 17 Qwen3.5-122B-A10B 36.2%
  18. 18 Qwen3.5-27B 36.2%
  19. 19 DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
  20. 20 Qwen3.5-Omni-Plus 34.4%
  21. 21 Qwen 3.6 35B-A3B 34.4%
  22. 22 Qwen3.5-35B-A3B 34.1%
  23. 23 Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
  24. 24 Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
  25. 25 Qwen3.5-9B 31.1%
  26. 26 Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
  27. 27 Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
  28. 28 Qwen3-VL-235B-A22B 27.7%
  29. 29 Qwen3.5-4B 26.3%
  30. 30 GLM-4.6V 25.8%
  31. 31 GLM-4.5V 23.4%
  32. 32 Step 3 23%
  33. 33 Gemini 3.5 Flash 22%
  34. 34 Seed2.1 Pro 22%
  35. 35 GPT-5.6 Luna max 21%
  36. 36 Gemini 3 Pro 20%
  37. 37 GPT-6 Luna max 20%
  38. 38 GLM-4.1V-9B-Thinking 19.2%
  39. 39 GPT-5.6 Terra max 19%
  40. 40 Nemotron Nano 12B v2 VL 18.3%
  41. 41 GPT-5.2 medium 18%
  42. 42 Gemini 3.6 Flash high 18%
  43. 43 Qwen3.5-2B 17.1%
  44. 44 Grok 4.6 xhigh 17%
  45. 45 Inkling 2 Mini 17%
  46. 46 Gemini 3 Flash 15%
  47. 47 Claude Opus 4.7 xhigh 15%
  48. 48 Grok 4.5 high 15%
  49. 49 Gemini 3.5 Flash-Lite high 14%
  50. 50 Grok 4.3 high 13%
  51. 51 Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
  52. 52 Claude Sonnet 5 max 13%
  53. 53 Qwen3.5-0.8B 12.9%
  54. 54 Claude Opus 4.6 12%
  55. 55 Seed2.0 Pro 12%
  56. 56 Claude Sonnet 4.6 12%
  57. 57 o4-mini 11%
  58. 58 Seed1.8 11%
  59. 59 GPT-5.4 Mini 11%
  60. 60 Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
  61. 61 Grok 4.20 reasoning 10%
  62. 62 GPT-5 mini high 9.00%
  63. 63 Claude Opus 4.5 9.00%
  64. 64 Kimi K2.5 9.00%
  65. 65 Claude Opus 4.1 thinking 8.00%
  66. 66 GPT-5 medium 7.00%
  67. 67 Claude 3.7 Sonnet thinking 6.00%
  68. 68 GPT-4.5 6.00%
  69. 69 o3 6.00%
  70. 70 Claude Opus 4 thinking 6.00%
  71. 71 Claude Sonnet 4 6.00%
  72. 72 Grok 4 6.00%
  73. 73 GPT-5.1 medium 6.00%
  74. 74 Gemini 2.0 Flash Thinking Jan 2025 5.00%
  75. 75 Gemini 2.5 Pro Jun 2025 5.00%
  76. 76 Claude Sonnet 4.5 5.00%
  77. 77 Gemini 3.1 Flash-Lite 5.00%
  78. 78 Claude 3.5 Sonnet (October 2024) 3.00%
  79. 79 Pixtral Large 3.00%
  80. 80 Gemini 2.0 Flash Feb 2025 3.00%
  81. 81 Llama 4 Scout 3.00%
  82. 82 GPT-4.1 3.00%
  83. 83 GPT-5 nano high 3.00%
  84. 84 Gemini 2.5 Flash Sep 2025 3.00%
  85. 85 Gemini 1.0 Pro Vision 2.00%
  86. 86 Claude 3.5 Sonnet 2.00%
  87. 87 GPT-4o mini 2.00%
  88. 88 Gemini 1.5 Flash Sep 2024 2.00%
  89. 89 Qwen2-VL-72B-Instruct original release evaluation 2.00%
  90. 90 Gemini 1.5 Pro Sep 2024 2.00%
  91. 91 Llama 4 Maverick 2.00%
  92. 92 Seed1.5-VL 2.00%
  93. 93 Gemini 2.5 Flash Jun 2025 2.00%
  94. 94 Claude 3 Sonnet 1.00%
  95. 95 NVLM-D-72B 1.00%
  96. 96 GPT-4o Mar 2025, externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
  97. 97 STEP3-VL-10B 1.00%
  98. 98 Claude 3 Opus 0%
  99. 99 Claude 3 Haiku 0%
  100. 100 o1-mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 0%
  101. 101 Llama 3.2 90B 0%
  102. 102 o1 0%
  103. 103 Reka Edge 0%

Sources: Official ZeroBench leaderboard and labs’ model cards

#EntryScoreSettingDateSource
1 GPT-5.6 Sol 54.6% max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
2 Muse Spark 1.2 54% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Aug 2026 Official leaderboard
3 Claude Opus 5 52% max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. Jul 2026 Official leaderboard
4 GPT-6 Astra 52% max Sep 2026 Official leaderboard
5 Qwen 3.8 Max 49% w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. Aug 2026 Official leaderboard
6 DeepSeek V4.1 Flash 49% w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. Sep 2026 Official leaderboard
7 Claude Fable 5 46% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jun 2026 Official leaderboard
8 Muse Spark 1.1 46% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
9 Claude Opus 5.5 43% max Sep 2026 Official leaderboard
10 GPT-5.4 41% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Mar 2026 Official leaderboard
11 GPT-5.5 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Apr 2026 Official leaderboard
12 Kimi K3 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jul 2026 Official leaderboard
13 GPT-6 Sol 41% max Sep 2026 Official leaderboard
14 DeepSeek-V4-Flash-Vision-Exp 35% externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. Aug 2026 Official leaderboard
15 Claude Opus 4.8 34% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. May 2026 Official leaderboard
16 Muse Spark 33% w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. Apr 2026 Official leaderboard
17 Gemini 3.7 Flash 30% high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. Aug 2026 Official leaderboard
18 Gemini 3.1 Pro 29% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Feb 2026 Official leaderboard
19 Gemini 3.5 Flash 22% May 2026 Official leaderboard
20 GPT-5.6 Luna 21% max Jul 2026 Official leaderboard
21 Gemini 3 Pro 20% Nov 2025 Official leaderboard
22 GPT-6 Luna 20% max Sep 2026 Official leaderboard
23 dots3-note Preview 19% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge; values from card image. Official leaderboard
24 GPT-5.6 Terra 19% max Jul 2026 Official leaderboard
25 Seed2.1 Turbo 18% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Official leaderboard
26 GPT-5.2 18% medium Dec 2025 Official leaderboard
27 Gemini 3.6 Flash 18% high Jul 2026 Official leaderboard
28 Grok 4.6 17% xhigh Aug 2026 Official leaderboard
29 Gemini 3 Flash 15% Dec 2025 Official leaderboard
30 Claude Opus 4.7 15% xhigh Apr 2026 Official leaderboard
31 Grok 4.5 15% high Jul 2026 Official leaderboard
32 Gemini 3.5 Flash-Lite 14% high Jul 2026 Official leaderboard
33 Grok 4.3 13% high Apr 2026 Official leaderboard
34 Qwen3.7-Plus 13% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jun 2026 Official leaderboard
35 Claude Sonnet 5 13% max Jun 2026 Official leaderboard
36 Claude Opus 4.6 12% Feb 2026 Official leaderboard
37 Claude Sonnet 4.6 12% Feb 2026 Official leaderboard
38 o4-mini 11% Apr 2025 Official leaderboard
39 GPT-5.4 Mini 11% Mar 2026 Official leaderboard
40 Inkling-Small 11% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jul 2026 Official leaderboard
41 Grok 4.20 10% reasoning Feb 2026 Official leaderboard
42 GPT-5 mini 9.00% high Aug 2025 Official leaderboard
43 Claude Opus 4.5 9.00% Nov 2025 Official leaderboard
44 Claude Opus 4.1 8.00% thinking Aug 2025 Official leaderboard
45 GPT-5 7.00% medium Aug 2025 Official leaderboard
46 Claude 3.7 Sonnet 6.00% thinking Feb 2025 Official leaderboard
47 GPT-4.5 6.00% Feb 2025 Official leaderboard
48 o3 6.00% Apr 2025 Official leaderboard
49 Claude Opus 4 6.00% thinking May 2025 Official leaderboard
50 Claude Sonnet 4 6.00% May 2025 Official leaderboard
51 Grok 4 6.00% Jul 2025 Official leaderboard
52 GPT-5.1 6.00% medium Nov 2025 Official leaderboard
53 Gemini 2.0 Flash Thinking Jan 2025 5.00% Jan 2025 Official leaderboard
54 Gemini 2.5 Pro Jun 2025 5.00% Jun 2025 Official leaderboard
55 Claude Sonnet 4.5 5.00% Sep 2025 Official leaderboard
56 Gemini 3.1 Flash-Lite 5.00% Mar 2026 Official leaderboard
57 QVQ 4.00% original release evaluation Official leaderboard
58 Claude 3.5 Sonnet (October 2024) 3.00% Oct 2024 Official leaderboard
59 Pixtral Large 3.00% Nov 2024 Official leaderboard
60 Gemini 2.0 Flash Feb 2025 3.00% Feb 2025 Official leaderboard
61 Llama 4 Scout 3.00% Apr 2025 Official leaderboard
62 GPT-4.1 3.00% Apr 2025 Official leaderboard
63 GPT-5 nano 3.00% high Aug 2025 Official leaderboard
64 Gemini 2.5 Flash Sep 2025 3.00% Sep 2025 Official leaderboard
65 Gemini 1.0 Pro Vision 2.00% Jan 2024 Official leaderboard
66 Claude 3.5 Sonnet 2.00% Jun 2024 Official leaderboard
67 GPT-4o mini 2.00% Jul 2024 Official leaderboard
68 Gemini 1.5 Flash Sep 2024 2.00% Aug 2024 Official leaderboard
69 Qwen2-VL-72B-Instruct 2.00% original release evaluation Aug 2024 Official leaderboard
70 Gemini 1.5 Pro Sep 2024 2.00% Sep 2024 Official leaderboard
71 Llama 4 Maverick 2.00% Apr 2025 Official leaderboard
72 Claude 3 Sonnet 1.00% Feb 2024 Official leaderboard
73 NVLM-D-72B 1.00% Sep 2024 Official leaderboard
74 GPT-4o Mar 2025 1.00% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Mar 2025 Official leaderboard
75 Claude 3 Opus 0% Feb 2024 Official leaderboard
76 Claude 3 Haiku 0% Mar 2024 Official leaderboard
77 o1-mini 0% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Sep 2024 Official leaderboard
78 Llama 3.2 90B 0% Sep 2024 Official leaderboard
79 o1 0% Dec 2024 Official leaderboard
80 Reka Edge 0% Mar 2026 Official leaderboard
#ModelScoreSettingDateSource
1 GPT-5.6 Sol 54.6% max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
2 Muse Spark 1.2 54% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Aug 2026 Official leaderboard
3 Claude Opus 5 52% max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. Jul 2026 Official leaderboard
4 GPT-6 Astra 52% max Sep 2026 Official leaderboard
5 Qwen 3.8 Max 49% w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. Aug 2026 Official leaderboard
6 DeepSeek V4.1 Flash 49% w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. Sep 2026 Official leaderboard
7 Claude Fable 5 46% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jun 2026 Official leaderboard
8 Muse Spark 1.1 46% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
9 Claude Opus 5.5 43% max Sep 2026 Official leaderboard
10 GPT-5.4 41% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Mar 2026 Official leaderboard
11 GPT-5.5 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Apr 2026 Official leaderboard
12 Kimi K3 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jul 2026 Official leaderboard
13 GPT-6 Sol 41% max Sep 2026 Official leaderboard
14 DeepSeek-V4-Flash-Vision-Exp 35% externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. Aug 2026 Official leaderboard
15 Claude Opus 4.8 34% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. May 2026 Official leaderboard
16 Muse Spark 33% w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. Apr 2026 Official leaderboard
17 Gemini 3.7 Flash 30% high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. Aug 2026 Official leaderboard
18 Gemini 3.1 Pro 29% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Feb 2026 Official leaderboard
19 Gemini 3.5 Flash 22% May 2026 Official leaderboard
20 GPT-5.6 Luna 21% max Jul 2026 Official leaderboard
21 Gemini 3 Pro 20% Nov 2025 Official leaderboard
22 GPT-6 Luna 20% max Sep 2026 Official leaderboard
23 GPT-5.6 Terra 19% max Jul 2026 Official leaderboard
24 GPT-5.2 18% medium Dec 2025 Official leaderboard
25 Gemini 3.6 Flash 18% high Jul 2026 Official leaderboard
26 Grok 4.6 17% xhigh Aug 2026 Official leaderboard
27 Gemini 3 Flash 15% Dec 2025 Official leaderboard
28 Claude Opus 4.7 15% xhigh Apr 2026 Official leaderboard
29 Grok 4.5 15% high Jul 2026 Official leaderboard
30 Gemini 3.5 Flash-Lite 14% high Jul 2026 Official leaderboard
31 Grok 4.3 13% high Apr 2026 Official leaderboard
32 Qwen3.7-Plus 13% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jun 2026 Official leaderboard
33 Claude Sonnet 5 13% max Jun 2026 Official leaderboard
34 Claude Opus 4.6 12% Feb 2026 Official leaderboard
35 Claude Sonnet 4.6 12% Feb 2026 Official leaderboard
36 o4-mini 11% Apr 2025 Official leaderboard
37 GPT-5.4 Mini 11% Mar 2026 Official leaderboard
38 Inkling-Small 11% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jul 2026 Official leaderboard
39 Grok 4.20 10% reasoning Feb 2026 Official leaderboard
40 GPT-5 mini 9.00% high Aug 2025 Official leaderboard
41 Claude Opus 4.5 9.00% Nov 2025 Official leaderboard
42 Claude Opus 4.1 8.00% thinking Aug 2025 Official leaderboard
43 GPT-5 7.00% medium Aug 2025 Official leaderboard
44 Claude 3.7 Sonnet 6.00% thinking Feb 2025 Official leaderboard
45 GPT-4.5 6.00% Feb 2025 Official leaderboard
46 o3 6.00% Apr 2025 Official leaderboard
47 Claude Opus 4 6.00% thinking May 2025 Official leaderboard
48 Claude Sonnet 4 6.00% May 2025 Official leaderboard
49 Grok 4 6.00% Jul 2025 Official leaderboard
50 GPT-5.1 6.00% medium Nov 2025 Official leaderboard
51 Gemini 2.0 Flash Thinking Jan 2025 5.00% Jan 2025 Official leaderboard
52 Gemini 2.5 Pro Jun 2025 5.00% Jun 2025 Official leaderboard
53 Claude Sonnet 4.5 5.00% Sep 2025 Official leaderboard
54 Gemini 3.1 Flash-Lite 5.00% Mar 2026 Official leaderboard
55 Claude 3.5 Sonnet (October 2024) 3.00% Oct 2024 Official leaderboard
56 Pixtral Large 3.00% Nov 2024 Official leaderboard
57 Gemini 2.0 Flash Feb 2025 3.00% Feb 2025 Official leaderboard
58 Llama 4 Scout 3.00% Apr 2025 Official leaderboard
59 GPT-4.1 3.00% Apr 2025 Official leaderboard
60 GPT-5 nano 3.00% high Aug 2025 Official leaderboard
61 Gemini 2.5 Flash Sep 2025 3.00% Sep 2025 Official leaderboard
62 Gemini 1.0 Pro Vision 2.00% Jan 2024 Official leaderboard
63 Claude 3.5 Sonnet 2.00% Jun 2024 Official leaderboard
64 GPT-4o mini 2.00% Jul 2024 Official leaderboard
65 Gemini 1.5 Flash Sep 2024 2.00% Aug 2024 Official leaderboard
66 Qwen2-VL-72B-Instruct 2.00% original release evaluation Aug 2024 Official leaderboard
67 Gemini 1.5 Pro Sep 2024 2.00% Sep 2024 Official leaderboard
68 Llama 4 Maverick 2.00% Apr 2025 Official leaderboard
69 Claude 3 Sonnet 1.00% Feb 2024 Official leaderboard
70 NVLM-D-72B 1.00% Sep 2024 Official leaderboard
71 GPT-4o Mar 2025 1.00% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Mar 2025 Official leaderboard
72 Claude 3 Opus 0% Feb 2024 Official leaderboard
73 Claude 3 Haiku 0% Mar 2024 Official leaderboard
74 o1-mini 0% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Sep 2024 Official leaderboard
75 Llama 3.2 90B 0% Sep 2024 Official leaderboard
76 o1 0% Dec 2024 Official leaderboard
77 Reka Edge 0% Mar 2026 Official leaderboard
#ModelScoreSettingDateSource
1 GPT-5.6 Sol 54.6% max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
2 Muse Spark 1.2 54% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Aug 2026 Official leaderboard
3 Claude Opus 5 52% max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. Jul 2026 Official leaderboard
4 GPT-6 Astra 52% max Sep 2026 Official leaderboard
5 DeepSeek V4.1 Flash 1017 51% Oct 2026 Lab model card
6 Muse Spark Open 50% Oct 2026 Lab model card
7 Qwen 3.8 Max 49% w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. Aug 2026 Official leaderboard
8 DeepSeek V4.1 Flash 49% w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. Sep 2026 Official leaderboard
9 Claude Fable 5 46% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jun 2026 Official leaderboard
10 Muse Spark 1.1 46% xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. Jul 2026 Official leaderboard
11 Claude Opus 5.5 43% max Sep 2026 Official leaderboard
12 GPT-5.4 41% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Mar 2026 Official leaderboard
13 GPT-5.5 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Apr 2026 Official leaderboard
14 Kimi K3 41% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. Jul 2026 Official leaderboard
15 GPT-6 Sol 41% max Sep 2026 Official leaderboard
16 Kimi K3.1 Pro 39% Oct 2026 Lab model card
17 Qwen3.5-122B-A10B 36.2% Feb 2026 Lab model card
18 Qwen3.5-27B 36.2% Feb 2026 Lab model card
19 DeepSeek-V4-Flash-Vision-Exp 35% externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. Aug 2026 Official leaderboard
20 Qwen3.5-Omni-Plus 34.4% Mar 2026 Lab model card
21 Qwen 3.6 35B-A3B 34.4% Apr 2026 Lab model card
22 Qwen3.5-35B-A3B 34.1% Feb 2026 Lab model card
23 Claude Opus 4.8 34% w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. May 2026 Official leaderboard
24 Muse Spark 33% w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. Apr 2026 Official leaderboard
25 Qwen3.5-9B 31.1% Mar 2026 Lab model card
26 Gemini 3.7 Flash 30% high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. Aug 2026 Official leaderboard
27 Gemini 3.1 Pro 29% w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. Feb 2026 Official leaderboard
28 Qwen3-VL-235B-A22B 27.7% Sep 2025 Lab model card
29 Qwen3.5-4B 26.3% Mar 2026 Lab model card
30 GLM-4.6V 25.8% Dec 2025 Lab model card
31 GLM-4.5V 23.4% Aug 2025 Lab model card
32 Step 3 23% Jul 2025 Lab model card
33 Gemini 3.5 Flash 22% May 2026 Official leaderboard
34 Seed2.1 Pro 22% Jun 2026 Lab model card
35 GPT-5.6 Luna 21% max Jul 2026 Official leaderboard
36 Gemini 3 Pro 20% Nov 2025 Official leaderboard
37 GPT-6 Luna 20% max Sep 2026 Official leaderboard
38 GLM-4.1V-9B-Thinking 19.2% Jul 2025 Lab model card
39 GPT-5.6 Terra 19% max Jul 2026 Official leaderboard
40 Nemotron Nano 12B v2 VL 18.3% Nov 2025 Lab model card
41 GPT-5.2 18% medium Dec 2025 Official leaderboard
42 Gemini 3.6 Flash 18% high Jul 2026 Official leaderboard
43 Qwen3.5-2B 17.1% Mar 2026 Lab model card
44 Grok 4.6 17% xhigh Aug 2026 Official leaderboard
45 Inkling 2 Mini 17% Nov 2026 Lab model card
46 Gemini 3 Flash 15% Dec 2025 Official leaderboard
47 Claude Opus 4.7 15% xhigh Apr 2026 Official leaderboard
48 Grok 4.5 15% high Jul 2026 Official leaderboard
49 Gemini 3.5 Flash-Lite 14% high Jul 2026 Official leaderboard
50 Grok 4.3 13% high Apr 2026 Official leaderboard
51 Qwen3.7-Plus 13% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jun 2026 Official leaderboard
52 Claude Sonnet 5 13% max Jun 2026 Official leaderboard
53 Qwen3.5-0.8B 12.9% Mar 2026 Lab model card
54 Claude Opus 4.6 12% Feb 2026 Official leaderboard
55 Seed2.0 Pro 12% Feb 2026 Lab model card
56 Claude Sonnet 4.6 12% Feb 2026 Official leaderboard
57 o4-mini 11% Apr 2025 Official leaderboard
58 Seed1.8 11% Dec 2025 Lab model card
59 GPT-5.4 Mini 11% Mar 2026 Official leaderboard
60 Inkling-Small 11% externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. Jul 2026 Official leaderboard
61 Grok 4.20 10% reasoning Feb 2026 Official leaderboard
62 GPT-5 mini 9.00% high Aug 2025 Official leaderboard
63 Claude Opus 4.5 9.00% Nov 2025 Official leaderboard
64 Kimi K2.5 9.00% Jan 2026 Lab model card
65 Claude Opus 4.1 8.00% thinking Aug 2025 Official leaderboard
66 GPT-5 7.00% medium Aug 2025 Official leaderboard
67 Claude 3.7 Sonnet 6.00% thinking Feb 2025 Official leaderboard
68 GPT-4.5 6.00% Feb 2025 Official leaderboard
69 o3 6.00% Apr 2025 Official leaderboard
70 Claude Opus 4 6.00% thinking May 2025 Official leaderboard
71 Claude Sonnet 4 6.00% May 2025 Official leaderboard
72 Grok 4 6.00% Jul 2025 Official leaderboard
73 GPT-5.1 6.00% medium Nov 2025 Official leaderboard
74 Gemini 2.0 Flash Thinking Jan 2025 5.00% Jan 2025 Official leaderboard
75 Gemini 2.5 Pro Jun 2025 5.00% Jun 2025 Official leaderboard
76 Claude Sonnet 4.5 5.00% Sep 2025 Official leaderboard
77 Gemini 3.1 Flash-Lite 5.00% Mar 2026 Official leaderboard
78 Claude 3.5 Sonnet (October 2024) 3.00% Oct 2024 Official leaderboard
79 Pixtral Large 3.00% Nov 2024 Official leaderboard
80 Gemini 2.0 Flash Feb 2025 3.00% Feb 2025 Official leaderboard
81 Llama 4 Scout 3.00% Apr 2025 Official leaderboard
82 GPT-4.1 3.00% Apr 2025 Official leaderboard
83 GPT-5 nano 3.00% high Aug 2025 Official leaderboard
84 Gemini 2.5 Flash Sep 2025 3.00% Sep 2025 Official leaderboard
85 Gemini 1.0 Pro Vision 2.00% Jan 2024 Official leaderboard
86 Claude 3.5 Sonnet 2.00% Jun 2024 Official leaderboard
87 GPT-4o mini 2.00% Jul 2024 Official leaderboard
88 Gemini 1.5 Flash Sep 2024 2.00% Aug 2024 Official leaderboard
89 Qwen2-VL-72B-Instruct 2.00% original release evaluation Aug 2024 Official leaderboard
90 Gemini 1.5 Pro Sep 2024 2.00% Sep 2024 Official leaderboard
91 Llama 4 Maverick 2.00% Apr 2025 Official leaderboard
92 Seed1.5-VL 2.00% May 2025 Lab model card
93 Gemini 2.5 Flash Jun 2025 2.00% Jul 2025 Lab model card
94 Claude 3 Sonnet 1.00% Feb 2024 Official leaderboard
95 NVLM-D-72B 1.00% Sep 2024 Official leaderboard
96 GPT-4o Mar 2025 1.00% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Mar 2025 Official leaderboard
97 STEP3-VL-10B 1.00% Jan 2026 Lab model card
98 Claude 3 Opus 0% Feb 2024 Official leaderboard
99 Claude 3 Haiku 0% Mar 2024 Official leaderboard
100 o1-mini 0% externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. Sep 2024 Official leaderboard
101 Llama 3.2 90B 0% Sep 2024 Official leaderboard
102 o1 0% Dec 2024 Official leaderboard
103 Reka Edge 0% Mar 2026 Official leaderboard

Labs report scores for 32 models in their cards. Switch the scores above to rank them together.

Results plot: tokens per question

Official runs only: pass@5 on the 100 main questions against the mean total (input and output) tokens per question, with the Pareto frontier. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 1001,00010K100K Mean total tokens per question (tokens), log scale, lower is better pass@5 GPT-6 Astra (max)GPT-5.2 (medium)GPT-4.5Gemini 2.5 ProGemini 1.0 Pro VisionReka Edge 0% 20% 40% 60% 1001,00010K100K Mean total tokens per question (tokens), log scale, lower is better pass@5 GPT-6 Astra (max)GPT-5.2 (medium)GPT-4.5Gemini 2.5 ProGemini 1.0 Pro VisionReka Edge
ModelSettingpass@5Mean total tokens per question
GPT-6 Astra max 52% 16,763 tokens
Claude Opus 5.5 max 43% 58,573 tokens
GPT-6 Sol max 41% 22,672 tokens
GPT-5.6 Sol max 30% 31,065 tokens
Claude Opus 5 max 26% 29,995 tokens
Claude Fable 5 max 24% 23,151 tokens
GPT-5.5 xhigh 23% 20,174 tokens
GPT-5.4 xhigh 23% 22,315 tokens
Gemini 3.5 Flash 22% 21,955 tokens
GPT-5.6 Luna max 21% 52,301 tokens
Gemini 3 Pro 20% 19,825 tokens
GPT-6 Luna max 20% 41,067 tokens
Gemini 3.1 Pro 19% 21,764 tokens
GPT-5.6 Terra max 19% 42,219 tokens
GPT-5.2 medium 18% 5,804 tokens
Gemini 3.6 Flash high 18% 20,539 tokens
Claude Opus 4.8 18% 35,501 tokens
Grok 4.6 xhigh 17% 26,165 tokens
Claude Opus 4.7 xhigh 15% 15,895 tokens
Gemini 3 Flash 15% 23,650 tokens
Grok 4.5 high 15% 25,758 tokens
Gemini 3.5 Flash-Lite high 14% 12,939 tokens
Grok 4.3 high 13% 19,309 tokens
Claude Sonnet 5 max 13% 72,104 tokens
Claude Sonnet 4.6 12% 24,824 tokens
Claude Opus 4.6 12% 30,000 tokens
o4-mini 11% 18,025 tokens
GPT-5.4 Mini 11% 44,650 tokens
Grok 4.20 non-reasoning 10% 12,893 tokens
Grok 4.20 reasoning 10% 20,485 tokens
GPT-5 mini high 9% 12,067 tokens
Claude Opus 4.5 9% 20,194 tokens
Claude Opus 4.1 thinking 8% 8,001 tokens
GPT-5 medium 7% 6,040 tokens
GPT-4.5 6% 1,509 tokens
Claude Sonnet 4 6% 2,179 tokens
GPT-5.1 medium 6% 6,206 tokens
o3 6% 6,454 tokens
Claude Opus 4 thinking 6% 7,486 tokens
Claude Sonnet 4 thinking 6% 12,654 tokens
Claude 3.7 Sonnet thinking 6% 13,853 tokens
Grok 4 6% 22,343 tokens
Gemini 2.5 Pro Jun 2025 5% 1,480 tokens
Claude Opus 4.1 5% 2,222 tokens
Claude Sonnet 4.5 5% 2,321 tokens
Gemini 2.0 Flash Thinking Jan 2025 5% 2,919 tokens
Gemini 2.0 Flash Thinking Jan 2025 original release evaluation 5% 2,919 tokens
Gemini 3.1 Flash-Lite 5% 14,139 tokens
Claude Opus 4 4% 2,181 tokens
QVQ original release evaluation 4% 4,138 tokens
GPT-4.1 3% 1,650 tokens
Claude 3.5 Sonnet (October 2024) 3% 1,905 tokens
Gemini 2.0 Flash Feb 2025 3% 2,798 tokens
Gemini 2.0 Flash Feb 2025 original release evaluation 3% 2,798 tokens
Llama 4 Scout 3% 3,150 tokens
Gemini 2.5 Flash Sep 2025 3% 3,226 tokens
Pixtral Large 3% 4,107 tokens
Pixtral Large original release evaluation 3% 4,107 tokens
GPT-5 nano high 3% 10,351 tokens
Claude Sonnet 4.5 thinking 3% 14,680 tokens
Gemini 1.0 Pro Vision 2% 610 tokens
Gemini 1.0 Pro Vision original release evaluation 2% 610 tokens
Gemini 1.5 Pro Sep 2024 2% 707 tokens
Gemini 1.5 Pro Sep 2024 original release evaluation 2% 707 tokens
Gemini 1.5 Flash Sep 2024 2% 728 tokens
Gemini 1.5 Flash Sep 2024 original release evaluation 2% 728 tokens
Llama 4 Maverick 2% 985 tokens
GPT-4o mini 2% 1,546 tokens
GPT-4o mini original release evaluation 2% 1,546 tokens
Claude 3.5 Sonnet (October 2024) original release evaluation 2% 1,905 tokens
Claude 3.5 Sonnet 2% 1,940 tokens
Claude 3.7 Sonnet 2% 1,981 tokens
Qwen2-VL-72B-Instruct original release evaluation 2% 3,313 tokens
QVQ 2% 4,138 tokens
Claude 3 Sonnet 1% 1,920 tokens
Claude 3 Sonnet original release evaluation 1% 1,920 tokens
Claude 3.5 Sonnet original release evaluation 1% 1,940 tokens
Qwen2-VL-72B-Instruct 1% 3,313 tokens
NVLM-D-72B 1% 3,728 tokens
NVLM-D-72B original release evaluation 1% 3,728 tokens
Reka Edge 0% 530 tokens
Reka Edge original release evaluation 0% 530 tokens
Llama 3.2 90B 0% 989 tokens
Llama 3.2 90B original release evaluation 0% 989 tokens
GPT-4o Mar 2025 0% 1,582 tokens
GPT-4o Mar 2025 original release evaluation 0% 1,582 tokens
Claude 3 Opus 0% 1,915 tokens
Claude 3 Opus original release evaluation 0% 1,915 tokens
Claude 3 Haiku 0% 1,919 tokens
Claude 3 Haiku original release evaluation 0% 1,919 tokens
o1 0% 8,206 tokens
o1 original release evaluation 0% 8,206 tokens

Results plot: tokens per question

Official runs only: pass@5 on the 100 main questions against the mean total (input and output) tokens per question, with the Pareto frontier. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 1001,00010K100K Mean total tokens per question (tokens), log scale, lower is better pass@5 GPT-6 Astra (max)GPT-5.2 (medium)GPT-4.5Gemini 2.5 ProGemini 1.0 Pro VisionReka Edge 0% 20% 40% 60% 1001,00010K100K Mean total tokens per question (tokens), log scale, lower is better pass@5 GPT-6 Astra (max)GPT-5.2 (medium)GPT-4.5Gemini 2.5 ProGemini 1.0 Pro VisionReka Edge
ModelSettingpass@5Mean total tokens per question
GPT-6 Astra max 52% 16,763 tokens
Claude Opus 5.5 max 43% 58,573 tokens
GPT-6 Sol max 41% 22,672 tokens
GPT-5.6 Sol max 30% 31,065 tokens
Claude Opus 5 max 26% 29,995 tokens
Claude Fable 5 max 24% 23,151 tokens
GPT-5.5 xhigh 23% 20,174 tokens
GPT-5.4 xhigh 23% 22,315 tokens
Gemini 3.5 Flash 22% 21,955 tokens
GPT-5.6 Luna max 21% 52,301 tokens
Gemini 3 Pro 20% 19,825 tokens
GPT-6 Luna max 20% 41,067 tokens
Gemini 3.1 Pro 19% 21,764 tokens
GPT-5.6 Terra max 19% 42,219 tokens
GPT-5.2 medium 18% 5,804 tokens
Gemini 3.6 Flash high 18% 20,539 tokens
Claude Opus 4.8 18% 35,501 tokens
Grok 4.6 xhigh 17% 26,165 tokens
Claude Opus 4.7 xhigh 15% 15,895 tokens
Gemini 3 Flash 15% 23,650 tokens
Grok 4.5 high 15% 25,758 tokens
Gemini 3.5 Flash-Lite high 14% 12,939 tokens
Grok 4.3 high 13% 19,309 tokens
Claude Sonnet 5 max 13% 72,104 tokens
Claude Sonnet 4.6 12% 24,824 tokens
Claude Opus 4.6 12% 30,000 tokens
o4-mini 11% 18,025 tokens
GPT-5.4 Mini 11% 44,650 tokens
Grok 4.20 non-reasoning 10% 12,893 tokens
Grok 4.20 reasoning 10% 20,485 tokens
GPT-5 mini high 9% 12,067 tokens
Claude Opus 4.5 9% 20,194 tokens
Claude Opus 4.1 thinking 8% 8,001 tokens
GPT-5 medium 7% 6,040 tokens
GPT-4.5 6% 1,509 tokens
Claude Sonnet 4 6% 2,179 tokens
GPT-5.1 medium 6% 6,206 tokens
o3 6% 6,454 tokens
Claude Opus 4 thinking 6% 7,486 tokens
Claude Sonnet 4 thinking 6% 12,654 tokens
Claude 3.7 Sonnet thinking 6% 13,853 tokens
Grok 4 6% 22,343 tokens
Gemini 2.5 Pro Jun 2025 5% 1,480 tokens
Claude Opus 4.1 5% 2,222 tokens
Claude Sonnet 4.5 5% 2,321 tokens
Gemini 2.0 Flash Thinking Jan 2025 5% 2,919 tokens
Gemini 2.0 Flash Thinking Jan 2025 original release evaluation 5% 2,919 tokens
Gemini 3.1 Flash-Lite 5% 14,139 tokens
Claude Opus 4 4% 2,181 tokens
GPT-4.1 3% 1,650 tokens
Claude 3.5 Sonnet (October 2024) 3% 1,905 tokens
Gemini 2.0 Flash Feb 2025 3% 2,798 tokens
Gemini 2.0 Flash Feb 2025 original release evaluation 3% 2,798 tokens
Llama 4 Scout 3% 3,150 tokens
Gemini 2.5 Flash Sep 2025 3% 3,226 tokens
Pixtral Large 3% 4,107 tokens
Pixtral Large original release evaluation 3% 4,107 tokens
GPT-5 nano high 3% 10,351 tokens
Claude Sonnet 4.5 thinking 3% 14,680 tokens
Gemini 1.0 Pro Vision 2% 610 tokens
Gemini 1.0 Pro Vision original release evaluation 2% 610 tokens
Gemini 1.5 Pro Sep 2024 2% 707 tokens
Gemini 1.5 Pro Sep 2024 original release evaluation 2% 707 tokens
Gemini 1.5 Flash Sep 2024 2% 728 tokens
Gemini 1.5 Flash Sep 2024 original release evaluation 2% 728 tokens
Llama 4 Maverick 2% 985 tokens
GPT-4o mini 2% 1,546 tokens
GPT-4o mini original release evaluation 2% 1,546 tokens
Claude 3.5 Sonnet (October 2024) original release evaluation 2% 1,905 tokens
Claude 3.5 Sonnet 2% 1,940 tokens
Claude 3.7 Sonnet 2% 1,981 tokens
Qwen2-VL-72B-Instruct original release evaluation 2% 3,313 tokens
Claude 3 Sonnet 1% 1,920 tokens
Claude 3 Sonnet original release evaluation 1% 1,920 tokens
Claude 3.5 Sonnet original release evaluation 1% 1,940 tokens
Qwen2-VL-72B-Instruct 1% 3,313 tokens
NVLM-D-72B 1% 3,728 tokens
NVLM-D-72B original release evaluation 1% 3,728 tokens
Reka Edge 0% 530 tokens
Reka Edge original release evaluation 0% 530 tokens
Llama 3.2 90B 0% 989 tokens
Llama 3.2 90B original release evaluation 0% 989 tokens
GPT-4o Mar 2025 0% 1,582 tokens
GPT-4o Mar 2025 original release evaluation 0% 1,582 tokens
Claude 3 Opus 0% 1,915 tokens
Claude 3 Opus original release evaluation 0% 1,915 tokens
Claude 3 Haiku 0% 1,919 tokens
Claude 3 Haiku original release evaluation 0% 1,919 tokens
o1 0% 8,206 tokens
o1 original release evaluation 0% 8,206 tokens

Of runs with a cost, the best scores 52% for $0.56 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind Anthropic OpenAI Cost frontier One model at each effort level
0% 20% 40% 60% $0.0001 $0.001 $0.01 $0.10 $1.00 $10 Cost per task, log scale GPT-6 Astra, max GPT-6 Sol, max GPT-6 Luna, max Claude Sonnet 4 Gemini 2.0 Flash 0% 20% 40% 60% $0.0001 $0.001 $0.01 $0.10 $1.00 $10 Cost per task, log scale GPT-6 Astra, max GPT-6 Sol, max

Source: Official ZeroBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
GPT-5.6 Sol max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 53%
Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
GPT-6 Astra max 52% $0.56
Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
Claude Opus 5 max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Opus 5 number; values from chart image. 47.5%
Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
Claude Opus 5.5 max 43% $1.11
GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
Kimi K3 w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; matches Moonshot’s own with-tools value; possibly relayed. 41%
GPT-6 Sol max 41% $0.17
GPT-5.6 Sol w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 35%
DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
Claude Opus 4.8 externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 34%
Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
GPT-5.6 Sol max 30% $0.76
Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
Claude Opus 5 max 26% $0.68
Claude Fable 5 max 24% $1.01
Kimi K3 externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge; independent re-run by dots-studio. 24%
Qwen 3.8 Max externally reported: Qwen3.8 release; 5 samplings. 24%
GPT-5.4 xhigh 23% $0.30
GPT-5.5 xhigh 23% $0.55
Claude Fable 5 externally reported: Kimi K3 report; 5 samplings. 23%
Kimi K3 externally reported: Kimi K3 report; 5 samplings. 23%
GPT-5.5 externally reported: Kimi K3 report; 5 samplings. 22%
Gemini 3.5 Flash 22% $0.18
Gemini 3.5 Flash externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 21%
GPT-5.6 Luna max 21% $0.28
Gemini 3 Pro 20% $0.22
GPT-6 Luna max 20% $0.02
Gemini 3.1 Pro 19% $0.25
GPT-5.6 Terra max 19% $0.55
GPT-5.2 medium 18% $0.06
Claude Opus 4.8 18% $0.81
Gemini 3.6 Flash high 18% $0.14
Claude Opus 4.8 externally reported: Kimi K3 report; 5 samplings. 17%
GPT-5.6 Sol externally reported: Kimi K3 report; 5 samplings. 17%
Grok 4.6 xhigh 17% $0.15
Gemini 3.1 Pro externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 16%
Gemini 3 Flash 15% $0.07
Claude Opus 4.7 xhigh 15% $0.32
Grok 4.5 high 15% $0.15
Gemini 3.5 Flash-Lite high 14% $0.03
Grok 4.3 high 13% $0.05
Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
Claude Sonnet 5 max 13% $0.69
Claude Opus 4.6 12% $0.72
Claude Sonnet 4.6 12% $0.35
o4-mini 11% $0.07
GPT-5.4 Mini 11% $0.19
Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
Grok 4.20 reasoning 10% $0.11
Grok 4.20 non-reasoning 10% $0.07
GPT-5 mini high 9.00% $0.02
Claude Opus 4.5 9.00% $0.47
Claude Opus 4.1 thinking 8.00% $0.50
GPT-5 medium 7.00% $0.05
Claude 3.7 Sonnet thinking 6.00% $0.19
GPT-4.5 6.00% $0.15
o3 6.00% $0.23
Claude Opus 4 thinking 6.00% $0.46
Claude Sonnet 4 6.00% $0.01
Claude Sonnet 4 thinking 6.00% $0.17
Grok 4 6.00% $0.31
GPT-5.1 medium 6.00% $0.05
Gemini 2.0 Flash Thinking Jan 2025 5.00%
Gemini 2.0 Flash Thinking Jan 2025 original release evaluation 5.00%
Gemini 2.5 Pro Jun 2025 5.00% $0.01
Claude Opus 4.1 5.00% $0.07
Claude Sonnet 4.5 5.00% $0.02
Gemini 3.1 Flash-Lite 5.00% $0.02
Claude Opus 4 4.00% $0.07
Claude 3.5 Sonnet (October 2024) 3.00% $0.009
Pixtral Large 3.00% $0.01
Pixtral Large original release evaluation 3.00% $0.01
Gemini 2.0 Flash Feb 2025 3.00% $0.0004
Gemini 2.0 Flash Feb 2025 original release evaluation 3.00% $0.0004
Llama 4 Scout 3.00% $0.0005
GPT-4.1 3.00% $0.007
GPT-5 nano high 3.00% $0.003
Gemini 2.5 Flash Sep 2025 3.00% $0.003
Claude Sonnet 4.5 thinking 3.00% $0.20
Gemini 1.0 Pro Vision 2.00% $0.0002
Gemini 1.0 Pro Vision original release evaluation 2.00% $0.0002
Claude 3.5 Sonnet 2.00% $0.009
GPT-4o mini 2.00% $0.0004
GPT-4o mini original release evaluation 2.00% $0.0004
Gemini 1.5 Flash Sep 2024 2.00% $6.05
Gemini 1.5 Flash Sep 2024 original release evaluation 2.00% $6.05
Qwen2-VL-72B-Instruct original release evaluation 2.00% $0.001
Gemini 1.5 Pro Sep 2024 2.00% $0.003
Gemini 1.5 Pro Sep 2024 original release evaluation 2.00% $0.003
Claude 3.5 Sonnet (October 2024) original release evaluation 2.00% $0.009
Claude 3.7 Sonnet 2.00% $0.01
Llama 4 Maverick 2.00% $0.0006
Claude 3 Sonnet 1.00% $0.009
Claude 3 Sonnet original release evaluation 1.00% $0.009
Claude 3.5 Sonnet original release evaluation 1.00% $0.009
GPT-4o mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
Qwen2-VL-72B-Instruct 1.00% $0.001
NVLM-D-72B 1.00%
NVLM-D-72B original release evaluation 1.00%
GPT-4o Mar 2025 externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
Claude 3 Opus 0% $0.05
Claude 3 Opus original release evaluation 0% $0.05
Claude 3 Haiku 0% $0.0008
Claude 3 Haiku original release evaluation 0% $0.0008
o1-mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 0%
Llama 3.2 90B 0% $0.0004
Llama 3.2 90B original release evaluation 0% $0.0004
o1 0% $0.45
o1 original release evaluation 0% $0.45
GPT-4o Mar 2025 0% $0.007
GPT-4o Mar 2025 original release evaluation 0% $0.007
Reka Edge 0% $5.30
Reka Edge original release evaluation 0% $5.30

Of runs with a cost, the best scores 52% for $0.56 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind Anthropic OpenAI Cost frontier One model at each effort level
0% 20% 40% 60% $0.0001 $0.001 $0.01 $0.10 $1.00 $10 Cost per task, log scale GPT-6 Astra, max GPT-6 Sol, max GPT-6 Luna, max Claude Sonnet 4 Gemini 2.0 Flash 0% 20% 40% 60% $0.0001 $0.001 $0.01 $0.10 $1.00 $10 Cost per task, log scale GPT-6 Astra, max GPT-6 Sol, max

Source: Official ZeroBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-5.6 Sol max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54.6%
Muse Spark 1.2 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 54%
GPT-5.6 Sol max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 53%
Claude Opus 5 max, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; comparison value, provenance not stated. 52%
GPT-6 Astra max 52% $0.56
Qwen 3.8 Max w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. 49%
DeepSeek V4.1 Flash w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. 49%
Claude Opus 5 max, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Opus 5 number; values from chart image. 47.5%
Claude Fable 5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 46%
Muse Spark 1.1 xhigh, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; values from chart image. 46%
Claude Opus 5.5 max 43% $1.11
GPT-5.4 w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 41%
GPT-5.5 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
Kimi K3 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 41%
Kimi K3 w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings; matches Moonshot’s own with-tools value; possibly relayed. 41%
GPT-6 Sol max 41% $0.17
GPT-5.6 Sol w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 35%
DeepSeek-V4-Flash-Vision-Exp externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 35%
Claude Opus 4.8 w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. 34%
Claude Opus 4.8 externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. 34%
Muse Spark w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. 33%
GPT-5.6 Sol max 30% $0.76
Gemini 3.7 Flash high, w/ tools; externally reported: Muse Spark 1.2 multimodal research post; Containerized coding+GUI tools; 5 samplings, graded by an LLM judge; first public Gemini 3.7 Flash number; values from chart image. 30%
Gemini 3.1 Pro w/ tools; externally reported: Muse Spark methodology; Sandboxed code-execution tools; 5 samplings, graded by an LLM judge. 29%
Claude Opus 5 max 26% $0.68
Claude Fable 5 max 24% $1.01
Kimi K3 externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge; independent re-run by dots-studio. 24%
Qwen 3.8 Max externally reported: Qwen3.8 release; 5 samplings. 24%
GPT-5.4 xhigh 23% $0.30
GPT-5.5 xhigh 23% $0.55
Claude Fable 5 externally reported: Kimi K3 report; 5 samplings. 23%
Kimi K3 externally reported: Kimi K3 report; 5 samplings. 23%
GPT-5.5 externally reported: Kimi K3 report; 5 samplings. 22%
Gemini 3.5 Flash 22% $0.18
Gemini 3.5 Flash externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 21%
GPT-5.6 Luna max 21% $0.28
Gemini 3 Pro 20% $0.22
GPT-6 Luna max 20% $0.02
Gemini 3.1 Pro 19% $0.25
GPT-5.6 Terra max 19% $0.55
GPT-5.2 medium 18% $0.06
Claude Opus 4.8 18% $0.81
Gemini 3.6 Flash high 18% $0.14
Claude Opus 4.8 externally reported: Kimi K3 report; 5 samplings. 17%
GPT-5.6 Sol externally reported: Kimi K3 report; 5 samplings. 17%
Grok 4.6 xhigh 17% $0.15
Gemini 3.1 Pro externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 16%
Gemini 3 Flash 15% $0.07
Claude Opus 4.7 xhigh 15% $0.32
Grok 4.5 high 15% $0.15
Gemini 3.5 Flash-Lite high 14% $0.03
Grok 4.3 high 13% $0.05
Qwen3.7-Plus externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 13%
Claude Sonnet 5 max 13% $0.69
Claude Opus 4.6 12% $0.72
Claude Sonnet 4.6 12% $0.35
o4-mini 11% $0.07
GPT-5.4 Mini 11% $0.19
Inkling-Small externally reported: dots3-note Preview model card; 5 samplings, graded by an LLM judge. 11%
Grok 4.20 reasoning 10% $0.11
Grok 4.20 non-reasoning 10% $0.07
GPT-5 mini high 9.00% $0.02
Claude Opus 4.5 9.00% $0.47
Claude Opus 4.1 thinking 8.00% $0.50
GPT-5 medium 7.00% $0.05
Claude 3.7 Sonnet thinking 6.00% $0.19
GPT-4.5 6.00% $0.15
o3 6.00% $0.23
Claude Opus 4 thinking 6.00% $0.46
Claude Sonnet 4 6.00% $0.01
Claude Sonnet 4 thinking 6.00% $0.17
Grok 4 6.00% $0.31
GPT-5.1 medium 6.00% $0.05
Gemini 2.0 Flash Thinking Jan 2025 5.00%
Gemini 2.0 Flash Thinking Jan 2025 original release evaluation 5.00%
Gemini 2.5 Pro Jun 2025 5.00% $0.01
Claude Opus 4.1 5.00% $0.07
Claude Sonnet 4.5 5.00% $0.02
Gemini 3.1 Flash-Lite 5.00% $0.02
Claude Opus 4 4.00% $0.07
Claude 3.5 Sonnet (October 2024) 3.00% $0.009
Pixtral Large 3.00% $0.01
Pixtral Large original release evaluation 3.00% $0.01
Gemini 2.0 Flash Feb 2025 3.00% $0.0004
Gemini 2.0 Flash Feb 2025 original release evaluation 3.00% $0.0004
Llama 4 Scout 3.00% $0.0005
GPT-4.1 3.00% $0.007
GPT-5 nano high 3.00% $0.003
Gemini 2.5 Flash Sep 2025 3.00% $0.003
Claude Sonnet 4.5 thinking 3.00% $0.20
Gemini 1.0 Pro Vision 2.00% $0.0002
Gemini 1.0 Pro Vision original release evaluation 2.00% $0.0002
Claude 3.5 Sonnet 2.00% $0.009
GPT-4o mini 2.00% $0.0004
GPT-4o mini original release evaluation 2.00% $0.0004
Gemini 1.5 Flash Sep 2024 2.00% $6.05
Gemini 1.5 Flash Sep 2024 original release evaluation 2.00% $6.05
Qwen2-VL-72B-Instruct original release evaluation 2.00% $0.001
Gemini 1.5 Pro Sep 2024 2.00% $0.003
Gemini 1.5 Pro Sep 2024 original release evaluation 2.00% $0.003
Claude 3.5 Sonnet (October 2024) original release evaluation 2.00% $0.009
Claude 3.7 Sonnet 2.00% $0.01
Llama 4 Maverick 2.00% $0.0006
Claude 3 Sonnet 1.00% $0.009
Claude 3 Sonnet original release evaluation 1.00% $0.009
Claude 3.5 Sonnet original release evaluation 1.00% $0.009
GPT-4o mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
Qwen2-VL-72B-Instruct 1.00% $0.001
NVLM-D-72B 1.00%
NVLM-D-72B original release evaluation 1.00%
GPT-4o Mar 2025 externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 1.00%
Claude 3 Opus 0% $0.05
Claude 3 Opus original release evaluation 0% $0.05
Claude 3 Haiku 0% $0.0008
Claude 3 Haiku original release evaluation 0% $0.0008
o1-mini externally reported: UK AISI inspect_evals ZeroBench README; Official-style pass@k protocol (1 greedy + 5 sampled); third-party harness run. 0%
Llama 3.2 90B 0% $0.0004
Llama 3.2 90B original release evaluation 0% $0.0004
o1 0% $0.45
o1 original release evaluation 0% $0.45
GPT-4o Mar 2025 0% $0.007
GPT-4o Mar 2025 original release evaluation 0% $0.007
Reka Edge 0% $5.30
Reka Edge original release evaluation 0% $5.30

8 models have held number one on the official leaderboard since its release in February 2025 12 models have held number one with claims included since its release in February 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Google DeepMind Anthropic OpenAI ByteDance Seed Z.ai StepFun Qwen Meta DeepSeek
Own leaderboard Not tested since Sep 2026
GPT-5.6 Sol 54.6%
Lab claims 9 leaders

Sources: Official ZeroBench leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) GPT-5.6 Sol Claude Fable 5 GPT-5.4 Qwen3.5-122B-A10B Gemini 3.1 Pro GLM-4.5V GLM-4.1V-9B-Thinking 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) GPT-5.6 Sol Claude Fable 5

Sources: Official ZeroBench leaderboard and labs’ model cards

6 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

ByteDance Seed
StepFun
Z.ai
Qwen
Moonshot AI
Meta
DeepSeek
Thinking Machines
DateLaunch postScoreAs the post writes it
Nov 20, 2026 Inkling 2 Mini 17 ZeroBench (main, original, 34.0 with Python)
Oct 24, 2026 Kimi K3.1 Pro 39 ZeroBench_main (avg@5 / pass@5) 29.8 / 39.0
Oct 17, 2026 DeepSeek V4.1 Flash 1017 51 ZeroBench-main (w/tools)
Oct 15, 2026 Muse Spark Open 50 ZeroBench (w/ tools)
Sep 10, 2026 DeepSeek V4.1 Flash 49 ZeroBench-main w/ tools (Pass@5)
Aug 21, 2026 DeepSeek-V4-Flash-Vision-Exp 35 ZeroBench (Pass@5)
Aug 3, 2026 Qwen 3.8 Max 49 Best of ZeroBench (Pass@5) (without CI / with CI) 24.0 / 49.0 / ZeroBench-Sub
Jul 16, 2026 Kimi K3 23 ZeroBench (pass@5) (without python)
Jun 23, 2026 Seed2.1 Pro 22 ZEROBench (w. Tool) 18.0 (22.0), main set
Apr 15, 2026 Qwen 3.6 35B-A3B 34.4 ZEROBench_sub
Apr 8, 2026 Muse Spark 33 ZeroBench (Multi-Step Visual Reasoning (pass@5) - With Python)
Mar 30, 2026 Qwen3.5-Omni-Plus 34.4 Best of ZEROBench 5 / ZEROBench_sub 34.4
Mar 2, 2026 Qwen3.5-0.8B 12.9 Best of ZEROBench / ZEROBench_sub 12.9/11.4
Mar 2, 2026 Qwen3.5-2B 17.1 Best of ZEROBench / ZEROBench_sub 17.1/18.6
Mar 2, 2026 Qwen3.5-4B 26.3 Best of ZEROBench / ZEROBench_sub 26.3
Mar 2, 2026 Qwen3.5-9B 31.1 Best of ZEROBench / ZEROBench_sub 31.1
Feb 24, 2026 Qwen3.5-35B-A3B 34.1 Best of ZEROBench / ZEROBench_sub 34.1
Feb 24, 2026 Qwen3.5-27B 36.2 Best of ZEROBench / ZEROBench_sub 36.2
Feb 24, 2026 Qwen3.5-122B-A10B 36.2 Best of ZEROBench 9 / ZEROBench_sub 36.2
Feb 16, 2026 Qwen3.5 397B-A17B 12 Best of ZEROBench / ZEROBench_sub
Jan 27, 2026 Kimi K2.5 9 ZeroBench
Dec 18, 2025 Seed1.8 11 ZeroBench (main)
Dec 8, 2025 GLM-4.6V 25.8 ZeroBench (sub)
Sep 23, 2025 Qwen3-VL-235B-A22B 27.7 Best of ZEROBench 4.0 / ZEROBench_Sub 27.7
Aug 11, 2025 GLM-4.5V 23.4 ZeroBench (sub)
Jul 31, 2025 Step 3 23 ZeroBench (sub)
May 12, 2025 Seed1.5-VL 2 ZeroBench (main) 2 (sub 30.8)

17 system cards report it, from 9 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Nov 20, 2026 Inkling 2 Mini model card 17 ZeroBench (main, original, 34.0 with Python) Capabilities
Sep 10, 2026 DeepSeek V4.1 Flash technical report 49 ZeroBench-main w/ tools (Pass@5) (Table 3) Capabilities
Jul 27, 2026 Kimi K3 technical report 23 ZeroBench-main (pass@5) without Python, cell 23.0 / 41.0, 5 runs (Table 2) Capabilities
Jun 23, 2026 Seed2.1 Pro model card 18 ZeroBench (main), no tools, 22.0 with tools. ZeroBench (sub) 49.4, 56.3 with tools (Table 9) Capabilities
Apr 17, 2026 Qwen3.5-Omni-Plus technical report 5 ZEROBench 5, ZEROBench_sub 34.4 (Table 6) Capabilities
Mar 21, 2026 Seed1.8 model card 11 ZeroBench (main) (Table 2) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 12 ZeroBench (main) 12.0, ZeroBench (sub) 47.6 (Table 8) Capabilities
Jan 30, 2026 Kimi K2.5 technical report 9 ZeroBench without tools, 11 with tools (Table 4) Capabilities
Jan 14, 2026 STEP3-VL-10B technical report 1 ZeroBench (main), SeRe, sub 27.54 (Tables 1 and 3). PaCoRe main 5.00, sub 29.94 Capabilities
Jan 1, 2026 GLM-4.6V technical report 25.8 ZeroBench (sub), GLM-4.6V thinking (Table 2) Capabilities
Nov 26, 2025 Qwen3-VL-235B-A22B technical report 4 ZeroBench (Table 2) Capabilities
Nov 6, 2025 Nemotron Nano 12B v2 VL technical report 18.3 ZeroBench (sub), reasoning on (Table 3). Reasoning off 15.0 Capabilities
Aug 13, 2025 GLM-4.5V technical report 23.4 ZeroBench (sub), GLM-4.5V thinking (Table 2) Capabilities
Jul 7, 2025 Gemini 2.5 Pro technical report 4.5 ZeroBench, Gemini as judge, average of 4 runs (Table 3) Capabilities
Jul 7, 2025 Gemini 2.5 Flash technical report 2 ZeroBench, Gemini as judge, average of 4 runs (Table 3) Capabilities
Jul 1, 2025 GLM-4.1V-9B-Thinking technical report 19.2 ZeroBench (sub), GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities
May 11, 2025 Seed1.5-VL technical report 2 ZeroBench (main) 2, ZeroBench (sub) 30.8, thinking (Table 6). Non-thinking main 0, sub 29.0 Capabilities

The lab's number comes out ahead in 0 of 2 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. 4 of them match, which usually means one side copied the other. Every benchmark.

Model Lab says Board says Gap What differs
Kimi K3 23% ZeroBench (pass@5) (without python) 41% Leaderboard, w/ tools; externally reported: Kimi K3 report; Python tools; 5 samplings. −18 pts Moonshot's 23.0 is the result without Python tools. The selected board result is its 41.0 run with Python tools.
Gemini 2.5 Pro Jun 2025 4.50% ZeroBench, Gemini as judge, average of 4 runs (Table 3) 5.00% Leaderboard −0.5 pts
Muse Spark 33% ZeroBench (Multi-Step Visual Reasoning (pass@5) - With Python) 33% Leaderboard, w/ tools; externally reported: Muse Spark methodology; Sandboxed Python tools; 5 samplings, graded by an LLM judge. Same
Qwen 3.8 Max 49% Best of ZeroBench (Pass@5) (without CI / with CI) 24.0 / 49.0 / ZeroBench-Sub 49% Leaderboard, w/ tools; externally reported: Qwen3.8 release; Code-interpreter tools; 5 samplings. Same
DeepSeek-V4-Flash-Vision-Exp 35% ZeroBench (Pass@5) 35% Leaderboard, externally reported: DeepSeek-V4-Flash-Vision-Exp release note; Tool use unclear; 5 samplings; values from chart image. Same
DeepSeek V4.1 Flash 49% ZeroBench-main w/ tools (Pass@5) 49% Leaderboard, w/ tools; externally reported: DeepSeek-V4.1-Flash model card; Claude Code harness tools; 5 samplings. Same

2 logged changes that move scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Scores recomputed August 2026

    The maintainers corrected 0.71% of grading decisions and updated official scores. Mean pass@5 increased by 0.79 percentage points without question changes. Source

  2. Questions corrected March 2025

    The maintainers revised questions after post-release review. Results should identify the question revision used for evaluation. Source

More in Vision

All 636 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. Qwen 3.6 Plus 86%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. Seed2.1 Pro 90.7%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%

Data audit

All 10 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (80 entries), launch posts (27 scores) and system cards (17 scores).
  • Pass Models scored 103 models we track, plus 3 other entries on its leaderboard.
  • Pass Newest score The newest is from November 2026, for Inkling 2 Mini.
  • Note Frontier models Only 2 of the top 10 have a score. The strongest tested is GPT-6 Astra (#4).
  • Pass Headroom The best is 54.6%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on zerobench.github.io: 124 rows for models we track and 3 other entries at the last read.
  • Pass Last read November 18th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims 1 large gap of 6 pairs, each explained in our notes.
  • Pass Known issues No published audit found problems.

How we check