Benchmark Almanac

HumanEval

Solar Pro 2 98.8% leads since July 2025
o1-mini 96.3% leads since September 2024
Solar Pro 2 98.8% leads since July 2025
Fading

164 hand-written Python function problems. Generated code is scored pass@1 against unit tests.

Saturated. Reported only for DeepSeek-V4 base model.

Made by
OpenAI
Released
July 2021
Kind
Function-level problems
Level
School
Scored
Checked automatically
Answers
Code
Tasks
All public
Strongest model tested
DeepSeek V4.1 Flash #34 on the capability index
Last in a launch post
September 2026, dropped after it saturated
Last model added
October 2026
Scores refreshed
November 12th, 2026

In the authors’ words

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

Abstract of Evaluating Large Language Models Trained on Code (arXiv:2107.03374, CC0)
101
Models scored in lab cards
65
Launch posts list it, from 22 labs
70
System cards report it, from 19 labs

Solar Pro 2 leads with 98.8%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

o1-mini leads with 96.3% across all independent runs

Best score per model across EvalPlus’s leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

Solar Pro 2 leads with 98.8%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

101 models. Scroll sideways for the rest.

  1. 98.8 Solar Pro 2
  2. 98.2 InternVL3.5-241B-A28B
  3. 96.3 Reka Flash 3
  4. 95.9 Tulu 3 405B
  5. 94.5 DiffusionGemma 26B A4B
  6. 93.7 Claude 3.5 Sonnet (October 2024)
  7. 92.1 Mistral Medium 3
  8. 92.1 Sarvam 30B
  9. 92.0 Claude 3.5 Sonnet
  10. 92.0 Mistral Large 2
  11. 91.0 Hunyuan-TurboS
  12. 90.2 GPT-4o
  13. 90.0 Mercury Coder Small
  14. 89.7 Granite 3.3 8B Instruct
  15. 89.3 Granite 3.2 8B Instruct
  16. 89.0 Llama 3.1-405B
  17. 89.0 DeepSeek-V2.5
  18. 88.4 Mistral Small 3.1
  19. 88.4 Granite 4.1 30B
  20. 88.4 Grok-2
  21. 88.4 Llama 3.3 70B Instruct
  22. 88.4 Qwen2.5-32B
  23. 88.4 Qwen2.5-14B-Instruct-1M
  24. 88.2 GPT-4 Turbo
  25. 88.1 Claude 3.5 Haiku
  26. 88.0 Sarvam-M
  27. 88.0 Granite 4.0 H Small
  28. 87.8 Qwen2.5-VL-72B
  29. 87.8 Gemma 3 27B
  30. 87.3 Beam 1.0 Base
  31. 87.2 GPT-4o mini
  32. 87.2 Falcon-H1-34B
  33. 86.9 MiniMax-Text-01
  34. 86.6 Qwen2.5-72B
  35. 86.6 Qwen-Turbo
  36. 86.6 Codestral
  37. 86.0 Qwen2-72B
  38. 85.4 Gemma 3 12B
  39. 85.4 Cohere Command A
  40. 85.0 Mercury
  41. 84.9 Claude 3 Opus
  42. 84.8 Qwen2.5-7B
  43. 84.8 Mistral Small 3
  44. 84.8 Seed-Coder-8B-Instruct
  45. 84.1 Gemini 1.5 Pro
  46. 83.8 Nemotron 3 Ultra 550B-A55B Base
  47. 83.5 Qwen2.5-14B
  48. 82.8 Seed Diffusion Preview
  49. 82.6 Phi-4
  50. 82.3 InternLM3-8B-Instruct
  51. 81.7 Llama 3-70B
  52. 81.3 Nemotron 3.5 Super 120B-A12B Base
  53. 81.1 DeepSeek-V2 (MoE-236B, May 2024)
  54. 80.5 Llama 3.1-70B
  55. 79.4 DeepSeek V4.1 Flash
  56. 79.4 DeepSeek-V4.1-Flash Base
  57. 78.7 Qwen2.5-Omni-7B
  58. 78.0 Nemotron 3 Nano 30B-A3B Base
  59. 76.9 Nemotron 3.5 Lightning 30B-A3B Base
  60. 76.8 DeepSeek-V4-Pro
  61. 75.9 Claude 3 Haiku
  62. 75.0 Gemma 3n E4B
  63. 74.4 Gemini 1.0 Ultra
  64. 74.4 Phi-4 Mini
  65. 74.1 Grok-1.5
  66. 74.0 granite-4.0-1b
  67. 73.8 DeepSeek LLM 67B
  68. 73.0 Claude 3 Sonnet
  69. 73.0 Apertus 70B
  70. 72.6 Llama 3.1-8B
  71. 72.0 GLM-4
  72. 71.9 Gemini 1.5 Pro
  73. 71.3 Gemma 3 4B
  74. 71.2 Claude 2
  75. 70.7 Phi-3.5-MoE
  76. 67.0 GPT-4
  77. 65.2 DeepSeek-V3
  78. 63.2 Grok-1
  79. 62.2 Llama 3-8B
  80. 62.2 phi-3-medium 14B
  81. 61.5 Phi-3.5-mini
  82. 61.0 phi-3-small 7.4B
  83. 58.5 ChatGLM3-6B
  84. 58.5 phi-3-mini 3.8B
  85. 51.8 Gemma 2 27B
  86. 48.1 Moonlight-16B-A3B
  87. 45.1 Mistral Large
  88. 43.9 Qwen-14B
  89. 41.5 Qwen1.5-72B
  90. 41.5 Gemma 3 1B
  91. 40.2 Mixtral 8x7B
  92. 40.2 Gemma 2 9B
  93. 37.2 Qwen-7B
  94. 32.3 Gemma 7B
  95. 30.5 Mistral 7B v0.1
  96. 29.9 Llama 2-70B
  97. 22.6 Llama 2-34B
  98. 22.0 Gemma 2B
  99. 18.3 Llama 2-13B
  100. 14.6 Qwen-1_8B
  101. 12.8 Llama 2-7B
  1. 1 Solar Pro 2 98.8%
  2. 2 InternVL3.5-241B-A28B 98.2%
  3. 3 Reka Flash 3 96.3%
  4. 4 Tulu 3 405B 95.9%
  5. 5 DiffusionGemma 26B A4B 94.5%
  6. 6 Claude 3.5 Sonnet (October 2024) 93.7%
  7. 7 Mistral Medium 3 92.1%
  8. 8 Sarvam 30B 92.1%
  9. 9 Claude 3.5 Sonnet 92%
  10. 10 Mistral Large 2 Jul 2024 92%
  11. 11 Hunyuan-TurboS 91%
  12. 12 GPT-4o May 2024 90.2%
  13. 13 Mercury Coder Small 90%
  14. 14 Granite 3.3 8B Instruct 89.7%
  15. 15 Granite 3.2 8B Instruct 89.3%
  16. 16 Llama 3.1-405B 89%
  17. 17 DeepSeek-V2.5 Sep 2024 89%
  18. 18 Mistral Small 3.1 88.4%
  19. 19 Granite 4.1 30B 88.4%
  20. 20 Grok-2 Aug 2024 88.4%
  21. 21 Llama 3.3 70B Instruct 88.4%
  22. 22 Qwen2.5-32B 88.4%
  23. 23 Qwen2.5-14B-Instruct-1M 88.4%
  24. 24 GPT-4 Turbo Apr 2024 88.2%
  25. 25 Claude 3.5 Haiku 88.1%
  26. 26 Sarvam-M 88%
  27. 27 Granite 4.0 H Small 88%
  28. 28 Qwen2.5-VL-72B 87.8%
  29. 29 Gemma 3 27B 87.8%
  30. 30 Beam 1.0 Base 87.3%
  31. 31 GPT-4o mini 87.2%
  32. 32 Falcon-H1-34B 87.2%
  33. 33 MiniMax-Text-01 86.9%
  34. 34 Qwen2.5-72B 86.6%
  35. 35 Qwen-Turbo 86.6%
  36. 36 Codestral 86.6%
  37. 37 Qwen2-72B 86%
  38. 38 Gemma 3 12B 85.4%
  39. 39 Cohere Command A 85.4%
  40. 40 Mercury 85%
  41. 41 Claude 3 Opus 84.9%
  42. 42 Qwen2.5-7B 84.8%
  43. 43 Mistral Small 3 84.8%
  44. 44 Seed-Coder-8B-Instruct 84.8%
  45. 45 Gemini 1.5 Pro May 2024 84.1%
  46. 46 Nemotron 3 Ultra 550B-A55B Base 83.8%
  47. 47 Qwen2.5-14B 83.5%
  48. 48 Seed Diffusion Preview 82.8%
  49. 49 Phi-4 82.6%
  50. 50 InternLM3-8B-Instruct 82.3%
  51. 51 Llama 3-70B 81.7%
  52. 52 Nemotron 3.5 Super 120B-A12B Base 81.3%
  53. 53 DeepSeek-V2 (MoE-236B, May 2024) 81.1%
  54. 54 Llama 3.1-70B 80.5%
  55. 55 DeepSeek V4.1 Flash 79.4%
  56. 56 DeepSeek-V4.1-Flash Base 79.4%
  57. 57 Qwen2.5-Omni-7B 78.7%
  58. 58 Nemotron 3 Nano 30B-A3B Base 78%
  59. 59 Nemotron 3.5 Lightning 30B-A3B Base 76.9%
  60. 60 DeepSeek-V4-Pro 76.8%
  61. 61 Claude 3 Haiku 75.9%
  62. 62 Gemma 3n E4B 75%
  63. 63 Gemini 1.0 Ultra 74.4%
  64. 64 Phi-4 Mini 74.4%
  65. 65 Grok-1.5 74.1%
  66. 66 granite-4.0-1b 74%
  67. 67 DeepSeek LLM 67B 73.8%
  68. 68 Claude 3 Sonnet 73%
  69. 69 Apertus 70B 73%
  70. 70 Llama 3.1-8B 72.6%
  71. 71 GLM-4 72%
  72. 72 Gemini 1.5 Pro Feb 2024 71.9%
  73. 73 Gemma 3 4B 71.3%
  74. 74 Claude 2 71.2%
  75. 75 Phi-3.5-MoE 70.7%
  76. 76 GPT-4 Mar 2023 67%
  77. 77 DeepSeek-V3 65.2%
  78. 78 Grok-1 63.2%
  79. 79 Llama 3-8B 62.2%
  80. 80 phi-3-medium 14B 62.2%
  81. 81 Phi-3.5-mini 61.5%
  82. 82 phi-3-small 7.4B 61%
  83. 83 ChatGLM3-6B 58.5%
  84. 84 phi-3-mini 3.8B 58.5%
  85. 85 Gemma 2 27B 51.8%
  86. 86 Moonlight-16B-A3B 48.1%
  87. 87 Mistral Large 45.1%
  88. 88 Qwen-14B 43.9%
  89. 89 Qwen1.5-72B 41.5%
  90. 90 Gemma 3 1B 41.5%
  91. 91 Mixtral 8x7B 40.2%
  92. 92 Gemma 2 9B 40.2%
  93. 93 Qwen-7B 37.2%
  94. 94 Gemma 7B 32.3%
  95. 95 Mistral 7B v0.1 30.5%
  96. 96 Llama 2-70B 29.9%
  97. 97 Llama 2-34B 22.6%
  98. 98 Gemma 2B 22%
  99. 99 Llama 2-13B 18.3%
  100. 100 Qwen-1_8B 14.6%
  101. 101 Llama 2-7B 12.8%

Source: Labs’ model cards

46 models. Scroll sideways for the rest.

  1. 96.3 o1-mini
  2. 96.3 o1-preview
  3. 92.7 GPT-4o
  4. 92.1 Qwen2.5-Coder-32B-Instruct
  5. 91.5 DeepSeek-V3
  6. 90.2 GPT-4 Turbo
  7. 90.2 DeepSeek-V2.5
  8. 89.0 Gemini 1.5 Pro
  9. 88.4 GPT-4
  10. 88.4 GPT-4o mini
  11. 87.2 Claude 3.5 Sonnet
  12. 85.4 GPT-4 Turbo
  13. 85.4 DeepSeek-Coder-V2 236B
  14. 82.9 Claude 3 Opus
  15. 82.3 Gemini 1.5 Flash
  16. 77.4 Llama 3-70B
  17. 76.8 GPT-3.5 Turbo
  18. 76.8 Claude 3 Haiku
  19. 76.2 Mixtral 8x22B
  20. 75.0 DBRX
  21. 73.2 GPT-3.5 Turbo (Mar 2023)
  22. 72.0 Code Llama 70B Instruct
  23. 70.7 Claude 3 Sonnet
  24. 69.5 Claude 2
  25. 69.5 Mistral Large
  26. 69.5 Llama 3.1-8B
  27. 68.3 Qwen1.5-72B
  28. 64.6 phi-3-mini 3.8B
  29. 63.4 Gemini 1.0 Pro
  30. 61.6 Llama 3-8B
  31. 57.3 Claude Instant
  32. 51.8 CodeQwen1.5-7B
  33. 49.4 Phi-2
  34. 46.3 StarCoder 2 15B
  35. 45.1 Mixtral 8x7B
  36. 43.3 SOLAR-10.7B-Instruct-v1.0
  37. 42.7 Gemma 1.1 7B IT
  38. 42.1 Mistral-7B-Instruct-v0.2
  39. 35.4 StarCoder 2 7B
  40. 35.4 Gemma 7B
  41. 31.7 StarCoder 2 3B
  42. 28.7 Mistral 7B v0.1
  43. 25.0 Gemma 2B
  44. 22.6 Gemma 1.1 2B IT
  45. 12.2 GPT-J-6B
  46. 7.90 GPT-Neo-2.7B
  1. 1 o1-mini 96.3%
  2. 2 o1-preview 96.3%
  3. 3 GPT-4o Aug 2024 92.7%
  4. 4 Qwen2.5-Coder-32B-Instruct 92.1%
  5. 5 DeepSeek-V3 91.5%
  6. 6 GPT-4 Turbo Apr 2024 90.2%
  7. 7 DeepSeek-V2.5 Sep 2024 90.2%
  8. 8 Gemini 1.5 Pro Sep 2024 89%
  9. 9 GPT-4 Mar 2023 88.4%
  10. 10 GPT-4o mini 88.4%
  11. 11 Claude 3.5 Sonnet 87.2%
  12. 12 GPT-4 Turbo Nov 2023 85.4%
  13. 13 DeepSeek-Coder-V2 236B 85.4%
  14. 14 Claude 3 Opus 82.9%
  15. 15 Gemini 1.5 Flash Sep 2024 82.3%
  16. 16 Llama 3-70B 77.4%
  17. 17 GPT-3.5 Turbo Nov 2023 76.8%
  18. 18 Claude 3 Haiku 76.8%
  19. 19 Mixtral 8x22B 76.2%
  20. 20 DBRX 75%
  21. 21 GPT-3.5 Turbo (Mar 2023) 73.2%
  22. 22 Code Llama 70B Instruct 72%
  23. 23 Claude 3 Sonnet 70.7%
  24. 24 Claude 2 69.5%
  25. 25 Mistral Large 69.5%
  26. 26 Llama 3.1-8B 69.5%
  27. 27 Qwen1.5-72B 68.3%
  28. 28 phi-3-mini 3.8B 64.6%
  29. 29 Gemini 1.0 Pro 63.4%
  30. 30 Llama 3-8B 61.6%
  31. 31 Claude Instant 57.3%
  32. 32 CodeQwen1.5-7B 51.8%
  33. 33 Phi-2 49.4%
  34. 34 StarCoder 2 15B 46.3%
  35. 35 Mixtral 8x7B 45.1%
  36. 36 SOLAR-10.7B-Instruct-v1.0 43.3%
  37. 37 Gemma 1.1 7B IT 42.7%
  38. 38 Mistral-7B-Instruct-v0.2 42.1%
  39. 39 StarCoder 2 7B 35.4%
  40. 40 Gemma 7B 35.4%
  41. 41 StarCoder 2 3B 31.7%
  42. 42 Mistral 7B v0.1 28.7%
  43. 43 Gemma 2B 25%
  44. 44 Gemma 1.1 2B IT 22.6%
  45. 45 GPT-J-6B 12.2%
  46. 46 GPT-Neo-2.7B 7.90%

Source: EvalPlus’s leaderboard

127 models. Scroll sideways for the rest.

  1. 98.8 Solar Pro 2
  2. 98.2 InternVL3.5-241B-A28B
  3. 96.3 o1-mini
  4. 96.3 o1-preview
  5. 96.3 Reka Flash 3
  6. 95.9 Tulu 3 405B
  7. 94.5 DiffusionGemma 26B A4B
  8. 93.7 Claude 3.5 Sonnet (October 2024)
  9. 92.7 GPT-4o
  10. 92.1 Qwen2.5-Coder-32B-Instruct
  11. 92.1 Mistral Medium 3
  12. 92.1 Sarvam 30B
  13. 92.0 Claude 3.5 Sonnet
  14. 92.0 Mistral Large 2
  15. 91.5 DeepSeek-V3
  16. 91.0 Hunyuan-TurboS
  17. 90.2 GPT-4 Turbo
  18. 90.2 GPT-4o
  19. 90.2 DeepSeek-V2.5
  20. 90.0 Mercury Coder Small
  21. 89.7 Granite 3.3 8B Instruct
  22. 89.3 Granite 3.2 8B Instruct
  23. 89.0 Llama 3.1-405B
  24. 89.0 Gemini 1.5 Pro
  25. 88.4 Mistral Small 3.1
  26. 88.4 Granite 4.1 30B
  27. 88.4 GPT-4
  28. 88.4 GPT-4o mini
  29. 88.4 Grok-2
  30. 88.4 Llama 3.3 70B Instruct
  31. 88.4 Qwen2.5-32B
  32. 88.4 Qwen2.5-14B-Instruct-1M
  33. 88.1 Claude 3.5 Haiku
  34. 88.0 Sarvam-M
  35. 88.0 Granite 4.0 H Small
  36. 87.8 Qwen2.5-VL-72B
  37. 87.8 Gemma 3 27B
  38. 87.3 Beam 1.0 Base
  39. 87.2 Falcon-H1-34B
  40. 86.9 MiniMax-Text-01
  41. 86.6 Qwen2.5-72B
  42. 86.6 Qwen-Turbo
  43. 86.6 Codestral
  44. 86.0 Qwen2-72B
  45. 85.4 GPT-4 Turbo
  46. 85.4 DeepSeek-Coder-V2 236B
  47. 85.4 Gemma 3 12B
  48. 85.4 Cohere Command A
  49. 85.0 Mercury
  50. 84.9 Claude 3 Opus
  51. 84.8 Qwen2.5-7B
  52. 84.8 Mistral Small 3
  53. 84.8 Seed-Coder-8B-Instruct
  54. 84.1 Gemini 1.5 Pro
  55. 83.8 Nemotron 3 Ultra 550B-A55B Base
  56. 83.5 Qwen2.5-14B
  57. 82.8 Seed Diffusion Preview
  58. 82.6 Phi-4
  59. 82.3 Gemini 1.5 Flash
  60. 82.3 InternLM3-8B-Instruct
  61. 81.7 Llama 3-70B
  62. 81.3 Nemotron 3.5 Super 120B-A12B Base
  63. 81.1 DeepSeek-V2 (MoE-236B, May 2024)
  64. 80.5 Llama 3.1-70B
  65. 79.4 DeepSeek V4.1 Flash
  66. 79.4 DeepSeek-V4.1-Flash Base
  67. 78.7 Qwen2.5-Omni-7B
  68. 78.0 Nemotron 3 Nano 30B-A3B Base
  69. 76.9 Nemotron 3.5 Lightning 30B-A3B Base
  70. 76.8 GPT-3.5 Turbo
  71. 76.8 Claude 3 Haiku
  72. 76.8 DeepSeek-V4-Pro
  73. 76.2 Mixtral 8x22B
  74. 75.0 DBRX
  75. 75.0 Gemma 3n E4B
  76. 74.4 Gemini 1.0 Ultra
  77. 74.4 Phi-4 Mini
  78. 74.1 Grok-1.5
  79. 74.0 granite-4.0-1b
  80. 73.8 DeepSeek LLM 67B
  81. 73.2 GPT-3.5 Turbo (Mar 2023)
  82. 73.0 Claude 3 Sonnet
  83. 73.0 Apertus 70B
  84. 72.6 Llama 3.1-8B
  85. 72.0 GLM-4
  86. 72.0 Code Llama 70B Instruct
  87. 71.9 Gemini 1.5 Pro
  88. 71.3 Gemma 3 4B
  89. 71.2 Claude 2
  90. 70.7 Phi-3.5-MoE
  91. 69.5 Mistral Large
  92. 68.3 Qwen1.5-72B
  93. 64.6 phi-3-mini 3.8B
  94. 63.4 Gemini 1.0 Pro
  95. 63.2 Grok-1
  96. 62.2 Llama 3-8B
  97. 62.2 phi-3-medium 14B
  98. 61.5 Phi-3.5-mini
  99. 61.0 phi-3-small 7.4B
  100. 58.5 ChatGLM3-6B
  101. 57.3 Claude Instant
  102. 51.8 CodeQwen1.5-7B
  103. 51.8 Gemma 2 27B
  104. 49.4 Phi-2
  105. 48.1 Moonlight-16B-A3B
  106. 46.3 StarCoder 2 15B
  107. 45.1 Mixtral 8x7B
  108. 43.9 Qwen-14B
  109. 43.3 SOLAR-10.7B-Instruct-v1.0
  110. 42.7 Gemma 1.1 7B IT
  111. 42.1 Mistral-7B-Instruct-v0.2
  112. 41.5 Gemma 3 1B
  113. 40.2 Gemma 2 9B
  114. 37.2 Qwen-7B
  115. 35.4 StarCoder 2 7B
  116. 35.4 Gemma 7B
  117. 31.7 StarCoder 2 3B
  118. 30.5 Mistral 7B v0.1
  119. 29.9 Llama 2-70B
  120. 25.0 Gemma 2B
  121. 22.6 Llama 2-34B
  122. 22.6 Gemma 1.1 2B IT
  123. 18.3 Llama 2-13B
  124. 14.6 Qwen-1_8B
  125. 12.8 Llama 2-7B
  126. 12.2 GPT-J-6B
  127. 7.90 GPT-Neo-2.7B
  1. 1 Solar Pro 2 98.8%
  2. 2 InternVL3.5-241B-A28B 98.2%
  3. 3 o1-mini 96.3%
  4. 4 o1-preview 96.3%
  5. 5 Reka Flash 3 96.3%
  6. 6 Tulu 3 405B 95.9%
  7. 7 DiffusionGemma 26B A4B 94.5%
  8. 8 Claude 3.5 Sonnet (October 2024) 93.7%
  9. 9 GPT-4o Aug 2024 92.7%
  10. 10 Qwen2.5-Coder-32B-Instruct 92.1%
  11. 11 Mistral Medium 3 92.1%
  12. 12 Sarvam 30B 92.1%
  13. 13 Claude 3.5 Sonnet 92%
  14. 14 Mistral Large 2 Jul 2024 92%
  15. 15 DeepSeek-V3 91.5%
  16. 16 Hunyuan-TurboS 91%
  17. 17 GPT-4 Turbo Apr 2024 90.2%
  18. 18 GPT-4o May 2024 90.2%
  19. 19 DeepSeek-V2.5 Sep 2024 90.2%
  20. 20 Mercury Coder Small 90%
  21. 21 Granite 3.3 8B Instruct 89.7%
  22. 22 Granite 3.2 8B Instruct 89.3%
  23. 23 Llama 3.1-405B 89%
  24. 24 Gemini 1.5 Pro Sep 2024 89%
  25. 25 Mistral Small 3.1 88.4%
  26. 26 Granite 4.1 30B 88.4%
  27. 27 GPT-4 Mar 2023 88.4%
  28. 28 GPT-4o mini 88.4%
  29. 29 Grok-2 Aug 2024 88.4%
  30. 30 Llama 3.3 70B Instruct 88.4%
  31. 31 Qwen2.5-32B 88.4%
  32. 32 Qwen2.5-14B-Instruct-1M 88.4%
  33. 33 Claude 3.5 Haiku 88.1%
  34. 34 Sarvam-M 88%
  35. 35 Granite 4.0 H Small 88%
  36. 36 Qwen2.5-VL-72B 87.8%
  37. 37 Gemma 3 27B 87.8%
  38. 38 Beam 1.0 Base 87.3%
  39. 39 Falcon-H1-34B 87.2%
  40. 40 MiniMax-Text-01 86.9%
  41. 41 Qwen2.5-72B 86.6%
  42. 42 Qwen-Turbo 86.6%
  43. 43 Codestral 86.6%
  44. 44 Qwen2-72B 86%
  45. 45 GPT-4 Turbo Nov 2023 85.4%
  46. 46 DeepSeek-Coder-V2 236B 85.4%
  47. 47 Gemma 3 12B 85.4%
  48. 48 Cohere Command A 85.4%
  49. 49 Mercury 85%
  50. 50 Claude 3 Opus 84.9%
  51. 51 Qwen2.5-7B 84.8%
  52. 52 Mistral Small 3 84.8%
  53. 53 Seed-Coder-8B-Instruct 84.8%
  54. 54 Gemini 1.5 Pro May 2024 84.1%
  55. 55 Nemotron 3 Ultra 550B-A55B Base 83.8%
  56. 56 Qwen2.5-14B 83.5%
  57. 57 Seed Diffusion Preview 82.8%
  58. 58 Phi-4 82.6%
  59. 59 Gemini 1.5 Flash Sep 2024 82.3%
  60. 60 InternLM3-8B-Instruct 82.3%
  61. 61 Llama 3-70B 81.7%
  62. 62 Nemotron 3.5 Super 120B-A12B Base 81.3%
  63. 63 DeepSeek-V2 (MoE-236B, May 2024) 81.1%
  64. 64 Llama 3.1-70B 80.5%
  65. 65 DeepSeek V4.1 Flash 79.4%
  66. 66 DeepSeek-V4.1-Flash Base 79.4%
  67. 67 Qwen2.5-Omni-7B 78.7%
  68. 68 Nemotron 3 Nano 30B-A3B Base 78%
  69. 69 Nemotron 3.5 Lightning 30B-A3B Base 76.9%
  70. 70 GPT-3.5 Turbo Nov 2023 76.8%
  71. 71 Claude 3 Haiku 76.8%
  72. 72 DeepSeek-V4-Pro 76.8%
  73. 73 Mixtral 8x22B 76.2%
  74. 74 DBRX 75%
  75. 75 Gemma 3n E4B 75%
  76. 76 Gemini 1.0 Ultra 74.4%
  77. 77 Phi-4 Mini 74.4%
  78. 78 Grok-1.5 74.1%
  79. 79 granite-4.0-1b 74%
  80. 80 DeepSeek LLM 67B 73.8%
  81. 81 GPT-3.5 Turbo (Mar 2023) 73.2%
  82. 82 Claude 3 Sonnet 73%
  83. 83 Apertus 70B 73%
  84. 84 Llama 3.1-8B 72.6%
  85. 85 GLM-4 72%
  86. 86 Code Llama 70B Instruct 72%
  87. 87 Gemini 1.5 Pro Feb 2024 71.9%
  88. 88 Gemma 3 4B 71.3%
  89. 89 Claude 2 71.2%
  90. 90 Phi-3.5-MoE 70.7%
  91. 91 Mistral Large 69.5%
  92. 92 Qwen1.5-72B 68.3%
  93. 93 phi-3-mini 3.8B 64.6%
  94. 94 Gemini 1.0 Pro 63.4%
  95. 95 Grok-1 63.2%
  96. 96 Llama 3-8B 62.2%
  97. 97 phi-3-medium 14B 62.2%
  98. 98 Phi-3.5-mini 61.5%
  99. 99 phi-3-small 7.4B 61%
  100. 100 ChatGLM3-6B 58.5%
  101. 101 Claude Instant 57.3%
  102. 102 CodeQwen1.5-7B 51.8%
  103. 103 Gemma 2 27B 51.8%
  104. 104 Phi-2 49.4%
  105. 105 Moonlight-16B-A3B 48.1%
  106. 106 StarCoder 2 15B 46.3%
  107. 107 Mixtral 8x7B 45.1%
  108. 108 Qwen-14B 43.9%
  109. 109 SOLAR-10.7B-Instruct-v1.0 43.3%
  110. 110 Gemma 1.1 7B IT 42.7%
  111. 111 Mistral-7B-Instruct-v0.2 42.1%
  112. 112 Gemma 3 1B 41.5%
  113. 113 Gemma 2 9B 40.2%
  114. 114 Qwen-7B 37.2%
  115. 115 StarCoder 2 7B 35.4%
  116. 116 Gemma 7B 35.4%
  117. 117 StarCoder 2 3B 31.7%
  118. 118 Mistral 7B v0.1 30.5%
  119. 119 Llama 2-70B 29.9%
  120. 120 Gemma 2B 25%
  121. 121 Llama 2-34B 22.6%
  122. 122 Gemma 1.1 2B IT 22.6%
  123. 123 Llama 2-13B 18.3%
  124. 124 Qwen-1_8B 14.6%
  125. 125 Llama 2-7B 12.8%
  126. 126 GPT-J-6B 12.2%
  127. 127 GPT-Neo-2.7B 7.90%

Sources: Labs’ model cards and EvalPlus’s leaderboard

#ModelScoreSettingDateSource
1 Solar Pro 2 98.8% Jul 2025 Lab model card
2 InternVL3.5-241B-A28B 98.2% Aug 2025 Lab model card
3 Reka Flash 3 96.3% Mar 2025 Lab model card
4 Tulu 3 405B 95.9% Jan 2025 Lab model card
5 DiffusionGemma 26B A4B 94.5% Jul 2026 Lab model card
6 Claude 3.5 Sonnet (October 2024) 93.7% Oct 2024 Lab model card
7 Mistral Medium 3 92.1% May 2025 Lab model card
8 Sarvam 30B 92.1% Mar 2026 Lab model card
9 Claude 3.5 Sonnet 92% Jun 2024 Lab model card
10 Mistral Large 2 Jul 2024 92% Jul 2024 Lab model card
11 Hunyuan-TurboS 91% Feb 2025 Lab model card
12 GPT-4o May 2024 90.2% May 2024 Lab model card
13 Mercury Coder Small 90% Feb 2025 Lab model card
14 Granite 3.3 8B Instruct 89.7% Apr 2025 Lab model card
15 Granite 3.2 8B Instruct 89.3% Feb 2025 Lab model card
16 Llama 3.1-405B 89% Jul 2024 Lab model card
17 DeepSeek-V2.5 Sep 2024 89% Sep 2024 Lab model card
18 Mistral Small 3.1 88.4% Mar 2025 Lab model card
19 Granite 4.1 30B 88.4% Apr 2026 Lab model card
20 Grok-2 Aug 2024 88.4% Aug 2024 Lab model card
21 Llama 3.3 70B Instruct 88.4% Dec 2024 Lab model card
22 Qwen2.5-32B 88.4% Dec 2024 Lab model card
23 Qwen2.5-14B-Instruct-1M 88.4% Jan 2025 Lab model card
24 GPT-4 Turbo Apr 2024 88.2% Apr 2024 Lab model card
25 Claude 3.5 Haiku 88.1% Oct 2024 Lab model card
26 Sarvam-M 88% May 2025 Lab model card
27 Granite 4.0 H Small 88% Oct 2025 Lab model card
28 Qwen2.5-VL-72B 87.8% Feb 2025 Lab model card
29 Gemma 3 27B 87.8% Mar 2025 Lab model card
30 Beam 1.0 Base 87.3% Oct 2026 Lab model card
31 GPT-4o mini 87.2% Jul 2024 Lab model card
32 Falcon-H1-34B 87.2% May 2025 Lab model card
33 MiniMax-Text-01 86.9% Jan 2025 Lab model card
34 Qwen2.5-72B 86.6% Sep 2024 Lab model card
35 Qwen-Turbo 86.6% Dec 2024 Lab model card
36 Codestral 86.6% Jan 2025 Lab model card
37 Qwen2-72B 86% Jun 2024 Lab model card
38 Gemma 3 12B 85.4% Mar 2025 Lab model card
39 Cohere Command A 85.4% Apr 2025 Lab model card
40 Mercury 85% Jun 2025 Lab model card
41 Claude 3 Opus 84.9% Mar 2024 Lab model card
42 Qwen2.5-7B 84.8% Dec 2024 Lab model card
43 Mistral Small 3 84.8% Jan 2025 Lab model card
44 Seed-Coder-8B-Instruct 84.8% May 2025 Lab model card
45 Gemini 1.5 Pro May 2024 84.1% May 2024 Lab model card
46 Nemotron 3 Ultra 550B-A55B Base 83.8% Jun 2026 Lab model card
47 Qwen2.5-14B 83.5% Dec 2024 Lab model card
48 Seed Diffusion Preview 82.8% Jul 2025 Lab model card
49 Phi-4 82.6% Dec 2024 Lab model card
50 InternLM3-8B-Instruct 82.3% Jan 2025 Lab model card
51 Llama 3-70B 81.7% Apr 2024 Lab model card
52 Nemotron 3.5 Super 120B-A12B Base 81.3% Oct 2026 Lab model card
53 DeepSeek-V2 (MoE-236B, May 2024) 81.1% May 2024 Lab model card
54 Llama 3.1-70B 80.5% Jul 2024 Lab model card
55 DeepSeek V4.1 Flash 79.4% Sep 2026 Lab model card
56 DeepSeek-V4.1-Flash Base 79.4% Sep 2026 Lab model card
57 Qwen2.5-Omni-7B 78.7% Mar 2025 Lab model card
58 Nemotron 3 Nano 30B-A3B Base 78% Dec 2025 Lab model card
59 Nemotron 3.5 Lightning 30B-A3B Base 76.9% Oct 2026 Lab model card
60 DeepSeek-V4-Pro 76.8% Apr 2026 Lab model card
61 Claude 3 Haiku 75.9% Mar 2024 Lab model card
62 Gemma 3n E4B 75% Jun 2025 Lab model card
63 Gemini 1.0 Ultra 74.4% Dec 2023 Lab model card
64 Phi-4 Mini 74.4% Mar 2025 Lab model card
65 Grok-1.5 74.1% Mar 2024 Lab model card
66 granite-4.0-1b 74% Oct 2025 Lab model card
67 DeepSeek LLM 67B 73.8% Nov 2023 Lab model card
68 Claude 3 Sonnet 73% Mar 2024 Lab model card
69 Apertus 70B 73% Sep 2025 Lab model card
70 Llama 3.1-8B 72.6% Jul 2024 Lab model card
71 GLM-4 72% Jan 2024 Lab model card
72 Gemini 1.5 Pro Feb 2024 71.9% Feb 2024 Lab model card
73 Gemma 3 4B 71.3% Mar 2025 Lab model card
74 Claude 2 71.2% Jul 2023 Lab model card
75 Phi-3.5-MoE 70.7% Aug 2024 Lab model card
76 GPT-4 Mar 2023 67% Mar 2023 Lab model card
77 DeepSeek-V3 65.2% Dec 2024 Lab model card
78 Grok-1 63.2% Nov 2023 Lab model card
79 Llama 3-8B 62.2% Apr 2024 Lab model card
80 phi-3-medium 14B 62.2% Aug 2024 Lab model card
81 Phi-3.5-mini 61.5% Aug 2024 Lab model card
82 phi-3-small 7.4B 61% Aug 2024 Lab model card
83 ChatGLM3-6B 58.5% Jun 2024 Lab model card
84 phi-3-mini 3.8B 58.5% Aug 2024 Lab model card
85 Gemma 2 27B 51.8% Jun 2024 Lab model card
86 Moonlight-16B-A3B 48.1% Feb 2025 Lab model card
87 Mistral Large 45.1% Feb 2024 Lab model card
88 Qwen-14B 43.9% Sep 2023 Lab model card
89 Qwen1.5-72B 41.5% Feb 2024 Lab model card
90 Gemma 3 1B 41.5% Mar 2025 Lab model card
91 Mixtral 8x7B 40.2% Dec 2023 Lab model card
92 Gemma 2 9B 40.2% Jul 2024 Lab model card
93 Qwen-7B 37.2% Sep 2023 Lab model card
94 Gemma 7B 32.3% Feb 2024 Lab model card
95 Mistral 7B v0.1 30.5% Sep 2023 Lab model card
96 Llama 2-70B 29.9% Jul 2023 Lab model card
97 Llama 2-34B 22.6% Jul 2023 Lab model card
98 Gemma 2B 22% Mar 2024 Lab model card
99 Llama 2-13B 18.3% Jul 2023 Lab model card
100 Qwen-1_8B 14.6% Sep 2023 Lab model card
101 Llama 2-7B 12.8% Jul 2023 Lab model card
#ModelScoreSettingDateSource
1 o1-mini 96.3% Sep 2024 Official leaderboard
2 o1-preview 96.3% Sep 2024 Official leaderboard
3 GPT-4o Aug 2024 92.7% Aug 2024 Official leaderboard
4 Qwen2.5-Coder-32B-Instruct 92.1% Nov 2024 Official leaderboard
5 DeepSeek-V3 91.5% Dec 2024 Official leaderboard
6 GPT-4 Turbo Apr 2024 90.2% Apr 2024 Official leaderboard
7 DeepSeek-V2.5 Sep 2024 90.2% Sep 2024 Official leaderboard
8 Gemini 1.5 Pro Sep 2024 89% Sep 2024 Official leaderboard
9 GPT-4 Mar 2023 88.4% Mar 2023 Official leaderboard
10 GPT-4o mini 88.4% Jul 2024 Official leaderboard
11 Claude 3.5 Sonnet 87.2% Jun 2024 Official leaderboard
12 GPT-4 Turbo Nov 2023 85.4% Nov 2023 Official leaderboard
13 DeepSeek-Coder-V2 236B 85.4% Jun 2024 Official leaderboard
14 Claude 3 Opus 82.9% Feb 2024 Official leaderboard
15 Gemini 1.5 Flash Sep 2024 82.3% Aug 2024 Official leaderboard
16 Llama 3-70B 77.4% Apr 2024 Official leaderboard
17 GPT-3.5 Turbo Nov 2023 76.8% Nov 2023 Official leaderboard
18 Claude 3 Haiku 76.8% Mar 2024 Official leaderboard
19 Mixtral 8x22B 76.2% Apr 2024 Official leaderboard
20 DBRX 75% Mar 2024 Official leaderboard
21 GPT-3.5 Turbo (Mar 2023) 73.2% Mar 2023 Official leaderboard
22 Code Llama 70B Instruct 72% Jan 2024 Official leaderboard
23 Claude 3 Sonnet 70.7% Feb 2024 Official leaderboard
24 Claude 2 69.5% Jul 2023 Official leaderboard
25 Mistral Large 69.5% Feb 2024 Official leaderboard
26 Llama 3.1-8B 69.5% Jul 2024 Official leaderboard
27 Qwen1.5-72B 68.3% Feb 2024 Official leaderboard
28 phi-3-mini 3.8B 64.6% Apr 2024 Official leaderboard
29 Gemini 1.0 Pro 63.4% Dec 2023 Official leaderboard
30 Llama 3-8B 61.6% Apr 2024 Official leaderboard
31 Claude Instant 57.3% Aug 2023 Official leaderboard
32 CodeQwen1.5-7B 51.8% Apr 2024 Official leaderboard
33 Phi-2 49.4% Dec 2023 Official leaderboard
34 StarCoder 2 15B 46.3% Feb 2024 Official leaderboard
35 Mixtral 8x7B 45.1% Dec 2023 Official leaderboard
36 SOLAR-10.7B-Instruct-v1.0 43.3% Dec 2023 Official leaderboard
37 Gemma 1.1 7B IT 42.7% Apr 2024 Official leaderboard
38 Mistral-7B-Instruct-v0.2 42.1% Dec 2023 Official leaderboard
39 StarCoder 2 7B 35.4% Feb 2024 Official leaderboard
40 Gemma 7B 35.4% Feb 2024 Official leaderboard
41 StarCoder 2 3B 31.7% Feb 2024 Official leaderboard
42 Mistral 7B v0.1 28.7% Sep 2023 Official leaderboard
43 Gemma 2B 25% Feb 2024 Official leaderboard
44 Gemma 1.1 2B IT 22.6% Apr 2024 Official leaderboard
45 GPT-J-6B 12.2% Aug 2021 Official leaderboard
46 GPT-Neo-2.7B 7.90% Mar 2021 Official leaderboard
#ModelScoreSettingDateSource
1 Solar Pro 2 98.8% Jul 2025 Lab model card
2 InternVL3.5-241B-A28B 98.2% Aug 2025 Lab model card
3 o1-mini 96.3% Sep 2024 Official leaderboard
4 o1-preview 96.3% Sep 2024 Official leaderboard
5 Reka Flash 3 96.3% Mar 2025 Lab model card
6 Tulu 3 405B 95.9% Jan 2025 Lab model card
7 DiffusionGemma 26B A4B 94.5% Jul 2026 Lab model card
8 Claude 3.5 Sonnet (October 2024) 93.7% Oct 2024 Lab model card
9 GPT-4o Aug 2024 92.7% Aug 2024 Official leaderboard
10 Qwen2.5-Coder-32B-Instruct 92.1% Nov 2024 Official leaderboard
11 Mistral Medium 3 92.1% May 2025 Lab model card
12 Sarvam 30B 92.1% Mar 2026 Lab model card
13 Claude 3.5 Sonnet 92% Jun 2024 Lab model card
14 Mistral Large 2 Jul 2024 92% Jul 2024 Lab model card
15 DeepSeek-V3 91.5% Dec 2024 Official leaderboard
16 Hunyuan-TurboS 91% Feb 2025 Lab model card
17 GPT-4 Turbo Apr 2024 90.2% Apr 2024 Official leaderboard
18 GPT-4o May 2024 90.2% May 2024 Lab model card
19 DeepSeek-V2.5 Sep 2024 90.2% Sep 2024 Official leaderboard
20 Mercury Coder Small 90% Feb 2025 Lab model card
21 Granite 3.3 8B Instruct 89.7% Apr 2025 Lab model card
22 Granite 3.2 8B Instruct 89.3% Feb 2025 Lab model card
23 Llama 3.1-405B 89% Jul 2024 Lab model card
24 Gemini 1.5 Pro Sep 2024 89% Sep 2024 Official leaderboard
25 Mistral Small 3.1 88.4% Mar 2025 Lab model card
26 Granite 4.1 30B 88.4% Apr 2026 Lab model card
27 GPT-4 Mar 2023 88.4% Mar 2023 Official leaderboard
28 GPT-4o mini 88.4% Jul 2024 Official leaderboard
29 Grok-2 Aug 2024 88.4% Aug 2024 Lab model card
30 Llama 3.3 70B Instruct 88.4% Dec 2024 Lab model card
31 Qwen2.5-32B 88.4% Dec 2024 Lab model card
32 Qwen2.5-14B-Instruct-1M 88.4% Jan 2025 Lab model card
33 Claude 3.5 Haiku 88.1% Oct 2024 Lab model card
34 Sarvam-M 88% May 2025 Lab model card
35 Granite 4.0 H Small 88% Oct 2025 Lab model card
36 Qwen2.5-VL-72B 87.8% Feb 2025 Lab model card
37 Gemma 3 27B 87.8% Mar 2025 Lab model card
38 Beam 1.0 Base 87.3% Oct 2026 Lab model card
39 Falcon-H1-34B 87.2% May 2025 Lab model card
40 MiniMax-Text-01 86.9% Jan 2025 Lab model card
41 Qwen2.5-72B 86.6% Sep 2024 Lab model card
42 Qwen-Turbo 86.6% Dec 2024 Lab model card
43 Codestral 86.6% Jan 2025 Lab model card
44 Qwen2-72B 86% Jun 2024 Lab model card
45 GPT-4 Turbo Nov 2023 85.4% Nov 2023 Official leaderboard
46 DeepSeek-Coder-V2 236B 85.4% Jun 2024 Official leaderboard
47 Gemma 3 12B 85.4% Mar 2025 Lab model card
48 Cohere Command A 85.4% Apr 2025 Lab model card
49 Mercury 85% Jun 2025 Lab model card
50 Claude 3 Opus 84.9% Mar 2024 Lab model card
51 Qwen2.5-7B 84.8% Dec 2024 Lab model card
52 Mistral Small 3 84.8% Jan 2025 Lab model card
53 Seed-Coder-8B-Instruct 84.8% May 2025 Lab model card
54 Gemini 1.5 Pro May 2024 84.1% May 2024 Lab model card
55 Nemotron 3 Ultra 550B-A55B Base 83.8% Jun 2026 Lab model card
56 Qwen2.5-14B 83.5% Dec 2024 Lab model card
57 Seed Diffusion Preview 82.8% Jul 2025 Lab model card
58 Phi-4 82.6% Dec 2024 Lab model card
59 Gemini 1.5 Flash Sep 2024 82.3% Aug 2024 Official leaderboard
60 InternLM3-8B-Instruct 82.3% Jan 2025 Lab model card
61 Llama 3-70B 81.7% Apr 2024 Lab model card
62 Nemotron 3.5 Super 120B-A12B Base 81.3% Oct 2026 Lab model card
63 DeepSeek-V2 (MoE-236B, May 2024) 81.1% May 2024 Lab model card
64 Llama 3.1-70B 80.5% Jul 2024 Lab model card
65 DeepSeek V4.1 Flash 79.4% Sep 2026 Lab model card
66 DeepSeek-V4.1-Flash Base 79.4% Sep 2026 Lab model card
67 Qwen2.5-Omni-7B 78.7% Mar 2025 Lab model card
68 Nemotron 3 Nano 30B-A3B Base 78% Dec 2025 Lab model card
69 Nemotron 3.5 Lightning 30B-A3B Base 76.9% Oct 2026 Lab model card
70 GPT-3.5 Turbo Nov 2023 76.8% Nov 2023 Official leaderboard
71 Claude 3 Haiku 76.8% Mar 2024 Official leaderboard
72 DeepSeek-V4-Pro 76.8% Apr 2026 Lab model card
73 Mixtral 8x22B 76.2% Apr 2024 Official leaderboard
74 DBRX 75% Mar 2024 Official leaderboard
75 Gemma 3n E4B 75% Jun 2025 Lab model card
76 Gemini 1.0 Ultra 74.4% Dec 2023 Lab model card
77 Phi-4 Mini 74.4% Mar 2025 Lab model card
78 Grok-1.5 74.1% Mar 2024 Lab model card
79 granite-4.0-1b 74% Oct 2025 Lab model card
80 DeepSeek LLM 67B 73.8% Nov 2023 Lab model card
81 GPT-3.5 Turbo (Mar 2023) 73.2% Mar 2023 Official leaderboard
82 Claude 3 Sonnet 73% Mar 2024 Lab model card
83 Apertus 70B 73% Sep 2025 Lab model card
84 Llama 3.1-8B 72.6% Jul 2024 Lab model card
85 GLM-4 72% Jan 2024 Lab model card
86 Code Llama 70B Instruct 72% Jan 2024 Official leaderboard
87 Gemini 1.5 Pro Feb 2024 71.9% Feb 2024 Lab model card
88 Gemma 3 4B 71.3% Mar 2025 Lab model card
89 Claude 2 71.2% Jul 2023 Lab model card
90 Phi-3.5-MoE 70.7% Aug 2024 Lab model card
91 Mistral Large 69.5% Feb 2024 Official leaderboard
92 Qwen1.5-72B 68.3% Feb 2024 Official leaderboard
93 phi-3-mini 3.8B 64.6% Apr 2024 Official leaderboard
94 Gemini 1.0 Pro 63.4% Dec 2023 Official leaderboard
95 Grok-1 63.2% Nov 2023 Lab model card
96 Llama 3-8B 62.2% Apr 2024 Lab model card
97 phi-3-medium 14B 62.2% Aug 2024 Lab model card
98 Phi-3.5-mini 61.5% Aug 2024 Lab model card
99 phi-3-small 7.4B 61% Aug 2024 Lab model card
100 ChatGLM3-6B 58.5% Jun 2024 Lab model card
101 Claude Instant 57.3% Aug 2023 Official leaderboard
102 CodeQwen1.5-7B 51.8% Apr 2024 Official leaderboard
103 Gemma 2 27B 51.8% Jun 2024 Lab model card
104 Phi-2 49.4% Dec 2023 Official leaderboard
105 Moonlight-16B-A3B 48.1% Feb 2025 Lab model card
106 StarCoder 2 15B 46.3% Feb 2024 Official leaderboard
107 Mixtral 8x7B 45.1% Dec 2023 Official leaderboard
108 Qwen-14B 43.9% Sep 2023 Lab model card
109 SOLAR-10.7B-Instruct-v1.0 43.3% Dec 2023 Official leaderboard
110 Gemma 1.1 7B IT 42.7% Apr 2024 Official leaderboard
111 Mistral-7B-Instruct-v0.2 42.1% Dec 2023 Official leaderboard
112 Gemma 3 1B 41.5% Mar 2025 Lab model card
113 Gemma 2 9B 40.2% Jul 2024 Lab model card
114 Qwen-7B 37.2% Sep 2023 Lab model card
115 StarCoder 2 7B 35.4% Feb 2024 Official leaderboard
116 Gemma 7B 35.4% Feb 2024 Official leaderboard
117 StarCoder 2 3B 31.7% Feb 2024 Official leaderboard
118 Mistral 7B v0.1 30.5% Sep 2023 Lab model card
119 Llama 2-70B 29.9% Jul 2023 Lab model card
120 Gemma 2B 25% Feb 2024 Official leaderboard
121 Llama 2-34B 22.6% Jul 2023 Lab model card
122 Gemma 1.1 2B IT 22.6% Apr 2024 Official leaderboard
123 Llama 2-13B 18.3% Jul 2023 Lab model card
124 Qwen-1_8B 14.6% Sep 2023 Lab model card
125 Llama 2-7B 12.8% Jul 2023 Lab model card
126 GPT-J-6B 12.2% Aug 2021 Official leaderboard
127 GPT-Neo-2.7B 7.90% Mar 2021 Official leaderboard

EvalPlus’s leaderboard has 125 entries, 46 of them models we track. Switch the scores above to rank them together.

12 models have held number one in labs’ own numbers since 2023 6 models have held number one across all independent runs since 2023 8 models have held number one with claims included since 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for EvalPlus’s leaderboard and what labs claim in their cards.

EleutherAI OpenAI Anthropic DeepSeek Google DeepMind Ai2 Reka Upstage
EvalPlus leaderboard Passed 85% in Mar 2023, by GPT-4
o1-mini 96.3%
Lab claims Passed 85% in Apr 2024, by GPT-4 Turbo
Solar Pro 2 98.8%

Sources: EvalPlus’s leaderboard and labs’ model cards

GPT-4 on the official leaderboard passed 85% in March 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On EvalPlus’s leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2021 2022 2023 2024 2025 2026 Near the ceiling (85%) Solar Pro 2 GPT-4o GPT-4 GPT-3.5 Turbo (Mar 2023) GPT-J-6B GPT-Neo-2.7B 0% 20% 40% 60% 80% 100% 2021 2023 2025 Near the ceiling (85%) Solar Pro 2 Claude 3.5 Sonnet

Sources: EvalPlus’s leaderboard and labs’ model cards

Labs headline it less often now, last in September 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
Anthropic
Qwen
Mistral AI
xAI
DeepSeek
Google DeepMind
Microsoft
Z.ai
Meta
Shanghai AI Lab
MiniMax
Ai2
Moonshot AI
Inception Labs
IBM
Tencent
Reka
ByteDance Seed
Technology Innovation Institute
Sarvam AI
Upstage
DateLaunch postScoreAs the post writes it
Sep 10, 2026 DeepSeek V4.1 Flash 79.4 HumanEval (Pass@1)
Apr 29, 2026 Granite 4.1 30B 88.4 HumanEval (pass@1)
Apr 24, 2026 DeepSeek-V4-Pro 76.8 HumanEval (Pass@1)
Mar 6, 2026 Sarvam 30B 92.1 Humaneval
Oct 28, 2025 granite-4.0-1b 74 HumanEval (pass@1)
Oct 2, 2025 Granite 4.0 H Small 88 HumanEval (pass@1)
Jul 31, 2025 Seed Diffusion Preview 82.8 Human Eval (Figure 1)
Jul 10, 2025 Solar Pro 2 98.8 HumanEval (0-shot)
Jun 26, 2025 Gemma 3n E4B 75 HumanEval pass@1 0-shot (IT)
Jun 26, 2025 Mercury 85 HumanEval (% accuracy)
May 23, 2025 Sarvam-M 88 HumanEval (0.88)
May 20, 2025 Falcon-H1-34B 87.2 HumanEval
May 8, 2025 Seed-Coder-8B-Instruct 84.8 HumanEval
May 7, 2025 Mistral Medium 3 92.1 HumanEval 0-shot
Apr 16, 2025 Granite 3.3 8B Instruct 89.7 HumanEval
Mar 27, 2025 Qwen2.5-Omni-7B 78.7 HumanEval
Mar 17, 2025 Mistral Small 3.1 88.4 HumanEval
Mar 12, 2025 Gemma 3 27B 87.8 HumanEval 0-shot (IT); PT = 48.8
Mar 10, 2025 Reka Flash 3 96.3 HumanEval
Feb 27, 2025 Hunyuan-TurboS 91 HumanEval
Feb 26, 2025 Granite 3.2 8B Instruct 89.3 HumanEval
Feb 26, 2025 Mercury Coder Small 90 HumanEval
Feb 22, 2025 Moonlight-16B-A3B 48.1 HumanEval
Jan 30, 2025 Tulu 3 405B 95.9 HumanEval (pass@10)
Jan 30, 2025 Mistral Small 3 84.8 humaneval_instruct_pass@1
Jan 28, 2025 Qwen2.5-Max HumanEval
Jan 15, 2025 MiniMax-Text-01 86.9 HumanEval
Jan 13, 2025 Codestral 86.6 HumanEval (Python)
Jan 13, 2025 InternLM3-8B-Instruct 82.3 HumanEval(Pass@1)
Dec 26, 2024 DeepSeek-V3 65.2 HumanEval (Pass@1)
Dec 12, 2024 Phi-4 82.6 HumanEval (simple-evals)
Dec 6, 2024 Llama 3.3 70B Instruct 88.4 HumanEval, 0-shot, pass@1
Oct 22, 2024 Claude 3.5 Sonnet (October 2024) 93.7 HumanEval (0-shot)
Sep 19, 2024 Qwen2.5-72B 86.6 Best of HumanEval / HumanEval+
Sep 5, 2024 DeepSeek-V2.5 89 HumanEval python
Aug 20, 2024 Phi-3.5-MoE 70.7 HumanEval (0-shot)
Aug 13, 2024 Grok-2 88.4 HumanEval (pass@1)
Jul 24, 2024 Mistral Large 2 92 Best of Human Eval / Human Eval Plus
Jul 23, 2024 Llama 3.1-405B 89 HumanEval (0-shot)
Jul 18, 2024 GPT-4o mini 87.2 HumanEval
Jun 27, 2024 Gemma 2 27B 51.8 HumanEval (pass@1)
Jun 20, 2024 Claude 3.5 Sonnet 92 HumanEval (0-shot)
Jun 7, 2024 Qwen2-72B 86 HumanEval
May 14, 2024 Gemini 1.5 Pro 84.1 HumanEval chat preamble, 0-shot
May 13, 2024 GPT-4o 90.2 HumanEval (0-shot)
May 7, 2024 DeepSeek-V2 (MoE-236B, May 2024) 81.1 HumanEval
Apr 22, 2024 phi-3-medium 14B 55.5 HumanEval (0-Shot)
Apr 18, 2024 Llama 3-70B 81.7 HumanEval (0-shot) [instruct]
Apr 11, 2024 GPT-4 Turbo 88.2 HumanEval%
Mar 28, 2024 Grok-1.5 74.1 HumanEval (0-shot, pass@1)
Mar 4, 2024 Claude 3 Opus 84.9 HumanEval (0-shot)
Feb 26, 2024 Mistral Large 45.1 HumanE (pass@1)
Feb 21, 2024 Gemma 7B 32.3 HumanEval (pass@1)
Feb 15, 2024 Gemini 1.5 Pro 71.9 HumanEval chat preamble, 0-shot
Feb 5, 2024 Qwen1.5-72B 41.5 HumanEval
Jan 16, 2024 GLM-4 72 HumanEval (0-shot)
Dec 12, 2023 Phi-2 HumanEval (in Coding average only)
Dec 11, 2023 Mixtral 8x7B 40.2 HumanE [detailed table]
Dec 6, 2023 Gemini 1.0 Ultra 74.4 HumanEval, 0-shot (IT)
Nov 29, 2023 DeepSeek LLM 67B 73.8 HumanEval
Nov 3, 2023 Grok-1 63.2 HumanEval (0-shot)
Sep 27, 2023 Mistral 7B v0.1 30.5 HumanEval
Sep 25, 2023 Qwen-14B 43.9 HumanEval (0-shot Pass@1)
Jul 11, 2023 Claude 2 71.2 Codex P@1 (0-shot)
Mar 14, 2023 GPT-4 67 HumanEval (0-shot)

70 system cards report it, from 19 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 15, 2026 Nemotron 3.5 Super 120B-A12B Base technical report 81.3 HumanEval, sampled pass@1, n=32, EvalPlus sanitized, Table 6 Capabilities
Oct 15, 2026 Nemotron 3.5 Lightning 30B-A3B Base technical report 76.9 HumanEval, sampled pass@1, n=32, EvalPlus sanitized, Table 6 Capabilities
Oct 5, 2026 Beam 1.0 Base technical report 87.3 HumanEval+, 0-shot temperature 0.7, Table 23 Capabilities
Sep 10, 2026 DeepSeek-V4.1-Flash Base model card 79.4 HumanEval, pass@1, 0-shot Capabilities
Jul 31, 2026 DiffusionGemma 26B A4B technical report 94.5 HumanEval, TD, 92.7 no-think (Table 3) Capabilities
Jun 4, 2026 Nemotron 3 Ultra 550B-A55B Base model card 83.8 HumanEval, sampled pass@1 n=32, EvalPlus sanitized Capabilities
Dec 23, 2025 Nemotron 3 Nano 30B-A3B Base technical report 78 HumanEval, 0-shot, base model Table 2 Capabilities
Sep 17, 2025 Apertus 70B technical report 73 HumanEval (Pass@10) (Table 18) Capabilities
Aug 25, 2025 InternVL3.5-241B-A28B technical report 98.2 HumanEval, OpenCompass (Table 14) Capabilities
Aug 4, 2025 Seed Diffusion Preview technical report 82.8 Human Eval (Figure 1) Capabilities
Jul 30, 2025 Falcon-H1-34B technical report 87.2 HumanEval pass@1 (Table 24) Capabilities
Jun 17, 2025 Mercury Coder Small technical report 90 HumanEval, pass@1, marked as reported by Artificial Analysis (Table 1) Capabilities
Jun 4, 2025 Seed-Coder-8B-Instruct technical report 84.8 HumanEval (HE), EvalPlus (Table 7) Capabilities
Apr 1, 2025 Cohere Command A technical report 85.4 HumanEval Python pass@1 (Table 26) Capabilities
Mar 26, 2025 Qwen2.5-Omni-7B technical report 78.7 HumanEval (Table 1) Capabilities
Mar 25, 2025 Gemma 3 4B technical report 71.3 HumanEval, IT, 0-shot pass@1 (Table 18) Capabilities
Mar 25, 2025 Gemma 3 27B technical report 87.8 HumanEval, IT, 0-shot pass@1 (Table 18) Capabilities
Mar 25, 2025 Gemma 3 1B technical report 41.5 HumanEval, IT, 0-shot pass@1 (Table 18) Capabilities
Mar 25, 2025 Gemma 3 12B technical report 85.4 HumanEval, IT, 0-shot pass@1 (Table 18) Capabilities
Mar 3, 2025 Phi-4 Mini technical report 74.4 HumanEval (0-Shot) (Tables 7 and 8) Capabilities
Feb 19, 2025 Qwen2.5-VL-72B technical report 87.8 HumanEval (Table 4) Capabilities
Jan 26, 2025 Qwen2.5-14B-Instruct-1M technical report 88.4 HumanEval (Table 6) Capabilities
Jan 14, 2025 MiniMax-Text-01 technical report 86.9 HumanEval (Table 8) Capabilities
Dec 19, 2024 Qwen2.5-7B technical report 84.8 HumanEval, Qwen2.5-7B-Instruct (Table 8) Capabilities
Dec 19, 2024 Qwen2.5-72B technical report 86.6 HumanEval, Qwen2.5-72B-Instruct (Table 6) Capabilities
Dec 19, 2024 Qwen2.5-32B technical report 88.4 HumanEval, Qwen2.5-32B-Instruct (Table 7) Capabilities
Dec 19, 2024 Qwen2.5-14B technical report 83.5 HumanEval, Qwen2.5-14B-Instruct (Table 7) Capabilities
Dec 19, 2024 Qwen-Turbo technical report 86.6 HumanEval, Qwen2.5-Turbo (Table 7) Capabilities
Dec 12, 2024 Phi-4 technical report 82.6 HumanEval, simple-evals (Table 1) Capabilities
Dec 6, 2024 Llama 3.3 70B Instruct model card 88.4 HumanEval, 0-shot, pass@1 (Instruction tuned models table) Capabilities
Nov 23, 2024 Llama 3.1-8B technical report 72.6 HumanEval (0-shot), post-trained (Table 2) Capabilities
Nov 23, 2024 Llama 3.1-70B technical report 80.5 HumanEval (0-shot), post-trained (Table 2) Capabilities
Nov 23, 2024 Llama 3.1-405B technical report 89 HumanEval (0-shot), post-trained (Table 2) Capabilities
Nov 22, 2024 Tulu 3 405B technical report 95.9 HumanEval (pass@10), RLVR (Table 4) Capabilities
Oct 22, 2024 Claude 3.5 Sonnet (October 2024) model card 93.7 HumanEval, 0-shot (Table 7) Capabilities
Oct 22, 2024 Claude 3.5 Haiku model card 88.1 HumanEval, 0-shot (Table 8) Capabilities
Aug 30, 2024 phi-3-small 7.4B technical report 61 HumanEval (0-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-mini 3.8B technical report 58.5 HumanEval (0-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-medium 14B technical report 62.2 HumanEval (0-Shot) (Section 3 table) Capabilities
Aug 30, 2024 Phi-3.5-MoE technical report 70.7 HumanEval (0-shot) (Table 3) Capabilities
Aug 30, 2024 Phi-3.5-mini technical report 61.5 HumanEval (0-shot) (Table 3) Capabilities
Jul 31, 2024 Gemma 2 9B technical report 40.2 HumanEval, PT, pass@1 (Table 13) Capabilities
Jul 31, 2024 Gemma 2 27B technical report 51.8 HumanEval, PT, pass@1 (Table 13) Capabilities
Jul 23, 2024 Llama 3.1-8B model card 72.6 HumanEval, 0-shot, pass@1, instruction tuned (Instruction tuned models table) Capabilities
Jul 23, 2024 Llama 3.1-70B model card 80.5 HumanEval, 0-shot, pass@1, instruction tuned (Instruction tuned models table) Capabilities
Jul 23, 2024 Llama 3.1-405B model card 89 HumanEval, 0-shot, pass@1, instruction tuned (Instruction tuned models table) Capabilities
Jul 15, 2024 Qwen2-72B technical report 86 HumanEval, Qwen2-72B-Instruct (Table 6). 87.0 on the non-contaminated subset (Table 15) Capabilities
Jun 20, 2024 Claude 3.5 Sonnet model card 92 HumanEval, 0-shot (Table 1) Capabilities
Jun 18, 2024 GLM-4 technical report 72 GLM-4 (0116), HumanEval (Table 2) Capabilities
Jun 18, 2024 ChatGLM3-6B technical report 58.5 ChatGLM3-6B-Base, HumanEval (Table 1) Capabilities
May 7, 2024 DeepSeek-V2 (MoE-236B, May 2024) technical report 81.1 HumanEval 0-shot, DeepSeek-V2 Chat (RL) (Table 3) Capabilities
Apr 18, 2024 Llama 3-8B model card 62.2 HumanEval (0-shot), instruction tuned (Instruction tuned models table) Capabilities
Apr 18, 2024 Llama 3-70B model card 81.7 HumanEval (0-shot), instruction tuned (Instruction tuned models table) Capabilities
Mar 13, 2024 Gemma 7B technical report 32.3 HumanEval, PT, pass@1 (Table 6) Capabilities
Mar 13, 2024 Gemma 2B technical report 22 HumanEval, PT, pass@1 (Table 6) Capabilities
Mar 4, 2024 Claude 3 Sonnet model card 73 HumanEval, 0-shot (Table 1) Capabilities
Mar 4, 2024 Claude 3 Opus model card 84.9 HumanEval, 0-shot (Table 1) Capabilities
Mar 4, 2024 Claude 3 Haiku model card 75.9 HumanEval, 0-shot (Table 1) Capabilities
Jan 8, 2024 Mixtral 8x7B technical report 40.2 HumanE, 0-shot (Table 2) Capabilities
Jan 5, 2024 DeepSeek LLM 67B technical report 73.8 HumanEval (Python), DeepSeek 67B Chat (Table 6) Capabilities
Oct 10, 2023 Mistral 7B v0.1 technical report 30.5 HumanEval, 0-shot (Table 2) Capabilities
Sep 28, 2023 Qwen-7B technical report 37.2 HumanEval 0-shot pass@1 (Table 5, Table 10) Capabilities
Sep 28, 2023 Qwen-14B technical report 43.9 HumanEval 0-shot pass@1 (Table 5, Table 10) Capabilities
Sep 28, 2023 Qwen-1_8B technical report 14.6 HumanEval 0-shot pass@1 (Table 5, Table 10) Capabilities
Jul 19, 2023 Llama 2-7B technical report 12.8 Human-Eval pass@1, 0-shot, pretrained (Table 21). pass@100 = 45.6 Capabilities
Jul 19, 2023 Llama 2-70B technical report 29.9 Human-Eval pass@1, 0-shot, pretrained (Table 21). pass@100 = 89.0 Capabilities
Jul 19, 2023 Llama 2-34B technical report 22.6 Human-Eval pass@1, 0-shot, pretrained (Table 21). pass@100 = 77.2 Capabilities
Jul 19, 2023 Llama 2-13B technical report 18.3 Human-Eval pass@1, 0-shot, pretrained (Table 21). pass@100 = 60.2 Capabilities
Jul 11, 2023 Claude 2 model card 71.2 Codex P@1 (0-shot) (4.3) Capabilities
Mar 14, 2023 GPT-4 technical report 67 HumanEval, 0-shot (Table 2, base model) Capabilities

The lab's number comes out ahead in 8 of 20 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap
Qwen1.5-72B 41.5% HumanEval 68.3% Leaderboard −27 pts
DeepSeek-V3 65.2% HumanEval (Pass@1) 91.5% Leaderboard −26 pts
Mistral Large 45.1% HumanE (pass@1) 69.5% Leaderboard −24 pts
GPT-4 Mar 2023 67% HumanEval (0-shot) 88.4% Leaderboard −21 pts
phi-3-mini 3.8B 58.5% HumanEval (0-Shot) (Section 3 table) 64.6% Leaderboard −6.1 pts
Mixtral 8x7B 40.2% HumanE [detailed table] 45.1% Leaderboard −4.9 pts
Claude 3.5 Sonnet 92% HumanEval (0-shot) 87.2% Leaderboard +4.8 pts
Llama 3-70B 81.7% HumanEval (0-shot) [instruct] 77.4% Leaderboard +4.3 pts
Gemma 7B 32.3% HumanEval (pass@1) 35.4% Leaderboard −3.1 pts
Llama 3.1-8B 72.6% HumanEval, 0-shot, pass@1, instruction tuned (Instruction tuned models table) 69.5% Leaderboard +3.1 pts
Gemma 2B 22% HumanEval, PT, pass@1 (Table 6) 25% Leaderboard −3.0 pts
Claude 3 Sonnet 73% HumanEval, 0-shot (Table 1) 70.7% Leaderboard +2.3 pts
Claude 3 Opus 84.9% HumanEval (0-shot) 82.9% Leaderboard +2.0 pts
GPT-4 Turbo Apr 2024 88.2% HumanEval% 90.2% Leaderboard −2.0 pts
Mistral 7B v0.1 30.5% HumanEval 28.7% Leaderboard +1.8 pts
Claude 2 71.2% Codex P@1 (0-shot) 69.5% Leaderboard +1.7 pts
GPT-4o mini 87.2% HumanEval 88.4% Leaderboard −1.2 pts
DeepSeek-V2.5 Sep 2024 89% HumanEval python 90.2% Leaderboard −1.2 pts
Claude 3 Haiku 75.9% HumanEval, 0-shot (Table 1) 76.8% Leaderboard −0.9 pts
Llama 3-8B 62.2% HumanEval (0-shot), instruction tuned (Instruction tuned models table) 61.6% Leaderboard +0.6 pts

Versions

Replaced by LiveCodeBench

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471