Benchmark Almanac

SciCode

Seed2.1 Pro 59.8% leads since June 2026
o3-mini 10.8% leads since January 2025
Seed2.1 Pro 59.8% leads since June 2026
In use

Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases.

Mistral cites 'SciCode-Verified' (pass@1, n=6). Kimi uses Artificial Analysis numbers. Test-set answers partly held out.

Made by
SciCode team
Released
July 2024
Kind
Scientific code
Level
Expert
Scored
Checked automatically
Answers
Code
Tasks
All public
Strongest model tested
Kimi K3 #24 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026

In the authors’ words

Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.

Abstract of SciCode: A Research Coding Benchmark Curated by Scientists (arXiv:2407.13168, CC0)
49
Models scored in lab cards
41
Launch posts list it, from 21 labs
26
System cards report it, from 10 labs

Seed2.1 Pro leads with 59.8%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

o3-mini leads with 10.8% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

Seed2.1 Pro leads with 59.8%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

49 models. Scroll sideways for the rest.

  1. 59.8 Seed2.1 Pro
  2. 59.0 Gemini 3.1 Pro
  3. 58.9 Step 5 Preview
  4. 58.7 Kimi K3
  5. 58.5 Muse Spark Open
  6. 53.5 Qwen3.7-Max
  7. 52.2 Kimi K2.6
  8. 51.3 Qwen3.7-Plus
  9. 50.3 KAT-Coder-Pro V2.5
  10. 49.7 Beam 1.0
  11. 48.7 Kimi K2.5
  12. 48.7 Inkling-Small
  13. 48.5 Seed2.0 Pro
  14. 47.6 Intern-S2-397B
  15. 47.6 Solar Mini 4
  16. 47.1 Nemotron 3.5 Super VL
  17. 47.0 Qwen 3.6 Max (Preview)
  18. 47.0 MiniMax-M2.7
  19. 44.8 Kimi K2 Thinking
  20. 44.6 Nemotron 3 Ultra
  21. 44.4 MiniMax-M2.5
  22. 44.2 KAT-Coder-V2.5-Dev
  23. 43.6 Muse Glimmer
  24. 43.0 Gemma 4 31B IT
  25. 42.0 Nemotron 3 Super
  26. 41.7 GLM-4.5
  27. 41.0 Nemotron 3.5 Bolt 74B-A9B
  28. 41.0 MiniMax-M2.1
  29. 40.1 K-EXAONE 2.0
  30. 40.0 Gemma 4 26B A4B
  31. 38.8 Granite 4.2 30B
  32. 38.2 North Mini Code 1.0
  33. 38.0 Mercury 2
  34. 38.0 Command A+
  35. 38.0 Mercury 2.5
  36. 37.3 GLM-4.5-Air
  37. 36.8 Spark-X2.6-4B
  38. 36.4 Nemotron-Cascade 2 30B-A3B
  39. 36.0 MiniMax-M2
  40. 33.3 Nemotron 3 Nano 30B-A3B
  41. 32.7 Nemotron 3.5 Lightning 30B-A3B
  42. 32.7 LFM2.5-24B-A2B
  43. 32.0 Nemotron 3 Nano Omni 30B-A3B
  44. 28.5 LFM3-1.7B
  45. 28.3 Falcon H1R 7B
  46. 24.0 Gemma 4 E4B
  47. 21.0 Gemma 4 E2B
  48. 18.0 Mercury
  49. 17.6 Nemotron Nano 12B v2 VL
  1. 1 Seed2.1 Pro 59.8%
  2. 2 Gemini 3.1 Pro 59%
  3. 3 Step 5 Preview 58.9%
  4. 4 Kimi K3 58.7%
  5. 5 Muse Spark Open 58.5%
  6. 6 Qwen3.7-Max 53.5%
  7. 7 Kimi K2.6 52.2%
  8. 8 Qwen3.7-Plus 51.3%
  9. 9 KAT-Coder-Pro V2.5 50.3%
  10. 10 Beam 1.0 49.7%
  11. 11 Kimi K2.5 48.7%
  12. 12 Inkling-Small 48.7%
  13. 13 Seed2.0 Pro 48.5%
  14. 14 Intern-S2-397B 47.6%
  15. 15 Solar Mini 4 47.6%
  16. 16 Nemotron 3.5 Super VL 47.1%
  17. 17 Qwen 3.6 Max (Preview) 47%
  18. 18 MiniMax-M2.7 47%
  19. 19 Kimi K2 Thinking 44.8%
  20. 20 Nemotron 3 Ultra 44.6%
  21. 21 MiniMax-M2.5 44.4%
  22. 22 KAT-Coder-V2.5-Dev 44.2%
  23. 23 Muse Glimmer 43.6%
  24. 24 Gemma 4 31B IT 43%
  25. 25 Nemotron 3 Super 42%
  26. 26 GLM-4.5 41.7%
  27. 27 Nemotron 3.5 Bolt 74B-A9B 41%
  28. 28 MiniMax-M2.1 41%
  29. 29 K-EXAONE 2.0 40.1%
  30. 30 Gemma 4 26B A4B 40%
  31. 31 Granite 4.2 30B 38.8%
  32. 32 North Mini Code 1.0 38.2%
  33. 33 Mercury 2 38%
  34. 34 Command A+ 38%
  35. 35 Mercury 2.5 38%
  36. 36 GLM-4.5-Air 37.3%
  37. 37 Spark-X2.6-4B 36.8%
  38. 38 Nemotron-Cascade 2 30B-A3B 36.4%
  39. 39 MiniMax-M2 36%
  40. 40 Nemotron 3 Nano 30B-A3B 33.3%
  41. 41 Nemotron 3.5 Lightning 30B-A3B 32.7%
  42. 42 LFM2.5-24B-A2B 32.7%
  43. 43 Nemotron 3 Nano Omni 30B-A3B 32%
  44. 44 LFM3-1.7B 28.5%
  45. 45 Falcon H1R 7B 28.3%
  46. 46 Gemma 4 E4B 24%
  47. 47 Gemma 4 E2B 21%
  48. 48 Mercury 18%
  49. 49 Nemotron Nano 12B v2 VL 17.6%

Source: Labs’ model cards

18 models. Scroll sideways for the rest.

  1. 10.8 o3-mini
  2. 7.70 o1-preview
  3. 4.60 Claude 3.5 Sonnet
  4. 4.60 Claude 3.5 Sonnet (October 2024)
  5. 4.60 DeepSeek-R1
  6. 3.10 DeepSeek-Coder-V2 236B
  7. 3.10 DeepSeek-V3
  8. 1.50 Claude 3 Opus
  9. 1.50 Claude 3 Sonnet
  10. 1.50 GPT-4 Turbo
  11. 1.50 GPT-4o
  12. 1.50 Gemini 1.5 Pro
  13. 1.50 Qwen2-72B
  14. 1.50 Llama 3.1-405B
  15. 1.50 o1-mini
  16. 0.0 Mixtral 8x22B
  17. 0.0 Llama 3-70B
  18. 0.0 Llama 3.1-70B
  1. 1 o3-mini low 10.8%
  2. 2 o1-preview 7.70%
  3. 3 Claude 3.5 Sonnet 4.60%
  4. 4 Claude 3.5 Sonnet (October 2024) 4.60%
  5. 5 DeepSeek-R1 4.60%
  6. 6 DeepSeek-Coder-V2 236B 3.10%
  7. 7 DeepSeek-V3 3.10%
  8. 8 Claude 3 Opus 1.50%
  9. 9 Claude 3 Sonnet 1.50%
  10. 10 GPT-4 Turbo Apr 2024 1.50%
  11. 11 GPT-4o May 2024 1.50%
  12. 12 Gemini 1.5 Pro May 2024 1.50%
  13. 13 Qwen2-72B 1.50%
  14. 14 Llama 3.1-405B 1.50%
  15. 15 o1-mini 1.50%
  16. 16 Mixtral 8x22B 0%
  17. 17 Llama 3-70B 0%
  18. 18 Llama 3.1-70B 0%

Source: Official SciCode leaderboard

67 models. Scroll sideways for the rest.

  1. 59.8 Seed2.1 Pro
  2. 59.0 Gemini 3.1 Pro
  3. 58.9 Step 5 Preview
  4. 58.7 Kimi K3
  5. 58.5 Muse Spark Open
  6. 53.5 Qwen3.7-Max
  7. 52.2 Kimi K2.6
  8. 51.3 Qwen3.7-Plus
  9. 50.3 KAT-Coder-Pro V2.5
  10. 49.7 Beam 1.0
  11. 48.7 Kimi K2.5
  12. 48.7 Inkling-Small
  13. 48.5 Seed2.0 Pro
  14. 47.6 Intern-S2-397B
  15. 47.6 Solar Mini 4
  16. 47.1 Nemotron 3.5 Super VL
  17. 47.0 Qwen 3.6 Max (Preview)
  18. 47.0 MiniMax-M2.7
  19. 44.8 Kimi K2 Thinking
  20. 44.6 Nemotron 3 Ultra
  21. 44.4 MiniMax-M2.5
  22. 44.2 KAT-Coder-V2.5-Dev
  23. 43.6 Muse Glimmer
  24. 43.0 Gemma 4 31B IT
  25. 42.0 Nemotron 3 Super
  26. 41.7 GLM-4.5
  27. 41.0 Nemotron 3.5 Bolt 74B-A9B
  28. 41.0 MiniMax-M2.1
  29. 40.1 K-EXAONE 2.0
  30. 40.0 Gemma 4 26B A4B
  31. 38.8 Granite 4.2 30B
  32. 38.2 North Mini Code 1.0
  33. 38.0 Mercury 2
  34. 38.0 Command A+
  35. 38.0 Mercury 2.5
  36. 37.3 GLM-4.5-Air
  37. 36.8 Spark-X2.6-4B
  38. 36.4 Nemotron-Cascade 2 30B-A3B
  39. 36.0 MiniMax-M2
  40. 33.3 Nemotron 3 Nano 30B-A3B
  41. 32.7 Nemotron 3.5 Lightning 30B-A3B
  42. 32.7 LFM2.5-24B-A2B
  43. 32.0 Nemotron 3 Nano Omni 30B-A3B
  44. 28.5 LFM3-1.7B
  45. 28.3 Falcon H1R 7B
  46. 24.0 Gemma 4 E4B
  47. 21.0 Gemma 4 E2B
  48. 18.0 Mercury
  49. 17.6 Nemotron Nano 12B v2 VL
  50. 10.8 o3-mini
  51. 7.70 o1-preview
  52. 4.60 Claude 3.5 Sonnet
  53. 4.60 Claude 3.5 Sonnet (October 2024)
  54. 4.60 DeepSeek-R1
  55. 3.10 DeepSeek-Coder-V2 236B
  56. 3.10 DeepSeek-V3
  57. 1.50 Claude 3 Opus
  58. 1.50 Claude 3 Sonnet
  59. 1.50 GPT-4 Turbo
  60. 1.50 GPT-4o
  61. 1.50 Gemini 1.5 Pro
  62. 1.50 Qwen2-72B
  63. 1.50 Llama 3.1-405B
  64. 1.50 o1-mini
  65. 0.0 Mixtral 8x22B
  66. 0.0 Llama 3-70B
  67. 0.0 Llama 3.1-70B
  1. 1 Seed2.1 Pro 59.8%
  2. 2 Gemini 3.1 Pro 59%
  3. 3 Step 5 Preview 58.9%
  4. 4 Kimi K3 58.7%
  5. 5 Muse Spark Open 58.5%
  6. 6 Qwen3.7-Max 53.5%
  7. 7 Kimi K2.6 52.2%
  8. 8 Qwen3.7-Plus 51.3%
  9. 9 KAT-Coder-Pro V2.5 50.3%
  10. 10 Beam 1.0 49.7%
  11. 11 Kimi K2.5 48.7%
  12. 12 Inkling-Small 48.7%
  13. 13 Seed2.0 Pro 48.5%
  14. 14 Intern-S2-397B 47.6%
  15. 15 Solar Mini 4 47.6%
  16. 16 Nemotron 3.5 Super VL 47.1%
  17. 17 Qwen 3.6 Max (Preview) 47%
  18. 18 MiniMax-M2.7 47%
  19. 19 Kimi K2 Thinking 44.8%
  20. 20 Nemotron 3 Ultra 44.6%
  21. 21 MiniMax-M2.5 44.4%
  22. 22 KAT-Coder-V2.5-Dev 44.2%
  23. 23 Muse Glimmer 43.6%
  24. 24 Gemma 4 31B IT 43%
  25. 25 Nemotron 3 Super 42%
  26. 26 GLM-4.5 41.7%
  27. 27 Nemotron 3.5 Bolt 74B-A9B 41%
  28. 28 MiniMax-M2.1 41%
  29. 29 K-EXAONE 2.0 40.1%
  30. 30 Gemma 4 26B A4B 40%
  31. 31 Granite 4.2 30B 38.8%
  32. 32 North Mini Code 1.0 38.2%
  33. 33 Mercury 2 38%
  34. 34 Command A+ 38%
  35. 35 Mercury 2.5 38%
  36. 36 GLM-4.5-Air 37.3%
  37. 37 Spark-X2.6-4B 36.8%
  38. 38 Nemotron-Cascade 2 30B-A3B 36.4%
  39. 39 MiniMax-M2 36%
  40. 40 Nemotron 3 Nano 30B-A3B 33.3%
  41. 41 Nemotron 3.5 Lightning 30B-A3B 32.7%
  42. 42 LFM2.5-24B-A2B 32.7%
  43. 43 Nemotron 3 Nano Omni 30B-A3B 32%
  44. 44 LFM3-1.7B 28.5%
  45. 45 Falcon H1R 7B 28.3%
  46. 46 Gemma 4 E4B 24%
  47. 47 Gemma 4 E2B 21%
  48. 48 Mercury 18%
  49. 49 Nemotron Nano 12B v2 VL 17.6%
  50. 50 o3-mini low 10.8%
  51. 51 o1-preview 7.70%
  52. 52 Claude 3.5 Sonnet 4.60%
  53. 53 Claude 3.5 Sonnet (October 2024) 4.60%
  54. 54 DeepSeek-R1 4.60%
  55. 55 DeepSeek-Coder-V2 236B 3.10%
  56. 56 DeepSeek-V3 3.10%
  57. 57 Claude 3 Opus 1.50%
  58. 58 Claude 3 Sonnet 1.50%
  59. 59 GPT-4 Turbo Apr 2024 1.50%
  60. 60 GPT-4o May 2024 1.50%
  61. 61 Gemini 1.5 Pro May 2024 1.50%
  62. 62 Qwen2-72B 1.50%
  63. 63 Llama 3.1-405B 1.50%
  64. 64 o1-mini 1.50%
  65. 65 Mixtral 8x22B 0%
  66. 66 Llama 3-70B 0%
  67. 67 Llama 3.1-70B 0%

Sources: Labs’ model cards and official SciCode leaderboard

#ModelScoreSettingDateSource
1 Seed2.1 Pro 59.8% Jun 2026 Lab model card
2 Gemini 3.1 Pro 59% Feb 2026 Lab model card
3 Step 5 Preview 58.9% Sep 2026 Lab model card
4 Kimi K3 58.7% Jul 2026 Lab model card
5 Muse Spark Open 58.5% Oct 2026 Lab model card
6 Qwen3.7-Max 53.5% May 2026 Lab model card
7 Kimi K2.6 52.2% Apr 2026 Lab model card
8 Qwen3.7-Plus 51.3% Jun 2026 Lab model card
9 KAT-Coder-Pro V2.5 50.3% Jul 2026 Lab model card
10 Beam 1.0 49.7% Oct 2026 Lab model card
11 Kimi K2.5 48.7% Jan 2026 Lab model card
12 Inkling-Small 48.7% Jul 2026 Lab model card
13 Seed2.0 Pro 48.5% Feb 2026 Lab model card
14 Intern-S2-397B 47.6% Sep 2026 Lab model card
15 Solar Mini 4 47.6% Oct 2026 Lab model card
16 Nemotron 3.5 Super VL 47.1% Oct 2026 Lab model card
17 Qwen 3.6 Max (Preview) 47% Apr 2026 Lab model card
18 MiniMax-M2.7 47% May 2026 Lab model card
19 Kimi K2 Thinking 44.8% Nov 2025 Lab model card
20 Nemotron 3 Ultra 44.6% Jun 2026 Lab model card
21 MiniMax-M2.5 44.4% Feb 2026 Lab model card
22 KAT-Coder-V2.5-Dev 44.2% Jul 2026 Lab model card
23 Muse Glimmer 43.6% Aug 2026 Lab model card
24 Gemma 4 31B IT 43% Jul 2026 Lab model card
25 Nemotron 3 Super 42% Mar 2026 Lab model card
26 GLM-4.5 41.7% Jul 2025 Lab model card
27 Nemotron 3.5 Bolt 74B-A9B 41% Oct 2026 Lab model card
28 MiniMax-M2.1 41% Dec 2025 Lab model card
29 K-EXAONE 2.0 40.1% Aug 2026 Lab model card
30 Gemma 4 26B A4B 40% Jul 2026 Lab model card
31 Granite 4.2 30B 38.8% Aug 2026 Lab model card
32 North Mini Code 1.0 38.2% Jun 2026 Lab model card
33 Mercury 2 38% Feb 2026 Lab model card
34 Command A+ 38% May 2026 Lab model card
35 Mercury 2.5 38% Sep 2026 Lab model card
36 GLM-4.5-Air 37.3% Aug 2025 Lab model card
37 Spark-X2.6-4B 36.8% Oct 2026 Lab model card
38 Nemotron-Cascade 2 30B-A3B 36.4% Mar 2026 Lab model card
39 MiniMax-M2 36% Oct 2025 Lab model card
40 Nemotron 3 Nano 30B-A3B 33.3% Dec 2025 Lab model card
41 Nemotron 3.5 Lightning 30B-A3B 32.7% Oct 2026 Lab model card
42 LFM2.5-24B-A2B 32.7% Oct 2026 Lab model card
43 Nemotron 3 Nano Omni 30B-A3B 32% Apr 2026 Lab model card
44 LFM3-1.7B 28.5% Oct 2026 Lab model card
45 Falcon H1R 7B 28.3% Jan 2026 Lab model card
46 Gemma 4 E4B 24% Jul 2026 Lab model card
47 Gemma 4 E2B 21% Jul 2026 Lab model card
48 Mercury 18% Jun 2025 Lab model card
49 Nemotron Nano 12B v2 VL 17.6% Nov 2025 Lab model card
#ModelScoreSettingDateSource
1 o3-mini 10.8% low Jan 2025 Official leaderboard
2 o1-preview 7.70% Sep 2024 Official leaderboard
3 Claude 3.5 Sonnet 4.60% Jun 2024 Official leaderboard
4 Claude 3.5 Sonnet (October 2024) 4.60% Oct 2024 Official leaderboard
5 DeepSeek-R1 4.60% Jan 2025 Official leaderboard
6 DeepSeek-Coder-V2 236B 3.10% Jun 2024 Official leaderboard
7 DeepSeek-V3 3.10% Dec 2024 Official leaderboard
8 Claude 3 Opus 1.50% Feb 2024 Official leaderboard
9 Claude 3 Sonnet 1.50% Feb 2024 Official leaderboard
10 GPT-4 Turbo Apr 2024 1.50% Apr 2024 Official leaderboard
11 GPT-4o May 2024 1.50% May 2024 Official leaderboard
12 Gemini 1.5 Pro May 2024 1.50% May 2024 Official leaderboard
13 Qwen2-72B 1.50% Jun 2024 Official leaderboard
14 Llama 3.1-405B 1.50% Jul 2024 Official leaderboard
15 o1-mini 1.50% Sep 2024 Official leaderboard
16 Mixtral 8x22B 0% Apr 2024 Official leaderboard
17 Llama 3-70B 0% Apr 2024 Official leaderboard
18 Llama 3.1-70B 0% Jul 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Seed2.1 Pro 59.8% Jun 2026 Lab model card
2 Gemini 3.1 Pro 59% Feb 2026 Lab model card
3 Step 5 Preview 58.9% Sep 2026 Lab model card
4 Kimi K3 58.7% Jul 2026 Lab model card
5 Muse Spark Open 58.5% Oct 2026 Lab model card
6 Qwen3.7-Max 53.5% May 2026 Lab model card
7 Kimi K2.6 52.2% Apr 2026 Lab model card
8 Qwen3.7-Plus 51.3% Jun 2026 Lab model card
9 KAT-Coder-Pro V2.5 50.3% Jul 2026 Lab model card
10 Beam 1.0 49.7% Oct 2026 Lab model card
11 Kimi K2.5 48.7% Jan 2026 Lab model card
12 Inkling-Small 48.7% Jul 2026 Lab model card
13 Seed2.0 Pro 48.5% Feb 2026 Lab model card
14 Intern-S2-397B 47.6% Sep 2026 Lab model card
15 Solar Mini 4 47.6% Oct 2026 Lab model card
16 Nemotron 3.5 Super VL 47.1% Oct 2026 Lab model card
17 Qwen 3.6 Max (Preview) 47% Apr 2026 Lab model card
18 MiniMax-M2.7 47% May 2026 Lab model card
19 Kimi K2 Thinking 44.8% Nov 2025 Lab model card
20 Nemotron 3 Ultra 44.6% Jun 2026 Lab model card
21 MiniMax-M2.5 44.4% Feb 2026 Lab model card
22 KAT-Coder-V2.5-Dev 44.2% Jul 2026 Lab model card
23 Muse Glimmer 43.6% Aug 2026 Lab model card
24 Gemma 4 31B IT 43% Jul 2026 Lab model card
25 Nemotron 3 Super 42% Mar 2026 Lab model card
26 GLM-4.5 41.7% Jul 2025 Lab model card
27 Nemotron 3.5 Bolt 74B-A9B 41% Oct 2026 Lab model card
28 MiniMax-M2.1 41% Dec 2025 Lab model card
29 K-EXAONE 2.0 40.1% Aug 2026 Lab model card
30 Gemma 4 26B A4B 40% Jul 2026 Lab model card
31 Granite 4.2 30B 38.8% Aug 2026 Lab model card
32 North Mini Code 1.0 38.2% Jun 2026 Lab model card
33 Mercury 2 38% Feb 2026 Lab model card
34 Command A+ 38% May 2026 Lab model card
35 Mercury 2.5 38% Sep 2026 Lab model card
36 GLM-4.5-Air 37.3% Aug 2025 Lab model card
37 Spark-X2.6-4B 36.8% Oct 2026 Lab model card
38 Nemotron-Cascade 2 30B-A3B 36.4% Mar 2026 Lab model card
39 MiniMax-M2 36% Oct 2025 Lab model card
40 Nemotron 3 Nano 30B-A3B 33.3% Dec 2025 Lab model card
41 Nemotron 3.5 Lightning 30B-A3B 32.7% Oct 2026 Lab model card
42 LFM2.5-24B-A2B 32.7% Oct 2026 Lab model card
43 Nemotron 3 Nano Omni 30B-A3B 32% Apr 2026 Lab model card
44 LFM3-1.7B 28.5% Oct 2026 Lab model card
45 Falcon H1R 7B 28.3% Jan 2026 Lab model card
46 Gemma 4 E4B 24% Jul 2026 Lab model card
47 Gemma 4 E2B 21% Jul 2026 Lab model card
48 Mercury 18% Jun 2025 Lab model card
49 Nemotron Nano 12B v2 VL 17.6% Nov 2025 Lab model card
50 o3-mini 10.8% low Jan 2025 Official leaderboard
51 o1-preview 7.70% Sep 2024 Official leaderboard
52 Claude 3.5 Sonnet 4.60% Jun 2024 Official leaderboard
53 Claude 3.5 Sonnet (October 2024) 4.60% Oct 2024 Official leaderboard
54 DeepSeek-R1 4.60% Jan 2025 Official leaderboard
55 DeepSeek-Coder-V2 236B 3.10% Jun 2024 Official leaderboard
56 DeepSeek-V3 3.10% Dec 2024 Official leaderboard
57 Claude 3 Opus 1.50% Feb 2024 Official leaderboard
58 Claude 3 Sonnet 1.50% Feb 2024 Official leaderboard
59 GPT-4 Turbo Apr 2024 1.50% Apr 2024 Official leaderboard
60 GPT-4o May 2024 1.50% May 2024 Official leaderboard
61 Gemini 1.5 Pro May 2024 1.50% May 2024 Official leaderboard
62 Qwen2-72B 1.50% Jun 2024 Official leaderboard
63 Llama 3.1-405B 1.50% Jul 2024 Official leaderboard
64 o1-mini 1.50% Sep 2024 Official leaderboard
65 Mixtral 8x22B 0% Apr 2024 Official leaderboard
66 Llama 3-70B 0% Apr 2024 Official leaderboard
67 Llama 3.1-70B 0% Jul 2024 Official leaderboard

The benchmark's own leaderboard has 18 models. Switch the scores above to rank them together.

6 models have held number one in labs’ own numbers since its release in July 2024 3 models have held number one across all independent runs since its release in July 2024 9 models have held number one with claims included since its release in July 2024

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Anthropic OpenAI Inception Labs Z.ai Moonshot AI Google DeepMind ByteDance Seed
Own leaderboard Not tested since Jan 2025
o3-mini 10.8%
Lab claims 6 leaders
Seed2.1 Pro 59.8%

Sources: Official SciCode leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Seed2.1 Pro Kimi K2.5 GLM-4.5 Mercury o3-mini Claude 3.5 Sonnet 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Seed2.1 Pro Kimi K2.5

Sources: Official SciCode leaderboard and labs’ model cards

17 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Inception Labs
Z.ai
MiniMax
Moonshot AI
NVIDIA
Technology Innovation Institute
ByteDance Seed
Google DeepMind
Qwen
Cohere
Kuaishou
Thinking Machines
LG AI Research
Meta
IBM
Shanghai AI Lab
StepFun
Upstage
Reflection
Liquid AI
iFlytek
DateLaunch postScoreAs the post writes it
Oct 27, 2026 LFM3-1.7B 28.5 SciCode
Oct 22, 2026 Spark-X2.6-4B 36.8 SciCode
Oct 15, 2026 Muse Spark Open 58.5 SciCode
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B 41 SciCode
Oct 15, 2026 Nemotron 3.5 Super VL 47.1 SciCode, subtask
Oct 13, 2026 LFM2.5-24B-A2B 32.7 SciCode
Oct 5, 2026 Beam 1.0 49.7 SciCode
Oct 1, 2026 Solar Mini 4 47.6 SciCode (Artificial Analysis score quoted by Upstage)
Sep 20, 2026 Step 5 Preview 58.9 SciCode
Sep 13, 2026 Intern-S2-397B 47.6 SciCode (Agentic Scientific Coding)
Sep 8, 2026 Mercury 2.5 38 SciCode
Aug 25, 2026 Granite 4.2 30B 38.8 SciCode
Aug 11, 2026 Nemotron 3.5 Lightning 30B-A3B 32.6 SciCode
Aug 10, 2026 Muse Glimmer 43.6 SciCode
Jul 31, 2026 K-EXAONE 2.0 37.4 SciCode
Jul 30, 2026 Inkling-Small 48.7 SciCode
Jul 23, 2026 KAT-Coder-V2.5-Dev 44.2 Scicode
Jul 16, 2026 Kimi K3 58.7 SciCode
Jul 6, 2026 KAT-Coder-Pro V2.5 50.3 SciCode (AA Coding Index category)
Jun 23, 2026 Seed2.1 Pro 59.8 SciCode
Jun 9, 2026 North Mini Code 1.0 38.2 SciCode
Jun 4, 2026 Nemotron 3 Ultra 44.6 SciCode (subtask)
Jun 1, 2026 Qwen3.7-Plus 51.3 SciCode
May 20, 2026 Command A+ 38 SciCode (65 problems, 288 subproblems, with background)
May 20, 2026 Qwen3.7-Max 53.5 SciCode
Apr 20, 2026 Qwen 3.6 Max (Preview) 47 SciCode Research Coding
Apr 20, 2026 Kimi K2.6 52.2 SciCode
Mar 19, 2026 Nemotron-Cascade 2 30B-A3B 36.4 SciCode
Mar 11, 2026 Nemotron 3 Super 42 SciCode (subtask)
Feb 24, 2026 Mercury 2 38 SciCode
Feb 19, 2026 Gemini 3.1 Pro 59 SciCode
Feb 14, 2026 Seed2.0 Pro 48.5 Scicode
Feb 12, 2026 MiniMax-M2.5 44.4 SciCode
Jan 27, 2026 Kimi K2.5 48.7 SciCode
Jan 5, 2026 Falcon H1R 7B 28.3 SciCode (sub-problem)
Dec 23, 2025 MiniMax-M2.1 41 SciCode
Dec 15, 2025 Nemotron 3 Nano 30B-A3B 33.3 SciCode (subtask)
Nov 6, 2025 Kimi K2 Thinking 44.8 SciCode, no tools
Oct 27, 2025 MiniMax-M2 36 SciCode (AA)
Jul 28, 2025 GLM-4.5 41.7 SciCode
Jun 26, 2025 Mercury 18 SciCode (% accuracy)

26 system cards report it, from 10 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 19, 2026 Muse Glimmer technical report 43.6 SciCode, high, 39.2 xhigh (Table 2) Capabilities
Oct 15, 2026 Nemotron 3.5 Super VL technical report 47.1 SciCode (subtask), problem accuracy 15.10 (Table 10) Capabilities
Oct 15, 2026 Nemotron 3.5 Lightning 30B-A3B technical report 32.7 SciCode (subtask), BF16 acc. (Table 22) Capabilities
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B technical report 41 SciCode, released 24GB checkpoint (Table 7). BF16 44.49 Capabilities
Aug 5, 2026 K-EXAONE 2.0 technical report 40.1 SciCode. The model card prints 37.4 (Table 6, Reasoning mode) Capabilities
Jul 30, 2026 Inkling-Small model card 48.7 SciCode (5. Evaluations table) Capabilities
Jul 27, 2026 Kimi K3 technical report 58.7 SciCode, cited from Artificial Analysis (Table 2) Capabilities
Jul 24, 2026 Gemma 4 E4B technical report 24 SciCode, thinking (Table 5) Capabilities
Jul 24, 2026 Gemma 4 E2B technical report 21 SciCode, thinking (Table 5) Capabilities
Jul 24, 2026 Gemma 4 31B IT technical report 43 SciCode, thinking (Table 5) Capabilities
Jul 24, 2026 Gemma 4 26B A4B technical report 40 SciCode, thinking (Table 5) Capabilities
Jun 23, 2026 Seed2.1 Pro model card 59.8 SciCode (Table 11) Capabilities
Jun 9, 2026 Nemotron 3 Ultra technical report 44.6 SciCode (subtask) (Table 10) Capabilities
May 26, 2026 MiniMax-M2.7 technical report 47 SciCode, AA Index v4.0 protocol (Table 4) Capabilities
May 26, 2026 MiniMax-M2.5 technical report 43 SciCode, AA Index v4.0 protocol (Table 4) Capabilities
Apr 27, 2026 Nemotron 3 Nano Omni 30B-A3B technical report 32 SciCode, 1 run (Table 10) Capabilities
Mar 19, 2026 Nemotron-Cascade 2 30B-A3B technical report 36.4 SciCode (Table 1) Capabilities
Mar 11, 2026 Nemotron 3 Super technical report 42 SciCode (subtask) (Table 5) Capabilities
Feb 19, 2026 Gemini 3.1 Pro model card 59 SciCode, printed as whole percent (results table, p.4) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 48.5 Scicode (Table 11). Table 13 prints 52.1 Capabilities
Jan 30, 2026 Kimi K2.5 technical report 48.7 SciCode (Table 4) Capabilities
Jan 5, 2026 Falcon H1R 7B technical report 28.3 SciCode sub-problem 28.3, main problem 3.9 (Table 5) Capabilities
Dec 23, 2025 Nemotron 3 Nano 30B-A3B technical report 33.3 SciCode (subtask) (Table 3) Capabilities
Nov 6, 2025 Nemotron Nano 12B v2 VL technical report 17.6 SciCode_subtask_accuracy, reasoning on, SFT stage 4 (Table 6). Problem accuracy 6.9 Capabilities
Aug 8, 2025 GLM-4.5 technical report 41.7 SciCode (Table 4) Capabilities
Aug 8, 2025 GLM-4.5-Air technical report 37.3 SciCode (Table 4) Capabilities

One logged change that moves scores

Changes by the maintainers, so a score from before a change may not compare with one from after it. Newest first, each with the maintainers' own source. The list may be incomplete. What we log.

  1. Harness changed July 24th, 2024

    The SciCode maintainers added scientist-written background and an evaluation setting that supplies it. Scores from this setting measure performance with extra scientific guidance and differ from scores without that guidance. Source

More in Coding

All 282 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471
  • NL2Repo-Bench In use From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. DeepSeek V4.1 Flash 1017 68.2%