Benchmark Almanac

KOR-Bench

o1-preview 72.9% leads since October 2024
Seed2.0 Pro 77.5% leads since February 2026
Fading

Models read a new rule set and apply it to questions in five task types: operation, logic, cipher, puzzle and counterfactual. Scored by accuracy against an answer key.

The rules are designed to be independent of domain knowledge. ByteDance Seed reports it in Seed-OSS and Seed 1.6 launch posts.

Released
October 2024
Kind
Abstract puzzles and logic
Level
University
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Strongest model tested
o1-mini #140 on the capability index
Last in a launch post
February 2026, dropped before it saturated
Last model added
March 2026
Scores refreshed
November 12th, 2026

In the authors’ words

In this paper, we introduce Knowledge-Orthogonal Reasoning (KOR), a concept aimed at minimizing reliance on domain-specific knowledge, enabling more accurate evaluation of models' reasoning abilities in out-of-distribution settings. Based on this concept, we propose the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), encompassing five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes models' effectiveness in applying new rule descriptions to solve novel rule-driven questions. O1-Preview and O1-Mini achieve accuracies of 72.88% and 70.16%, surpassing Claude-3.5-Sonnet and GPT-4o (58.96% and 58.00%), highlighting the effectiveness of KOR-Bench. We perform detailed analyses, identifying bottlenecks in the Cipher task with Stepwise Prompting, where two rounds of Self-Correction yield optimal results. We evaluate performance across three integrated tasks, explore the impact of Tricks on the Puzzle task, and visualize rule-focused attention. Additionally, we conduct an ablation study on dataset size, benchmark correlations, and zero-shot and three-shot "only questions" experiments. KOR-Bench aims to enhance reasoning evaluation and support further research in this area.

Abstract of KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks (arXiv:2410.06526, CC0)
62
Entries on its own leaderboard, 33 models we track
4
Launch posts list it, from 2 labs
2
System cards report it, from 1 lab

o1-preview leads with 72.9% among models released by September 2024

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

Seed2.0 Pro leads with 77.5%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in September 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

62 entries. Scroll sideways for the rest.

  1. 72.9 o1-preview
  2. 70.2 o1-mini
  3. 59.0 Claude 3.5 Sonnet
  4. 58.0 GPT-4o
  5. 55.4 Llama 3.1-405B
  6. 54.7 Qwen2.5-32B
  7. 53.5 GPT-4 Turbo
  8. 53.1 Mistral Large 2
  9. 52.2 Qwen2.5-72B
  10. 50.0 Yi-Large
  11. 50.0 Llama 3.1-70B
  12. 49.4 Qwen2.5-14B
  13. 49.2 Llama 3-70B
  14. 48.1 Doubao-Pro-128k
  15. 47.8 DeepSeek-V2.5
  16. 47.0 Qwen2-72B
  17. 44.5 Gemma 2 27B
  18. 43.9 Phi-3.5-MoE
  19. 43.4 Gemini 1.5 Pro
  20. 41.6 Gemma 2 9B
  21. 39.8 Yi-1.5-34B
  22. 39.7 Meta-Llama-3.1-405B (base)
  23. 39.0 Phi-3.5-mini
  24. 38.6 Qwen2.5-7B
  25. 37.3 Qwen2.5-32B (base)
  26. 37.3 Qwen2.5-72B (base)
  27. 37.2 Llama 3.1-8B
  28. 35.2 Yi-1.5-9B-Chat
  29. 35.2 Meta-Llama-3-70B (base)
  30. 34.3 Qwen2-72B (base)
  31. 33.8 Meta-Llama-3.1-70B (base)
  32. 33.4 Gemma-2-27b (base)
  33. 33.3 Qwen2.5-14B (base)
  34. 32.8 Llama 3-8B
  35. 32.7 Command R+
  36. 32.5 Yi-1.5-6B-Chat
  37. 31.1 c4ai-command-r-08-2024
  38. 30.7 Qwen2-7B-Instruct
  39. 30.1 Yi-1.5-34B (base)
  40. 29.2 Yi-1.5-9B (base)
  41. 28.8 Qwen2.5-7B (base)
  42. 27.4 Qwen2-7B (base)
  43. 26.0 Meta-Llama-3.1-8B (base)
  44. 25.5 Gemma-2-9b (base)
  45. 25.0 Meta-Llama-3-8B (base)
  46. 24.3 Gemma 2 2B IT
  47. 24.2 Mistral 7B v0.3
  48. 21.6 Mistral-7B-v0.1 (base)
  49. 20.9 Yi-1.5-6B (base)
  50. 20.4 Qwen2.5-1.5B-Instruct
  51. 18.5 OLMo-7B-0724-Instruct-hf
  52. 18.2 MAP-Neo-7B-Instruct-v0.1
  53. 15.6 MAP-Neo-7B (base)
  54. 15.1 Qwen2.5-1.5B (base)
  55. 14.8 OLMo-7B-0724-hf (base)
  56. 14.3 Qwen2-1.5B-Instruct
  57. 13.2 Gemma-2-2b (base)
  58. 12.3 Qwen2-1.5B (base)
  59. 9.92 Qwen2-0.5B (base)
  60. 9.12 Qwen2.5-0.5B (base)
  61. 9.04 Qwen2.5-0.5B-Instruct
  62. 3.52 Qwen2-0.5B-Instruct
  1. 1 o1-preview chat model 72.9%
  2. 2 o1-mini chat model 70.2%
  3. 3 Claude 3.5 Sonnet chat model 59%
  4. 4 GPT-4o May 2024, chat model 58%
  5. 5 Llama 3.1-405B chat model 55.4%
  6. 6 Qwen2.5-32B chat model 54.7%
  7. 7 GPT-4 Turbo Apr 2024, chat model 53.5%
  8. 8 Mistral Large 2 Jul 2024, chat model 53.1%
  9. 9 Qwen2.5-72B chat model 52.2%
  10. 10 Yi-Large chat model 50%
  11. 11 Llama 3.1-70B chat model 50%
  12. 12 Qwen2.5-14B chat model 49.4%
  13. 13 Llama 3-70B chat model 49.2%
  14. 14 Doubao-Pro-128k chat model 48.1%
  15. 15 DeepSeek-V2.5 Sep 2024, chat model 47.8%
  16. 16 Qwen2-72B chat model 47%
  17. 17 Gemma 2 27B chat model 44.5%
  18. 18 Phi-3.5-MoE chat model 43.9%
  19. 19 Gemini 1.5 Pro May 2024, chat model 43.4%
  20. 20 Gemma 2 9B chat model 41.6%
  21. 21 Yi-1.5-34B chat model 39.8%
  22. 22 Meta-Llama-3.1-405B (base) base model 39.7%
  23. 23 Phi-3.5-mini chat model 39%
  24. 24 Qwen2.5-7B chat model 38.6%
  25. 25 Qwen2.5-32B (base) base model 37.3%
  26. 26 Qwen2.5-72B (base) base model 37.3%
  27. 27 Llama 3.1-8B chat model 37.2%
  28. 28 Yi-1.5-9B-Chat chat model 35.2%
  29. 29 Meta-Llama-3-70B (base) base model 35.2%
  30. 30 Qwen2-72B (base) base model 34.3%
  31. 31 Meta-Llama-3.1-70B (base) base model 33.8%
  32. 32 Gemma-2-27b (base) base model 33.4%
  33. 33 Qwen2.5-14B (base) base model 33.3%
  34. 34 Llama 3-8B chat model 32.8%
  35. 35 Command R+ chat model 32.7%
  36. 36 Yi-1.5-6B-Chat chat model 32.5%
  37. 37 c4ai-command-r-08-2024 chat model 31.1%
  38. 38 Qwen2-7B-Instruct chat model 30.7%
  39. 39 Yi-1.5-34B (base) base model 30.1%
  40. 40 Yi-1.5-9B (base) base model 29.2%
  41. 41 Qwen2.5-7B (base) base model 28.8%
  42. 42 Qwen2-7B (base) base model 27.4%
  43. 43 Meta-Llama-3.1-8B (base) base model 26%
  44. 44 Gemma-2-9b (base) base model 25.5%
  45. 45 Meta-Llama-3-8B (base) base model 25%
  46. 46 Gemma 2 2B IT chat model 24.3%
  47. 47 Mistral 7B v0.3 chat model 24.2%
  48. 48 Mistral-7B-v0.1 (base) base model 21.6%
  49. 49 Yi-1.5-6B (base) base model 20.9%
  50. 50 Qwen2.5-1.5B-Instruct chat model 20.4%
  51. 51 OLMo-7B-0724-Instruct-hf chat model 18.5%
  52. 52 MAP-Neo-7B-Instruct-v0.1 chat model 18.2%
  53. 53 MAP-Neo-7B (base) base model 15.6%
  54. 54 Qwen2.5-1.5B (base) base model 15.1%
  55. 55 OLMo-7B-0724-hf (base) base model 14.8%
  56. 56 Qwen2-1.5B-Instruct chat model 14.3%
  57. 57 Gemma-2-2b (base) base model 13.2%
  58. 58 Qwen2-1.5B (base) base model 12.3%
  59. 59 Qwen2-0.5B (base) base model 9.92%
  60. 60 Qwen2.5-0.5B (base) base model 9.12%
  61. 61 Qwen2.5-0.5B-Instruct chat model 9.04%
  62. 62 Qwen2-0.5B-Instruct chat model 3.52%

Source: Official KOR-Bench leaderboard

37 models. Scroll sideways for the rest.

  1. 77.5 Seed2.0 Pro
  2. 76.2 Seed1.8
  3. 72.9 o1-preview
  4. 70.6 seed-oss-36b-instruct
  5. 70.2 o1-mini
  6. 59.0 Claude 3.5 Sonnet
  7. 58.0 GPT-4o
  8. 56.4 InternLM3-8B-Instruct
  9. 55.4 Llama 3.1-405B
  10. 54.7 Qwen2.5-32B
  11. 53.5 GPT-4 Turbo
  12. 53.1 Mistral Large 2
  13. 52.2 Qwen2.5-72B
  14. 50.0 Yi-Large
  15. 50.0 Llama 3.1-70B
  16. 49.4 Qwen2.5-14B
  17. 49.2 Llama 3-70B
  18. 47.8 DeepSeek-V2.5
  19. 47.0 Qwen2-72B
  20. 44.5 Gemma 2 27B
  21. 43.9 Phi-3.5-MoE
  22. 43.4 Gemini 1.5 Pro
  23. 41.6 Gemma 2 9B
  24. 39.8 Yi-1.5-34B
  25. 39.0 Phi-3.5-mini
  26. 38.6 Qwen2.5-7B
  27. 37.2 Llama 3.1-8B
  28. 35.2 Yi-1.5-9B-Chat
  29. 32.8 Llama 3-8B
  30. 32.7 Command R+
  31. 32.5 Yi-1.5-6B-Chat
  32. 31.1 c4ai-command-r-08-2024
  33. 30.7 Qwen2-7B-Instruct
  34. 24.3 Gemma 2 2B IT
  35. 24.2 Mistral 7B v0.3
  36. 14.3 Qwen2-1.5B-Instruct
  37. 3.52 Qwen2-0.5B-Instruct
  1. 1 Seed2.0 Pro 77.5%
  2. 2 Seed1.8 76.2%
  3. 3 o1-preview chat model 72.9%
  4. 4 seed-oss-36b-instruct 70.6%
  5. 5 o1-mini chat model 70.2%
  6. 6 Claude 3.5 Sonnet chat model 59%
  7. 7 GPT-4o May 2024, chat model 58%
  8. 8 InternLM3-8B-Instruct 56.4%
  9. 9 Llama 3.1-405B chat model 55.4%
  10. 10 Qwen2.5-32B chat model 54.7%
  11. 11 GPT-4 Turbo Apr 2024, chat model 53.5%
  12. 12 Mistral Large 2 Jul 2024, chat model 53.1%
  13. 13 Qwen2.5-72B chat model 52.2%
  14. 14 Yi-Large chat model 50%
  15. 15 Llama 3.1-70B chat model 50%
  16. 16 Qwen2.5-14B chat model 49.4%
  17. 17 Llama 3-70B chat model 49.2%
  18. 18 DeepSeek-V2.5 Sep 2024, chat model 47.8%
  19. 19 Qwen2-72B chat model 47%
  20. 20 Gemma 2 27B chat model 44.5%
  21. 21 Phi-3.5-MoE chat model 43.9%
  22. 22 Gemini 1.5 Pro May 2024, chat model 43.4%
  23. 23 Gemma 2 9B chat model 41.6%
  24. 24 Yi-1.5-34B chat model 39.8%
  25. 25 Phi-3.5-mini chat model 39%
  26. 26 Qwen2.5-7B chat model 38.6%
  27. 27 Llama 3.1-8B chat model 37.2%
  28. 28 Yi-1.5-9B-Chat chat model 35.2%
  29. 29 Llama 3-8B chat model 32.8%
  30. 30 Command R+ chat model 32.7%
  31. 31 Yi-1.5-6B-Chat chat model 32.5%
  32. 32 c4ai-command-r-08-2024 chat model 31.1%
  33. 33 Qwen2-7B-Instruct chat model 30.7%
  34. 34 Gemma 2 2B IT chat model 24.3%
  35. 35 Mistral 7B v0.3 chat model 24.2%
  36. 36 Qwen2-1.5B-Instruct chat model 14.3%
  37. 37 Qwen2-0.5B-Instruct chat model 3.52%

Sources: Labs’ model cards and official KOR-Bench leaderboard

#EntryScoreSettingDateSource
1 o1-preview 72.9% chat model Sep 2024 Official leaderboard
2 o1-mini 70.2% chat model Sep 2024 Official leaderboard
3 Claude 3.5 Sonnet 59% chat model Jun 2024 Official leaderboard
4 GPT-4o May 2024 58% chat model May 2024 Official leaderboard
5 Llama 3.1-405B 55.4% chat model Jul 2024 Official leaderboard
6 Qwen2.5-32B 54.7% chat model Sep 2024 Official leaderboard
7 GPT-4 Turbo Apr 2024 53.5% chat model Apr 2024 Official leaderboard
8 Mistral Large 2 Jul 2024 53.1% chat model Jul 2024 Official leaderboard
9 Qwen2.5-72B 52.2% chat model Sep 2024 Official leaderboard
10 Yi-Large 50% chat model May 2024 Official leaderboard
11 Llama 3.1-70B 50% chat model Jul 2024 Official leaderboard
12 Qwen2.5-14B 49.4% chat model Sep 2024 Official leaderboard
13 Llama 3-70B 49.2% chat model Apr 2024 Official leaderboard
14 Doubao-Pro-128k 48.1% chat model Jun 2024 Official leaderboard
15 DeepSeek-V2.5 Sep 2024 47.8% chat model Sep 2024 Official leaderboard
16 Qwen2-72B 47% chat model Jun 2024 Official leaderboard
17 Gemma 2 27B 44.5% chat model Jun 2024 Official leaderboard
18 Phi-3.5-MoE 43.9% chat model Aug 2024 Official leaderboard
19 Gemini 1.5 Pro May 2024 43.4% chat model May 2024 Official leaderboard
20 Gemma 2 9B 41.6% chat model Jun 2024 Official leaderboard
21 Yi-1.5-34B 39.8% chat model May 2024 Official leaderboard
22 Meta-Llama-3.1-405B (base) 39.7% base model Sep 2024 Official leaderboard
23 Phi-3.5-mini 39% chat model Aug 2024 Official leaderboard
24 Qwen2.5-7B 38.6% chat model Sep 2024 Official leaderboard
25 Qwen2.5-32B (base) 37.3% base model Sep 2024 Official leaderboard
26 Qwen2.5-72B (base) 37.3% base model Sep 2024 Official leaderboard
27 Llama 3.1-8B 37.2% chat model Jul 2024 Official leaderboard
28 Yi-1.5-9B-Chat 35.2% chat model May 2024 Official leaderboard
29 Meta-Llama-3-70B (base) 35.2% base model Sep 2024 Official leaderboard
30 Qwen2-72B (base) 34.3% base model Sep 2024 Official leaderboard
31 Meta-Llama-3.1-70B (base) 33.8% base model Sep 2024 Official leaderboard
32 Gemma-2-27b (base) 33.4% base model Sep 2024 Official leaderboard
33 Qwen2.5-14B (base) 33.3% base model Sep 2024 Official leaderboard
34 Llama 3-8B 32.8% chat model Apr 2024 Official leaderboard
35 Command R+ 32.7% chat model Aug 2024 Official leaderboard
36 Yi-1.5-6B-Chat 32.5% chat model May 2024 Official leaderboard
37 c4ai-command-r-08-2024 31.1% chat model Aug 2024 Official leaderboard
38 Qwen2-7B-Instruct 30.7% chat model Jun 2024 Official leaderboard
39 Yi-1.5-34B (base) 30.1% base model Sep 2024 Official leaderboard
40 Yi-1.5-9B (base) 29.2% base model Sep 2024 Official leaderboard
41 Qwen2.5-7B (base) 28.8% base model Sep 2024 Official leaderboard
42 Qwen2-7B (base) 27.4% base model Sep 2024 Official leaderboard
43 Meta-Llama-3.1-8B (base) 26% base model Sep 2024 Official leaderboard
44 Gemma-2-9b (base) 25.5% base model Sep 2024 Official leaderboard
45 Meta-Llama-3-8B (base) 25% base model Sep 2024 Official leaderboard
46 Gemma 2 2B IT 24.3% chat model Jul 2024 Official leaderboard
47 Mistral 7B v0.3 24.2% chat model Sep 2023 Official leaderboard
48 Mistral-7B-v0.1 (base) 21.6% base model Sep 2024 Official leaderboard
49 Yi-1.5-6B (base) 20.9% base model Sep 2024 Official leaderboard
50 Qwen2.5-1.5B-Instruct 20.4% chat model Sep 2024 Official leaderboard
51 OLMo-7B-0724-Instruct-hf 18.5% chat model Sep 2024 Official leaderboard
52 MAP-Neo-7B-Instruct-v0.1 18.2% chat model Sep 2024 Official leaderboard
53 MAP-Neo-7B (base) 15.6% base model Sep 2024 Official leaderboard
54 Qwen2.5-1.5B (base) 15.1% base model Sep 2024 Official leaderboard
55 OLMo-7B-0724-hf (base) 14.8% base model Sep 2024 Official leaderboard
56 Qwen2-1.5B-Instruct 14.3% chat model Jun 2024 Official leaderboard
57 Gemma-2-2b (base) 13.2% base model Sep 2024 Official leaderboard
58 Qwen2-1.5B (base) 12.3% base model Sep 2024 Official leaderboard
59 Qwen2-0.5B (base) 9.92% base model Sep 2024 Official leaderboard
60 Qwen2.5-0.5B (base) 9.12% base model Sep 2024 Official leaderboard
61 Qwen2.5-0.5B-Instruct 9.04% chat model Sep 2024 Official leaderboard
62 Qwen2-0.5B-Instruct 3.52% chat model Jun 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Seed2.0 Pro 77.5% Feb 2026 Lab model card
2 Seed1.8 76.2% Dec 2025 Lab model card
3 o1-preview 72.9% chat model Sep 2024 Official leaderboard
4 seed-oss-36b-instruct 70.6% Aug 2025 Lab model card
5 o1-mini 70.2% chat model Sep 2024 Official leaderboard
6 Claude 3.5 Sonnet 59% chat model Jun 2024 Official leaderboard
7 GPT-4o May 2024 58% chat model May 2024 Official leaderboard
8 InternLM3-8B-Instruct 56.4% Jan 2025 Lab model card
9 Llama 3.1-405B 55.4% chat model Jul 2024 Official leaderboard
10 Qwen2.5-32B 54.7% chat model Sep 2024 Official leaderboard
11 GPT-4 Turbo Apr 2024 53.5% chat model Apr 2024 Official leaderboard
12 Mistral Large 2 Jul 2024 53.1% chat model Jul 2024 Official leaderboard
13 Qwen2.5-72B 52.2% chat model Sep 2024 Official leaderboard
14 Yi-Large 50% chat model May 2024 Official leaderboard
15 Llama 3.1-70B 50% chat model Jul 2024 Official leaderboard
16 Qwen2.5-14B 49.4% chat model Sep 2024 Official leaderboard
17 Llama 3-70B 49.2% chat model Apr 2024 Official leaderboard
18 DeepSeek-V2.5 Sep 2024 47.8% chat model Sep 2024 Official leaderboard
19 Qwen2-72B 47% chat model Jun 2024 Official leaderboard
20 Gemma 2 27B 44.5% chat model Jun 2024 Official leaderboard
21 Phi-3.5-MoE 43.9% chat model Aug 2024 Official leaderboard
22 Gemini 1.5 Pro May 2024 43.4% chat model May 2024 Official leaderboard
23 Gemma 2 9B 41.6% chat model Jun 2024 Official leaderboard
24 Yi-1.5-34B 39.8% chat model May 2024 Official leaderboard
25 Phi-3.5-mini 39% chat model Aug 2024 Official leaderboard
26 Qwen2.5-7B 38.6% chat model Sep 2024 Official leaderboard
27 Llama 3.1-8B 37.2% chat model Jul 2024 Official leaderboard
28 Yi-1.5-9B-Chat 35.2% chat model May 2024 Official leaderboard
29 Llama 3-8B 32.8% chat model Apr 2024 Official leaderboard
30 Command R+ 32.7% chat model Aug 2024 Official leaderboard
31 Yi-1.5-6B-Chat 32.5% chat model May 2024 Official leaderboard
32 c4ai-command-r-08-2024 31.1% chat model Aug 2024 Official leaderboard
33 Qwen2-7B-Instruct 30.7% chat model Jun 2024 Official leaderboard
34 Gemma 2 2B IT 24.3% chat model Jul 2024 Official leaderboard
35 Mistral 7B v0.3 24.2% chat model Sep 2023 Official leaderboard
36 Qwen2-1.5B-Instruct 14.3% chat model Jun 2024 Official leaderboard
37 Qwen2-0.5B-Instruct 3.52% chat model Jun 2024 Official leaderboard

Labs report scores for 4 models in their cards. Switch the scores above to rank them together.

Number one over time 3 models have held number one with claims included since its release in October 2024

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Shanghai AI Lab ByteDance Seed
Own leaderboard Not tested since Oct 2024
o1-preview 72.9%
Lab claims Not tested since Mar 2026
Seed2.0 Pro 77.5%

Sources: Official KOR-Bench leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Seed2.0 Pro o1-preview Claude 3.5 Sonnet GPT-4 Turbo Mistral 7B v0.3 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Seed2.0 Pro Claude 3.5 Sonnet

Sources: Official KOR-Bench leaderboard and labs’ model cards

Labs headline it less often now, last in February 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Shanghai AI Lab
ByteDance Seed
DateLaunch postScoreAs the post writes it
Feb 14, 2026 Seed2.0 Pro 77.5 KORBench
Dec 18, 2025 Seed1.8 76.2 KOR-Bench
Aug 20, 2025 seed-oss-36b-instruct 70.6 KORBench
Jan 13, 2025 InternLM3-8B-Instruct 56.4 KOR-Bench(0-shot)

2 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Mar 21, 2026 Seed1.8 model card 76.2 KOR-Bench (Table 1) Capabilities
Feb 14, 2026 Seed2.0 Pro model card 77.5 KORBench (Table 3). Table 13 prints 77.2 Capabilities

More in Science and reasoning

All 92 in Science and reasoning
  • GPQA Diamond In use 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. Gemini 4 Carbon 96.5%
  • Terminal-Bench-Science In use 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. GPT-6 Astra 68.1%
  • FrontierScience In use Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. Intern-S2-397B 87%
  • BioMysteryBench In use Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. Claude Fable 5.2 83.7%
  • HiPhO In use Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. Gemini 3 Pro 78.3%
  • LABBench2 In use 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. GPT-5.2 60.9%