Benchmark Almanac

PinchBench

KAT-Coder-Pro V2.5 94.9% leads since July 2026
Gemini 3 Flash 87% leads since February 2026
KAT-Coder-Pro V2.5 94.9% leads since July 2026
In use

Real-world tasks (scheduling, email triage, coding, research, files) for an LLM driving an OpenClaw agent. Graded by automated checks and LLM judge.

v1.x grew from 23 to 53 tasks. PinchBench 2.0 (May 6th, 2026) expanded to 148 tasks and changed the default judge, so it has its own record.

Made by
Kilo AI
Released
February 2026
Kind
General assistants
Level
Everyday
Scored
Graded by a model
Answers
Agent task
Tasks
All public
Strongest model tested
GPT-5.4 Pro #19 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 12th, 2026
14
Models scored in lab cards
13
Launch posts list it, from 7 labs
6
System cards report it, from 3 labs

KAT-Coder-Pro V2.5 leads with 94.9%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Gemini 3 Flash leads with 87% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Each bar and row names its source.

KAT-Coder-Pro V2.5 leads with 94.9%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

14 models. Scroll sideways for the rest.

  1. 94.9 KAT-Coder-Pro V2.5
  2. 93.4 KAT-Coder-V2.5-Dev
  3. 91.9 Trinity Large Thinking
  4. 90.5 Muse Glimmer
  5. 90.0 Nemotron 3 Ultra
  6. 87.8 Nemotron 3.5 Bolt 74B-A9B
  7. 87.6 Ring-2.6-1T
  8. 87.4 Ling-2.6-1T
  9. 85.4 Nemotron 3.5 Lightning 30B-A3B
  10. 83.2 LFM2.5-24B-A2B
  11. 81.2 MiMo-V2-Omni
  12. 81.0 MiMo-V2-Pro
  13. 80.7 GLM-5V-Turbo
  14. 68.2 LFM2.5-2.6B
  1. 1 KAT-Coder-Pro V2.5 94.9%
  2. 2 KAT-Coder-V2.5-Dev 93.4%
  3. 3 Trinity Large Thinking 91.9%
  4. 4 Muse Glimmer 90.5%
  5. 5 Nemotron 3 Ultra 90%
  6. 6 Nemotron 3.5 Bolt 74B-A9B 87.8%
  7. 7 Ring-2.6-1T 87.6%
  8. 8 Ling-2.6-1T 87.4%
  9. 9 Nemotron 3.5 Lightning 30B-A3B 85.4%
  10. 10 LFM2.5-24B-A2B 83.2%
  11. 11 MiMo-V2-Omni 81.2%
  12. 12 MiMo-V2-Pro 81%
  13. 13 GLM-5V-Turbo 80.7%
  14. 14 LFM2.5-2.6B 68.2%

Source: Labs’ model cards

48 models. Scroll sideways for the rest.

  1. 87.0 Gemini 3 Flash
  2. 80.9 GLM-5.1
  3. 80.7 Gemini 3.1 Pro
  4. 80.6 MiniMax-M2.7
  5. 80.4 MiniMax-M2.1
  6. 79.8 Seed2.0 Lite (0215)
  7. 78.3 MiMo-V2-Pro
  8. 78.2 MiMo-V2-Omni
  9. 76.4 Claude Sonnet 4.6
  10. 76.2 GLM-5
  11. 75.2 Kimi K2.5
  12. 74.4 Claude Opus 4.6
  13. 74.1 Gemma 4 26B A4B
  14. 74.0 Claude Sonnet 4
  15. 73.4 GLM-5-Turbo
  16. 72.5 Claude Opus 4.5
  17. 71.6 Claude Haiku 4.5
  18. 71.4 GPT-5.4 Mini
  19. 71.3 Claude Sonnet 4.5
  20. 71.3 Mistral Small 4
  21. 71.2 Grok 4.20
  22. 70.5 Step 3.5 Flash
  23. 69.5 GPT-5.4 Nano
  24. 69.4 GLM-4.5-Air
  25. 67.5 MiMo-V2-Flash
  26. 67.2 MiniMax-M2.5
  27. 67.0 GLM-5V-Turbo
  28. 67.0 Devstral 2
  29. 66.6 Grok 4.1 Fast
  30. 66.5 Gemma 4 31B IT
  31. 65.9 Mistral Large 3
  32. 65.4 GPT-4o
  33. 65.2 GPT-5 mini
  34. 64.4 GPT-4o mini
  35. 63.3 Qwen 3.6 Plus
  36. 63.1 Gemini 2.5 Pro
  37. 63.1 Nova 2 Lite
  38. 59.9 GPT-5.4
  39. 59.9 DeepSeek-V3.2
  40. 59.7 gpt-oss-120b
  41. 53.9 Nemotron 3 Super
  42. 50.6 Gemini 2.5 Flash
  43. 48.4 gpt-oss-20b
  44. 41.5 GPT-5 nano
  45. 40.7 Trinity Large Preview
  46. 34.8 Qwen3.5-9B
  47. 12.0 GPT-5.4 Pro
  48. 11.8 Gemini 2.5 Flash-Lite
  1. 1 Gemini 3 Flash 87%
  2. 2 GLM-5.1 80.9%
  3. 3 Gemini 3.1 Pro 80.7%
  4. 4 MiniMax-M2.7 80.6%
  5. 5 MiniMax-M2.1 80.4%
  6. 6 Seed2.0 Lite (0215) 79.8%
  7. 7 MiMo-V2-Pro 78.3%
  8. 8 MiMo-V2-Omni 78.2%
  9. 9 Claude Sonnet 4.6 76.4%
  10. 10 GLM-5 76.2%
  11. 11 Kimi K2.5 75.2%
  12. 12 Claude Opus 4.6 74.4%
  13. 13 Gemma 4 26B A4B 74.1%
  14. 14 Claude Sonnet 4 74%
  15. 15 GLM-5-Turbo 73.4%
  16. 16 Claude Opus 4.5 72.5%
  17. 17 Claude Haiku 4.5 71.6%
  18. 18 GPT-5.4 Mini 71.4%
  19. 19 Claude Sonnet 4.5 71.3%
  20. 20 Mistral Small 4 71.3%
  21. 21 Grok 4.20 71.2%
  22. 22 Step 3.5 Flash 70.5%
  23. 23 GPT-5.4 Nano 69.5%
  24. 24 GLM-4.5-Air 69.4%
  25. 25 MiMo-V2-Flash 67.5%
  26. 26 MiniMax-M2.5 67.2%
  27. 27 GLM-5V-Turbo 67%
  28. 28 Devstral 2 67%
  29. 29 Grok 4.1 Fast 66.6%
  30. 30 Gemma 4 31B IT 66.5%
  31. 31 Mistral Large 3 65.9%
  32. 32 GPT-4o Mar 2025 65.4%
  33. 33 GPT-5 mini 65.2%
  34. 34 GPT-4o mini 64.4%
  35. 35 Qwen 3.6 Plus 63.3%
  36. 36 Gemini 2.5 Pro Jun 2025 63.1%
  37. 37 Nova 2 Lite 63.1%
  38. 38 GPT-5.4 59.9%
  39. 39 DeepSeek-V3.2 59.9%
  40. 40 gpt-oss-120b 59.7%
  41. 41 Nemotron 3 Super 53.9%
  42. 42 Gemini 2.5 Flash Sep 2025 50.6%
  43. 43 gpt-oss-20b 48.4%
  44. 44 GPT-5 nano 41.5%
  45. 45 Trinity Large Preview 40.7%
  46. 46 Qwen3.5-9B 34.8%
  47. 47 GPT-5.4 Pro 12%
  48. 48 Gemini 2.5 Flash-Lite Sep 2025 11.8%

Source: Official PinchBench leaderboard

59 models. Scroll sideways for the rest.

  1. 94.9 KAT-Coder-Pro V2.5
  2. 93.4 KAT-Coder-V2.5-Dev
  3. 91.9 Trinity Large Thinking
  4. 90.5 Muse Glimmer
  5. 90.0 Nemotron 3 Ultra
  6. 87.8 Nemotron 3.5 Bolt 74B-A9B
  7. 87.6 Ring-2.6-1T
  8. 87.4 Ling-2.6-1T
  9. 87.0 Gemini 3 Flash
  10. 85.4 Nemotron 3.5 Lightning 30B-A3B
  11. 83.2 LFM2.5-24B-A2B
  12. 81.2 MiMo-V2-Omni
  13. 81.0 MiMo-V2-Pro
  14. 80.9 GLM-5.1
  15. 80.7 Gemini 3.1 Pro
  16. 80.7 GLM-5V-Turbo
  17. 80.6 MiniMax-M2.7
  18. 80.4 MiniMax-M2.1
  19. 79.8 Seed2.0 Lite (0215)
  20. 76.4 Claude Sonnet 4.6
  21. 76.2 GLM-5
  22. 75.2 Kimi K2.5
  23. 74.4 Claude Opus 4.6
  24. 74.1 Gemma 4 26B A4B
  25. 74.0 Claude Sonnet 4
  26. 73.4 GLM-5-Turbo
  27. 72.5 Claude Opus 4.5
  28. 71.6 Claude Haiku 4.5
  29. 71.4 GPT-5.4 Mini
  30. 71.3 Claude Sonnet 4.5
  31. 71.3 Mistral Small 4
  32. 71.2 Grok 4.20
  33. 70.5 Step 3.5 Flash
  34. 69.5 GPT-5.4 Nano
  35. 69.4 GLM-4.5-Air
  36. 68.2 LFM2.5-2.6B
  37. 67.5 MiMo-V2-Flash
  38. 67.2 MiniMax-M2.5
  39. 67.0 Devstral 2
  40. 66.6 Grok 4.1 Fast
  41. 66.5 Gemma 4 31B IT
  42. 65.9 Mistral Large 3
  43. 65.4 GPT-4o
  44. 65.2 GPT-5 mini
  45. 64.4 GPT-4o mini
  46. 63.3 Qwen 3.6 Plus
  47. 63.1 Gemini 2.5 Pro
  48. 63.1 Nova 2 Lite
  49. 59.9 GPT-5.4
  50. 59.9 DeepSeek-V3.2
  51. 59.7 gpt-oss-120b
  52. 53.9 Nemotron 3 Super
  53. 50.6 Gemini 2.5 Flash
  54. 48.4 gpt-oss-20b
  55. 41.5 GPT-5 nano
  56. 40.7 Trinity Large Preview
  57. 34.8 Qwen3.5-9B
  58. 12.0 GPT-5.4 Pro
  59. 11.8 Gemini 2.5 Flash-Lite
  1. 1 KAT-Coder-Pro V2.5 94.9%
  2. 2 KAT-Coder-V2.5-Dev 93.4%
  3. 3 Trinity Large Thinking 91.9%
  4. 4 Muse Glimmer 90.5%
  5. 5 Nemotron 3 Ultra 90%
  6. 6 Nemotron 3.5 Bolt 74B-A9B 87.8%
  7. 7 Ring-2.6-1T 87.6%
  8. 8 Ling-2.6-1T 87.4%
  9. 9 Gemini 3 Flash 87%
  10. 10 Nemotron 3.5 Lightning 30B-A3B 85.4%
  11. 11 LFM2.5-24B-A2B 83.2%
  12. 12 MiMo-V2-Omni 81.2%
  13. 13 MiMo-V2-Pro 81%
  14. 14 GLM-5.1 80.9%
  15. 15 Gemini 3.1 Pro 80.7%
  16. 16 GLM-5V-Turbo 80.7%
  17. 17 MiniMax-M2.7 80.6%
  18. 18 MiniMax-M2.1 80.4%
  19. 19 Seed2.0 Lite (0215) 79.8%
  20. 20 Claude Sonnet 4.6 76.4%
  21. 21 GLM-5 76.2%
  22. 22 Kimi K2.5 75.2%
  23. 23 Claude Opus 4.6 74.4%
  24. 24 Gemma 4 26B A4B 74.1%
  25. 25 Claude Sonnet 4 74%
  26. 26 GLM-5-Turbo 73.4%
  27. 27 Claude Opus 4.5 72.5%
  28. 28 Claude Haiku 4.5 71.6%
  29. 29 GPT-5.4 Mini 71.4%
  30. 30 Claude Sonnet 4.5 71.3%
  31. 31 Mistral Small 4 71.3%
  32. 32 Grok 4.20 71.2%
  33. 33 Step 3.5 Flash 70.5%
  34. 34 GPT-5.4 Nano 69.5%
  35. 35 GLM-4.5-Air 69.4%
  36. 36 LFM2.5-2.6B 68.2%
  37. 37 MiMo-V2-Flash 67.5%
  38. 38 MiniMax-M2.5 67.2%
  39. 39 Devstral 2 67%
  40. 40 Grok 4.1 Fast 66.6%
  41. 41 Gemma 4 31B IT 66.5%
  42. 42 Mistral Large 3 65.9%
  43. 43 GPT-4o Mar 2025 65.4%
  44. 44 GPT-5 mini 65.2%
  45. 45 GPT-4o mini 64.4%
  46. 46 Qwen 3.6 Plus 63.3%
  47. 47 Gemini 2.5 Pro Jun 2025 63.1%
  48. 48 Nova 2 Lite 63.1%
  49. 49 GPT-5.4 59.9%
  50. 50 DeepSeek-V3.2 59.9%
  51. 51 gpt-oss-120b 59.7%
  52. 52 Nemotron 3 Super 53.9%
  53. 53 Gemini 2.5 Flash Sep 2025 50.6%
  54. 54 gpt-oss-20b 48.4%
  55. 55 GPT-5 nano 41.5%
  56. 56 Trinity Large Preview 40.7%
  57. 57 Qwen3.5-9B 34.8%
  58. 58 GPT-5.4 Pro 12%
  59. 59 Gemini 2.5 Flash-Lite Sep 2025 11.8%

Sources: Labs’ model cards and official PinchBench leaderboard

#ModelScoreSettingDateSource
1 KAT-Coder-Pro V2.5 94.9% Jul 2026 Lab model card
2 KAT-Coder-V2.5-Dev 93.4% Jul 2026 Lab model card
3 Trinity Large Thinking 91.9% Apr 2026 Lab model card
4 Muse Glimmer 90.5% Oct 2026 Lab model card
5 Nemotron 3 Ultra 90% Jun 2026 Lab model card
6 Nemotron 3.5 Bolt 74B-A9B 87.8% Oct 2026 Lab model card
7 Ring-2.6-1T 87.6% May 2026 Lab model card
8 Ling-2.6-1T 87.4% Apr 2026 Lab model card
9 Nemotron 3.5 Lightning 30B-A3B 85.4% Aug 2026 Lab model card
10 LFM2.5-24B-A2B 83.2% Oct 2026 Lab model card
11 MiMo-V2-Omni 81.2% Mar 2026 Lab model card
12 MiMo-V2-Pro 81% Mar 2026 Lab model card
13 GLM-5V-Turbo 80.7% Apr 2026 Lab model card
14 LFM2.5-2.6B 68.2% Aug 2026 Lab model card
#ModelScoreSettingDateSource
1 Gemini 3 Flash 87% OpenClaw Dec 2025 Official leaderboard
2 GLM-5.1 80.9% OpenClaw Apr 2026 Official leaderboard
3 Gemini 3.1 Pro 80.7% OpenClaw Feb 2026 Official leaderboard
4 MiniMax-M2.7 80.6% OpenClaw Mar 2026 Official leaderboard
5 MiniMax-M2.1 80.4% OpenClaw Dec 2025 Official leaderboard
6 Seed2.0 Lite (0215) 79.8% OpenClaw Feb 2026 Official leaderboard
7 MiMo-V2-Pro 78.3% OpenClaw Mar 2026 Official leaderboard
8 MiMo-V2-Omni 78.2% OpenClaw Mar 2026 Official leaderboard
9 Claude Sonnet 4.6 76.4% OpenClaw Feb 2026 Official leaderboard
10 GLM-5 76.2% OpenClaw Feb 2026 Official leaderboard
11 Kimi K2.5 75.2% OpenClaw Jan 2026 Official leaderboard
12 Claude Opus 4.6 74.4% OpenClaw Feb 2026 Official leaderboard
13 Gemma 4 26B A4B 74.1% OpenClaw Apr 2026 Official leaderboard
14 Claude Sonnet 4 74% OpenClaw May 2025 Official leaderboard
15 GLM-5-Turbo 73.4% OpenClaw Mar 2026 Official leaderboard
16 Claude Opus 4.5 72.5% OpenClaw Nov 2025 Official leaderboard
17 Claude Haiku 4.5 71.6% OpenClaw Oct 2025 Official leaderboard
18 GPT-5.4 Mini 71.4% OpenClaw Mar 2026 Official leaderboard
19 Claude Sonnet 4.5 71.3% OpenClaw Sep 2025 Official leaderboard
20 Mistral Small 4 71.3% OpenClaw Mar 2026 Official leaderboard
21 Grok 4.20 71.2% OpenClaw Feb 2026 Official leaderboard
22 Step 3.5 Flash 70.5% OpenClaw Feb 2026 Official leaderboard
23 GPT-5.4 Nano 69.5% OpenClaw Mar 2026 Official leaderboard
24 GLM-4.5-Air 69.4% OpenClaw Jul 2025 Official leaderboard
25 MiMo-V2-Flash 67.5% OpenClaw Dec 2025 Official leaderboard
26 MiniMax-M2.5 67.2% OpenClaw Feb 2026 Official leaderboard
27 GLM-5V-Turbo 67% OpenClaw Apr 2026 Official leaderboard
28 Devstral 2 67% OpenClaw Dec 2025 Official leaderboard
29 Grok 4.1 Fast 66.6% OpenClaw Nov 2025 Official leaderboard
30 Gemma 4 31B IT 66.5% OpenClaw Apr 2026 Official leaderboard
31 Mistral Large 3 65.9% OpenClaw Dec 2025 Official leaderboard
32 GPT-4o Mar 2025 65.4% OpenClaw Mar 2025 Official leaderboard
33 GPT-5 mini 65.2% OpenClaw Aug 2025 Official leaderboard
34 GPT-4o mini 64.4% OpenClaw Jul 2024 Official leaderboard
35 Qwen 3.6 Plus 63.3% OpenClaw Mar 2026 Official leaderboard
36 Gemini 2.5 Pro Jun 2025 63.1% OpenClaw Jun 2025 Official leaderboard
37 Nova 2 Lite 63.1% OpenClaw Dec 2025 Official leaderboard
38 GPT-5.4 59.9% OpenClaw Mar 2026 Official leaderboard
39 DeepSeek-V3.2 59.9% OpenClaw Dec 2025 Official leaderboard
40 gpt-oss-120b 59.7% OpenClaw Aug 2025 Official leaderboard
41 Nemotron 3 Super 53.9% OpenClaw Mar 2026 Official leaderboard
42 Gemini 2.5 Flash Sep 2025 50.6% OpenClaw Sep 2025 Official leaderboard
43 gpt-oss-20b 48.4% OpenClaw Aug 2025 Official leaderboard
44 GPT-5 nano 41.5% OpenClaw Aug 2025 Official leaderboard
45 Trinity Large Preview 40.7% OpenClaw Jan 2026 Official leaderboard
46 Qwen3.5-9B 34.8% OpenClaw Feb 2026 Official leaderboard
47 GPT-5.4 Pro 12% OpenClaw Mar 2026 Official leaderboard
48 Gemini 2.5 Flash-Lite Sep 2025 11.8% OpenClaw Sep 2025 Official leaderboard
#ModelScoreSettingDateSource
1 KAT-Coder-Pro V2.5 94.9% Jul 2026 Lab model card
2 KAT-Coder-V2.5-Dev 93.4% Jul 2026 Lab model card
3 Trinity Large Thinking 91.9% Apr 2026 Lab model card
4 Muse Glimmer 90.5% Oct 2026 Lab model card
5 Nemotron 3 Ultra 90% Jun 2026 Lab model card
6 Nemotron 3.5 Bolt 74B-A9B 87.8% Oct 2026 Lab model card
7 Ring-2.6-1T 87.6% May 2026 Lab model card
8 Ling-2.6-1T 87.4% Apr 2026 Lab model card
9 Gemini 3 Flash 87% OpenClaw Dec 2025 Official leaderboard
10 Nemotron 3.5 Lightning 30B-A3B 85.4% Aug 2026 Lab model card
11 LFM2.5-24B-A2B 83.2% Oct 2026 Lab model card
12 MiMo-V2-Omni 81.2% Mar 2026 Lab model card
13 MiMo-V2-Pro 81% Mar 2026 Lab model card
14 GLM-5.1 80.9% OpenClaw Apr 2026 Official leaderboard
15 Gemini 3.1 Pro 80.7% OpenClaw Feb 2026 Official leaderboard
16 GLM-5V-Turbo 80.7% Apr 2026 Lab model card
17 MiniMax-M2.7 80.6% OpenClaw Mar 2026 Official leaderboard
18 MiniMax-M2.1 80.4% OpenClaw Dec 2025 Official leaderboard
19 Seed2.0 Lite (0215) 79.8% OpenClaw Feb 2026 Official leaderboard
20 Claude Sonnet 4.6 76.4% OpenClaw Feb 2026 Official leaderboard
21 GLM-5 76.2% OpenClaw Feb 2026 Official leaderboard
22 Kimi K2.5 75.2% OpenClaw Jan 2026 Official leaderboard
23 Claude Opus 4.6 74.4% OpenClaw Feb 2026 Official leaderboard
24 Gemma 4 26B A4B 74.1% OpenClaw Apr 2026 Official leaderboard
25 Claude Sonnet 4 74% OpenClaw May 2025 Official leaderboard
26 GLM-5-Turbo 73.4% OpenClaw Mar 2026 Official leaderboard
27 Claude Opus 4.5 72.5% OpenClaw Nov 2025 Official leaderboard
28 Claude Haiku 4.5 71.6% OpenClaw Oct 2025 Official leaderboard
29 GPT-5.4 Mini 71.4% OpenClaw Mar 2026 Official leaderboard
30 Claude Sonnet 4.5 71.3% OpenClaw Sep 2025 Official leaderboard
31 Mistral Small 4 71.3% OpenClaw Mar 2026 Official leaderboard
32 Grok 4.20 71.2% OpenClaw Feb 2026 Official leaderboard
33 Step 3.5 Flash 70.5% OpenClaw Feb 2026 Official leaderboard
34 GPT-5.4 Nano 69.5% OpenClaw Mar 2026 Official leaderboard
35 GLM-4.5-Air 69.4% OpenClaw Jul 2025 Official leaderboard
36 LFM2.5-2.6B 68.2% Aug 2026 Lab model card
37 MiMo-V2-Flash 67.5% OpenClaw Dec 2025 Official leaderboard
38 MiniMax-M2.5 67.2% OpenClaw Feb 2026 Official leaderboard
39 Devstral 2 67% OpenClaw Dec 2025 Official leaderboard
40 Grok 4.1 Fast 66.6% OpenClaw Nov 2025 Official leaderboard
41 Gemma 4 31B IT 66.5% OpenClaw Apr 2026 Official leaderboard
42 Mistral Large 3 65.9% OpenClaw Dec 2025 Official leaderboard
43 GPT-4o Mar 2025 65.4% OpenClaw Mar 2025 Official leaderboard
44 GPT-5 mini 65.2% OpenClaw Aug 2025 Official leaderboard
45 GPT-4o mini 64.4% OpenClaw Jul 2024 Official leaderboard
46 Qwen 3.6 Plus 63.3% OpenClaw Mar 2026 Official leaderboard
47 Gemini 2.5 Pro Jun 2025 63.1% OpenClaw Jun 2025 Official leaderboard
48 Nova 2 Lite 63.1% OpenClaw Dec 2025 Official leaderboard
49 GPT-5.4 59.9% OpenClaw Mar 2026 Official leaderboard
50 DeepSeek-V3.2 59.9% OpenClaw Dec 2025 Official leaderboard
51 gpt-oss-120b 59.7% OpenClaw Aug 2025 Official leaderboard
52 Nemotron 3 Super 53.9% OpenClaw Mar 2026 Official leaderboard
53 Gemini 2.5 Flash Sep 2025 50.6% OpenClaw Sep 2025 Official leaderboard
54 gpt-oss-20b 48.4% OpenClaw Aug 2025 Official leaderboard
55 GPT-5 nano 41.5% OpenClaw Aug 2025 Official leaderboard
56 Trinity Large Preview 40.7% OpenClaw Jan 2026 Official leaderboard
57 Qwen3.5-9B 34.8% OpenClaw Feb 2026 Official leaderboard
58 GPT-5.4 Pro 12% OpenClaw Mar 2026 Official leaderboard
59 Gemini 2.5 Flash-Lite Sep 2025 11.8% OpenClaw Sep 2025 Official leaderboard

The benchmark's own leaderboard has 49 entries, 48 of them models we track. Switch the scores above to rank them together.

Success Rate vs. Execution Time

Each point is one model: its average score against its average execution time for a full run, over its official submissions. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% 100% 101001,00010K Execution time per run (s), log scale, lower is better Success rate GPT-4o mini: Success rate 64.4%, execution time per run 2,510 s GPT-4o: Success rate 65.4%, execution time per run 1,945 s Gemini 2.5 Pro: Success rate 63.1%, execution time per run 3,518 s GLM-4.5-Air: Success rate 69.4%, execution time per run 3,306 s gpt-oss-120b: Success rate 59.7%, execution time per run 2,355 s gpt-oss-20b: Success rate 48.4%, execution time per run 2,951 s GPT-5 mini: Success rate 65.2%, execution time per run 2,263 s GPT-5 nano: Success rate 41.5%, execution time per run 3,400 s Gemini 2.5 Flash-Lite: Success rate 11.8%, execution time per run 2,328 s Gemini 2.5 Flash: Success rate 50.6%, execution time per run 2,386 s Claude Sonnet 4.5: Success rate 71.3%, execution time per run 2,842 s Claude Haiku 4.5: Success rate 71.6%, execution time per run 2,838 s Grok 4.1 Fast: Success rate 66.6%, execution time per run 3,264 s Claude Opus 4.5: Success rate 72.5%, execution time per run 3,040 s DeepSeek-V3.2: Success rate 59.9%, execution time per run 4,798 s Mistral Large 3: Success rate 65.9%, execution time per run 3,116 s Nova 2 Lite: Success rate 63.1%, execution time per run 2,746 s Devstral 2: Success rate 67%, execution time per run 3,141 s MiMo-V2-Flash: Success rate 67.5%, execution time per run 3,216 s MiniMax-M2.1: Success rate 80.4%, execution time per run 2,367 s Kimi K2.5: Success rate 75.2%, execution time per run 2,891 s Trinity Large Preview: Success rate 40.7%, execution time per run 4,674 s Step 3.5 Flash: Success rate 70.5%, execution time per run 2,797 s Claude Opus 4.6: Success rate 74.4%, execution time per run 2,579 s GLM-5: Success rate 76.2%, execution time per run 4,069 s MiniMax-M2.5: Success rate 67.2%, execution time per run 3,433 s Claude Sonnet 4.6: Success rate 76.4%, execution time per run 2,701 s Grok 4.20: Success rate 71.2%, execution time per run 1,402 s Gemini 3.1 Pro: Success rate 80.7%, execution time per run 2,252 s Qwen3.5-9B: Success rate 34.8%, execution time per run 872 s GPT-5.4: Success rate 59.9%, execution time per run 3,670 s GPT-5.4 Pro: Success rate 12%, execution time per run 5,297 s Nemotron 3 Super: Success rate 53.9%, execution time per run 4,652 s GLM-5-Turbo: Success rate 73.4%, execution time per run 2,309 s Mistral Small 4: Success rate 71.3%, execution time per run 1,364 s GPT-5.4 Mini: Success rate 71.4%, execution time per run 1,361 s GPT-5.4 Nano: Success rate 69.5%, execution time per run 1,481 s MiMo-V2-Omni: Success rate 78.2%, execution time per run 2,639 s MiMo-V2-Pro: Success rate 78.3%, execution time per run 2,047 s MiniMax-M2.7: Success rate 80.6%, execution time per run 2,761 s Qwen 3.6 Plus: Success rate 63.3%, execution time per run 1,844 s GLM-5V-Turbo: Success rate 67%, execution time per run 2,931 s Gemma 4 26B A4B: Success rate 74.1%, execution time per run 2,337 s Gemma 4 31B IT: Success rate 66.5%, execution time per run 3,325 s deepseek-chat: Success rate 53.1%, execution time per run 2,800 s. Not in our model list Claude Sonnet 4: Success rate 74%, execution time per run 86.8 s Gemini 3 Flash: Success rate 87%, execution time per run 2,180 s Seed2.0 Lite (0215): Success rate 79.8%, execution time per run 1,755 s GLM-5.1: Success rate 80.9%, execution time per run 1,880 s Gemini 3 FlashGLM-5.1Seed2.0 Lite (0215)Claude Sonnet 4 0% 20% 40% 60% 80% 100% 101001,00010K Execution time per run (s), log scale, lower is better Success rate GPT-4o mini: Success rate 64.4%, execution time per run 2,510 s GPT-4o: Success rate 65.4%, execution time per run 1,945 s Gemini 2.5 Pro: Success rate 63.1%, execution time per run 3,518 s GLM-4.5-Air: Success rate 69.4%, execution time per run 3,306 s gpt-oss-120b: Success rate 59.7%, execution time per run 2,355 s gpt-oss-20b: Success rate 48.4%, execution time per run 2,951 s GPT-5 mini: Success rate 65.2%, execution time per run 2,263 s GPT-5 nano: Success rate 41.5%, execution time per run 3,400 s Gemini 2.5 Flash-Lite: Success rate 11.8%, execution time per run 2,328 s Gemini 2.5 Flash: Success rate 50.6%, execution time per run 2,386 s Claude Sonnet 4.5: Success rate 71.3%, execution time per run 2,842 s Claude Haiku 4.5: Success rate 71.6%, execution time per run 2,838 s Grok 4.1 Fast: Success rate 66.6%, execution time per run 3,264 s Claude Opus 4.5: Success rate 72.5%, execution time per run 3,040 s DeepSeek-V3.2: Success rate 59.9%, execution time per run 4,798 s Mistral Large 3: Success rate 65.9%, execution time per run 3,116 s Nova 2 Lite: Success rate 63.1%, execution time per run 2,746 s Devstral 2: Success rate 67%, execution time per run 3,141 s MiMo-V2-Flash: Success rate 67.5%, execution time per run 3,216 s MiniMax-M2.1: Success rate 80.4%, execution time per run 2,367 s Kimi K2.5: Success rate 75.2%, execution time per run 2,891 s Trinity Large Preview: Success rate 40.7%, execution time per run 4,674 s Step 3.5 Flash: Success rate 70.5%, execution time per run 2,797 s Claude Opus 4.6: Success rate 74.4%, execution time per run 2,579 s GLM-5: Success rate 76.2%, execution time per run 4,069 s MiniMax-M2.5: Success rate 67.2%, execution time per run 3,433 s Claude Sonnet 4.6: Success rate 76.4%, execution time per run 2,701 s Grok 4.20: Success rate 71.2%, execution time per run 1,402 s Gemini 3.1 Pro: Success rate 80.7%, execution time per run 2,252 s Qwen3.5-9B: Success rate 34.8%, execution time per run 872 s GPT-5.4: Success rate 59.9%, execution time per run 3,670 s GPT-5.4 Pro: Success rate 12%, execution time per run 5,297 s Nemotron 3 Super: Success rate 53.9%, execution time per run 4,652 s GLM-5-Turbo: Success rate 73.4%, execution time per run 2,309 s Mistral Small 4: Success rate 71.3%, execution time per run 1,364 s GPT-5.4 Mini: Success rate 71.4%, execution time per run 1,361 s GPT-5.4 Nano: Success rate 69.5%, execution time per run 1,481 s MiMo-V2-Omni: Success rate 78.2%, execution time per run 2,639 s MiMo-V2-Pro: Success rate 78.3%, execution time per run 2,047 s MiniMax-M2.7: Success rate 80.6%, execution time per run 2,761 s Qwen 3.6 Plus: Success rate 63.3%, execution time per run 1,844 s GLM-5V-Turbo: Success rate 67%, execution time per run 2,931 s Gemma 4 26B A4B: Success rate 74.1%, execution time per run 2,337 s Gemma 4 31B IT: Success rate 66.5%, execution time per run 3,325 s deepseek-chat: Success rate 53.1%, execution time per run 2,800 s. Not in our model list Claude Sonnet 4: Success rate 74%, execution time per run 86.8 s Gemini 3 Flash: Success rate 87%, execution time per run 2,180 s Seed2.0 Lite (0215): Success rate 79.8%, execution time per run 1,755 s GLM-5.1: Success rate 80.9%, execution time per run 1,880 s Gemini 3 FlashGLM-5.1Seed2.0 Lite (0215)Claude Sonnet 4
ModelSettingSuccess rateExecution time per run
Gemini 3 Flash OpenClaw 87.1% 2,180 s
GLM-5.1 OpenClaw 80.9% 1,880 s
Gemini 3.1 Pro OpenClaw 80.7% 2,252 s
MiniMax-M2.7 OpenClaw 80.6% 2,761 s
MiniMax-M2.1 OpenClaw 80.4% 2,367 s
Seed2.0 Lite (0215) OpenClaw 79.8% 1,755 s
MiMo-V2-Pro OpenClaw 78.3% 2,047 s
MiMo-V2-Omni OpenClaw 78.2% 2,639 s
Claude Sonnet 4.6 OpenClaw 76.4% 2,701 s
GLM-5 OpenClaw 76.2% 4,069 s
Kimi K2.5 OpenClaw 75.2% 2,891 s
Claude Opus 4.6 OpenClaw 74.4% 2,579 s
Gemma 4 26B A4B OpenClaw 74.1% 2,337 s
Claude Sonnet 4 OpenClaw 74% 86.76 s
GLM-5-Turbo OpenClaw 73.4% 2,309 s
Claude Opus 4.5 OpenClaw 72.5% 3,040 s
Claude Haiku 4.5 OpenClaw 71.6% 2,838 s
GPT-5.4 Mini OpenClaw 71.4% 1,361 s
Claude Sonnet 4.5 OpenClaw 71.3% 2,842 s
Mistral Small 4 OpenClaw 71.3% 1,364 s
Grok 4.20 OpenClaw 71.2% 1,402 s
Step 3.5 Flash OpenClaw 70.5% 2,797 s
GPT-5.4 Nano OpenClaw 69.5% 1,481 s
GLM-4.5-Air OpenClaw 69.4% 3,306 s
MiMo-V2-Flash OpenClaw 67.5% 3,216 s
MiniMax-M2.5 OpenClaw 67.2% 3,433 s
GLM-5V-Turbo OpenClaw 67% 2,931 s
Devstral 2 OpenClaw 67% 3,141 s
Grok 4.1 Fast OpenClaw 66.6% 3,264 s
Gemma 4 31B IT OpenClaw 66.5% 3,325 s
Mistral Large 3 OpenClaw 65.9% 3,116 s
GPT-4o Mar 2025 OpenClaw 65.4% 1,945 s
GPT-5 mini OpenClaw 65.2% 2,263 s
GPT-4o mini OpenClaw 64.4% 2,510 s
Qwen 3.6 Plus OpenClaw 63.3% 1,844 s
Gemini 2.5 Pro Jun 2025 OpenClaw 63.1% 3,518 s
Nova 2 Lite OpenClaw 63.1% 2,746 s
GPT-5.4 OpenClaw 59.9% 3,670 s
DeepSeek-V3.2 OpenClaw 59.9% 4,798 s
gpt-oss-120b OpenClaw 59.7% 2,355 s
Nemotron 3 Super OpenClaw 53.9% 4,652 s
deepseek-chat OpenClaw 53.1% 2,800 s
Gemini 2.5 Flash Sep 2025 OpenClaw 50.6% 2,386 s
gpt-oss-20b OpenClaw 48.4% 2,951 s
GPT-5 nano OpenClaw 41.5% 3,400 s
Trinity Large Preview OpenClaw 40.7% 4,674 s
Qwen3.5-9B OpenClaw 34.8% 872.16 s
GPT-5.4 Pro OpenClaw 12% 5,297 s
Gemini 2.5 Flash-Lite Sep 2025 OpenClaw 11.8% 2,328 s

The best score, 87%, costs $1.23 for the full run

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind MiniMax ByteDance Seed Qwen Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 $100 Cost of the full run, log scale Gemini 3 Flash MiniMax-M2.7 Gemma 4 26B A4B Qwen3.5-9B 0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 $100 Cost of the full run, log scale Gemini 3 Flash MiniMax-M2.7

Source: Official PinchBench leaderboard

ModelEffortHarnessScoreCost of the full runOutput tokens
Gemini 3 Flash OpenClaw 87% $1.23
GLM-5.1 OpenClaw 80.9%
Gemini 3.1 Pro OpenClaw 80.7% $1.84
MiniMax-M2.7 OpenClaw 80.6% $0.41
MiniMax-M2.1 OpenClaw 80.4% $0.24
Seed2.0 Lite (0215) OpenClaw 79.8% $0.21
MiMo-V2-Pro OpenClaw 78.3% $0.98
MiMo-V2-Omni OpenClaw 78.2% $0.52
Claude Sonnet 4.6 OpenClaw 76.4% $5.37
GLM-5 OpenClaw 76.2% $1.01
Kimi K2.5 OpenClaw 75.2% $0.42
Claude Opus 4.6 OpenClaw 74.4% $5.98
Gemma 4 26B A4B OpenClaw 74.1% $0.03
Claude Sonnet 4 OpenClaw 74% $0.24
GLM-5-Turbo OpenClaw 73.4% $0.73
Claude Opus 4.5 OpenClaw 72.5% $7.59
Claude Haiku 4.5 OpenClaw 71.6% $1.43
GPT-5.4 Mini OpenClaw 71.4% $0.18
Claude Sonnet 4.5 OpenClaw 71.3% $4.26
Mistral Small 4 OpenClaw 71.3% $0.05
Grok 4.20 OpenClaw 71.2% $1.16
Step 3.5 Flash OpenClaw 70.5% $0.25
GPT-5.4 Nano OpenClaw 69.5% $0.05
GLM-4.5-Air OpenClaw 69.4% $0.19
MiMo-V2-Flash OpenClaw 67.5% $0.27
MiniMax-M2.5 OpenClaw 67.2% $0.30
GLM-5V-Turbo OpenClaw 67% $2.06
Devstral 2 OpenClaw 67% $0.82
Grok 4.1 Fast OpenClaw 66.6% $0.42
Gemma 4 31B IT OpenClaw 66.5% $0.04
Mistral Large 3 OpenClaw 65.9% $0.82
GPT-4o Mar 2025 OpenClaw 65.4% $2.33
GPT-5 mini OpenClaw 65.2% $0.15
GPT-4o mini OpenClaw 64.4% $0.26
Qwen 3.6 Plus OpenClaw 63.3%
Gemini 2.5 Pro Jun 2025 OpenClaw 63.1% $2.66
Nova 2 Lite OpenClaw 63.1% $0.89
GPT-5.4 OpenClaw 59.9% $2.61
DeepSeek-V3.2 OpenClaw 59.9% $1.56
gpt-oss-120b OpenClaw 59.7% $0.07
Nemotron 3 Super OpenClaw 53.9%
Gemini 2.5 Flash Sep 2025 OpenClaw 50.6% $0.52
gpt-oss-20b OpenClaw 48.4% $0.05
GPT-5 nano OpenClaw 41.5% $0.04
Trinity Large Preview OpenClaw 40.7%
Qwen3.5-9B OpenClaw 34.8% $0.03
GPT-5.4 Pro OpenClaw 12% $15
Gemini 2.5 Flash-Lite Sep 2025 OpenClaw 11.8% $0.05

3 models have held number one in labs’ own numbers since its release in February 2026 Number one over time 3 models have held number one with claims included since its release in February 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Google DeepMind Xiaomi Corp Arcee AI Kuaishou
Own leaderboard Passed 85% in Feb 2026, by Gemini 3 Flash
Gemini 3 Flash 87%
Lab claims Passed 85% in Apr 2026, by Trinity Large Thinking
KAT-Coder-Pro V2.5 94.9%

Sources: Official PinchBench leaderboard and labs’ model cards

Gemini 3 Flash on the official leaderboard passed 85% in December 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) KAT-Coder-Pro V2.5 Gemini 3 Flash Claude Sonnet 4 GPT-4o GPT-4o mini 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) KAT-Coder-Pro V2.5 Gemini 3 Flash

Sources: Official PinchBench leaderboard and labs’ model cards

3 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Xiaomi Corp
Z.ai
Arcee AI
Ant Group
NVIDIA
Kuaishou
Liquid AI
DateLaunch postScoreAs the post writes it
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B 87.8 PinchBench
Oct 13, 2026 LFM2.5-24B-A2B 83.2 PinchBench
Aug 11, 2026 Nemotron 3.5 Lightning 30B-A3B 85.4 PinchBench
Aug 4, 2026 LFM2.5-2.6B 68.2 PinchBench
Jul 23, 2026 KAT-Coder-V2.5-Dev 93.4 PinchBench
Jul 6, 2026 KAT-Coder-Pro V2.5 94.9 PinchBench(Avg)
Jun 4, 2026 Nemotron 3 Ultra 90 PinchBench
May 14, 2026 Ring-2.6-1T 87.6 PinchBench (high)
Apr 29, 2026 Ling-2.6-1T 87.4 PinchBench (Avg. Score)
Apr 1, 2026 Trinity Large Thinking 91.9 PinchBench
Apr 1, 2026 GLM-5V-Turbo 80.7 PinchBench Best / Avg 87.0 / 80.7
Mar 18, 2026 MiMo-V2-Omni 81.2 PinchBench (avg.)
Mar 18, 2026 MiMo-V2-Pro 81 PinchBench (avg.)

6 system cards report it, from 3 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Oct 19, 2026 Muse Glimmer technical report 90.5 PinchBench, 24 tasks, high (Table 5) Capabilities
Oct 15, 2026 Nemotron 3.5 Lightning 30B-A3B technical report 83.9 PinchBench, BF16 acc. (Table 22) Capabilities
Oct 15, 2026 Nemotron 3.5 Bolt 74B-A9B technical report 87.8 PinchBench, released 24GB checkpoint (Table 7). BF16 90.35 Capabilities
Jun 13, 2026 Ring-2.6-1T technical report 87.6 PinchBench (Avg@3), high, no xhigh value (Table 6) Capabilities
Jun 13, 2026 Ling-2.6-1T technical report 85.2 PinchBench, Mean@5 (Table 5) Capabilities
Jun 9, 2026 Nemotron 3 Ultra technical report 90 PinchBench, held-out gate (Table 10) Capabilities

The lab's number comes out ahead in 3 of 3 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap
GLM-5V-Turbo 80.7% PinchBench Best / Avg 87.0 / 80.7 67% Leaderboard, OpenClaw +14 pts
MiMo-V2-Omni 81.2% PinchBench (avg.) 78.2% Leaderboard, OpenClaw +3.0 pts
MiMo-V2-Pro 81% PinchBench (avg.) 78.3% Leaderboard, OpenClaw +2.7 pts

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

Xiaomi Corp Arcee AI Kuaishou Qwen Anthropic
PinchBench 2.0 Passed 85% in May 2026, by MiMo-V2.5
Claude Opus 4.8 93.5%
PinchBench Passed 85% in Apr 2026, by Trinity Large Thinking
KAT-Coder-Pro V2.5 94.9%
PinchBench 2.0 Passed 85% in May 2026, by MiMo-V2.5
Claude Opus 4.8 93.5%

Sources: Official leaderboards and labs’ model cards

Replaced by PinchBench 2.0

More in Tool use

All 144 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%