Benchmark Almanac

APEX-Accounting

Gemini 4 Argon 71.5% leads since September 2026
In use

Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria.

Ten worlds are held out and an eleventh is open source. Mercor reports mean score and Pass@1.

Released
July 2026
Kind
Simulated companies
Level
Expert
Scored
Graded by a model
Answers
Agent task
Tasks
Public, plus a private test set
Strongest model tested
Claude Fable 5.2 #1 on the capability index
Last in a launch post
November 2026
Last model added
November 2026
Scores refreshed
November 13th, 2026

In the authors’ words

We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants. Tasks include reconciling accounts, accruing expenses, posting transactions, and producing reports. The private eval set comprises 160 tasks, split across 10 worlds. Each world contains an accounting system, as well as spreadsheets, PDFs, and other files. Every task was authored and solved by experts in accounting and bookkeeping, who also wrote grading rubrics. Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%. No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)) and the highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)). We experiment with increasing the token budget from $1 to $50 and observe an instance of Simpson's paradox: scores increase as the token budget increases but within a given budget-constrained harness, scores are lower on tasks where the model spends more tokens. As APEX-Accounting is a closed benchmark, leaderboard evals can be run for any frontier model on request.

Abstract of APEX-Accounting (arXiv:2607.27189, CC0)
50
Entries on its own leaderboard, 49 models we track
1
Launch post lists it, from 1 lab

Gemini 4 Argon leads with 71.5%

Best score per entry from the benchmark's own leaderboard, read on November 13th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

50 entries. Scroll sideways for the rest.

  1. 71.5 Gemini 4 Argon
  2. 70.7 Gemini 4 Carbon
  3. 70.4 Claude Fable 5.2
  4. 67.6 Gemini 4 Hydrogen
  5. 67.1 Muse Wonder
  6. 66.7 Claude Fable 5.1
  7. 65.5 Claude Opus 5.5
  8. 64.9 Muse Spark 1.3
  9. 64.7 GPT-6 Astra
  10. 64.5 Claude Sonnet 5.5
  11. 64.3 Claude Opus 5
  12. 64.3 GPT-6.1 Sol
  13. 63.7 GPT-6.1 Terra
  14. 62.4 Kimi K3.1 Pro
  15. 62.2 GPT-5.6 Sol
  16. 62.2 Claude Haiku 5.5
  17. 62.0 Inkling 2 Small
  18. 61.0 Claude Fable 5
  19. 60.3 Grok 4.6
  20. 60.0 MiniMax-M3.1-Flash
  21. 59.8 GPT-5.6 Terra
  22. 59.7 GPT-6 Sol
  23. 59.5 GLM-5.3
  24. 59.3 DeepSeek V4.1 Flash 1017
  25. 58.8 Gemini 3.8 Flash
  26. 58.0 Muse Spark Open
  27. 57.4 Grok 4.5
  28. 57.2 Qwen 3.8 Max
  29. 57.2 Gemini 3.7 Flash
  30. 56.7 Kimi K3
  31. 56.5 Claude Opus 4.7
  32. 56.5 Mistral Large 4
  33. 56.3 GPT-5.5
  34. 55.7 Claude Sonnet 5
  35. 55.3 Grok 4.7
  36. 54.3 GPT-5.6 Luna
  37. 53.9 GPT-6 Luna
  38. 53.7 GLM-5.3-Flash
  39. 53.4 DeepSeek V4 Pro 0813
  40. 53.3 DeepSeek-V4-Flash
  41. 52.5 Gemini 3.6 Flash
  42. 50.8 Mercury 3
  43. 50.1 DeepSeek V4.1 Flash
  44. 41.8 MiniMax-M3
  45. 38.1 Nemotron 3.5 Super VL
  46. 35.7 Nemotron 3.5 Bolt
  47. 31.4 Gemini 3.5 Flash
  48. 31.2 Inkling
  49. 29.4 Nemotron 3 Ultra
  50. 5.20 Gemini 3.5 Flash-Lite
  1. 1 Gemini 4 Argon high 71.5%
  2. 2 Gemini 4 Carbon high 70.7%
  3. 3 Claude Fable 5.2 max 70.4%
  4. 4 Gemini 4 Hydrogen xhigh 67.6%
  5. 5 Muse Wonder max 67.1%
  6. 6 Claude Fable 5.1 max 66.7%
  7. 7 Claude Opus 5.5 max 65.5%
  8. 8 Muse Spark 1.3 max 64.9%
  9. 9 GPT-6 Astra max 64.7%
  10. 10 Claude Sonnet 5.5 max 64.5%
  11. 11 Claude Opus 5 max 64.3%
  12. 12 GPT-6.1 Sol max 64.3%
  13. 13 GPT-6.1 Terra max 63.7%
  14. 14 Kimi K3.1 Pro max 62.4%
  15. 15 GPT-5.6 Sol max 62.2%
  16. 16 Claude Haiku 5.5 max 62.2%
  17. 17 Inkling 2 Small 62%
  18. 18 Claude Fable 5 max 61%
  19. 19 Grok 4.6 xhigh 60.3%
  20. 20 MiniMax-M3.1-Flash max 60%
  21. 21 GPT-5.6 Terra max 59.8%
  22. 22 GPT-6 Sol max 59.7%
  23. 23 GLM-5.3 max 59.5%
  24. 24 DeepSeek V4.1 Flash 1017 max 59.3%
  25. 25 Gemini 3.8 Flash high 58.8%
  26. 26 Muse Spark Open xhigh 58%
  27. 27 Grok 4.5 high 57.4%
  28. 28 Qwen 3.8 Max xhigh 57.2%
  29. 29 Gemini 3.7 Flash high 57.2%
  30. 30 Kimi K3 max 56.7%
  31. 31 Claude Opus 4.7 max 56.5%
  32. 32 Mistral Large 4 high 56.5%
  33. 33 GPT-5.5 xhigh 56.3%
  34. 34 Claude Sonnet 5 max 55.7%
  35. 35 Grok 4.7 xhigh 55.3%
  36. 36 GPT-5.6 Luna max 54.3%
  37. 37 GPT-6 Luna max 53.9%
  38. 38 GLM-5.3-Flash max 53.7%
  39. 39 DeepSeek V4 Pro 0813 max 53.4%
  40. 40 DeepSeek-V4-Flash max 53.3%
  41. 41 Gemini 3.6 Flash high 52.5%
  42. 42 Mercury 3 high 50.8%
  43. 43 DeepSeek V4.1 Flash max 50.1%
  44. 44 MiniMax-M3 high 41.8%
  45. 45 Nemotron 3.5 Super VL high 38.1%
  46. 46 Nemotron 3.5 Bolt high 35.7%
  47. 47 Gemini 3.5 Flash high 31.4%
  48. 48 Inkling high 31.2%
  49. 49 Nemotron 3 Ultra high 29.4%
  50. 50 Gemini 3.5 Flash-Lite high 5.20%

Source: Official APEX-Accounting leaderboard

#EntryScoreSettingDateSource
1 Gemini 4 Argon 71.5% high, loop_agent Sep 2026 Official leaderboard
2 Gemini 4 Carbon 70.7% high, loop_agent Oct 2026 Official leaderboard
3 Claude Fable 5.2 70.4% max, loop_agent Nov 2026 Official leaderboard
4 Gemini 4 Hydrogen 67.6% xhigh, loop_agent Nov 2026 Official leaderboard
5 Muse Wonder 67.1% max, loop_agent Oct 2026 Official leaderboard
6 Claude Fable 5.1 66.7% max, loop_agent Sep 2026 Official leaderboard
7 Claude Opus 5.5 65.5% max, loop_agent Sep 2026 Official leaderboard
8 Muse Spark 1.3 64.9% max, loop_agent Sep 2026 Official leaderboard
9 GPT-6 Astra 64.7% max, loop_agent Sep 2026 Official leaderboard
10 Claude Sonnet 5.5 64.5% max, loop_agent Sep 2026 Official leaderboard
11 Claude Opus 5 64.3% max, loop_agent Jul 2026 Official leaderboard
12 GPT-6.1 Sol 64.3% max, loop_agent Sep 2026 Official leaderboard
13 GPT-6.1 Terra 63.7% max, loop_agent Oct 2026 Official leaderboard
14 Kimi K3.1 Pro 62.4% max, loop_agent Oct 2026 Official leaderboard
15 GPT-5.6 Sol 62.2% max, loop_agent Jul 2026 Official leaderboard
16 Claude Haiku 5.5 62.2% max, loop_agent Oct 2026 Official leaderboard
17 Inkling 2 Small 62% loop_agent Nov 2026 Official leaderboard
18 Claude Fable 5 61% max, loop_agent Jun 2026 Official leaderboard
19 Grok 4.6 60.3% xhigh, loop_agent Aug 2026 Official leaderboard
20 MiniMax-M3.1-Flash 60% max, loop_agent Oct 2026 Official leaderboard
21 GPT-5.6 Terra 59.8% max, loop_agent Jul 2026 Official leaderboard
22 GPT-6 Sol 59.7% max, loop_agent Sep 2026 Official leaderboard
23 GLM-5.3 59.5% max, loop_agent Aug 2026 Official leaderboard
24 DeepSeek V4.1 Flash 1017 59.3% max, loop_agent Oct 2026 Official leaderboard
25 Gemini 3.8 Flash 58.8% high, loop_agent Sep 2026 Official leaderboard
26 Muse Spark Open 58% xhigh, loop_agent Oct 2026 Official leaderboard
27 Grok 4.5 57.4% high, loop_agent Jul 2026 Official leaderboard
28 Qwen 3.8 Max 57.2% xhigh, loop_agent Aug 2026 Official leaderboard
29 Gemini 3.7 Flash 57.2% high, loop_agent Aug 2026 Official leaderboard
30 Kimi K3 56.7% max, loop_agent Jul 2026 Official leaderboard
31 Claude Opus 4.7 56.5% max, loop_agent Apr 2026 Official leaderboard
32 Mistral Large 4 56.5% high, loop_agent Oct 2026 Official leaderboard
33 GPT-5.5 56.3% xhigh, loop_agent Apr 2026 Official leaderboard
34 Claude Sonnet 5 55.7% max, loop_agent Jun 2026 Official leaderboard
35 Grok 4.7 55.3% xhigh, loop_agent Sep 2026 Official leaderboard
36 GPT-5.6 Luna 54.3% max, loop_agent Jul 2026 Official leaderboard
37 GPT-6 Luna 53.9% max, loop_agent Sep 2026 Official leaderboard
38 GLM-5.3-Flash 53.7% max, loop_agent Aug 2026 Official leaderboard
39 DeepSeek V4 Pro 0813 53.4% max, loop_agent Aug 2026 Official leaderboard
40 DeepSeek-V4-Flash 53.3% max, loop_agent Apr 2026 Official leaderboard
41 Gemini 3.6 Flash 52.5% high, loop_agent Jul 2026 Official leaderboard
42 Mercury 3 50.8% high, loop_agent Oct 2026 Official leaderboard
43 DeepSeek V4.1 Flash 50.1% max, loop_agent Sep 2026 Official leaderboard
44 MiniMax-M3 41.8% high, loop_agent Jun 2026 Official leaderboard
45 Nemotron 3.5 Super VL 38.1% high, loop_agent Oct 2026 Official leaderboard
46 Nemotron 3.5 Bolt 35.7% high, loop_agent Official leaderboard
47 Gemini 3.5 Flash 31.4% high, loop_agent May 2026 Official leaderboard
48 Inkling 31.2% high, loop_agent Jul 2026 Official leaderboard
49 Nemotron 3 Ultra 29.4% high, loop_agent Jun 2026 Official leaderboard
50 Gemini 3.5 Flash-Lite 5.20% high, loop_agent Jul 2026 Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

Cost vs. Performance

Mercor costPerRun values are preserved without assuming a per-task denominator. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0% 20% 40% 60% 80% $0$2$4$6$8 Source cost per run ($), lower is better Mean score Claude Opus 4.7 (max): Mean score 56.5%, source cost per run $4.1 GPT-5.5 (xhigh): Mean score 56.3%, source cost per run $2.81 DeepSeek-V4-Flash (max): Mean score 53.3%, source cost per run $0.16 Gemini 3.5 Flash (high): Mean score 31.4%, source cost per run $3.56 MiniMax-M3 (high): Mean score 41.8%, source cost per run $0.15 Nemotron 3 Ultra (high): Mean score 29.4%, source cost per run $0.24 Claude Fable 5 (max): Mean score 61%, source cost per run $7.14 Claude Sonnet 5 (max): Mean score 55.7%, source cost per run $2.25 Grok 4.5 (high): Mean score 57.4%, source cost per run $0.81 GPT-5.6 Luna (max): Mean score 54.3%, source cost per run $0.23 GPT-5.6 Sol (max): Mean score 62.2%, source cost per run $3.04 GPT-5.6 Terra (max): Mean score 59.8%, source cost per run $2.33 Inkling (high): Mean score 31.2%, source cost per run $0.5 Kimi K3 (max): Mean score 56.7%, source cost per run $1.66 Gemini 3.5 Flash-Lite (high): Mean score 5.2%, source cost per run $0.07 Gemini 3.6 Flash (high): Mean score 52.5%, source cost per run $0.57 Claude Opus 5 (max): Mean score 64.3%, source cost per run $3.29 Qwen 3.8 Max (xhigh): Mean score 57.2%, source cost per run $1.77 Grok 4.6 (xhigh): Mean score 60.3%, source cost per run $2.08 DeepSeek V4 Pro 0813 (max): Mean score 53.4%, source cost per run $1.04 Gemini 3.7 Flash (high): Mean score 57.2%, source cost per run $0.75 GLM-5.3 (max): Mean score 59.5%, source cost per run $1.98 GLM-5.3-Flash (max): Mean score 53.7%, source cost per run $0.24 Claude Fable 5.1 (max): Mean score 66.7%, source cost per run $5.73 Gemini 3.8 Flash (high): Mean score 58.8%, source cost per run $0.79 Muse Spark 1.3 (max): Mean score 64.9%, source cost per run $2.39 GPT-6 Astra (max): Mean score 64.7%, source cost per run $3.07 DeepSeek V4.1 Flash (max): Mean score 50.1%, source cost per run $0.29 Grok 4.7 (xhigh): Mean score 55.3%, source cost per run $2.63 Claude Opus 5.5 (max): Mean score 65.5%, source cost per run $3.54 GPT-6 Sol (max): Mean score 59.7%, source cost per run $1.24 Claude Sonnet 5.5 (max): Mean score 64.5%, source cost per run $2.09 Mistral Large 4 (high): Mean score 56.5%, source cost per run $0.73 Muse Wonder (max): Mean score 67.1%, source cost per run $4.91 Muse Spark Open (xhigh): Mean score 58%, source cost per run $1.1 Nemotron 3.5 Super VL (high): Mean score 38.1%, source cost per run $0.14 DeepSeek V4.1 Flash 1017 (max): Mean score 59.3%, source cost per run $0.18 MiniMax-M3.1-Flash (max): Mean score 60%, source cost per run $0.16 Kimi K3.1 Pro (max): Mean score 62.4%, source cost per run $2.98 Nemotron 3.5 Bolt (high): Mean score 35.7%, source cost per run $0.11. Not in our model list GPT-6 Luna (max): Mean score 53.9%, source cost per run $0.14 GPT-6.1 Sol (max): Mean score 64.3%, source cost per run $0.63 Gemini 4 Argon (high): Mean score 71.5%, source cost per run $2.04 Claude Haiku 5.5 (max): Mean score 62.2%, source cost per run $0.15 Mercury 3 (high): Mean score 50.8%, source cost per run $0.05 GPT-6.1 Terra (max): Mean score 63.7%, source cost per run $0.46 Gemini 4 Argon (high)GPT-6.1 Sol (max)GPT-6.1 Terra (max)Claude Haiku 5.5 (max)GPT-6 Luna (max)Mercury 3 (high) 0% 20% 40% 60% 80% $0$2$4$6$8 Source cost per run ($), lower is better Mean score Claude Opus 4.7 (max): Mean score 56.5%, source cost per run $4.1 GPT-5.5 (xhigh): Mean score 56.3%, source cost per run $2.81 DeepSeek-V4-Flash (max): Mean score 53.3%, source cost per run $0.16 Gemini 3.5 Flash (high): Mean score 31.4%, source cost per run $3.56 MiniMax-M3 (high): Mean score 41.8%, source cost per run $0.15 Nemotron 3 Ultra (high): Mean score 29.4%, source cost per run $0.24 Claude Fable 5 (max): Mean score 61%, source cost per run $7.14 Claude Sonnet 5 (max): Mean score 55.7%, source cost per run $2.25 Grok 4.5 (high): Mean score 57.4%, source cost per run $0.81 GPT-5.6 Luna (max): Mean score 54.3%, source cost per run $0.23 GPT-5.6 Sol (max): Mean score 62.2%, source cost per run $3.04 GPT-5.6 Terra (max): Mean score 59.8%, source cost per run $2.33 Inkling (high): Mean score 31.2%, source cost per run $0.5 Kimi K3 (max): Mean score 56.7%, source cost per run $1.66 Gemini 3.5 Flash-Lite (high): Mean score 5.2%, source cost per run $0.07 Gemini 3.6 Flash (high): Mean score 52.5%, source cost per run $0.57 Claude Opus 5 (max): Mean score 64.3%, source cost per run $3.29 Qwen 3.8 Max (xhigh): Mean score 57.2%, source cost per run $1.77 Grok 4.6 (xhigh): Mean score 60.3%, source cost per run $2.08 DeepSeek V4 Pro 0813 (max): Mean score 53.4%, source cost per run $1.04 Gemini 3.7 Flash (high): Mean score 57.2%, source cost per run $0.75 GLM-5.3 (max): Mean score 59.5%, source cost per run $1.98 GLM-5.3-Flash (max): Mean score 53.7%, source cost per run $0.24 Claude Fable 5.1 (max): Mean score 66.7%, source cost per run $5.73 Gemini 3.8 Flash (high): Mean score 58.8%, source cost per run $0.79 Muse Spark 1.3 (max): Mean score 64.9%, source cost per run $2.39 GPT-6 Astra (max): Mean score 64.7%, source cost per run $3.07 DeepSeek V4.1 Flash (max): Mean score 50.1%, source cost per run $0.29 Grok 4.7 (xhigh): Mean score 55.3%, source cost per run $2.63 Claude Opus 5.5 (max): Mean score 65.5%, source cost per run $3.54 GPT-6 Sol (max): Mean score 59.7%, source cost per run $1.24 Claude Sonnet 5.5 (max): Mean score 64.5%, source cost per run $2.09 Mistral Large 4 (high): Mean score 56.5%, source cost per run $0.73 Muse Wonder (max): Mean score 67.1%, source cost per run $4.91 Muse Spark Open (xhigh): Mean score 58%, source cost per run $1.1 Nemotron 3.5 Super VL (high): Mean score 38.1%, source cost per run $0.14 DeepSeek V4.1 Flash 1017 (max): Mean score 59.3%, source cost per run $0.18 MiniMax-M3.1-Flash (max): Mean score 60%, source cost per run $0.16 Kimi K3.1 Pro (max): Mean score 62.4%, source cost per run $2.98 Nemotron 3.5 Bolt (high): Mean score 35.7%, source cost per run $0.11. Not in our model list GPT-6 Luna (max): Mean score 53.9%, source cost per run $0.14 GPT-6.1 Sol (max): Mean score 64.3%, source cost per run $0.63 Gemini 4 Argon (high): Mean score 71.5%, source cost per run $2.04 Claude Haiku 5.5 (max): Mean score 62.2%, source cost per run $0.15 Mercury 3 (high): Mean score 50.8%, source cost per run $0.05 GPT-6.1 Terra (max): Mean score 63.7%, source cost per run $0.46 Gemini 4 Argon (high)GPT-6.1 Sol (max)GPT-6.1 Terra (max)Claude Haiku 5.5 (max)GPT-6 Luna (max)Mercury 3 (high)
ModelSettingMean scoreSource cost per run
Gemini 4 Argon high, loop_agent 71.5% $2.04
Muse Wonder max, loop_agent 67.1% $4.91
Claude Fable 5.1 max, loop_agent 66.7% $5.73
Claude Opus 5.5 max, loop_agent 65.5% $3.54
Muse Spark 1.3 max, loop_agent 64.9% $2.39
GPT-6 Astra max, loop_agent 64.7% $3.07
Claude Sonnet 5.5 max, loop_agent 64.5% $2.09
GPT-6.1 Sol max, loop_agent 64.3% $0.63
Claude Opus 5 max, loop_agent 64.3% $3.29
GPT-6.1 Terra max, loop_agent 63.7% $0.46
Kimi K3.1 Pro max, loop_agent 62.4% $2.98
Claude Haiku 5.5 max, loop_agent 62.2% $0.15
GPT-5.6 Sol max, loop_agent 62.2% $3.04
Claude Fable 5 max, loop_agent 61% $7.14
Grok 4.6 xhigh, loop_agent 60.3% $2.08
MiniMax-M3.1-Flash max, loop_agent 60% $0.16
GPT-5.6 Terra max, loop_agent 59.8% $2.33
GPT-6 Sol max, loop_agent 59.7% $1.24
GLM-5.3 max, loop_agent 59.5% $1.98
DeepSeek V4.1 Flash 1017 max, loop_agent 59.3% $0.18
Gemini 3.8 Flash high, loop_agent 58.8% $0.79
Muse Spark Open xhigh, loop_agent 58% $1.1
Grok 4.5 high, loop_agent 57.4% $0.81
Gemini 3.7 Flash high, loop_agent 57.2% $0.75
Qwen 3.8 Max xhigh, loop_agent 57.2% $1.77
Kimi K3 max, loop_agent 56.7% $1.66
Mistral Large 4 high, loop_agent 56.5% $0.73
Claude Opus 4.7 max, loop_agent 56.5% $4.1
GPT-5.5 xhigh, loop_agent 56.3% $2.81
Claude Sonnet 5 max, loop_agent 55.7% $2.25
Grok 4.7 xhigh, loop_agent 55.3% $2.63
GPT-5.6 Luna max, loop_agent 54.3% $0.23
GPT-6 Luna max, loop_agent 53.9% $0.14
GLM-5.3-Flash max, loop_agent 53.7% $0.24
DeepSeek V4 Pro 0813 max, loop_agent 53.4% $1.04
DeepSeek-V4-Flash max, loop_agent 53.3% $0.16
Gemini 3.6 Flash high, loop_agent 52.5% $0.57
Mercury 3 high, loop_agent 50.8% $0.05
DeepSeek V4.1 Flash max, loop_agent 50.1% $0.29
MiniMax-M3 high, loop_agent 41.8% $0.15
Nemotron 3.5 Super VL high, loop_agent 38.1% $0.14
Nemotron 3.5 Bolt high, loop_agent 35.7% $0.11
Gemini 3.5 Flash high, loop_agent 31.4% $3.56
Inkling high, loop_agent 31.2% $0.5
Nemotron 3 Ultra high, loop_agent 29.4% $0.24
Gemini 3.5 Flash-Lite high, loop_agent 5.2% $0.07

5 models have held number one on the official leaderboard since its release in July 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI Google DeepMind
Own leaderboard 5 leaders
Gemini 4 Argon 71.5%

Source: Official APEX-Accounting leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Gemini 4 Argon Claude Fable 5.1 GPT-5.6 Sol Claude Opus 4.7 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Gemini 4 Argon Claude Opus 5

Sources: Official APEX-Accounting leaderboard and labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Thinking Machines
DateLaunch postScoreAs the post writes it
Nov 5, 2026 Inkling 2 Small 14.5 Apex-Accounting v1.1 pass@1 (mean score 62.0)

One lab claim next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap
Inkling 2 Small 14.5% Apex-Accounting v1.1 pass@1 (mean score 62.0) 62% Leaderboard, loop_agent −48 pts

More in Economic work

All 97 in Economic work
  • JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
  • GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Seed2.1 Pro 87.9%
  • GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
  • AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
  • CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Claude Fable 5 75.9%
  • OfficeQA In use Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. Claude Opus 4.7 86.3%