Benchmark Almanac

CommerceAgentBench

Claude Opus 5 61.7% leads since August 2026
Not in model cards

Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds.

107 tasks: 53 CLI, 28 browser, 16 file and 10 API or MCP. Tasks were distilled from Accio merchant traces. Verifiers are deterministic or LLM-assisted per task. Formerly RealReplicaBench. Repo has 1,301 stars.

Released
August 2026
Kind
Running a business
Level
Expert
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
Claude Opus 5 #10 on the capability index
Last model added
August 2026
Scores refreshed
November 12th, 2026
13
Models scored on its own leaderboard

Claude Opus 5 leads with 61.7% among models released by August 2026

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

The newest model with a score here came out in August 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

13 models. Scroll sideways for the rest.

  1. 61.7 Claude Opus 5
  2. 55.1 Claude Opus 4.8
  3. 52.3 Claude Opus 4.7
  4. 52.3 Qwen 3.8 Max
  5. 51.4 GPT-5.6 Sol
  6. 50.5 DeepSeek-V4-Pro
  7. 47.7 GPT-5.5
  8. 46.7 DeepSeek-V4-Flash
  9. 46.7 GLM-5.2
  10. 46.7 Gemini 3.6 Flash
  11. 44.9 GPT-5.6 Luna
  12. 43.0 Gemini 3.5 Flash
  13. 29.0 Gemini 3 Flash
  1. 1 Claude Opus 5 61.7%
  2. 2 Claude Opus 4.8 55.1%
  3. 3 Claude Opus 4.7 52.3%
  4. 4 Qwen 3.8 Max 52.3%
  5. 5 GPT-5.6 Sol 51.4%
  6. 6 DeepSeek-V4-Pro 50.5%
  7. 7 GPT-5.5 47.7%
  8. 8 DeepSeek-V4-Flash 46.7%
  9. 9 GLM-5.2 46.7%
  10. 10 Gemini 3.6 Flash 46.7%
  11. 11 GPT-5.6 Luna 44.9%
  12. 12 Gemini 3.5 Flash 43%
  13. 13 Gemini 3 Flash 29%

Source: Official CommerceAgentBench leaderboard

#ModelScoreSettingDateSource
1 Claude Opus 5 61.7% Accio Jul 2026 Official leaderboard
2 Claude Opus 4.8 55.1% Accio May 2026 Official leaderboard
3 Claude Opus 4.7 52.3% Accio Apr 2026 Official leaderboard
4 Qwen 3.8 Max 52.3% Accio Aug 2026 Official leaderboard
5 GPT-5.6 Sol 51.4% Accio Jul 2026 Official leaderboard
6 DeepSeek-V4-Pro 50.5% Accio Apr 2026 Official leaderboard
7 GPT-5.5 47.7% OpenClaw Apr 2026 Official leaderboard
8 DeepSeek-V4-Flash 46.7% Accio Apr 2026 Official leaderboard
9 GLM-5.2 46.7% Pi Jun 2026 Official leaderboard
10 Gemini 3.6 Flash 46.7% Accio Jul 2026 Official leaderboard
11 GPT-5.6 Luna 44.9% Accio Jul 2026 Official leaderboard
12 Gemini 3.5 Flash 43% Accio May 2026 Official leaderboard
13 Gemini 3 Flash 29% Pi Dec 2025 Official leaderboard

Pass rate vs. task cost

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

40% 60% 80% $0$1$2$3$4$5 Average task cost ($), lower is better Pass rate Gemini 3 Flash: Pass rate 29%, average task cost $0.61 Gemini 3 Flash: Pass rate 29%, average task cost $0.51 Gemini 3 Flash: Pass rate 29%, average task cost $0.38 Claude Opus 4.7: Pass rate 48.6%, average task cost $1.86 Claude Opus 4.7: Pass rate 45.8%, average task cost $3.39 Claude Opus 4.7: Pass rate 52.3%, average task cost $3.3 GPT-5.5: Pass rate 43.9%, average task cost $1.31 GPT-5.5: Pass rate 47.7%, average task cost $2.73 GPT-5.5: Pass rate 44.9%, average task cost $2 DeepSeek-V4-Flash: Pass rate 43%, average task cost $0.06 DeepSeek-V4-Flash: Pass rate 43%, average task cost $0.07 DeepSeek-V4-Pro: Pass rate 43.9%, average task cost $0.31 Gemini 3.5 Flash: Pass rate 41.1%, average task cost $3.27 Gemini 3.5 Flash: Pass rate 36.5%, average task cost $2.11 Gemini 3.5 Flash: Pass rate 43%, average task cost $2.05 Claude Opus 4.8: Pass rate 52.3%, average task cost $1.83 Claude Opus 4.8: Pass rate 51.4%, average task cost $3.57 Claude Opus 4.8: Pass rate 55.1%, average task cost $4.17 GLM-5.2: Pass rate 46.7%, average task cost $0.28 GLM-5.2: Pass rate 39.3%, average task cost $1.07 GLM-5.2: Pass rate 46.7%, average task cost $1.25 GPT-5.6 Luna: Pass rate 33.6%, average task cost $0.48 GPT-5.6 Luna: Pass rate 44.9%, average task cost $0.73 GPT-5.6 Sol: Pass rate 48.6%, average task cost $1.08 GPT-5.6 Sol: Pass rate 49.5%, average task cost $3.16 GPT-5.6 Sol: Pass rate 51.4%, average task cost $2.79 Gemini 3.6 Flash: Pass rate 43.9%, average task cost $2.03 Gemini 3.6 Flash: Pass rate 44.9%, average task cost $2.92 Gemini 3.6 Flash: Pass rate 46.7%, average task cost $1.34 Claude Opus 5: Pass rate 56.1%, average task cost $2.7 Qwen 3.8 Max: Pass rate 49.5%, average task cost $0.72 Qwen 3.8 Max: Pass rate 43.9%, average task cost $0.83 DeepSeek-V4-Flash: Pass rate 46.7%, average task cost $0.04 DeepSeek-V4-Pro: Pass rate 49.5%, average task cost $0.2 DeepSeek-V4-Pro: Pass rate 50.5%, average task cost $0.24 GPT-5.6 Luna: Pass rate 39.3%, average task cost $0.03 Claude Opus 5: Pass rate 60.8%, average task cost $1.79 Claude Opus 5: Pass rate 61.7%, average task cost $3.5 Qwen 3.8 Max: Pass rate 52.3%, average task cost $1.28 Claude Opus 5Claude Opus 5Qwen 3.8 MaxDeepSeek-V4-ProDeepSeek-V4-ProDeepSeek-V4-FlashGPT-5.6 Luna 40% 60% 80% $0$1$2$3$4$5 Average task cost ($), lower is better Pass rate Gemini 3 Flash: Pass rate 29%, average task cost $0.61 Gemini 3 Flash: Pass rate 29%, average task cost $0.51 Gemini 3 Flash: Pass rate 29%, average task cost $0.38 Claude Opus 4.7: Pass rate 48.6%, average task cost $1.86 Claude Opus 4.7: Pass rate 45.8%, average task cost $3.39 Claude Opus 4.7: Pass rate 52.3%, average task cost $3.3 GPT-5.5: Pass rate 43.9%, average task cost $1.31 GPT-5.5: Pass rate 47.7%, average task cost $2.73 GPT-5.5: Pass rate 44.9%, average task cost $2 DeepSeek-V4-Flash: Pass rate 43%, average task cost $0.06 DeepSeek-V4-Flash: Pass rate 43%, average task cost $0.07 DeepSeek-V4-Pro: Pass rate 43.9%, average task cost $0.31 Gemini 3.5 Flash: Pass rate 41.1%, average task cost $3.27 Gemini 3.5 Flash: Pass rate 36.5%, average task cost $2.11 Gemini 3.5 Flash: Pass rate 43%, average task cost $2.05 Claude Opus 4.8: Pass rate 52.3%, average task cost $1.83 Claude Opus 4.8: Pass rate 51.4%, average task cost $3.57 Claude Opus 4.8: Pass rate 55.1%, average task cost $4.17 GLM-5.2: Pass rate 46.7%, average task cost $0.28 GLM-5.2: Pass rate 39.3%, average task cost $1.07 GLM-5.2: Pass rate 46.7%, average task cost $1.25 GPT-5.6 Luna: Pass rate 33.6%, average task cost $0.48 GPT-5.6 Luna: Pass rate 44.9%, average task cost $0.73 GPT-5.6 Sol: Pass rate 48.6%, average task cost $1.08 GPT-5.6 Sol: Pass rate 49.5%, average task cost $3.16 GPT-5.6 Sol: Pass rate 51.4%, average task cost $2.79 Gemini 3.6 Flash: Pass rate 43.9%, average task cost $2.03 Gemini 3.6 Flash: Pass rate 44.9%, average task cost $2.92 Gemini 3.6 Flash: Pass rate 46.7%, average task cost $1.34 Claude Opus 5: Pass rate 56.1%, average task cost $2.7 Qwen 3.8 Max: Pass rate 49.5%, average task cost $0.72 Qwen 3.8 Max: Pass rate 43.9%, average task cost $0.83 DeepSeek-V4-Flash: Pass rate 46.7%, average task cost $0.04 DeepSeek-V4-Pro: Pass rate 49.5%, average task cost $0.2 DeepSeek-V4-Pro: Pass rate 50.5%, average task cost $0.24 GPT-5.6 Luna: Pass rate 39.3%, average task cost $0.03 Claude Opus 5: Pass rate 60.8%, average task cost $1.79 Claude Opus 5: Pass rate 61.7%, average task cost $3.5 Qwen 3.8 Max: Pass rate 52.3%, average task cost $1.28 Claude Opus 5Claude Opus 5Qwen 3.8 MaxDeepSeek-V4-ProDeepSeek-V4-ProDeepSeek-V4-FlashGPT-5.6 Luna
ModelSettingPass rateAverage task cost
Claude Opus 5 Accio 61.7% $3.51
Claude Opus 5 Pi 60.8% $1.79
Claude Opus 5 OpenClaw 56.1% $2.7
Claude Opus 4.8 Accio 55.1% $4.17
Qwen 3.8 Max Accio 52.3% $1.28
Claude Opus 4.8 Pi 52.3% $1.83
Claude Opus 4.7 Accio 52.3% $3.3
GPT-5.6 Sol Accio 51.4% $2.79
Claude Opus 4.8 OpenClaw 51.4% $3.57
DeepSeek-V4-Pro Accio 50.5% $0.24
DeepSeek-V4-Pro Pi 49.5% $0.2
Qwen 3.8 Max Pi 49.5% $0.72
GPT-5.6 Sol OpenClaw 49.5% $3.16
GPT-5.6 Sol Pi 48.6% $1.08
Claude Opus 4.7 Pi 48.6% $1.86
GPT-5.5 OpenClaw 47.7% $2.73
DeepSeek-V4-Flash Accio 46.7% $0.04
GLM-5.2 Pi 46.7% $0.28
GLM-5.2 Accio 46.7% $1.25
Gemini 3.6 Flash Accio 46.7% $1.35
Claude Opus 4.7 OpenClaw 45.8% $3.39
GPT-5.6 Luna Accio 44.9% $0.73
GPT-5.5 Accio 44.9% $2
Gemini 3.6 Flash OpenClaw 44.9% $2.93
DeepSeek-V4-Pro OpenClaw 43.9% $0.31
Qwen 3.8 Max OpenClaw 43.9% $0.83
GPT-5.5 Pi 43.9% $1.31
Gemini 3.6 Flash Pi 43.9% $2.03
DeepSeek-V4-Flash Pi 43% $0.06
DeepSeek-V4-Flash OpenClaw 43% $0.07
Gemini 3.5 Flash Accio 43% $2.05
Gemini 3.5 Flash Pi 41.1% $3.27
GPT-5.6 Luna Pi 39.3% $0.03
GLM-5.2 OpenClaw 39.3% $1.08
Gemini 3.5 Flash OpenClaw 36.5% $2.11
GPT-5.6 Luna OpenClaw 33.6% $0.48
Gemini 3 Flash Accio 29% $0.38
Gemini 3 Flash OpenClaw 29% $0.51
Gemini 3 Flash Pi 29% $0.61

The best score, 61.7%, costs $3.50 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

DeepSeek OpenAI Anthropic Qwen Cost frontier One model at each effort level
0% 20% 40% 60% 80% $0.01 $0.10 $1.00 $10 Cost per task, log scale Claude Opus 5 Claude Opus 5 Qwen 3.8 Max DeepSeek-V4-Pro DeepSeek-V4-Flash GPT-5.6 Luna 0% 20% 40% 60% 80% $0.01 $0.10 $1.00 $10 Cost per task, log scale Claude Opus 5 Qwen 3.8 Max

Source: Official CommerceAgentBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 5 Accio 61.7% $3.50
Claude Opus 5 Pi 60.8% $1.79
Claude Opus 5 OpenClaw 56.1% $2.70
Claude Opus 4.8 Accio 55.1% $4.17
Claude Opus 4.7 Accio 52.3% $3.30
Claude Opus 4.8 Pi 52.3% $1.83
Qwen 3.8 Max Accio 52.3% $1.28
Claude Opus 4.8 OpenClaw 51.4% $3.57
GPT-5.6 Sol Accio 51.4% $2.79
DeepSeek-V4-Pro Accio 50.5% $0.24
DeepSeek-V4-Pro Pi 49.5% $0.20
GPT-5.6 Sol OpenClaw 49.5% $3.16
Qwen 3.8 Max Pi 49.5% $0.72
Claude Opus 4.7 Pi 48.6% $1.86
GPT-5.6 Sol Pi 48.6% $1.08
GPT-5.5 OpenClaw 47.7% $2.73
DeepSeek-V4-Flash Accio 46.7% $0.04
GLM-5.2 Pi 46.7% $0.28
GLM-5.2 Accio 46.7% $1.25
Gemini 3.6 Flash Accio 46.7% $1.34
Claude Opus 4.7 OpenClaw 45.8% $3.39
GPT-5.5 Accio 44.9% $2.00
GPT-5.6 Luna Accio 44.9% $0.73
Gemini 3.6 Flash OpenClaw 44.9% $2.92
GPT-5.5 Pi 43.9% $1.31
DeepSeek-V4-Pro OpenClaw 43.9% $0.31
Gemini 3.6 Flash Pi 43.9% $2.03
Qwen 3.8 Max OpenClaw 43.9% $0.83
DeepSeek-V4-Flash Pi 43% $0.06
DeepSeek-V4-Flash OpenClaw 43% $0.07
Gemini 3.5 Flash Accio 43% $2.05
Gemini 3.5 Flash Pi 41.1% $3.27
GLM-5.2 OpenClaw 39.3% $1.07
GPT-5.6 Luna Pi 39.3% $0.03
Gemini 3.5 Flash OpenClaw 36.5% $2.11
GPT-5.6 Luna OpenClaw 33.6% $0.48
Gemini 3 Flash Pi 29% $0.61
Gemini 3 Flash OpenClaw 29% $0.51
Gemini 3 Flash Accio 29% $0.38

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Not tested since Aug 2026
Claude Opus 5 61.7%

Source: Official CommerceAgentBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5 Claude Opus 4.8 Gemini 3 Flash 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5 Claude Opus 4.8

Source: Official CommerceAgentBench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Economic work

All 97 in Economic work
  • JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
  • GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Seed2.1 Pro 87.9%
  • GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
  • AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
  • CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Claude Fable 5 75.9%
  • OfficeQA In use Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. Claude Opus 4.7 86.3%