Benchmark Almanac

SMB Bench

GPT-6 Astra $7,749 leads since November 2026
Not in model cards

An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule.

Six businesses, each played in normal, customer and supplier variants where the cost of shortfalls arrives weeks later. The fixed rule earns $6,412 a quarter and a scripted expert buyer $10,795. The simulator stays on ORO's hosted service, which needs a grant, and the grader is gated. Runs and traces are public on Hugging Face.

Made by
ORO AI
Released
November 2026
Kind
Running a business
Level
University
Scored
Checked automatically
Answers
Agent task
Tasks
Kept private
Measures
Capability
Language
English
In and out
Text, Tables → Actions
Interaction
Agentic
Items from
Generated
Refresh
One fixed set
Run by
A host
Occupations
Business and financial operations
Strongest model tested
Claude Fable 5.2 #2 on the capability index
Last model added
November 2026
Scores refreshed
November 20th, 2026
16
Models scored on its own leaderboard

GPT-6 Astra leads with $7,749

Best score per model from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

  1. 1 GPT-6 Astra gpt-6-astra; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $7,749
  2. 2 Gemini 4 Carbon gemini-4-carbon; thinking level high; 65,536; runtime/v1.11-carbon; 3437a5c1 / 575d06be; 5d582f7f; GeminiAdapter, 300 s idle timeout; 2026-11-10 to 2026-11-12 $7,703
  3. 3 Claude Fable 5.2 claude-fable-5-2; effort high, adaptive thinking; 65,536; runtime/v1.11-fable52; 95bb6628 / bda9748d; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-10 $7,275
  4. 4 Claude Fable 5.1 claude-fable-5-1; effort high, adaptive thinking; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $5,427
  5. 5 GPT-6.1 Sol gpt-6.1-sol; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $5,404
  6. 6 Gemini 4 Hydrogen gemini-4-hydrogen; thinking level high; 65,536; runtime/v1.11-hydrogen; d490ef8c / b3b3b2ed; 40031d67; GeminiAdapter, 300 s idle timeout; 2026-11-12 to 2026-11-13 $2,560
  7. 7 Gemini 4 Argon gemini-4-argon; thinking level high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $1,501
  8. 8 Claude Opus 5.5 claude-opus-5-5; effort high, adaptive thinking; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $907
  9. 9 DeepSeek V4.1 Flash deepseek-ai/DeepSeek-V4.1-Flash-1017-Fast; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $-3,036
  10. 10 Grok 4.7 grok-4.7; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $-4,722
  11. 11 Muse Ember muse-ember-1.0; reasoning effort high; 65,536; runtime/v1.11-ember; 1be19945 / f8506595; 5d582f7f; Responses via a gateway, effort high, 90 s idle timeout; 2026-11-09 $-5,383
  12. 12 Muse Spark 1.3 meta/muse-spark-1.3; reasoning effort high; 65,536; runtime/v1.11-spark; c2304c55 / f791742a; 5d582f7f; Responses via a gateway, effort high, summary auto, 90 s idle timeout; 2026-11-08 to 2026-11-09 $-5,669
  13. 13 GLM-5.3 zai-org/GLM-5.3; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $-6,454
  14. 14 Claude Haiku 5.5 claude-haiku-5-5; effort high, adaptive thinking; 65,536; runtime/v1.11-haiku; 83e2e25d / 2a2fde45; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-09 to 2026-11-10 $-7,440
  15. 15 Muse Wonder meta/muse-wonder-1.0; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 $-8,291
  16. 16 GPT-6 Luna gpt-6-luna; reasoning effort high; 65,536; runtime/v1.11-luna; six reruns runtime/v1.11-luna(-null) rr1; ad7b46dc / c4127aed; reruns 65cc113 / 1d5bd29; 5d582f7f; reruns e2bb04c1; Responses, effort high, summary auto, 90 s idle timeout; 2026-11-09 to 2026-11-11 $-8,667

Source: Official SMB Bench leaderboard

#ModelScoreSettingAgent byReleasedRun onLinkSource
1 GPT-6 Astra $7,749 gpt-6-astra; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
2 Gemini 4 Carbon $7,703 gemini-4-carbon; thinking level high; 65,536; runtime/v1.11-carbon; 3437a5c1 / 575d06be; 5d582f7f; GeminiAdapter, 300 s idle timeout; 2026-11-10 to 2026-11-12, ORO buyer loop ORO Oct 2026 Nov 2026 oroagents.com Official leaderboard
3 Claude Fable 5.2 $7,275 claude-fable-5-2; effort high, adaptive thinking; 65,536; runtime/v1.11-fable52; 95bb6628 / bda9748d; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-10, ORO buyer loop ORO Nov 2026 Nov 2026 oroagents.com Official leaderboard
4 Claude Fable 5.1 $5,427 claude-fable-5-1; effort high, adaptive thinking; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
5 GPT-6.1 Sol $5,404 gpt-6.1-sol; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
6 Gemini 4 Hydrogen $2,560 gemini-4-hydrogen; thinking level high; 65,536; runtime/v1.11-hydrogen; d490ef8c / b3b3b2ed; 40031d67; GeminiAdapter, 300 s idle timeout; 2026-11-12 to 2026-11-13, ORO buyer loop ORO Nov 2026 Nov 2026 oroagents.com Official leaderboard
7 Gemini 4 Argon $1,501 gemini-4-argon; thinking level high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
8 Claude Opus 5.5 $907 claude-opus-5-5; effort high, adaptive thinking; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
9 DeepSeek V4.1 Flash $-3,036 deepseek-ai/DeepSeek-V4.1-Flash-1017-Fast; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
10 Grok 4.7 $-4,722 grok-4.7; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
11 Muse Ember $-5,383 muse-ember-1.0; reasoning effort high; 65,536; runtime/v1.11-ember; 1be19945 / f8506595; 5d582f7f; Responses via a gateway, effort high, 90 s idle timeout; 2026-11-09, ORO buyer loop ORO Nov 2026 Nov 2026 oroagents.com Official leaderboard
12 Muse Spark 1.3 $-5,669 meta/muse-spark-1.3; reasoning effort high; 65,536; runtime/v1.11-spark; c2304c55 / f791742a; 5d582f7f; Responses via a gateway, effort high, summary auto, 90 s idle timeout; 2026-11-08 to 2026-11-09, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard
13 GLM-5.3 $-6,454 zai-org/GLM-5.3; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Aug 2026 Nov 2026 oroagents.com Official leaderboard
14 Claude Haiku 5.5 $-7,440 claude-haiku-5-5; effort high, adaptive thinking; 65,536; runtime/v1.11-haiku; 83e2e25d / 2a2fde45; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-09 to 2026-11-10, ORO buyer loop ORO Oct 2026 Nov 2026 oroagents.com Official leaderboard
15 Muse Wonder $-8,291 meta/muse-wonder-1.0; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05, ORO buyer loop ORO Oct 2026 Nov 2026 oroagents.com Official leaderboard
16 GPT-6 Luna $-8,667 gpt-6-luna; reasoning effort high; 65,536; runtime/v1.11-luna; six reruns runtime/v1.11-luna(-null) rr1; ad7b46dc / c4127aed; reruns 65cc113 / 1d5bd29; 5d582f7f; reruns e2bb04c1; Responses, effort high, summary auto, 90 s idle timeout; 2026-11-09 to 2026-11-11, ORO buyer loop ORO Sep 2026 Nov 2026 oroagents.com Official leaderboard

Rankings by part

We read 21 parts from the benchmark’s own leaderboard in 2 groups: category and task type. Each ranks models on one part of the test. Claude Fable 5.2 leads 9 of them, and 3 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

Customer

16 models we track have a score here.

  1. 1 Gemini 4 Carbon ↑1 $9,706
  2. 2 GPT-6 Astra ↓1 $8,809
  3. 3 Claude Fable 5.2 $6,526
  4. 4 Claude Fable 5.1 $5,121
  5. 5 GPT-6.1 Sol $4,596
  6. 6 Gemini 4 Argon ↑1 $2,509
  7. 7 Gemini 4 Hydrogen ↓1 $2,374
  8. 8 Claude Opus 5.5 $1,185
  9. 9 DeepSeek V4.1 Flash $-6,051
  10. 10 Muse Ember ↑1 $-7,216

The best score, $7,749, costs $213 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Anthropic Google DeepMind Meta Cost frontier One model at each effort level
$0 $2,000 $4,000 $6,000 $8,000 $0.10 $1.00 $10 $100 $1,000 Cost per task, log scale GPT-6 Astra, gpt-6-astra; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 Claude Fable 5.2, claude-fable-5-2; effort high, adaptive thinking; 65,536; runtime/v1.11-fable52; 95bb6628 / bda9748d; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-10 GPT-6.1 Sol, gpt-6.1-sol; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 Gemini 4 Hydrogen, gemini-4-hydrogen; thinking level high; 65,536; runtime/v1.11-hydrogen; d490ef8c / b3b3b2ed; 40031d67; GeminiAdapter, 300 s idle timeout; 2026-11-12 to 2026-11-13 Claude Opus 5.5, claude-opus-5-5; effort high, adaptive thinking; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 Muse Ember, muse-ember-1.0; reasoning effort high; 65,536; runtime/v1.11-ember; 1be19945 / f8506595; 5d582f7f; Responses via a gateway, effort high, 90 s idle timeout; 2026-11-09 $0 $2,000 $4,000 $6,000 $8,000 $0.10 $1.00 $10 $100 $1,000 Cost per task, log scale GPT-6 Astra, gpt-6-astra; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 Claude Fable 5.2, claude-fable-5-2; effort high, adaptive thinking; 65,536; runtime/v1.11-fable52; 95bb6628 / bda9748d; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-10

Source: Official SMB Bench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra gpt-6-astra; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $7,749 $213
Gemini 4 Carbon gemini-4-carbon; thinking level high; 65,536; runtime/v1.11-carbon; 3437a5c1 / 575d06be; 5d582f7f; GeminiAdapter, 300 s idle timeout; 2026-11-10 to 2026-11-12 ORO buyer loop $7,703 $134
Claude Fable 5.2 claude-fable-5-2; effort high, adaptive thinking; 65,536; runtime/v1.11-fable52; 95bb6628 / bda9748d; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-10 ORO buyer loop $7,275 $94
Claude Fable 5.1 claude-fable-5-1; effort high, adaptive thinking; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $5,427 $106
GPT-6.1 Sol gpt-6.1-sol; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $5,404 $32
Gemini 4 Hydrogen gemini-4-hydrogen; thinking level high; 65,536; runtime/v1.11-hydrogen; d490ef8c / b3b3b2ed; 40031d67; GeminiAdapter, 300 s idle timeout; 2026-11-12 to 2026-11-13 ORO buyer loop $2,560 $25
Gemini 4 Argon gemini-4-argon; thinking level high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $1,501 $70
Claude Opus 5.5 claude-opus-5-5; effort high, adaptive thinking; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $907 $23
DeepSeek V4.1 Flash deepseek-ai/DeepSeek-V4.1-Flash-1017-Fast; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $-3,036 $34
Grok 4.7 grok-4.7; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $-4,722 $63
Muse Ember muse-ember-1.0; reasoning effort high; 65,536; runtime/v1.11-ember; 1be19945 / f8506595; 5d582f7f; Responses via a gateway, effort high, 90 s idle timeout; 2026-11-09 ORO buyer loop $-5,383 $0.65
Muse Spark 1.3 meta/muse-spark-1.3; reasoning effort high; 65,536; runtime/v1.11-spark; c2304c55 / f791742a; 5d582f7f; Responses via a gateway, effort high, summary auto, 90 s idle timeout; 2026-11-08 to 2026-11-09 ORO buyer loop $-5,669 $31
GLM-5.3 zai-org/GLM-5.3; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $-6,454 $29
Claude Haiku 5.5 claude-haiku-5-5; effort high, adaptive thinking; 65,536; runtime/v1.11-haiku; 83e2e25d / 2a2fde45; 5d582f7f; Anthropic Messages, adaptive thinking summarized, 300 s idle timeout; 2026-11-09 to 2026-11-10 ORO buyer loop $-7,440 $1.26
Muse Wonder meta/muse-wonder-1.0; reasoning effort high; 65,536; campaign build, source revision b17b864b; per the release; 02a1d1d8; per model, see Models; 2026-11-03 to 2026-11-05 ORO buyer loop $-8,291 $42
GPT-6 Luna gpt-6-luna; reasoning effort high; 65,536; runtime/v1.11-luna; six reruns runtime/v1.11-luna(-null) rr1; ad7b46dc / c4127aed; reruns 65cc113 / 1d5bd29; 5d582f7f; reruns e2bb04c1; Responses, effort high, summary auto, 90 s idle timeout; 2026-11-09 to 2026-11-11 ORO buyer loop $-8,667 $2.09

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard 1 leader
GPT-6 Astra $7,749

Source: Official SMB Bench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
$-10,000 $-5,000 $0 $5,000 $10,000 2027 $-10,000 $-5,000 $0 $5,000 $10,000 2027

Source: Official SMB Bench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Economic work

All 210 in Economic work
  • JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
  • GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Seed2.1 Pro 87.9%
  • GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
  • AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
  • Agents' Last Exam v1.1 In use Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. GPT-6.2 Astra 47.5%
  • CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Claude Fable 5 75.9%

Data audit

All 10 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access The tasks are private.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (16 entries).
  • Pass Models scored 16 models we track.
  • Pass Newest score The newest is from November 2026, for Gemini 4 Hydrogen.
  • Pass Frontier models 7 of the top 10 have a score, led by Claude Fable 5.2 (#2).
  • Note Headroom Scored as US dollars, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on huggingface.co: 16 rows for models we track at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check