Benchmark Almanac

ContractorOS Quantity Takeoff

Gemini 4 Carbon 60% leads since October 2026
Not in model cards

Models fill two bills of quantities from electrical and plumbing drawings. Counts must match the estimator exactly. Measured lengths must fall within 5%.

ContractorOS ranks 38 raw models on the same 130 lines. It also reports app runs, which allow blank answers. ContractorOS withholds the drawings.

Made by
ContractorOS
Released
September 2026
Kind
Tasks from real jobs
Level
University
Scored
Checked automatically
Answers
Short answer
Tasks
Kept private
Measures
Capability
Language
English
In and out
Text, Documents, Tables → Tables
Interaction
Single turn
Items from
Real work and use
Refresh
One fixed set
Run by
A host
Occupations
Business and financial operations
Strongest model tested
GPT-6.2 Astra #1 on the capability index
Last model added
November 2026
Scores refreshed
November 20th, 2026
43
Models scored on its own leaderboard

Gemini 4 Carbon leads with 60%

Best score per model from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

  1. 1 Gemini 4 Carbon electrical + plumbing · both sets in one session 60%
  2. 2 Claude Fable 5.2 electrical + plumbing · both sets in one session 59.2%
  3. 3 GPT-6.2 Astra electrical + plumbing · both sets in one session 53.8%
  4. 4 Claude Opus 5.5 electrical + plumbing · both sets in one session 53.1%
  5. 5 Gemini 4 Argon electrical + plumbing · Task mode 52.3%
  6. 6 Gemini 4 Neon electrical + plumbing · Task mode 51.5%
  7. 7 Claude Sonnet 5.5 electrical + plumbing · both sets in one session 49.2%
  8. 8 GPT-6.2 Sol electrical + plumbing · both sets in one session 49.2%
  9. 9 GPT-6 Astra Required answers 48.5%
  10. 10 GPT-6.1 Sol electrical + plumbing · both sets in one session · self-built tooling 46.2%
  11. 11 Claude Fable 5.1 electrical + plumbing · both sets in one session 43.9%
  12. 12 Claude Opus 5 electrical + plumbing · 1 session each 43.1%
  13. 13 Gemini 4 Hydrogen electrical + plumbing · both sets in one session 43.1%
  14. 14 GPT-6.1 Terra Required answers 42.3%
  15. 15 GPT-5.6 Sol electrical + plumbing · 1 session each 35.4%
  16. 16 Claude Opus 4.7 Required answers 33.9%
  17. 17 GPT-6 Luna Required answers 32.3%
  18. 18 GPT-5.6 Terra Required answers 31.5%
  19. 19 Muse Wonder Required answers 31.5%
  20. 20 Gemini 3.6 Flash Required answers 31.1%
  21. 21 Claude Fable 5 Required answers 30.8%
  22. 22 Grok 4.6 electrical + plumbing · 1 session each 30.8%
  23. 23 GPT-6 Sol electrical + plumbing · both sets in one session 30.8%
  24. 24 Gemini 2.5 Flash-Lite Sep 2025, May leave lines blank 30%
  25. 25 Claude Opus 4.8 Required answers 30%
  26. 26 Claude Haiku 4.5 Required answers 29.2%
  27. 27 GPT-5.4 Mini Required answers 28.5%
  28. 28 Claude Sonnet 5 Required answers 28.5%
  29. 29 Gemini 3.1 Flash-Lite Required answers 27.7%
  30. 30 Gemini 3.7 Flash Required answers 27.7%
  31. 31 Gemini 2.5 Pro Jun 2025, Required answers 26.9%
  32. 32 Gemini 3.1 Pro Required answers 26.9%
  33. 33 GPT-5.5 Required answers 26.5%
  34. 34 Mistral Large 3 Required answers 26.2%
  35. 35 Gemini 2.5 Flash Sep 2025, Required answers 25.4%
  36. 36 GPT-4.1 May leave lines blank 24.6%
  37. 37 Claude Sonnet 4.5 Required answers 24.6%
  38. 38 GPT-5.6 Luna Required answers 24.6%
  39. 39 Claude Haiku 5.5 Required answers 22.3%
  40. 40 Muse Spark 1.3 electrical + plumbing · both sets in one session 16.2%
  41. 41 GPT-4o Mar 2025, May leave lines blank 14.6%
  42. 42 Mistral Medium 3.1 Required answers 2.30%
  43. 43 GPT-5 nano Required answers 0%

Source: Official ContractorOS Quantity Takeoff leaderboard

#ModelScoreSettingDateSource
1 Gemini 4 Carbon 60% electrical + plumbing · both sets in one session, Gemini app Oct 2026 Official leaderboard
2 Claude Fable 5.2 59.2% electrical + plumbing · both sets in one session, Claude desktop app Nov 2026 Official leaderboard
3 GPT-6.2 Astra 53.8% electrical + plumbing · both sets in one session, ChatGPT desktop app Nov 2026 Official leaderboard
4 Claude Opus 5.5 53.1% electrical + plumbing · both sets in one session, Claude desktop app Sep 2026 Official leaderboard
5 Gemini 4 Argon 52.3% electrical + plumbing · Task mode, Gemini app Sep 2026 Official leaderboard
6 Gemini 4 Neon 51.5% electrical + plumbing · Task mode, Gemini app Nov 2026 Official leaderboard
7 Claude Sonnet 5.5 49.2% electrical + plumbing · both sets in one session, Claude desktop app Sep 2026 Official leaderboard
8 GPT-6.2 Sol 49.2% electrical + plumbing · both sets in one session, ChatGPT desktop app Nov 2026 Official leaderboard
9 GPT-6 Astra 48.5% Required answers, Raw API Sep 2026 Official leaderboard
10 GPT-6.1 Sol 46.2% electrical + plumbing · both sets in one session · self-built tooling, ChatGPT desktop app Sep 2026 Official leaderboard
11 Claude Fable 5.1 43.9% electrical + plumbing · both sets in one session, Claude desktop app Sep 2026 Official leaderboard
12 Claude Opus 5 43.1% electrical + plumbing · 1 session each, Claude desktop app Jul 2026 Official leaderboard
13 Gemini 4 Hydrogen 43.1% electrical + plumbing · both sets in one session, Gemini app Nov 2026 Official leaderboard
14 GPT-6.1 Terra 42.3% Required answers, Raw API Oct 2026 Official leaderboard
15 GPT-5.6 Sol 35.4% electrical + plumbing · 1 session each, ChatGPT desktop app Jul 2026 Official leaderboard
16 Claude Opus 4.7 33.9% Required answers, Raw API Apr 2026 Official leaderboard
17 GPT-6 Luna 32.3% Required answers, Raw API Sep 2026 Official leaderboard
18 GPT-5.6 Terra 31.5% Required answers, Raw API Jul 2026 Official leaderboard
19 Muse Wonder 31.5% Required answers, Raw API Oct 2026 Official leaderboard
20 Gemini 3.6 Flash 31.1% Required answers, Raw API Jul 2026 Official leaderboard
21 Claude Fable 5 30.8% Required answers, Raw API Jun 2026 Official leaderboard
22 Grok 4.6 30.8% electrical + plumbing · 1 session each, Cursor Aug 2026 Official leaderboard
23 GPT-6 Sol 30.8% electrical + plumbing · both sets in one session, ChatGPT desktop app Sep 2026 Official leaderboard
24 Gemini 2.5 Flash-Lite Sep 2025 30% May leave lines blank, Raw API Sep 2025 Official leaderboard
25 Claude Opus 4.8 30% Required answers, Raw API May 2026 Official leaderboard
26 Claude Haiku 4.5 29.2% Required answers, Raw API Oct 2025 Official leaderboard
27 GPT-5.4 Mini 28.5% Required answers, Raw API Mar 2026 Official leaderboard
28 Claude Sonnet 5 28.5% Required answers, Raw API Jun 2026 Official leaderboard
29 Gemini 3.1 Flash-Lite 27.7% Required answers, Raw API Mar 2026 Official leaderboard
30 Gemini 3.7 Flash 27.7% Required answers, Raw API Aug 2026 Official leaderboard
31 Gemini 2.5 Pro Jun 2025 26.9% Required answers, Raw API Jun 2025 Official leaderboard
32 Gemini 3.1 Pro 26.9% Required answers, Raw API Feb 2026 Official leaderboard
33 GPT-5.5 26.5% Required answers, Raw API Apr 2026 Official leaderboard
34 Mistral Large 3 26.2% Required answers, Raw API Dec 2025 Official leaderboard
35 Gemini 2.5 Flash Sep 2025 25.4% Required answers, Raw API Sep 2025 Official leaderboard
36 GPT-4.1 24.6% May leave lines blank, Raw API Apr 2025 Official leaderboard
37 Claude Sonnet 4.5 24.6% Required answers, Raw API Sep 2025 Official leaderboard
38 GPT-5.6 Luna 24.6% Required answers, Raw API Jul 2026 Official leaderboard
39 Claude Haiku 5.5 22.3% Required answers, Raw API Oct 2026 Official leaderboard
40 Muse Spark 1.3 16.2% electrical + plumbing · both sets in one session, Muse Sep 2026 Official leaderboard
41 GPT-4o Mar 2025 14.6% May leave lines blank, Raw API Mar 2025 Official leaderboard
42 Mistral Medium 3.1 2.30% Required answers, Raw API Aug 2025 Official leaderboard
43 GPT-5 nano 0% Required answers, Raw API Aug 2025 Official leaderboard

Rankings by part

We read 4 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Counts, Measured and Performing arts centre electrical. Gemini 4 Carbon leads 2 of them, and 2 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

Counts

43 models we track have a score here.

  1. 1 Gemini 4 Carbon 77%
  2. 2 Gemini 4 Argon ↑3 73%
  3. 3 GPT-6.1 Terra ↑11 72%
  4. 4 Claude Fable 5.2 ↓2 71%
  5. 5 Gemini 4 Neon ↑1 70%
  6. 6 Claude Opus 5.5 ↓2 69%
  7. 7 Claude Sonnet 5.5 69%
  8. 8 GPT-6.2 Astra ↓5 69%
  9. 9 GPT-6.2 Sol ↓1 68%
  10. 10 Claude Opus 5 ↑2 67%

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
GPT-6.2 Astra Raw API
May leave l… Required an…
51.5%
GPT-6 Astra Raw API
May leave l… Required an…
48.5%
GPT-6.2 Sol Raw API
Required an…
44.6%
GPT-6.1 Terra Raw API
May leave l… Required an…
42.3%
GPT-6.1 Sol Raw API
May leave l…
40%
Claude Opus 5.5 Raw API
May leave l… Required an…
36.9%
Claude Fable 5.2 Raw API
Required an…
35.4%
GPT-5.6 Sol Raw API
Required an…
35%
Claude Opus 4.7 Raw API
May leave l… Required an…
33.9%
Claude Opus 5 Raw API
May leave l…
32.3%
GPT-6 Luna Raw API
May leave l…
32.3%
GPT-5.6 Terra Raw API
May leave l… Required an…
31.5%

Source: Official ContractorOS Quantity Takeoff leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Gemini 4 Carbon electrical + plumbing · both sets in one session Gemini app 60%
Claude Fable 5.2 electrical + plumbing · both sets in one session Claude desktop app 59.2%
GPT-6.2 Astra electrical + plumbing · both sets in one session ChatGPT desktop app 53.8%
Claude Opus 5.5 electrical + plumbing · both sets in one session Claude desktop app 53.1%
Gemini 4 Argon electrical + plumbing · Task mode Gemini app 52.3%
GPT-6.2 Astra Required answers Raw API 51.5%
Gemini 4 Neon electrical + plumbing · Task mode Gemini app 51.5%
Claude Sonnet 5.5 electrical + plumbing · both sets in one session Claude desktop app 49.2%
GPT-6.2 Sol electrical + plumbing · both sets in one session ChatGPT desktop app 49.2%
GPT-6 Astra Required answers Raw API 48.5%
GPT-6 Astra electrical + plumbing · both sets in one session · self-built tooling ChatGPT desktop app 46.2%
GPT-6.1 Sol electrical + plumbing · both sets in one session · self-built tooling ChatGPT desktop app 46.2%
GPT-6.2 Sol Required answers Raw API 44.6%
GPT-6.2 Sol May leave lines blank Raw API 44.6%
Claude Fable 5.1 electrical + plumbing · both sets in one session Claude desktop app 43.9%
GPT-6.2 Astra May leave lines blank Raw API 43.9%
Claude Opus 5 electrical + plumbing · 1 session each Claude desktop app 43.1%
Gemini 4 Hydrogen electrical + plumbing · both sets in one session Gemini app 43.1%
GPT-6.1 Terra Required answers Raw API 42.3%
Claude Opus 5 electrical + plumbing · 1 session each Cursor 40.8%
GPT-6.1 Sol Required answers Raw API 40%
GPT-6 Astra May leave lines blank Raw API 38.5%
Claude Opus 5.5 Required answers Raw API 36.9%
GPT-6.1 Sol May leave lines blank Raw API 36.9%
GPT-5.6 Sol electrical + plumbing · 1 session each ChatGPT desktop app 35.4%
Claude Fable 5.2 May leave lines blank Raw API 35.4%
GPT-5.6 Sol May leave lines blank Raw API 35%
GPT-6.1 Terra May leave lines blank Raw API 34.6%
Claude Opus 4.7 Required answers Raw API 33.9%
Gemini 4 Carbon Required answers Raw API 33.9%
Claude Fable 5.2 Required answers Raw API 33.9%
GPT-5.6 Sol Required answers Raw API 32.7%
Claude Opus 5 Required answers Raw API 32.3%
GPT-6 Luna Required answers Raw API 32.3%
GPT-5.6 Terra Required answers Raw API 31.5%
Muse Wonder Required answers Raw API 31.5%
Gemini 3.6 Flash Required answers Raw API 31.1%
Claude Fable 5 Required answers Raw API 30.8%
Grok 4.6 electrical + plumbing · 1 session each Cursor 30.8%
GPT-6 Sol electrical + plumbing · both sets in one session ChatGPT desktop app 30.8%
Claude Opus 5 May leave lines blank Raw API 30.4%
Gemini 2.5 Flash-Lite Sep 2025 May leave lines blank Raw API 30%
Claude Opus 4.8 Required answers Raw API 30%
Claude Haiku 4.5 Required answers Raw API 29.2%
GPT-6 Luna May leave lines blank Raw API 29.2%
GPT-5.4 Mini Required answers Raw API 28.5%
Claude Sonnet 5 Required answers Raw API 28.5%
Claude Opus 5.5 May leave lines blank Raw API 28.5%
GPT-6 Sol Required answers Raw API 28.5%
Gemini 3.1 Flash-Lite Required answers Raw API 27.7%
Gemini 3.6 Flash May leave lines blank Raw API 27.7%
Gemini 3.7 Flash Required answers Raw API 27.7%
Gemini 3.7 Flash May leave lines blank Raw API 27.7%
GPT-6 Sol May leave lines blank Raw API 27.7%
Gemini 2.5 Pro Jun 2025 Required answers Raw API 26.9%
Gemini 3.1 Pro Required answers Raw API 26.9%
Claude Sonnet 5.5 Required answers Raw API 26.9%
Gemini 4 Argon Required answers Raw API 26.9%
Gemini 4 Argon May leave lines blank Raw API 26.9%
GPT-5.5 Required answers Raw API 26.5%
Gemini 2.5 Pro Jun 2025 May leave lines blank Raw API 26.2%
Mistral Large 3 Required answers Raw API 26.2%
GPT-5.5 May leave lines blank Raw API 26.2%
Gemini 2.5 Flash Sep 2025 Required answers Raw API 25.4%
GPT-4.1 May leave lines blank Raw API 24.6%
Claude Sonnet 4.5 Required answers Raw API 24.6%
GPT-5.6 Luna Required answers Raw API 24.6%
GPT-5.6 Luna May leave lines blank Raw API 24.6%
Muse Wonder May leave lines blank Raw API 24.6%
Gemini 2.5 Flash Sep 2025 May leave lines blank Raw API 23.8%
Gemini 3.1 Flash-Lite May leave lines blank Raw API 23.1%
GPT-4.1 Required answers Raw API 22.3%
Claude Haiku 5.5 Required answers Raw API 22.3%
GPT-5.6 Terra May leave lines blank Raw API 21.5%
Claude Opus 4.8 May leave lines blank Raw API 20.8%
Claude Sonnet 4.5 May leave lines blank Raw API 20%
Claude Opus 4.7 May leave lines blank Raw API 20%
Mistral Large 3 May leave lines blank Raw API 19.2%
Claude Fable 5 May leave lines blank Raw API 18.5%
Gemini 2.5 Flash-Lite Sep 2025 Required answers Raw API 16.9%
Muse Spark 1.3 electrical + plumbing · both sets in one session Muse 16.2%
GPT-5.4 Mini May leave lines blank Raw API 15.4%
GPT-4o Mar 2025 May leave lines blank Raw API 14.6%
Claude Haiku 5.5 electrical + plumbing · both sets in one session Claude desktop app 14.6%
Claude Haiku 4.5 May leave lines blank Raw API 13.9%
GPT-5.6 Terra electrical + plumbing · 1 session each ChatGPT desktop app 13.1%
Claude Sonnet 5 May leave lines blank Raw API 11.5%
Gemini 3.1 Pro May leave lines blank Raw API 10.8%
GPT-6.1 Terra electrical + plumbing · both sets in one session ChatGPT desktop app 10%
Claude Haiku 5.5 May leave lines blank Raw API 6.90%
Claude Sonnet 5.5 May leave lines blank Raw API 3.90%
GPT-4o Mar 2025 Required answers Raw API 3.10%
Mistral Medium 3.1 Required answers Raw API 2.30%
GPT-5 nano Required answers Raw API 0%
GPT-5 nano May leave lines blank Raw API 0%
Mistral Medium 3.1 May leave lines blank Raw API 0%
GPT-6 Luna electrical + plumbing · both sets in one session ChatGPT desktop app 0%

4 models have held number one on the official leaderboard since its release in September 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI Google DeepMind
Own leaderboard 4 leaders

Source: Official ContractorOS Quantity Takeoff leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Gemini 4 Carbon Claude Opus 5.5 GPT-6 Astra Claude Fable 5.1 GPT-5.6 Sol Gemini 2.5 Flash-Lite GPT-4.1 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Gemini 4 Carbon Claude Opus 5.5

Source: Official ContractorOS Quantity Takeoff leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Economic work

All 210 in Economic work
  • JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
  • GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Seed2.1 Pro 87.9%
  • GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
  • AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
  • Agents' Last Exam v1.1 In use Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. GPT-6.2 Astra 47.5%
  • CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Claude Fable 5 75.9%

Data audit

All 11 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access The tasks are private.
  • Pass Confidence Checked against a primary source.
Scores 5 pass
  • Pass Independent runs Scores from the official leaderboard (43 entries).
  • Pass Models scored 43 models we track.
  • Pass Newest score The newest is from November 2026, for Gemini 4 Neon.
  • Pass Frontier models 10 of the top 10 have a score, led by GPT-6.2 Astra (#1).
  • Pass Headroom The best is 60%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on contractoros.build: 97 rows for models we track at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check