ContractorOS Quantity Takeoff
Models fill two bills of quantities from electrical and plumbing drawings. Counts must match the estimator exactly. Measured lengths must fall within 5%.
ContractorOS ranks 38 raw models on the same 130 lines. It also reports app runs, which allow blank answers. ContractorOS withholds the drawings.
- Made by
- ContractorOS
- Released
- September 2026
- Kind
- Tasks from real jobs
- Level
- University
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- Kept private
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- One fixed set
- Run by
- A host
- Occupations
- Business and financial operations
- Strongest model tested
- GPT-6.2 Astra #1 on the capability index
- Last model added
- November 2026
- Scores refreshed
- November 20th, 2026
Gemini 4 Carbon leads with 60%
Best score per model from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
- 60.0
Gemini 4 Carbon
- 59.2 Claude Fable 5.2
- 53.8 GPT-6.2 Astra
- 53.1 Claude Opus 5.5
- 52.3
Gemini 4 Argon
- 51.5
Gemini 4 Neon
- 49.2 Claude Sonnet 5.5
- 49.2 GPT-6.2 Sol
- 48.5 GPT-6 Astra
- 46.2 GPT-6.1 Sol
- 43.9 Claude Fable 5.1
- 43.1 Claude Opus 5
- 43.1
Gemini 4 Hydrogen
- 42.3 GPT-6.1 Terra
- 35.4 GPT-5.6 Sol
- 33.9 Claude Opus 4.7
- 32.3 GPT-6 Luna
- 31.5 GPT-5.6 Terra
- 31.5
Muse Wonder
- 31.1
Gemini 3.6 Flash
- 30.8 Claude Fable 5
- 30.8 Grok 4.6
- 30.8 GPT-6 Sol
- 30.0
Gemini 2.5 Flash-Lite
- 30.0 Claude Opus 4.8
- 29.2 Claude Haiku 4.5
- 28.5 GPT-5.4 Mini
- 28.5 Claude Sonnet 5
- 27.7
Gemini 3.1 Flash-Lite
- 27.7
Gemini 3.7 Flash
- 26.9
Gemini 2.5 Pro
- 26.9
Gemini 3.1 Pro
- 26.5 GPT-5.5
- 26.2
Mistral Large 3
- 25.4
Gemini 2.5 Flash
- 24.6 GPT-4.1
- 24.6 Claude Sonnet 4.5
- 24.6 GPT-5.6 Luna
- 22.3 Claude Haiku 5.5
- 16.2
Muse Spark 1.3
- 14.6 GPT-4o
- 2.30
Mistral Medium 3.1
- 0.0 GPT-5 nano
- 1
Gemini 4 Carbon electrical + plumbing · both sets in one session 60%
- 2 Claude Fable 5.2 electrical + plumbing · both sets in one session 59.2%
- 3 GPT-6.2 Astra electrical + plumbing · both sets in one session 53.8%
- 4 Claude Opus 5.5 electrical + plumbing · both sets in one session 53.1%
- 5
Gemini 4 Argon electrical + plumbing · Task mode 52.3%
- 6
Gemini 4 Neon electrical + plumbing · Task mode 51.5%
- 7 Claude Sonnet 5.5 electrical + plumbing · both sets in one session 49.2%
- 8 GPT-6.2 Sol electrical + plumbing · both sets in one session 49.2%
- 9 GPT-6 Astra Required answers 48.5%
- 10 GPT-6.1 Sol electrical + plumbing · both sets in one session · self-built tooling 46.2%
- 11 Claude Fable 5.1 electrical + plumbing · both sets in one session 43.9%
- 12 Claude Opus 5 electrical + plumbing · 1 session each 43.1%
- 13
Gemini 4 Hydrogen electrical + plumbing · both sets in one session 43.1%
- 14 GPT-6.1 Terra Required answers 42.3%
- 15 GPT-5.6 Sol electrical + plumbing · 1 session each 35.4%
- 16 Claude Opus 4.7 Required answers 33.9%
- 17 GPT-6 Luna Required answers 32.3%
- 18 GPT-5.6 Terra Required answers 31.5%
- 19
Muse Wonder Required answers 31.5%
- 20
Gemini 3.6 Flash Required answers 31.1%
- 21 Claude Fable 5 Required answers 30.8%
- 22 Grok 4.6 electrical + plumbing · 1 session each 30.8%
- 23 GPT-6 Sol electrical + plumbing · both sets in one session 30.8%
- 24
Gemini 2.5 Flash-Lite Sep 2025, May leave lines blank 30%
- 25 Claude Opus 4.8 Required answers 30%
- 26 Claude Haiku 4.5 Required answers 29.2%
- 27 GPT-5.4 Mini Required answers 28.5%
- 28 Claude Sonnet 5 Required answers 28.5%
- 29
Gemini 3.1 Flash-Lite Required answers 27.7%
- 30
Gemini 3.7 Flash Required answers 27.7%
- 31
Gemini 2.5 Pro Jun 2025, Required answers 26.9%
- 32
Gemini 3.1 Pro Required answers 26.9%
- 33 GPT-5.5 Required answers 26.5%
- 34
Mistral Large 3 Required answers 26.2%
- 35
Gemini 2.5 Flash Sep 2025, Required answers 25.4%
- 36 GPT-4.1 May leave lines blank 24.6%
- 37 Claude Sonnet 4.5 Required answers 24.6%
- 38 GPT-5.6 Luna Required answers 24.6%
- 39 Claude Haiku 5.5 Required answers 22.3%
- 40
Muse Spark 1.3 electrical + plumbing · both sets in one session 16.2%
- 41 GPT-4o Mar 2025, May leave lines blank 14.6%
- 42
Mistral Medium 3.1 Required answers 2.30%
- 43 GPT-5 nano Required answers 0%
Source: Official ContractorOS Quantity Takeoff leaderboard
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 60% | electrical + plumbing · both sets in one session, Gemini app | Oct 2026 | Official leaderboard |
| 2 | Claude Fable 5.2 | 59.2% | electrical + plumbing · both sets in one session, Claude desktop app | Nov 2026 | Official leaderboard |
| 3 | GPT-6.2 Astra | 53.8% | electrical + plumbing · both sets in one session, ChatGPT desktop app | Nov 2026 | Official leaderboard |
| 4 | Claude Opus 5.5 | 53.1% | electrical + plumbing · both sets in one session, Claude desktop app | Sep 2026 | Official leaderboard |
| 5 | | 52.3% | electrical + plumbing · Task mode, Gemini app | Sep 2026 | Official leaderboard |
| 6 | | 51.5% | electrical + plumbing · Task mode, Gemini app | Nov 2026 | Official leaderboard |
| 7 | Claude Sonnet 5.5 | 49.2% | electrical + plumbing · both sets in one session, Claude desktop app | Sep 2026 | Official leaderboard |
| 8 | GPT-6.2 Sol | 49.2% | electrical + plumbing · both sets in one session, ChatGPT desktop app | Nov 2026 | Official leaderboard |
| 9 | GPT-6 Astra | 48.5% | Required answers, Raw API | Sep 2026 | Official leaderboard |
| 10 | GPT-6.1 Sol | 46.2% | electrical + plumbing · both sets in one session · self-built tooling, ChatGPT desktop app | Sep 2026 | Official leaderboard |
| 11 | Claude Fable 5.1 | 43.9% | electrical + plumbing · both sets in one session, Claude desktop app | Sep 2026 | Official leaderboard |
| 12 | Claude Opus 5 | 43.1% | electrical + plumbing · 1 session each, Claude desktop app | Jul 2026 | Official leaderboard |
| 13 | | 43.1% | electrical + plumbing · both sets in one session, Gemini app | Nov 2026 | Official leaderboard |
| 14 | GPT-6.1 Terra | 42.3% | Required answers, Raw API | Oct 2026 | Official leaderboard |
| 15 | GPT-5.6 Sol | 35.4% | electrical + plumbing · 1 session each, ChatGPT desktop app | Jul 2026 | Official leaderboard |
| 16 | Claude Opus 4.7 | 33.9% | Required answers, Raw API | Apr 2026 | Official leaderboard |
| 17 | GPT-6 Luna | 32.3% | Required answers, Raw API | Sep 2026 | Official leaderboard |
| 18 | GPT-5.6 Terra | 31.5% | Required answers, Raw API | Jul 2026 | Official leaderboard |
| 19 | | 31.5% | Required answers, Raw API | Oct 2026 | Official leaderboard |
| 20 | | 31.1% | Required answers, Raw API | Jul 2026 | Official leaderboard |
| 21 | Claude Fable 5 | 30.8% | Required answers, Raw API | Jun 2026 | Official leaderboard |
| 22 | Grok 4.6 | 30.8% | electrical + plumbing · 1 session each, Cursor | Aug 2026 | Official leaderboard |
| 23 | GPT-6 Sol | 30.8% | electrical + plumbing · both sets in one session, ChatGPT desktop app | Sep 2026 | Official leaderboard |
| 24 | | 30% | May leave lines blank, Raw API | Sep 2025 | Official leaderboard |
| 25 | Claude Opus 4.8 | 30% | Required answers, Raw API | May 2026 | Official leaderboard |
| 26 | Claude Haiku 4.5 | 29.2% | Required answers, Raw API | Oct 2025 | Official leaderboard |
| 27 | GPT-5.4 Mini | 28.5% | Required answers, Raw API | Mar 2026 | Official leaderboard |
| 28 | Claude Sonnet 5 | 28.5% | Required answers, Raw API | Jun 2026 | Official leaderboard |
| 29 | | 27.7% | Required answers, Raw API | Mar 2026 | Official leaderboard |
| 30 | | 27.7% | Required answers, Raw API | Aug 2026 | Official leaderboard |
| 31 | | 26.9% | Required answers, Raw API | Jun 2025 | Official leaderboard |
| 32 | | 26.9% | Required answers, Raw API | Feb 2026 | Official leaderboard |
| 33 | GPT-5.5 | 26.5% | Required answers, Raw API | Apr 2026 | Official leaderboard |
| 34 | | 26.2% | Required answers, Raw API | Dec 2025 | Official leaderboard |
| 35 | | 25.4% | Required answers, Raw API | Sep 2025 | Official leaderboard |
| 36 | GPT-4.1 | 24.6% | May leave lines blank, Raw API | Apr 2025 | Official leaderboard |
| 37 | Claude Sonnet 4.5 | 24.6% | Required answers, Raw API | Sep 2025 | Official leaderboard |
| 38 | GPT-5.6 Luna | 24.6% | Required answers, Raw API | Jul 2026 | Official leaderboard |
| 39 | Claude Haiku 5.5 | 22.3% | Required answers, Raw API | Oct 2026 | Official leaderboard |
| 40 | | 16.2% | electrical + plumbing · both sets in one session, Muse | Sep 2026 | Official leaderboard |
| 41 | GPT-4o Mar 2025 | 14.6% | May leave lines blank, Raw API | Mar 2025 | Official leaderboard |
| 42 | | 2.30% | Required answers, Raw API | Aug 2025 | Official leaderboard |
| 43 | GPT-5 nano | 0% | Required answers, Raw API | Aug 2025 | Official leaderboard |
Rankings by part
We read 4 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Counts, Measured and Performing arts centre electrical. Gemini 4 Carbon leads 2 of them, and 2 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.
Counts
43 models we track have a score here.
- 1
Gemini 4 Carbon 77%
- 2
Gemini 4 Argon ↑3 73%
- 3 GPT-6.1 Terra ↑11 72%
- 4 Claude Fable 5.2 ↓2 71%
- 5
Gemini 4 Neon ↑1 70%
- 6 Claude Opus 5.5 ↓2 69%
- 7 Claude Sonnet 5.5 69%
- 8 GPT-6.2 Astra ↓5 69%
- 9 GPT-6.2 Sol ↓1 68%
- 10 Claude Opus 5 ↑2 67%
Measured
43 models we track have a score here.
- 1 Claude Fable 5.2 ↑1 41%
- 2
Gemini 4 Carbon ↓1 38%
- 3 GPT-6.2 Astra 38%
- 4 GPT-6 Sol ↑19 33%
- 5 GPT-6.2 Sol ↑3 32%
- 6 Claude Opus 5.5 ↓2 30%
- 7 GPT-6.1 Sol ↑3 30%
- 8 GPT-6 Astra ↑1 28%
- 9
Gemini 4 Neon ↓3 24%
- 10
Gemini 4 Argon ↓5 22%
Performing arts centre electrical
43 models we track have a score here.
- 1 Claude Sonnet 5.5 ↑6 75%
- 2
Gemini 4 Carbon ↓1 75%
- 3
Gemini 4 Argon ↑2 72.9%
- 4 Claude Fable 5.2 ↓2 72.9%
- 5 Claude Opus 5 ↑7 70.8%
- 6 Claude Opus 5.5 ↓2 70.8%
- 7
Gemini 4 Neon ↓1 70.8%
- 8 Claude Fable 5.1 ↑3 68.8%
- 9 GPT-6 Astra 68.8%
- 10 GPT-6.2 Astra ↓7 68.8%
Supermarket plumbing
43 models we track have a score here.
- 1
Gemini 4 Carbon 51.2%
- 2 Claude Fable 5.2 51.2%
- 3 GPT-6.2 Astra 47.6%
- 4 Claude Opus 5.5 42.7%
- 5 GPT-6.2 Sol ↑3 42.7%
- 6
Gemini 4 Argon ↓1 40.2%
- 7
Gemini 4 Neon ↓1 40.2%
- 8 GPT-6 Astra ↑1 37.8%
- 9 GPT-6.1 Sol ↑1 35.4%
- 10 Claude Sonnet 5.5 ↓3 34.2%
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Source: Official ContractorOS Quantity Takeoff leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| electrical + plumbing · both sets in one session | Gemini app | 60% | |||
| Claude Fable 5.2 | electrical + plumbing · both sets in one session | Claude desktop app | 59.2% | ||
| GPT-6.2 Astra | electrical + plumbing · both sets in one session | ChatGPT desktop app | 53.8% | ||
| Claude Opus 5.5 | electrical + plumbing · both sets in one session | Claude desktop app | 53.1% | ||
| electrical + plumbing · Task mode | Gemini app | 52.3% | |||
| GPT-6.2 Astra | Required answers | Raw API | 51.5% | ||
| electrical + plumbing · Task mode | Gemini app | 51.5% | |||
| Claude Sonnet 5.5 | electrical + plumbing · both sets in one session | Claude desktop app | 49.2% | ||
| GPT-6.2 Sol | electrical + plumbing · both sets in one session | ChatGPT desktop app | 49.2% | ||
| GPT-6 Astra | Required answers | Raw API | 48.5% | ||
| GPT-6 Astra | electrical + plumbing · both sets in one session · self-built tooling | ChatGPT desktop app | 46.2% | ||
| GPT-6.1 Sol | electrical + plumbing · both sets in one session · self-built tooling | ChatGPT desktop app | 46.2% | ||
| GPT-6.2 Sol | Required answers | Raw API | 44.6% | ||
| GPT-6.2 Sol | May leave lines blank | Raw API | 44.6% | ||
| Claude Fable 5.1 | electrical + plumbing · both sets in one session | Claude desktop app | 43.9% | ||
| GPT-6.2 Astra | May leave lines blank | Raw API | 43.9% | ||
| Claude Opus 5 | electrical + plumbing · 1 session each | Claude desktop app | 43.1% | ||
| electrical + plumbing · both sets in one session | Gemini app | 43.1% | |||
| GPT-6.1 Terra | Required answers | Raw API | 42.3% | ||
| Claude Opus 5 | electrical + plumbing · 1 session each | Cursor | 40.8% | ||
| GPT-6.1 Sol | Required answers | Raw API | 40% | ||
| GPT-6 Astra | May leave lines blank | Raw API | 38.5% | ||
| Claude Opus 5.5 | Required answers | Raw API | 36.9% | ||
| GPT-6.1 Sol | May leave lines blank | Raw API | 36.9% | ||
| GPT-5.6 Sol | electrical + plumbing · 1 session each | ChatGPT desktop app | 35.4% | ||
| Claude Fable 5.2 | May leave lines blank | Raw API | 35.4% | ||
| GPT-5.6 Sol | May leave lines blank | Raw API | 35% | ||
| GPT-6.1 Terra | May leave lines blank | Raw API | 34.6% | ||
| Claude Opus 4.7 | Required answers | Raw API | 33.9% | ||
| Required answers | Raw API | 33.9% | |||
| Claude Fable 5.2 | Required answers | Raw API | 33.9% | ||
| GPT-5.6 Sol | Required answers | Raw API | 32.7% | ||
| Claude Opus 5 | Required answers | Raw API | 32.3% | ||
| GPT-6 Luna | Required answers | Raw API | 32.3% | ||
| GPT-5.6 Terra | Required answers | Raw API | 31.5% | ||
| Required answers | Raw API | 31.5% | |||
| Required answers | Raw API | 31.1% | |||
| Claude Fable 5 | Required answers | Raw API | 30.8% | ||
| Grok 4.6 | electrical + plumbing · 1 session each | Cursor | 30.8% | ||
| GPT-6 Sol | electrical + plumbing · both sets in one session | ChatGPT desktop app | 30.8% | ||
| Claude Opus 5 | May leave lines blank | Raw API | 30.4% | ||
| May leave lines blank | Raw API | 30% | |||
| Claude Opus 4.8 | Required answers | Raw API | 30% | ||
| Claude Haiku 4.5 | Required answers | Raw API | 29.2% | ||
| GPT-6 Luna | May leave lines blank | Raw API | 29.2% | ||
| GPT-5.4 Mini | Required answers | Raw API | 28.5% | ||
| Claude Sonnet 5 | Required answers | Raw API | 28.5% | ||
| Claude Opus 5.5 | May leave lines blank | Raw API | 28.5% | ||
| GPT-6 Sol | Required answers | Raw API | 28.5% | ||
| Required answers | Raw API | 27.7% | |||
| May leave lines blank | Raw API | 27.7% | |||
| Required answers | Raw API | 27.7% | |||
| May leave lines blank | Raw API | 27.7% | |||
| GPT-6 Sol | May leave lines blank | Raw API | 27.7% | ||
| Required answers | Raw API | 26.9% | |||
| Required answers | Raw API | 26.9% | |||
| Claude Sonnet 5.5 | Required answers | Raw API | 26.9% | ||
| Required answers | Raw API | 26.9% | |||
| May leave lines blank | Raw API | 26.9% | |||
| GPT-5.5 | Required answers | Raw API | 26.5% | ||
| May leave lines blank | Raw API | 26.2% | |||
| Required answers | Raw API | 26.2% | |||
| GPT-5.5 | May leave lines blank | Raw API | 26.2% | ||
| Required answers | Raw API | 25.4% | |||
| GPT-4.1 | May leave lines blank | Raw API | 24.6% | ||
| Claude Sonnet 4.5 | Required answers | Raw API | 24.6% | ||
| GPT-5.6 Luna | Required answers | Raw API | 24.6% | ||
| GPT-5.6 Luna | May leave lines blank | Raw API | 24.6% | ||
| May leave lines blank | Raw API | 24.6% | |||
| May leave lines blank | Raw API | 23.8% | |||
| May leave lines blank | Raw API | 23.1% | |||
| GPT-4.1 | Required answers | Raw API | 22.3% | ||
| Claude Haiku 5.5 | Required answers | Raw API | 22.3% | ||
| GPT-5.6 Terra | May leave lines blank | Raw API | 21.5% | ||
| Claude Opus 4.8 | May leave lines blank | Raw API | 20.8% | ||
| Claude Sonnet 4.5 | May leave lines blank | Raw API | 20% | ||
| Claude Opus 4.7 | May leave lines blank | Raw API | 20% | ||
| May leave lines blank | Raw API | 19.2% | |||
| Claude Fable 5 | May leave lines blank | Raw API | 18.5% | ||
| Required answers | Raw API | 16.9% | |||
| electrical + plumbing · both sets in one session | Muse | 16.2% | |||
| GPT-5.4 Mini | May leave lines blank | Raw API | 15.4% | ||
| GPT-4o Mar 2025 | May leave lines blank | Raw API | 14.6% | ||
| Claude Haiku 5.5 | electrical + plumbing · both sets in one session | Claude desktop app | 14.6% | ||
| Claude Haiku 4.5 | May leave lines blank | Raw API | 13.9% | ||
| GPT-5.6 Terra | electrical + plumbing · 1 session each | ChatGPT desktop app | 13.1% | ||
| Claude Sonnet 5 | May leave lines blank | Raw API | 11.5% | ||
| May leave lines blank | Raw API | 10.8% | |||
| GPT-6.1 Terra | electrical + plumbing · both sets in one session | ChatGPT desktop app | 10% | ||
| Claude Haiku 5.5 | May leave lines blank | Raw API | 6.90% | ||
| Claude Sonnet 5.5 | May leave lines blank | Raw API | 3.90% | ||
| GPT-4o Mar 2025 | Required answers | Raw API | 3.10% | ||
| Required answers | Raw API | 2.30% | |||
| GPT-5 nano | Required answers | Raw API | 0% | ||
| GPT-5 nano | May leave lines blank | Raw API | 0% | ||
| May leave lines blank | Raw API | 0% | |||
| GPT-6 Luna | electrical + plumbing · both sets in one session | ChatGPT desktop app | 0% |
4 models have held number one on the official leaderboard since its release in September 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official ContractorOS Quantity Takeoff leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official ContractorOS Quantity Takeoff leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Economic work
- JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
- GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate.
Seed2.1 Pro 87.9%
- GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
- AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
- Agents' Last Exam v1.1 In use Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. GPT-6.2 Astra 47.5%
- CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Claude Fable 5 75.9%
Data audit
All 11 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access The tasks are private.
- Pass Confidence Checked against a primary source.
Scores 5 pass
- Pass Independent runs Scores from the official leaderboard (43 entries).
- Pass Models scored 43 models we track.
- Pass Newest score The newest is from November 2026, for Gemini 4 Neon.
- Pass Frontier models 10 of the top 10 have a score, led by GPT-6.2 Astra (#1).
- Pass Headroom The best is 60%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on contractoros.build: 97 rows for models we track at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.