SnorkelUnderwrite 2.0
Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows.
Scores combine answer checks with safety, process and final state checks. At launch, no model passed over 41% of answer checks.
- Made by
- Snorkel AI
- Released
- October 2026
- Kind
- Finance
- Level
- Expert
- Scored
- Graded by a model
- Answers
- Agent task
- Tasks
- Kept private
- Strongest model tested
- GPT-6 Astra #3 on the capability index
- Last model added
- October 2026
- Scores refreshed
- November 12th, 2026
Grok 4.7 leads with 30.5%
Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
- 1 Grok 4.7 30.5%
- 2 GLM-5.3 30.4%
- 3
Nemotron 3 Ultra 30% - 4
DeepSeek-V4-Pro 28.5% - 5 Kimi K3 27.9%
- 6 GPT-6.1 Sol 25%
- 7 Claude Fable 5.1 24%
- 8
Gemini 4 Argon 23.3% - 9
Muse Wonder 23% - 10 Claude Opus 5.5 22.7%
- 11 GPT-6 Astra 21.2%
- 12
Qwen 3.8 Max 20% - 13 Muse Spark 1.3 19.6%
- 14 Gemini 3.8 Flash 19.1%
- 15 Claude Opus 5 18.3%
- 16 Grok 4.6 17.8%
Source: Official SnorkelUnderwrite 2.0 leaderboard
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Grok 4.7 | 30.5% | Sep 2026 | Official leaderboard | |
| 2 | GLM-5.3 | 30.4% | Aug 2026 | Official leaderboard | |
| 3 | Nemotron 3 Ultra | 30% | Jun 2026 | Official leaderboard | |
| 4 | DeepSeek-V4-Pro | 28.5% | Apr 2026 | Official leaderboard | |
| 5 | Kimi K3 | 27.9% | Jul 2026 | Official leaderboard | |
| 6 | GPT-6.1 Sol | 25% | Sep 2026 | Official leaderboard | |
| 7 | Claude Fable 5.1 | 24% | Sep 2026 | Official leaderboard | |
| 8 | Gemini 4 Argon | 23.3% | Sep 2026 | Official leaderboard | |
| 9 | Muse Wonder | 23% | Oct 2026 | Official leaderboard | |
| 10 | Claude Opus 5.5 | 22.7% | Sep 2026 | Official leaderboard | |
| 11 | GPT-6 Astra | 21.2% | Sep 2026 | Official leaderboard | |
| 12 | Qwen 3.8 Max | 20% | Aug 2026 | Official leaderboard | |
| 13 | Muse Spark 1.3 | 19.6% | Sep 2026 | Official leaderboard | |
| 14 | Gemini 3.8 Flash | 19.1% | Sep 2026 | Official leaderboard | |
| 15 | Claude Opus 5 | 18.3% | Jul 2026 | Official leaderboard | |
| 16 | Grok 4.6 | 17.8% | Aug 2026 | Official leaderboard |
Pass@1 vs. cost per trial
The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.
| Model | Setting | Pass@1 | Cost per trial |
|---|---|---|---|
| Grok 4.7 | 30.5% | $1.21 | |
| GLM-5.3 | 30.4% | $0.36 | |
| Nemotron 3 Ultra | 30% | $0.32 | |
| DeepSeek-V4-Pro | 28.5% | $0.61 | |
| Kimi K3 | 27.9% | $0.57 | |
| GPT-6.1 Sol | 25% | $1.02 | |
| Claude Fable 5.1 | 24% | $1.92 | |
| Gemini 4 Argon | 23.3% | $5.03 | |
| Muse Wonder | 23% | $8.48 | |
| Claude Opus 5.5 | 22.7% | $0.76 | |
| GPT-6 Astra | 21.2% | $1.71 | |
| Qwen 3.8 Max | 20% | $0.69 | |
| Muse Spark 1.3 | 19.6% | $0.33 | |
| Gemini 3.8 Flash | 19.1% | $0.58 | |
| Claude Opus 5 | 18.3% | $0.82 | |
| Grok 4.6 | 17.8% | $0.66 |
The best score, 30.5%, costs $1.21 a task
Runs on the cost frontier score higher than any run that's cheaper. Nemotron 3 Ultra comes within 0.5 points of the best for $0.32 a task, 4 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official SnorkelUnderwrite 2.0 leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Grok 4.7 | 30.5% | $1.21 | |||
| GLM-5.3 | 30.4% | $0.36 | |||
| Nemotron 3 Ultra | 30% | $0.32 | |||
| DeepSeek-V4-Pro | 28.5% | $0.61 | |||
| Kimi K3 | 27.9% | $0.57 | |||
| GPT-6.1 Sol | 25% | $1.02 | |||
| Claude Fable 5.1 | 24% | $1.92 | |||
| Gemini 4 Argon | 23.3% | $5.03 | |||
| Muse Wonder | 23% | $8.48 | |||
| Claude Opus 5.5 | 22.7% | $0.76 | |||
| GPT-6 Astra | 21.2% | $1.71 | |||
| Qwen 3.8 Max | 20% | $0.69 | |||
| Muse Spark 1.3 | 19.6% | $0.33 | |||
| Gemini 3.8 Flash | 19.1% | $0.58 | |||
| Claude Opus 5 | 18.3% | $0.82 | |||
| Grok 4.6 | 17.8% | $0.66 |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official SnorkelUnderwrite 2.0 leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official SnorkelUnderwrite 2.0 leaderboard
No lab model card reports it yet
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.
Versions
We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.
Source: Official leaderboards
More in Law, finance and medicine
- MedXpertQA-MM In use Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. Qwen 3.8 Max 80.4%
- HealthBench Professional In use 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. Claude Sonnet 5.5 69.2%
- HealthBench In use 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. GPT-5 67.2%
- FinSearchComp In use Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy.
Ling-3.1-flash 81.7% - KBL In use Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items.
Solar Pro 4 77.5% - KorMedMCQA In use Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. Solar Pro 4 93.2%