Snorkel AI
We track 29 benchmarks that Snorkel AI made or helped make.
0 in use 10 with public code 25 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| SnorkelSpatial Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning. | Language reasoning | Oct 2025 | 99% | Not in cards |
| SnorkelUnderwrite Agents help simulated small business insurance underwriters across six task types in multi-turn chats, using tools and asking the underwriter for missing facts. | Finance | May 2025 | 91% | Not in cards |
| SnorkelSequences Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math. | Word problems | Jun 2025 | 77.6% | Not in cards |
| SnorkelGraph Models solve 200 procedurally generated graph problems, such as minimum vertex cover, given as node and edge lists in text. Answers are checked against verifiable ground truth. | Word problems | Sep 2025 | 84.5% | Not in cards |
| Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. | Fixing real repositories | Jun 2026 | 39.3% | Not in cards |
| FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. | Finance | Mar 2026 | 81% | Not in cards |
| SnorkelFinance ReAct agents answer expert-verified financial questions built from 10-K filings, using SQL, code and other MCP tools without seeing the documents. A judge scores full traces. | Finance | 2025 | 81% | Not in cards |
| Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. | Finance | Jun 2025 | 53.1% | Not in cards |
| SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. | Puzzles and chess | Jul 2025 | 94% | Not in cards |
| Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. | Composite work indices | Oct 2026 | | Not in cards |
| Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. | Building whole projects | Oct 2026 | 47.6% | Not in cards |
| SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. | Finance | Oct 2026 | 25.1% | Not in cards |
| SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. | Law | Oct 2026 | | Not in cards |
| SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. | Other professions | Oct 2026 | | Not in cards |
| SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. | Running a business | Oct 2026 | | Not in cards |
| SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. | Finance | Oct 2026 | 30.5% | Not in cards |
| SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. | Fixing real repositories | Oct 2026 | 16.6% | Not in cards |
| WorkplaceAgents Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics. | Tasks from real jobs | Oct 2026 | | Not in cards |
| Snorkel Agentic Coding Agents complete 100 multi-step coding tasks in sandboxed containers, spread evenly over four difficulty tiers. Unit tests and rubrics grade both outputs and the agent trajectory. | Building whole projects | Jan 2026 | 65.2% | Not in cards |
| Cua-Bench EDA (KiCad v1) Agents build or edit KiCad schematics through the desktop interface. The grader checks the resulting netlist and reports full passes, partial passes, and mean reward. | Electronics | Jun 2026 | 24% | Not in cards |
| JudgmentBench Models produce legal work for 30 tasks. Practicing lawyers grade individual answers and compare pairs of answers. | Law | May 2026 | 89% | Not in cards |
| OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. | Terminal tasks | Feb 2026 | 75.1% | Not in cards |
| Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. | Deception and alignment | 0.59% | Not in cards | |
| Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. | Learning from the context | Jun 2026 | 0.20 | Not in cards |
| Snorkel Working Memory Test Models answer questions from a document placed among distractors. The evaluation varies the document position and the length of the input. | Finding facts in long inputs | Jul 2024 | 87.6% | Not in cards |
| BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. | Law | Mar 2026 | Not in cards | |
| GDPWorks Agents complete 83 professional tasks across 36 occupations, using source files to produce work such as documents, spreadsheets, and code. The Joules-to-GDP framework compares outputs with expert deliverables through a blinded model judge. Task success measures the share of wins and ties against the expert. | Tasks from real jobs | Nov 2026 | Not in cards | |
| Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | Customer service | Nov 2025 | Not in cards | |
| Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. | Customer service | Dec 2025 | Not in cards |