Benchmark Almanac

Snorkel AI

We track 29 benchmarks that Snorkel AI made or helped make.

0 in use 10 with public code 25 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
SnorkelSpatial Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning. Language reasoning Oct 2025 99% Not in cards
SnorkelUnderwrite Agents help simulated small business insurance underwriters across six task types in multi-turn chats, using tools and asking the underwriter for missing facts. Finance May 2025 91% Not in cards
SnorkelSequences Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math. Word problems Jun 2025 77.6% Not in cards
SnorkelGraph Models solve 200 procedurally generated graph problems, such as minimum vertex cover, given as node and edge lists in text. Answers are checked against verifiable ground truth. Word problems Sep 2025 84.5% Not in cards
Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. Fixing real repositories Jun 2026 39.3% Not in cards
FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. Finance Mar 2026 81% Not in cards
SnorkelFinance ReAct agents answer expert-verified financial questions built from 10-K filings, using SQL, code and other MCP tools without seeing the documents. A judge scores full traces. Finance 2025 81% Not in cards
Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. Finance Jun 2025 53.1% Not in cards
SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. Puzzles and chess Jul 2025 94% Not in cards
Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. Composite work indices Oct 2026 28.3 Not in cards
Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. Building whole projects Oct 2026 47.6% Not in cards
SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. Finance Oct 2026 25.1% Not in cards
SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. Law Oct 2026 46% Not in cards
SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. Other professions Oct 2026 27.8% Not in cards
SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. Running a business Oct 2026 18.4% Not in cards
SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. Finance Oct 2026 30.5% Not in cards
SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. Fixing real repositories Oct 2026 16.6% Not in cards
WorkplaceAgents Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics. Tasks from real jobs Oct 2026 21.2% Not in cards
Snorkel Agentic Coding Agents complete 100 multi-step coding tasks in sandboxed containers, spread evenly over four difficulty tiers. Unit tests and rubrics grade both outputs and the agent trajectory. Building whole projects Jan 2026 65.2% Not in cards
Cua-Bench EDA (KiCad v1) Agents build or edit KiCad schematics through the desktop interface. The grader checks the resulting netlist and reports full passes, partial passes, and mean reward. Electronics Jun 2026 24% Not in cards
JudgmentBench Models produce legal work for 30 tasks. Practicing lawyers grade individual answers and compare pairs of answers. Law May 2026 89% Not in cards
OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. Terminal tasks Feb 2026 75.1% Not in cards
Ping or Peek Coding agents receive an ambiguous request. A judge checks whether they ask the user for clarification or consult the grader. Deception and alignment 0.59% Not in cards
Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. Learning from the context Jun 2026 0.20 Not in cards
Snorkel Working Memory Test Models answer questions from a document placed among distractors. The evaluation varies the document position and the length of the input. Finding facts in long inputs Jul 2024 87.6% Not in cards
BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. Law Mar 2026 Not in cards
GDPWorks Agents complete 83 professional tasks across 36 occupations, using source files to produce work such as documents, spreadsheets, and code. The Joules-to-GDP framework compares outputs with expert deliverables through a blinded model judge. Task success measures the share of wins and ties against the expert. Tasks from real jobs Nov 2026 Not in cards
Tau2 Bench Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. Customer service Nov 2025 Not in cards
Tau2 Bench Verified Airline With Code Agents Agents handle airline customer requests by writing Python against a simulated database. The evaluator checks database updates offline against the expected state. Customer service Dec 2025 Not in cards