Benchmark Almanac

Law and finance

Tasks that need legal, economic or business knowledge. We count 592 benchmarks here, 4 of them arenas. A benchmark belongs here when subject includes law or economics and business.

How fast it grows

225 were released since July, more than the 105 in the first half of 2026.

  1. Earlier 69
  2. 2024 H1 27
  3. 2024 H2 37
  4. 2025 H1 39
  5. 2025 H2 60
  6. 2026 H1 105
  7. 2026 H2 225

How much is still open

No model has passed 85% on 221 of the 288 benchmarks here that score in percent, 77% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
Access to Justice Benchmark Models interview a simulated client about a civil legal problem, explain their options, and direct them to local help. Expert Nov 2026 76.4% Not in cards
AgenticFinBench Agents complete 230 financial requests in two languages. Graders apply task-specific rubrics and audit unsupported claims against external evidence. Expert Nov 2026 89.2 Not in cards
AlphaSense Financial Research Benchmark Agents answer 245 multi-step finance questions across company research, financial modeling, earnings, deals, markets and regulatory analysis using source retrieval. Expert Nov 2026 Not in cards
ChJGBench Models draft the reasoning and orders in Chinese civil, criminal, and administrative judgments. The evaluator compares their work with 3,462 real judgments. Expert Nov 2026 0.74 Not in cards
codex of veritas Models answer ledger questions when the supplied quantities determine the result. They must abstain when a necessary quantity is missing. School Nov 2026 100% Not in cards
Coverage Cat AI Insurance Benchmark (fictional insurance tasks) Models read fictional policy documents, find missing facts, and complete insurance applications. The scorer checks source quotes, required fields, and authorized actions. University Nov 2026 100% Not in cards
Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. University Nov 2026 -7.45% Not in cards
DeepFinance Agents do private equity and investment banking work on 21 anonymized real deals: valuation, due diligence, underwriting and financial modeling. Each task comes with a data room of about 19 files and would take a professional about 27 hours. Rubrics of about 84 criteria grade the output, the workflow and the judgment behind each assumption. Expert Nov 2026 62% Not in cards
EnterpriseOps-Gym 1.1 Agents complete 296 workflows across customer service, human resources, and IT service management. SQL checks grade the resulting database state. Expert Nov 2026 Not in cards
Financial Coaching Bench Models interview a simulated client about a household financial problem and propose a plan. Expert Nov 2026 76.2% Not in cards
FinDendro Models recall dated US financial facts and identify the limits of their knowledge. Four prompt styles test value recall, self-knowledge, and historical timing. Expert Nov 2026 0.76 Not in cards
FinDesk-Bench Agents find Treasury figures and use them to handle client requests across ten business services. A model grades API calls and the records agents leave behind. University Nov 2026 90.4% Not in cards
FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. University Nov 2026 74.5% Not in cards
GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. Expert Nov 2026 50.4% In use
GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. Expert Nov 2026 16.4% Not in cards
GPBam Models write legal opinions on German public law cases and retrieve the statutes they need. Three judges grade essays against reference solutions. Expert Nov 2026 76.4 Not in cards
Gulf Text-to-SQL Bench Models write SQL for the same 30 business questions in English, Modern Standard Arabic, and Gulf Arabic. The harness executes each query. University Nov 2026 96.7% Not in cards
HalachaBench Models answer 60 Hebrew rabbinical qualification exam questions. A model judge awards rubric credit on a scale from 0 to 100. Expert Nov 2026 85.8 Not in cards
HireWatch Models select questions for job interviews under different workplace pressures. The benchmark measures how often they select questions classified as unlawful under US employment law. University Nov 2026 0.05% Not in cards
How frontier models miss risk in routine enterprise work Agents handle workplace requests whose risks appear in the prompt or in supporting files. Judges check whether the agent completes an unsafe action. Expert Nov 2026 Not in cards
Invoice Approval (Ali Siala) Agents process 60 supplier invoice work orders under a German company’s approval policy. The evaluator compares the resulting database with a reference state. University Nov 2026 Not in cards
JFinBench Models answer questions from six Japanese certification exams in accounting, tax, financial planning, and banking. The authors report scores by exam. University Nov 2026 97.3% Not in cards
Legal Information Retrieval Benchmark Search APIs retrieve the official document behind legal research requests. Exact URL matching scores whether the official page appears first or in the top ten. Expert Nov 2026 Not in cards
Legal World sample Agents act as a legal assistant in an anonymized law practice. Four tasks grade the drafts, mail, calendar changes, and files they leave behind. Expert Nov 2026 Not in cards
LevelBench-100 v5.0 Models use ledger exports and supporting workbooks to close small business accounts. They reconcile transactions, apply accounting rules, and identify missing records. Expert Nov 2026 95.2% Not in cards
Moneda AdviceBench Models answer personal finance requests and paired education questions. Moneda’s default board scores paired pass rate on 50 EU personal requests with education twins. Expert Nov 2026 Not in cards
Moneda Bench Models answer finance questions across seven separate benchmarks. AdviceBench, CiteBench, FilingsBench, GroundBench and PortfolioBench have their own records. KnowledgeBench and MarketBench have no published results yet. Moneda publishes no combined score. Expert Nov 2026 Not in cards
Moneda CiteBench Models write company research notes with linked sources. Moneda scores the weighted share of linked claims that the cited pages support across 100 notes. Expert Nov 2026 Not in cards
Moneda FilingsBench Models answer questions about SEC filings without receiving documents. Moneda scores accuracy on the private questions using filing data and judges for segment questions. Expert Nov 2026 Not in cards
Moneda GroundBench Models answer live questions about market figures. Moneda’s default board reports the share of correct answers against reference data, with incorrect answers reported separately. Expert Nov 2026 Not in cards
Moneda PortfolioBench Models solve hypothetical portfolio tasks involving rebalancing, risk, position sizing, and constraints. Moneda’s default board scores accuracy on 150 tasks with fixed answers. Expert Nov 2026 Not in cards
NAFD E1 Financial Reporting Models sit professional finance exams on topics including financial reporting, planning, and taxation. A grader compares their answers with the official keys. University Nov 2026 0.93 Not in cards
OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. University Nov 2026 76.4% Not in cards
OpenLegalRetrieval-DE Models retrieve German statutes and court decisions across seven tasks. Each task scores where relevant documents appear in the first ten results. Each task carries equal weight. University Nov 2026 0.44 Not in cards
Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. Expert Nov 2026 Not in cards
PWS-Legal Web research systems answer dated questions about companies, laws, regulations, intellectual property, court decisions, and legal documents. Expert Nov 2026 Not in cards
Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. Expert Nov 2026 61.1% Not in cards
SG-LegalClarify Models answer Singapore legal questions and ask for missing facts. Three model judges grade each conversation against rubrics written by lawyers. Expert Nov 2026 77.8% Not in cards
SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. University Nov 2026 $7,749 Not in cards
StrataBench Land v1 Models classify land instruments, extract lease terms, and distinguish petroleum terms in three separate benchmarks. Classify and Extract have their own records. Define publishes three measures without one headline score. Expert Nov 2026 0.94% Not in cards
StrataBench Land v1 Classify Models identify the legal act performed by each of 66 synthetic land instruments. Collide scores label accuracy under open and closed prompt protocols. Expert Nov 2026 93.9% Not in cards
StrataBench Land v1 Extract Models extract ten shared lease fields from 628 synthetic instruments. Collide divides correct field decisions by correct, omitted, and wrong decisions, excluding unscored fields. Expert Nov 2026 88.1% Not in cards
Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. Expert Nov 2026 71.2% Not in cards
Veris FinAgent Arena v1.1 Agents investigate bank alerts, read Treasury scans, and complete research requests. The index combines accuracy and repeatability across three benchmarks, then applies a cost penalty. University Nov 2026 757.0 Not in cards
WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. Expert Nov 2026 37% Not in cards
AA-Omniscience v1.1 Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. Expert Oct 2026 16.1 In use
Agentic Credit Analysis Models extract and standardize income statements and balance sheets. Ramp compares their figures with credit analysts’ figures using a five percent tolerance. Expert Oct 2026 80.2% Not in cards
AGORA (Agentic Global Open Research Arena) Autonomous AI systems get one secret open finance question, 72 hours and a fixed budget to write a paper. Five finance scholars judge it. Research frontier Oct 2026 Not in cards
AI Professions Prompt Benchmark Models draft everyday work documents from fictional inputs: 100 prompts for tasks in 10 professions, such as accounting, law, healthcare administration and real estate, plus Excel and Microsoft 365, in English, Spanish, French and Portuguese. A judge model from another family checks 5 pass or fail criteria per prompt, fixed before any model ran. University Oct 2026 85.9% Not in cards
AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. Expert Oct 2026 70.8% Not in cards
AMLworld-Compact Models inspect transaction graphs and estimate whether a marked target is suspicious. The grader thresholds the probabilities and computes a weighted F1 score. University Oct 2026 24% Not in cards
Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. Expert Oct 2026 70.2 Not in cards
Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. Everyday Oct 2026 Not in cards
ATLAS-Finance v1.1.0 Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level. Expert Oct 2026 10.7% Not in cards
AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. University Oct 2026 309.4 Not in cards
BankerToolBench v1.1 Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes. Expert Oct 2026 32.6% Not in cards
BizBench Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success. Expert Oct 2026 70% Not in cards
Business Evidence Judgment v4 Models label business claims as supported, contradicted, or unknown under three document conditions. The grader scores accuracy and compliance separately. University Oct 2026 90.7% Not in cards
Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. Expert Oct 2026 37.8% In use
Can AI close the books? FP&A reasoning Models complete synthetic financial close tasks, including accruals and reconciliations. The grader checks calculations and whether reviewer claims agree with the books. University Oct 2026 100% Not in cards
Cents Matter Models solve Brazilian Portuguese accounting cases in matched pairs. The grader checks amounts to the cent across rounding, cash flow signs, and cases with insufficient evidence. University Oct 2026 97.5% Not in cards
ClaimBench Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency. University Oct 2026 Not in cards
Company Enrichment API Benchmark Company data APIs fill profile fields from a website. Code and expert checks score headquarters, employees, founding year, social links, industry, ticker and revenue. Expert Oct 2026 Not in cards
Credit Firewall Benchmark Models decide whether evidence still authorizes a credit decision after roles, targets, timing, or conversation context change. University Oct 2026 100% Not in cards
CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. University Oct 2026 83.6% Not in cards
DataAnalyticsBench Agents query a synthetic data warehouse to answer business questions. A judge model checks answers against a rubric and an answer key. University Oct 2026 43.0 Not in cards
DAYJOB: Accounting Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Oct 2026 49.6% Not in cards
DAYJOB: Law Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Oct 2026 41.4% Not in cards
DAYJOB: Tax Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria. Expert Oct 2026 10.8% Not in cards
Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. Expert Oct 2026 28.3 Not in cards
DOE sample pack Agents complete five simulated workplace assignments in law, private equity, and finance operations. Deterministic checks grade figures and application state, while model judges grade written work. Expert Oct 2026 Not in cards
DualEntry Accounting AI Benchmark Models use accounting tools to look up accounts, reconcile balances, and draft entries for a test company. DualEntry checks their answers against fixed references. University Oct 2026 91.1% Not in cards
EGI-BENCH Agents work across eight enterprise evaluations. The suite combines their results into six scores covering activities such as planning, retrieving evidence, and using tools. University Oct 2026 65.6 Not in cards
Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. Expert Oct 2026 81.7% System cards only
Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. Expert Oct 2026 77.4% In use
FairScreen: Testing LLM Hiring Decisions Models screen job applicants under a stated policy. Paired cases change identity cues, qualifications, or the completeness of the evidence. University Oct 2026 98.4% Not in cards
FinGREAT Models identify accounting relationships in financial passages. The authors compare 44 models against human annotations across 483 passages. University Oct 2026 92.8% Not in cards
FinHopper Agents research financial facts through up to six dependent searches. Intermediate answer keys show where an incorrect company, date, or indicator derails the search. University Oct 2026 67.3% Not in cards
Harvey LAB v1.1 (HLAB-Vals-1.1) Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes. Expert Oct 2026 27.1% In use
Harvey LAB-AA v1.1 Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. Expert Oct 2026 15% In use
Hebbia Financial Services Benchmark Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion. Expert Oct 2026 74.6% Not in cards
Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. Expert Oct 2026 Not in cards
Holdings-IndividualTaxBench 88 real client document packages turned into Form 1040 fields. Fact-level accuracy against practitioner-reviewed returns, with an LLM judge matching values. Expert Oct 2026 76.3% Not in cards
Hype Trap Models assess anonymized news about small stocks. The grader compares their predictions with stock movements over the next five trading days. University Oct 2026 0.01 Not in cards
Immigration Bench Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric. Expert Oct 2026 75.4% Not in cards
IndicIPR-QA Models extract answers from passages in seven Indian intellectual property laws. The benchmark compares predicted spans with reference answers across 4,462 questions. University Oct 2026 77.6 Not in cards
InsureBench Decision Models (Broader Comparison) Models read text evidence and return structured findings about insurance scenarios. The scorer compares 150 findings from 50 packets with the author's labels. University Oct 2026 Not in cards
Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. Expert Oct 2026 Not in cards
Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. Expert Oct 2026 Not in cards
Investor's Last Exam Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria. Expert Oct 2026 68.1 Not in cards
Ivo-micro1 ContractBench Models review contracts and return redlines across 350 attorney-written tasks in 35 scenarios and five agreement types. Tasks cover first review and replies to a counterparty. Expert Oct 2026 64% Not in cards
KMMLU-Pro 2025 Models answer questions from the 2025 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. University Oct 2026 Not in cards
KMMLU-Pro 2026 Models answer questions from the 2026 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. University Oct 2026 78.1% Not in cards
LAB: US Case Law Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings. Expert Oct 2026 Not in cards
Lakh, Crore and Carpet Area Models solve Indian property and finance problems involving local land units, loan payments, and lakh and crore notation. School Oct 2026 97% Not in cards
Legal Consent Sweep Agents review an acquisition data room, reconcile consent requirements, and prepare notices and closing advice. A separate verifier grades their work. Expert Oct 2026 0.38 Not in cards
LegalReasonBench Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings. Expert Oct 2026 50.8% Not in cards
LegalRewardBench Models choose which of two legal responses is better. The benchmark compares their choices with human-verified preferences across legal knowledge, analysis, decisions, and writing. Expert Oct 2026 86.8% Not in cards
LevelBench-100 v4.1 Models take over the books of small businesses using QuickBooks exports. They reconcile balances, identify missing evidence, and prepare closing entries across 12 workflows. Expert Oct 2026 81% Not in cards
LexPilot Models interview a user about a Chinese civil case and predict the ruling on each claim. Paired consultations reveal the effect of missing facts. Expert Oct 2026 72.1% Not in cards
LiveMACE Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns. Expert Oct 2026 Not in cards
LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. University Oct 2026 76.7% Not in cards
Maydan HF RL Sampler Agents reconcile synthetic Saudi social insurance contributions across ten payroll tasks. SQLite checks the final records and audit log. Expert Oct 2026 Not in cards
MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. Expert Oct 2026 15.8% Not in cards
MultiFinGround Models identify accounting concepts and extract every matching fact from English, French, and Greek financial reports. The benchmark checks their selections against annotated disclosures. University Oct 2026 95.4% Not in cards
norregel Models supply the numeric value of a Norwegian legal provision on a specified date. The evaluator distinguishes current values from superseded ones. University Oct 2026 45.1% Not in cards
OR-License Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem. Expert Oct 2026 88.4% Not in cards
PARCEL Models decide whether a New York court opinion supports, contradicts, or fails to address a cited claim. The benchmark contains 3,396 labeled claims. University Oct 2026 97% Not in cards
Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. Expert Oct 2026 In use
PWS-Finance Web research systems answer dated finance questions about company profiles, funding, ownership, markets, regulation, and recent developments. Expert Oct 2026 Not in cards
Quant-Arena Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window. Expert Oct 2026 51.3 Not in cards
Questflow Finance Benchmark Agents query financial filings and market data with a fixed set of tools. They calculate metrics, rank securities, and submit answers that software checks against frozen references. University Oct 2026 100% Not in cards
RealWorkDecision Agents review a frozen company workspace and predict the decision an employee made. Expert-written rubrics allow valid alternatives. University Oct 2026 43.8% Not in cards
ScopeCheck: evidence before numbers Models calculate an operating metric when the data covers the requested population. Paired cases test whether they abstain when essential evidence is missing. University Oct 2026 0.94 Not in cards
SEC filing claim verification (400 items) Models classify whether an excerpt from a corporate filing supports a claim, contradicts it, or provides insufficient evidence. University Oct 2026 99.5% Not in cards
Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. Expert Oct 2026 $2,949 Not in cards
SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. Expert Oct 2026 25.1% Not in cards
SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. Expert Oct 2026 46% Not in cards
SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. Expert Oct 2026 27.8% Not in cards
SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. Expert Oct 2026 18.4% Not in cards
SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. Expert Oct 2026 30.5% Not in cards
Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. Expert Oct 2026 6.11 Not in cards
StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. University Oct 2026 0.83 Not in cards
Tavily Company Search Benchmark Search providers return evidence for company research questions, and an answer agent responds from those results. A judge scores answer completeness, relevance and freshness. Expert Oct 2026 Not in cards
TreasuryQA Bench Agents search scanned Treasury volumes and read tables to find or calculate a number. A script checks whether the answer falls within the accepted range. School Oct 2026 82.4% Not in cards
Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. Expert Oct 2026 69.5% In use
Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. Expert Oct 2026 68.3% In use
Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. Oct 2026 71.3% In use
Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. Expert Oct 2026 78.4% In use
Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. Expert Oct 2026 Not in cards
VisR-RAG-Fin Models retrieve pages from financial documents and answer questions using their contents. The evaluator measures page ranking and checks answers against references. University Oct 2026 Not in cards
Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. Expert Oct 2026 80% Not in cards
AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Expert Sep 2026 1,822 In use
APEX-Agents 1.1 Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero. Expert Sep 2026 90.4% Not in cards
ApprenticeBench Agent joins a simulated company and processes 100 accounts-payable bills in Odoo via GUI or API, learning from records and feedback. Cumulative success rate. University Sep 2026 77% Not in cards
ATLAS-Finance Agents do 100 investment banking tasks inside 13 simulated firms. They gather facts from email, chat, calendar and files over MCP, then build an Excel model graded against expert rubrics. Expert Sep 2026 Not in cards
Australian BAS Reconciliation Models reconcile a synthetic Australian goods and services tax ledger, correct a Business Activity Statement, and identify bookkeeping errors. The grader weighs totals and fault detection. Expert Sep 2026 Not in cards
Beancount Ledger Agents reconcile bank statements, repair a Beancount ledger, and allocate payments to invoices. The evaluator checks balances and payment records against the underlying transactions. University Sep 2026 Not in cards
Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. Expert Sep 2026 65.1% Not in cards
CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. Everyday Sep 2026 92.7 Not in cards
DataBench v1.1 Agents answer ambiguous business questions and make recommendations from a synthetic company warehouse. Model judges assess the reasoning, supporting evidence, artifacts, and final recommendation against task rubrics. Expert Sep 2026 70.5% Not in cards
DAYJOB: Finance Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. Expert Sep 2026 42.5% In use
DealBench-100 Agents perform 100 investment banking workflows across ten synthetic transactions. Deterministic checks grade calculations, decisions, committed changes, and consistency between models and presentations. Expert Sep 2026 Not in cards
DELTA Models research Dutch legal questions and apply the law to facts. Lawyers judge accuracy and professional usefulness. Expert Sep 2026 15.0 Not in cards
DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. University Sep 2026 57.9% Not in cards
ERPBench-100 Agents execute 100 business workflows across ten simulated enterprise resource planning systems. Checks grade decisions, calculations, committed state, and handoffs on a 100-point scale. Expert Sep 2026 Not in cards
FinCrime Agent Benchmark Agents screen customers for sanctions, political exposure, and adverse media. A case passes when agents find the relevant alerts and resolve them under the bank's policy. University Sep 2026 71.1% Not in cards
FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. Expert Sep 2026 71.5% Not in cards
FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. Expert Sep 2026 63.6% In use
FinStepBench CorporateValuation Models value companies using financial evidence and calculations. The score is the published CorporateValuation percentage. Expert Sep 2026 77.3% Not in cards
FinStepBench DeepResearch Models write finance research reports. The score is the published DeepResearch percentage. Expert Sep 2026 59.1% Not in cards
FinStepBench LiveSearch Models retrieve and verify live financial facts. The score is the published LiveSearch percentage. Expert Sep 2026 76.7% Not in cards
Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. Expert Sep 2026 Not in cards
GDP.xlsx Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria. Expert Sep 2026 43.1% Not in cards
GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Expert Sep 2026 1,846 In use
HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Research frontier Sep 2026 77.9% In use
Insurance Agent Benchmark Models read commercial insurance documents and answer questions about their contents. Cooper compares each model alone and with its document processing system. University Sep 2026 77.6% Not in cards
JusticeAxis Models assess evidence and produce judgments for 256 criminal cases from 18 countries. The evaluator checks charges and outcomes against lawyer-written references. Expert Sep 2026 84.4% Not in cards
Legora BAR Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers. Expert Sep 2026 106.0 Not in cards
LEXam-hard Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed. Research frontier Sep 2026 40.8% Not in cards
LifeSci Diamond Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires. Expert Sep 2026 20% Not in cards
Long-Horizon Freight AuditBench Challenge Agents price 100 difficult freight invoices and flag incorrect charges. The cases include incomplete information and exceptions to ordinary contract terms. Expert Sep 2026 10% Not in cards
Long-Horizon Freight AuditBench Production Agents price 200 freight invoices from contracts and carrier documents, then flag billing errors. Loop weights accuracy to reflect a 24 percent billing error rate. University Sep 2026 16.1% Not in cards
Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. Expert Sep 2026 Not in cards
MM-FinEval Models forecast financial outcomes from earnings call transcripts, slides, and audio. The authors compare 19 models using directional accuracy and absolute error. University Sep 2026 88% Not in cards
Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. University Sep 2026 4497.2 Not in cards
Redtape Models assess eligibility for public benefits and identify when a missing fact prevents a decision. PolicyEngine supplies the reference determinations. Expert Sep 2026 0.79 Not in cards
Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. Expert Sep 2026 52.3% In use
TMMLU+ 1.1 Models answer traditional Chinese subject questions from a revised exam set. The revision corrects 197 answers and removes 539 questions from the original release. University Sep 2026 96.2% Not in cards
Typed Decision Bench Models assign probabilities to decisions about a given state. The benchmark scores accuracy and calibration across five suites. Expert Sep 2026 Not in cards
Vals Index v2.1 Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector. Sep 2026 68.9% In use
VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. University Sep 2026 40.1% Not in cards
ViLegalExpert Models retrieve Vietnamese law and answer questions from citizen consultations. Professional answers and expert-verified provisions supply references across 34 legal domains. University Sep 2026 Not in cards
AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. Expert Aug 2026 62.5% In use
AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. Research frontier Aug 2026 Not in cards
Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. Expert Aug 2026 Not in cards
Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. Expert Aug 2026 90.7% In use
CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. Expert Aug 2026 Not in cards
CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. Expert Aug 2026 Not in cards
DuMateBench Agents complete real-world productivity and office-work tasks in sandboxed workspaces with tools. Task evaluators check the requested artifact and record a reward. Expert Aug 2026 Not in cards
E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. University Aug 2026 14.3 Not in cards
Enterprise Task Pareto Frontier (Glean) Models complete 1,000 workplace tasks. A model compares paired answers for correctness, task completion, and readiness for use. Glean estimates preference ratings and cost per task. University Aug 2026 Not in cards
FactoryBench-100 Agents complete 100 manufacturing workflows through simulated ERP, email, spreadsheet, and chat tools. FactoryScore checks their decisions and the state left in each system. Expert Aug 2026 76.2% Not in cards
FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. University Aug 2026 39% Not in cards
GDPval Extended Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores. Expert Aug 2026 7.10% Not in cards
GreekBarRetrieval Models retrieve Greek statutory articles for bar exam questions and case facts. The authors compare ranked results with the statutes relevant to each case. University Aug 2026 0.43 Not in cards
Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. Expert Aug 2026 79.7% In use
Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. Expert Aug 2026 57% In use
Insight Consolidation v1 Agents read customer account documents, infer why the customer acted, and cite supporting records. The evaluator combines a model judgment with citation accuracy. University Aug 2026 Not in cards
LAB: Law Firm Knowledge Agents search a simulated law firm’s matter files and deliver work that satisfies legal task criteria. Expert Aug 2026 Not in cards
LedgerBench-100 Agents perform 100 corporate finance tasks through 66 tools in a shared enterprise system. SQL checks grade the changes they make. Expert Aug 2026 Not in cards
Loss Run Extraction (Bevaya) Models extract claims and policy tables from insurance loss reports into JSON. The main metric counts documents where every field matches the reference. University Aug 2026 85.3% Not in cards
MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. University Aug 2026 61.3% Not in cards
OfficeQA Pro V2 Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match. University Aug 2026 54.4% In use
Palmyra X6 Enterprise Workflow Evaluations Agents perform production workflows that require retrieval, tool use, content generation, or delegation. Writer compares capability and cost. Expert Aug 2026 Not in cards
SalesBench-100 Agents reconcile sales evidence, update authorized records, and deliver executive briefs across 100 workflows. Checks grade procedure, system state, and outputs. Expert Aug 2026 Not in cards
Singapore Factuality Models answer questions about Singapore's laws, economy, and public policies. A model judge checks whether each answer contradicts the reference. University Aug 2026 Not in cards
Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. Expert Aug 2026 71.6 Not in cards
Vals Index v2 Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration. Aug 2026 69.7% Not in cards
Agent Liability Gym Agents review case files and write legal memos about decisions made by autonomous agents. The grader checks their findings, evidence, and unsupported claims. Expert Jul 2026 Not in cards
Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. University Jul 2026 79.4% Not in cards
AI Law Legal Agent Eval Agents review five synthetic export control matters and write risk memos. The grader checks the deliverable, seven findings per matter, and unsupported claims. Expert Jul 2026 Not in cards
AMLBench v0.2 (Superjective) Models decide whether to escalate transaction alerts. The score measures how often they escalate reportable alerts under changes to analyst instructions and case narratives. University Jul 2026 100% Not in cards
APEX-Accounting Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria. Expert Jul 2026 71.5% In use
Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. University Jul 2026 Not in cards
CLBench-V Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition. Everyday Jul 2026 28.5% Not in cards
Cybersecurity and Privacy Legal Agent Eval Agents write legal risk memos for five synthetic privacy and incident reporting matters. The grader checks the completed memo and its coverage of legal issues. Expert Jul 2026 Not in cards
DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. Expert Jul 2026 58.1% In use
Employment Law Legal Agent Eval Agents review employment case files and write risk memos. The grader checks whether the memos identify planted legal issues and avoid unsupported findings. Expert Jul 2026 Not in cards
Enterprise Vendor Eval Agents review vendor documents, question an internal IT team, and recommend whether to approve a purchase. The grader checks their decision and identified requirements. University Jul 2026 Not in cards
Enterprise-Bench L1-L2 Agents answer 14 operational questions across connected enterprise records. Tasks require either retrieving facts or combining evidence under business rules. University Jul 2026 82.9% Not in cards
Excel Modeling Benchmark Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting. Expert Jul 2026 76.7% Not in cards
Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. Expert Jul 2026 62.6% System cards only
Financial Statement Review (fs-review) Models review eight sets of financial statements and flag inconsistencies in cash, retained earnings, or the balance sheet. Some sets contain no errors. University Jul 2026 Not in cards
Founder Bench Agents run four matched businesses per model through the acoco platform. The evaluation tracks real advertising, customer attention, and website traffic after seven days. Its headline customer-impressions metric adds ad impressions to organic site visits. Expert Jul 2026 127902.0 Not in cards
FrontierFinance Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics. Expert Jul 2026 55.9% In use
Hy-FinModelBench Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published. Expert Jul 2026 69% In use
LAB: Diligence Agent researches a synthetic M&A virtual data room (thousands of documents, up to ~80M tokens) and drafts a diligence memo scored against planted-issue rubric criteria. Expert Jul 2026 Not in cards
Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. Expert Jul 2026 43.2% In use
MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. Expert Jul 2026 59.5 Not in cards
Nodal Pricing LMP Models calculate locational marginal electricity prices for congested transmission networks. The grader checks their answers against a power flow optimization model. Expert Jul 2026 Not in cards
PolicyBench No-tools estimate of US taxes and benefits for 100 households across 18 programs. Headline metric is answers within $1 of PolicyEngine-US. Expert Jul 2026 95.2% Not in cards
Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. Expert Jul 2026 57.5% Not in cards
VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. University Jul 2026 62.3% Not in cards
Verifying the Verifiers (Project APE CRED benchmark) Verifier LLMs must find planted errors (prose-table, paper-code, execution, data) in AI-generated economics papers. Credit needs a verbatim quote of the edit. Expert Jul 2026 99.5% Not in cards
Artificial Analysis Intelligence Index v4.1 Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents. Jun 2026 61.0 In use
BigFinanceBench Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference. Expert Jun 2026 58.8% In use
CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. Expert Jun 2026 $22,148,357 Not in cards
CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. University Jun 2026 Not in cards
FinRED Financial Safety Models answer prompts about financial risks. A finance-specific judge grades each response and reports the percentage that assist harmful conduct. University Jun 2026 0.15% Not in cards
FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. University Jun 2026 77.8% In use
GDPval-Rubrics (MiniMax) MiniMax rubric grading of model deliverables on public GDPval tasks, in an environment matched to GDPval-AA. Expert Jun 2026 74.8% In use
Hedge-Bench Agents analyze supplied company and investment documents for 102 open-ended financial research tasks. A model judge applies rubrics that check whether the reasoning covers the analytical steps identified by investment professionals. Expert Jun 2026 31.5 Not in cards
InsureBench Multi-functional LLMs Models read insurance files and answer questions about claims, coverage, and actuarial decisions. InsureThing combines automatic answer checks with rubric grades for judgment tasks. University Jun 2026 96.3% Not in cards
LAB: Contracts 500 in-house contracting tasks (drafting, redlines, issues lists, escalation memos) across 10 contract categories. All-pass expert-rubric grading. Expert Jun 2026 Not in cards
Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. Expert Jun 2026 56.3% In use
LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. Expert Jun 2026 0.66 Not in cards
MacroLens Models use company and economic data to make financial estimates under different scenarios. The authors compare forecasting models with language models and simple baselines. University Jun 2026 Not in cards
PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. Expert Jun 2026 95.3% Not in cards
Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. Expert Jun 2026 76.9% Not in cards
Questflow Traders Arena Agents and people trade on Hyperliquid and Polymarket in a public competition. The leaderboard reports weekly portfolio returns alongside gains and trading volume. University Jun 2026 Not in cards
Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. Expert Jun 2026 62.1% Not in cards
RedlineBench Agent acts as counsel at one turn of a SaaS contract negotiation and returns tracked-change redlines. Three LLM judges grade attorney-written rubrics. Expert Jun 2026 50.5% Not in cards
RedlineBench v2 (Fa1zvn) Agents negotiate eight contract terms with a simulated vendor. The grader compares the agreement with the best deal both sides could accept. University Jun 2026 Not in cards
SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. Expert Jun 2026 34.9% In use
SWM-Bench (Social World Models) Predict how Kalshi and Polymarket belief trajectories shift after news events. 12k+ points across politics, finance and crypto, chronologically split. Expert Jun 2026 Not in cards
TaxCalcBench v2 Models read W-2, 1099 and other PDFs and compute full federal and state returns line by line. A return counts only when every evaluated line is exact. Expert Jun 2026 86% Not in cards
Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. University Jun 2026 60.2% Not in cards
AlexandriaX 2026 Models translate English dialogue into 13 Arabic varieties, translate financial text between six dialects and mark translation errors across five varieties. University May 2026 Not in cards
AvalancheBench (May study) Agents recover underlying patterns from synthetic enterprise data and report their findings. Expert May 2026 Not in cards
BenGER Models write German legal analyses for 596 exam-style cases and 531 doctrinal reasoning tasks. A rubric judge scores legal conclusions, reasoning, and presentation against expert references. Case scores map to the German 0-to-18 grading scale, with four points required to pass. Expert May 2026 71.1 Not in cards
DataPrep-Bench Finance Models prepare training data for the finance domain. The score is the published downstream average after backbone training. Expert May 2026 59.3 Not in cards
DataPrep-Bench Law Models prepare training data for the law domain. The score is the published downstream average after backbone training. Expert May 2026 76.4 Not in cards
DGCL Reward Hacking Models decide whether corporate actions comply with Delaware law and provide supporting calculations. Fifty evaluation cases distinguish correct headline decisions from fully correct answers. Expert May 2026 Not in cards
Envisioning Signals Benchmark Models identify early signs of change for an industry brief. Scores combine verifiable evidence, specific claims, recent sources, and coverage across the requested topics. University May 2026 86.0 Not in cards
ERP-Bench Agents complete 300 procurement and manufacturing workflows in Odoo 19. A constraint solver generates the tasks and optimal solutions, and the verifier checks final business state. Expert May 2026 43.4% Not in cards
ExpertEval Models answer cases in medicine, finance and law. Practicing professionals write the cases and supply weighted grading rubrics. Expert May 2026 66.9% Not in cards
Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. Expert May 2026 57% System cards only
FINESSE-Bench Models answer financial certification questions and interpret trading indicators. The benchmark checks numerical and multiple-choice answers and uses a model judge for free responses. University May 2026 0.90 Not in cards
Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. University May 2026 95.7% In use
HarnessAudit-Bench Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors. Expert May 2026 0.41 Not in cards
Harvey LAB (Legal Agent Benchmark) Agent works through legal matter files across 24+ practice areas and produces work product. Expert rubric, task passes only if every criterion passes. Expert May 2026 25.4% In use
HELM Arabic Enterprise Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute. Expert May 2026 77.9 Not in cards
High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. Expert May 2026 4.10% System cards only
IndustryBench Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses. Expert May 2026 69.4% Not in cards
JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. Expert May 2026 61.4% In use
JudgmentBench Models produce legal work for 30 tasks. Practicing lawyers grade individual answers and compare pairs of answers. Expert May 2026 89% Not in cards
MBABench Agents build complete financial models in Excel from a case prompt and documents. An LLM judge grades accuracy, formulas and formatting on a 0 to 100 scale. Expert May 2026 31.2 Not in cards
MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. Competition May 2026 63.3% System cards only
QF-Bench Agents write and run code for 87 quantitative finance tasks in a network-isolated sandbox with real data. Tasks cover derivatives pricing, fixed income, credit, risk, factor research, backtesting and market microstructure. More than 3,300 assertions check numeric outputs against finance rules such as no-arbitrage bounds and put-call parity. Scored by pass@1 over three runs. Expert May 2026 61.7% Not in cards
Snowflake Text-to-SQL HARD Models generate SQL for difficult questions about unfamiliar enterprise schemas. Snowflake compares models by query accuracy. Expert May 2026 47.9% Not in cards
Vals Finance Agent v2 927 expert-reviewed research questions on public-company filings. Partial credit only after required checks pass. Private test split. Expert May 2026 65.4% In use
Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. Expert May 2026 74.2% Not in cards
Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. Expert May 2026 70.7% Not in cards
Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. University Apr 2026 Not in cards
BankerToolBench 100 end-to-end junior-banker tasks producing Excel, PowerPoint and Word deliverables. Agentic grader scores ~150 weighted expert rubric criteria. Expert Apr 2026 16% In use
Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. University Apr 2026 $283,903 Not in cards
KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. University Apr 2026 42% Fading
NEPABench Models process public comments, extract information from federal documents and answer questions about environmental reviews. Each task has its own scoring rules. University Apr 2026 Not in cards
SAHM Models answer Arabic questions on financial standards, Islamic finance, and accounting. The authors report multiple-choice accuracy and grade open-ended answers separately. University Apr 2026 80.5% Not in cards
Swedish Benchmarks V1.1 Models answer Swedish language, document, and knowledge questions. The overall score combines normalized task performance and response time. School Apr 2026 85.5 Not in cards
$OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. Expert Mar 2026 42.1% In use
Accounting Autocoding Models select a bookkeeping category for each expense. Ramp checks the first choice against the assigned category. University Mar 2026 92.6% Not in cards
BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. Expert Mar 2026 Not in cards
BullshitBench V2 Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback. Everyday Mar 2026 95% Not in cards
Contextual Invoice OCR Models extract invoice fields using a business’s past bills. Ramp counts invoices where every field matches the submitted bill. University Mar 2026 Not in cards
Data Agent Benchmark Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers. Expert Mar 2026 70.9% Not in cards
Echo General AI Prediction Leaderboard Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. Expert Mar 2026 1,018 Not in cards
EnterpriseOps-Gym Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task. Expert Mar 2026 45.9% In use
Financial Statement OCR Models extract financial figures from profit and loss statements. Ramp counts values within one percent of a risk operator’s transcription. University Mar 2026 Not in cards
FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. University Mar 2026 81% Not in cards
FinToolBench Agents answer 295 finance questions by selecting and executing tools from a library of 760 financial interfaces. The evaluation separates tool execution, answer correctness, and mismatches in timeliness, intent, and regulatory domain. Answer checks combine model judging with exact matching. Expert Mar 2026 0.82 Not in cards
Fund Smart Routing Agent Models choose a fund for a transaction. Ramp measures how often a user overrides that choice. University Mar 2026 100% Not in cards
OfficeQA Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. University Mar 2026 86.3% In use
OfficeQA Pro Models answer the 133 hardest OfficeQA questions by finding tables in U.S. Treasury Bulletin documents and computing exact numeric answers. University Mar 2026 60.9% In use
Partner Restrictions Compliance Models identify bank partner restrictions that apply to businesses. Ramp scores missed restrictions, coverage across categories, and false positives. Expert Mar 2026 Not in cards
PMBench Agents search product managers’ notes to identify evidence across customer conversations and answer questions about product requirements. University Mar 2026 46.1 Not in cards
Policy Agent Models approve expenses, reject them, or request human review. Ramp compares their decisions with human expense reviews. University Mar 2026 100% Not in cards
RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. University Mar 2026 $24,351 Not in cards
τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Everyday Mar 2026 55.1% In use
AD-Bench Agents answer advertising analytics requests with production-platform tools. A model judge checks final answers against refreshed reference answers and measures coverage of expert tool-call trajectories. Results report success over one or three attempts. University Feb 2026 83% Not in cards
AgentSettlementBench Models decide whether to release or hold a payment under 20 settlement scenarios. The benchmark tracks accuracy and the cost of approving an unsafe settlement. Expert Feb 2026 Not in cards
BigLaw Bench: Global Models perform legal work specific to the UK, Australia and Spain. Local legal experts designed tasks and grading criteria for each jurisdiction. Expert Feb 2026 Not in cards
Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. Expert Feb 2026 39.6% System cards only
DraftNEPABench Agents draft sections of environmental impact statements using technical and regulatory source documents. Model judges score the drafts against rubrics, and specialists assess a sample. Expert Feb 2026 2.41 Not in cards
Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. Expert Feb 2026 21.6% Fading
GreekMMLU Models answer questions originally written in Greek for academic, professional, and government exams across 45 subjects. University Feb 2026 93.2% Not in cards
JFinQA Models calculate financial ratios, check consistency, and reason about changes over time using Japanese securities reports. University Feb 2026 Not in cards
LABBench2 PatentQA Models locate and interpret evidence in patent documents. Expert Feb 2026 90.9% Not in cards
LegalCiteBench Models recover and verify citations from United States judicial opinions without external lookup. The benchmark also tests whether models match cases and correct citation errors. University Feb 2026 0.78 Not in cards
Malt Freelancer Project Matching Models rank synthetic freelancer profiles against a search query or a project brief. Each task has 200 English prompts and 4,019 profiles across five languages. University Feb 2026 50.4% Not in cards
OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. Expert Feb 2026 23.3% Not in cards
Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. Expert Feb 2026 70% System cards only
ROI Calculator (dsm) Models calculate total and annualized investment returns, decide whether to invest, and explain the decision. The evaluator checks each answer against a bundled reference. School Feb 2026 Not in cards
SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. University Feb 2026 Not in cards
SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. University Feb 2026 55.8% Not in cards
Snowflake Financial PDF Table Parsing Models reconstruct tables from 50 manually annotated financial documents. Evaluators compare complete tables, table dimensions and individual cells with reference Markdown. University Feb 2026 Not in cards
TerraFirm REIT Models choose which properties to sell from a portfolio of 12 office buildings. The evaluator checks whether the remaining portfolio meets five debt and valuation constraints. Expert Feb 2026 Not in cards
Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. Everyday Feb 2026 69% System cards only
Vals Finance Agent Agents answer 537 financial analyst questions on public company SEC filings with search and filing tools. A three-vote GPT-5.2 judge grades each final answer. Expert Feb 2026 64.4% Fading
YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. University Feb 2026 $3,445,495 In use
APEX-Agents 480 long-horizon tasks in banking, consulting and law worlds with docs, sheets, email and chat. Rubric-graded, Pass@1. Expert Jan 2026 67.2% In use
BizFinBench v2 Models answer expert financial questions grounded in business data. The score is average accuracy on eight offline tasks. University Jan 2026 61.5% Not in cards
PLawBench Model handles 850 realistic legal consultation, case-analysis and document-drafting tasks. LLM judge scores against about 12,500 expert rubric items. Expert Jan 2026 69.7% In use
Prediction Arena Agents research events and trade contracts on Kalshi and Polymarket. The leaderboard ranks accounts by cash plus the market value of open positions. University Jan 2026 $138,945 Not in cards
prinzbench Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99. Expert Jan 2026 91.0 Not in cards
QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. School Jan 2026 69.2% Not in cards
QuantEval Models answer quantitative finance questions, solve mathematical problems, and implement trading strategies. The authors report answer accuracy and test the generated code. University Jan 2026 86% Not in cards
TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. University Jan 2026 0% Not in cards
AA-Briefcase Artificial Analysis knowledge-work evaluation of professional deliverables. Models ranked by Elo. Expert 2026 1,720 In use
CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. Research frontier Dec 2025 66.5 Not in cards
CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. Expert Dec 2025 83.7 Not in cards
COLIEE 2026 Models retrieve Canadian precedents and apply Japanese statutes. The statute retrieval track also requires a correct yes or no answer with sufficient supporting articles. University Dec 2025 Not in cards
Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. Everyday Dec 2025 64.7% Fading
CorpusQA 1M Models answer computation-heavy questions whose evidence is spread across hundreds of finance, education, and real estate documents in a 1M-token context. Scored by accuracy. University Dec 2025 62% Fading
DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. Expert Dec 2025 47% Not in cards
Finch (FinWorkBench) Agent completes 172 enterprise finance and accounting spreadsheet workflows (384 tasks) from real data. Human plus calibrated LLM-judge grading. Expert Dec 2025 43% In use
MAWARITH Models identify eligible heirs and calculate each share of an estate from an Arabic case. The MIR-E score gives credit for intermediate decisions. University Dec 2025 Not in cards
MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. University Dec 2025 Not in cards
Pieverse Agent Prediction Arena Agents trade event contracts on Polymarket with $1,000 each in real money. Pieverse tracks account value as market prices change. University Dec 2025 $981 Not in cards
Research Plan Generation Models write research plans from goals derived from scientific papers. Judges grade each plan against criteria specific to its source paper. Expert Dec 2025 Not in cards
VLegal-Bench Models answer questions and solve legal scenarios under Vietnamese law. Legal experts check that reference answers follow authoritative documents. University Dec 2025 72% Not in cards
XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. Expert Dec 2025 64.5% Fading
AA-Omniscience 41-topic closed-book knowledge test. Net score rewards correct answers, penalizes wrong ones and gives zero for abstentions. Expert Nov 2025 42.4 In use
BigLaw Bench: Arena Legal experts compare two answers to a legal task. The benchmark reports pairwise win percentages. Expert Nov 2025 Not in cards
Contract Intelligence Models answer questions about contract terms. Harvey compares answers with attorney reviewed labels. Expert Nov 2025 Not in cards
LexGenius Models answer Chinese legal questions drawn from cases and exams. The authors compare results across areas including legal reasoning, ethics, and judicial practice. University Nov 2025 59.7% Not in cards
Noor Models answer filtered academic questions chosen to distinguish early training checkpoints. The benchmark measures progress where standard MMLU scores are noisy. University Nov 2025 Not in cards
PRBench Model answers 1,100 expert-written law and finance questions. LLM judge grades against weighted expert rubrics of 10-30 criteria each. Expert Nov 2025 61.6% In use
TalentCLEF 2026 Contextualized Job-Person Matching Models rank candidate resumes for a job description in English or Spanish. Organizers score retrieval within and across languages and compare rankings across gender variants. University Nov 2025 Not in cards
TalentCLEF 2026 Job-Skill Matching with Skill Type Classification Models rank skills for an English job title. The main ranking metric gives core skills more weight than contextual skills. University Nov 2025 Not in cards
Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. Everyday Nov 2025 $16,843 In use
Vending-Bench Arena Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance. Expert Nov 2025 $15,962 Not in cards
Alpha Arena Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns. Expert Oct 2025 Not in cards
HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. Research frontier Oct 2025 Not in cards
Korean Canonical Legal Benchmark Models answer Korean legal questions with supporting precedents. The benchmark includes multiple-choice questions and essays with individual grading criteria. University Oct 2025 Not in cards
ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. Expert Oct 2025 66.2% In use
Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. Expert Oct 2025 41.3% In use
RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. University Oct 2025 69.4% Not in cards
StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. Expert Oct 2025 1.00 Not in cards
Vals Index v1 Vals AI's first index of agentic work in finance, law and coding, with each sector weighted by its share of U.S. GDP. Oct 2025 72.2% System cards only
Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. Expert Oct 2025 71.1% Not in cards
VLAIR Legal Research Legal AI tools answer 200 US legal research questions. Reviewers score legal accuracy, citation authority, and suitability for sharing. Expert Oct 2025 Not in cards
APEX-1 Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics. Expert Sep 2025 81.1% Not in cards
BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. Expert Sep 2025 64.6 Not in cards
DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. University Sep 2025 63.8 Not in cards
Financial Retrieval (Daloopa) Chatbots retrieve historical financial figures. Daloopa compares their answers with normalized reference values. University Sep 2025 91.6% Not in cards
FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. Expert Sep 2025 81.7% In use
FinSearchComp Global Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset. Expert Sep 2025 65.5% Dropped
FinSearchComp Greater China Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset. Expert Sep 2025 Not in cards
GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Expert Sep 2025 87.9% In use
Hebbia Financial Services Benchmark (2025) Models extract facts, summarize documents and reason about financial information. Hebbia evaluates responses to more than 600 questions. Expert Sep 2025 Not in cards
SCUBA Agents work in a Salesforce sandbox as administrators, sales representatives, or service agents. Evaluators check milestones such as creating accounts and routing support cases. University Sep 2025 53.9% Not in cards
SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. Everyday Sep 2025 74.3% System cards only
Trading Bot Bench Models write Python trading strategies for historical stock prices. The evaluator combines return, Sharpe ratio, and drawdown on held-out scenarios. University Sep 2025 Not in cards
VCBench Models predict venture outcomes from founder profiles. The benchmark emphasizes precision through the F0.5 score. The hub adaptation substitutes synthetic profiles for the original data. Expert Sep 2025 33.7% Not in cards
Arctic-Extract Business Document Benchmark Models answer questions about business documents and extract information from varied layouts. The internal visual tasks include contracts, signatures and checkboxes. University Aug 2025 58.5 Not in cards
CaseLaw v2 Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents. Expert Aug 2025 79.3% Not in cards
LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. University Aug 2025 12.1% Not in cards
R-ConstraintBench Models produce feasible operational plans as constraints accumulate. The score gives more weight to harder levels. Expert Aug 2025 79.2% Not in cards
IFScale Models write business reports while including up to 500 specified keywords. The benchmark measures how many instructions they follow at each density. University Jul 2025 68.9% Not in cards
Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. Expert Jul 2025 88.6% Fading
JamC-QA Models answer multiple-choice questions about Japan, including regional identity, customs, geography, and law. School Jul 2025 57.1% Not in cards
KBank-MMLU (Upstage) Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house. Expert Jul 2025 80.8% Fading
KMMLU-Pro Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Expert Jul 2025 79.2% In use
KMMLU-Redux Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed. University Jul 2025 81.1% Dropped
LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. University Jul 2025 Not in cards
Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. Expert Jul 2025 System cards only
TaxCalcBench Models compute complete US federal tax returns line by line from taxpayer data, with no tax engine. A return counts only when every evaluated line is exact. Expert Jul 2025 62.8% Not in cards
WISER-Search Web research agents answer fresh and business questions that require current sources, multi-hop search, and evidence synthesis. Expert Jul 2025 Not in cards
DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. Expert Jun 2025 Not in cards
EDINET-Bench Models read Japanese financial disclosures to detect fraud, forecast earnings and identify industries. Expert Jun 2025 0.73 Not in cards
ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. Expert Jun 2025 33.4% Not in cards
Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. Expert Jun 2025 53.1% Not in cards
FinanceReasoning Models solve financial numerical reasoning problems that need one or more finance formulas. Answers are numbers checked against reference Python solutions. University Jun 2025 86.9% System cards only
GerLeRB Models rank German federal statute sections for 367 legal questions. Legal experts checked the questions against the cited provisions. University Jun 2025 0.70 Not in cards
IITP-Legal Models translate 5,000 English legal sentences into Hindi. The suite includes 100 sentences at each length from five through 54 words. The original ranking uses BLEU. University Jun 2025 33.4 Not in cards
MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. University Jun 2025 33.5 Not in cards
WMT25 EEA Terminology Models translate English regulatory sentences into Icelandic. The scorer checks whether they use the official Icelandic terms from the European Economic Area agreement. University Jun 2025 46.5% Not in cards
xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. Expert Jun 2025 50.8 Not in cards
xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. Expert Jun 2025 78.5 Not in cards
BizFinBench Models answer 6,781 Chinese finance queries drawn from real business scenarios. The queries cover numerical calculation, reasoning, information extraction, prediction recognition and knowledge questions, in nine categories. Some tasks are scored against answers and others by IteraJudge, the authors' LLM judging method. University May 2025 73.9% Not in cards
CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. Everyday May 2025 56.2% Not in cards
DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. Expert May 2025 61.1% Fading
FinMaster Models complete financial workflows using simulated company records. The benchmark includes single calculations and tasks that require several financial metrics. University May 2025 73% Not in cards
GreekBarBench Models answer Greek bar exam questions in writing. A model judge grades their use of case facts, statutory articles, and legal analysis. University May 2025 8.35 Not in cards
LEXam Models answer 7,537 questions from 340 law exams at Swiss, EU and international law courses, in English and German. Open answers are judged against reference answers, and multiple-choice items are exact match. Expert May 2025 70.2% Not in cards
MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. University May 2025 74.1% Not in cards
QIAS 2025 Islamic Inheritance Reasoning Models choose the correct distribution of an estate under Islamic inheritance law. The Arabic test contains 500 beginner questions and 500 advanced questions. University May 2025 Not in cards
SnorkelUnderwrite Agents help simulated small business insurance underwriters across six task types in multi-turn chats, using tools and asking the underwriter for missing facts. Expert May 2025 91% Not in cards
Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. University Apr 2025 52.9% System cards only
OAB-Bench Models write answers for the second phase of the Brazilian bar examination. The port averages model judges’ scores against grading guidelines. Expert Apr 2025 9.39 Not in cards
TRLawBench Models answer 297 questions on Turkish law in three stages: 97 questions from official ÖSYM legal exams and 200 harder questions across 14 branches of law. Answers are checked against the answer key. A judge model also grades the reasoning from 0 to 10. University Apr 2025 9.85 Not in cards
Vals Finance Agent v1 Agents answer 537 financial analyst questions on public company SEC filings using search and filing tools. An LLM judge grades each final answer. Expert Apr 2025 60.6% Dropped
WISER-Atomic Web research systems answer 121 customer-inspired questions across finance, sales, recruitment, and other domains. Expert Apr 2025 69% Not in cards
HELM Capabilities Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public. University Mar 2025 81.9 Not in cards
Pre-Flight Models answer questions about aviation regulations, dispatch, pilot procedures and airport operations. The evaluator compares each choice with the answer key. Expert Mar 2025 84% Not in cards
Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. School Mar 2025 System cards only
Snowflake Financial Filings RAG Evaluation Systems answer about 500 questions over 23,000 SEC filings. Annotators identify supporting passages, pages and answers. Expert Mar 2025 Not in cards
FailSafeQA Models answer finance questions over long documents while the query or the document is degraded, irrelevant or missing. An LLM judge scores robustness, context grounding and compliance. Expert Feb 2025 0.76 Not in cards
MMLU-Redux 2.0 Spanish Models answer corrected MMLU questions in Spanish across 57 subjects. The evaluation compares each selected option with the answer key. University Feb 2025 Not in cards
NitiBench Models retrieve Thai statutes and answer legal questions. The authors score retrieval separately and use a model judge to check coverage and contradictions. University Feb 2025 70.3 Not in cards
SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Expert Feb 2025 73.9% In use
Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. Expert Feb 2025 Not in cards
Vending-Bench Agents run a simulated vending machine business over long horizons, ordering stock, setting prices, and paying daily fees. Scoring is final net worth in dollars. Everyday Feb 2025 $4,694 Dropped
COLIEE 2025 Models retrieve Canadian cases and answer Japanese bar exam questions using statutes. A new pilot asks which tort claims succeed and which arguments courts accept. University Jan 2025 Not in cards
LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. University Jan 2025 69.1% Not in cards
Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. University Jan 2025 88.7% In use
SnorkelFinance ReAct agents answer expert-verified financial questions built from 10-K filings, using SQL, code and other MCP tools without seeing the documents. A judge scores full traces. University 2025 81% Not in cards
Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. University Dec 2024 96% Dropped
Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. University Dec 2024 95.7% Dropped
LegalBench.PT Models answer questions about Portuguese law derived from university exams. The benchmark includes case analysis, multiple selection, and true-or-false questions. University Dec 2024 85.4% Not in cards
MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. University Dec 2024 73.4% Not in cards
BigLaw Bench Retrieval Retrieval systems find the most relevant passages in legal documents. Harvey compares retrieval results with attorney annotations. Expert Nov 2024 Not in cards
CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. Everyday Nov 2024 64.3% Not in cards
INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. University Nov 2024 89.8% Fading
MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. University Nov 2024 94.1% Dropped
PenguinScrolls Models answer user-style questions over long natural documents such as books, financial reports and legal papers, up to 128K tokens. Tasks cover extraction, localization, qualitative analysis and numerical reasoning. Everyday Nov 2024 87.7% System cards only
AMELIA Models classify passages from Italian tax decisions as premises or conclusions. They distinguish factual from legal premises and identify the reasoning behind legal premises. University Oct 2024 0.76 Not in cards
ChouBun Models answer questions about long Japanese documents and summarize company securities reports. University Oct 2024 17.4 Not in cards
KBL Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items. Expert Oct 2024 77.5% In use
Long-document QA (Kensho) Models answer questions by reasoning over multi-hundred-page financial filings and other long documents. Expert Oct 2024 Not in cards
MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. University Oct 2024 Not in cards
Multilingual Entity Linking of Occupations Models map occupational titles from national classifications to European occupation codes. The benchmark scores the rank of the first correct match. University Oct 2024 Not in cards
xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. University Oct 2024 41.4% System cards only
ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. University Sep 2024 72.2% Not in cards
BigLaw Bench Workflows: SPA Deal Points Models extract deal terms from stock purchase agreements. Harvey measures accuracy against terms extracted and checked by attorneys. Expert Sep 2024 Not in cards
ContextualBench CRM agents answer enterprise questions against contextual business data and receive accuracy, cost, speed, trust and safety measurements. Expert Sep 2024 Not in cards
FinDVer Models verify claims using text and tables in financial reports. The authors score the final true or false judgment across three kinds of reasoning. University Sep 2024 Not in cards
FinMTEB Models represent English and Chinese financial texts as vectors. The suite tests seven abilities, including retrieving definitions, grouping complaints, and comparing financial statements. University Sep 2024 Not in cards
Generating Confusion: Voting with a Disability Models answer questions about accessible voting in the United States. Reviewers measure how often answers to standard voter questions contain incorrect information. Everyday Sep 2024 Not in cards
MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. University Sep 2024 84.6% In use
TalentCLEF 2025 Multilingual Job Title Matching Models rank job titles by how closely they match a query title within or across languages. Organizers score mean average precision. Everyday Sep 2024 Not in cards
BigLaw Bench Models review legal documents and receive rubric scores for substantive accuracy and editing quality. Research frontier Aug 2024 91% Fading
MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. University Aug 2024 Not in cards
ComplaintsClustering Models group consumer complaints by financial product. The score measures whether each group contains one product and whether complaints about that product stay together. Everyday Jul 2024 Not in cards
DISCFinLLMReranking Models rank Chinese news articles for questions about financial events. Each question has two or three relevant articles and ten distractors. University Jul 2024 Not in cards
FinEvaReranking Models rank Chinese financial news passages for an investor question. Each question has one matching passage and nine or ten distractors. University Jul 2024 Not in cards
FinFactReranking Models rank fact checking articles for a claim. Each test row includes one designated article and nine or ten distractors. Everyday Jul 2024 Not in cards
FinMTEB FINAL Models compare passages from financial reports. The score measures how well their similarity rankings distinguish revised passages from mismatched passages. University Jul 2024 Not in cards
FiQA2018Reranking Models rank eleven candidate answers to personal finance questions. Each test row has one designated answer. Everyday Jul 2024 Not in cards
HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. University Jul 2024 78.8% Not in cards
HC3Reranking Models rank answers to personal finance questions. Each test question has one relevant human answer and one relevant ChatGPT answer. Everyday Jul 2024 Not in cards
PiiClustering Models group synthetic English financial documents by document type. The labels include bank statements, payment confirmations, and insurance claim forms. Everyday Jul 2024 Not in cards
TheGoldman Retrieval Models retrieve the Goldman Sachs dictionary definition for a financial term. The score rewards placing the correct definition near the top of the first ten results. Everyday Jul 2024 Not in cards
WikiCompany2IndustryClustering Models group company descriptions by industry. The score measures how well the groups recover the industry labels. Everyday Jul 2024 Not in cards
CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. Everyday Jun 2024 0.20 Not in cards
MMLU-Redux Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy. University Jun 2024 95% Fading
MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. University Jun 2024 65% Not in cards
Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. University Jun 2024 48.0 Not in cards
CFLUE Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation. University May 2024 53.8% Not in cards
CFLUE Application Models complete Chinese financial language tasks. The score is the published average of the application task metrics. University May 2024 44.2 Not in cards
CFLUE Knowledge Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test. University May 2024 53.8% Not in cards
Japanese Tort-case Dataset Models predict whether a Japanese court found a tort from the facts and the parties' arguments. They also label which arguments the court accepted. Expert May 2024 Not in cards
NIFTY Market Movement Models use financial news and market history to predict whether the market will rise, fall or remain neutral. University May 2024 Not in cards
Finance Fundamentals Models answer curated finance questions covering quantitative reasoning, financial domain knowledge and document-based questions. University Apr 2024 Not in cards
Snowflake Arctic Enterprise Intelligence Models write code, generate SQL and follow instructions. Snowflake averages the coding, SQL and instruction-following scores. University Apr 2024 Not in cards
SuperCLUE-Fin Models answer Chinese finance questions across 25 tasks in six categories. The evaluation covers financial knowledge and its application in practice. University Apr 2024 88.7 Not in cards
FinSTS Models compare statements from different annual reports by the same company and score whether their financial meaning changed. People labeled 370 test pairs. University Mar 2024 Not in cards
GerDaLIR Small Models match German court passages to cited decisions. This version keeps only decisions that answer at least one test query. VAGO scores the top ten results with nDCG@10. University Mar 2024 0.45 Not in cards
NTCIR-18 MedNLP-CHAT Models flag medical, legal, and ethical risks in answers to patient questions. The organizers compare the labels with expert judgments using macro F1. Expert Mar 2024 Not in cards
FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. University Feb 2024 0.25 Not in cards
GerLayQA Models find German Civil Code sections that answer questions from the public. The authors also evaluated generated answers against lawyers' answers. University Feb 2024 0.27 Not in cards
KMMLU Models answer 35,030 Korean multiple-choice questions across 45 subjects, collected from original Korean exams. Scoring is accuracy. University Feb 2024 77% Fading
Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. University Feb 2024 78.7% Dropped
Artificial Analysis Intelligence Index v1 Artificial Analysis's first composite score, called the Quality Index in 2024, which normalized and combined a few public benchmarks into one number. Jan 2024 Not in cards
CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. University Jan 2024 36.8% Not in cards
Economics Arena for Large Language Models Models choose numbers in beauty contests and place sealed bids in auctions. Researchers compare payoffs and measure how strategies change when players see previous rounds. University Jan 2024 0.95 Not in cards
Enterprise Scenarios Leaderboard Models complete six tasks drawn from workplace settings. The leaderboard combines their scores across all six tasks. University Jan 2024 Not in cards
KMMLU-Hard Models answer about 5,000 Korean multiple-choice questions from KMMLU that most models got wrong, across many expert subjects. University Jan 2024 30.5% Not in cards
Norwegian Courts Bitext Mining Models match court sentences written in Norwegian Bokmål and Nynorsk. The test contains 228 aligned pairs and scores matches with F1. School Jan 2024 100% Not in cards
OAB Exams Models answer Portuguese multiple-choice questions from the Brazilian bar examination. University Jan 2024 Not in cards
OpenFinData Models answer Chinese financial questions, extract entities and values, and check calculations or data consistency. OpenCompass scores answer accuracy, using required-keyword matching for entity extraction. University Jan 2024 Not in cards
HELM Lite A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014. University Dec 2023 93.8 Not in cards
SuperCLUE-Auto Models answer Chinese automotive questions through multiple turns. A judging model scores each answer against criteria for its task. University Dec 2023 76.6 Not in cards
TMMLU+ Models answer traditional Chinese questions across 66 subjects. The evaluation includes elementary material and professional examinations. University Dec 2023 88.6% Not in cards
FinanceBench Models answer open-book questions about public companies using their filings, such as 10-Ks and earnings reports. The full set has 10,231 questions with answers and evidence strings, and 150 of them are public. The makers graded answers by hand, and later users often grade with a model judge. Expert Nov 2023 41% Not in cards
FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. University Nov 2023 1.63% Not in cards
Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. University Nov 2023 4.67% Not in cards
MegaOath Forecasting Survey Models predict how each of 21 honesty oaths changes the share of earned income people report in a tax game. Researchers score correlation with observed effects. University Nov 2023 Not in cards
MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. University Nov 2023 86% In use
COLIEE 2024 Models retrieve Canadian precedents and supporting paragraphs. They also retrieve Japanese Civil Code articles and answer yes or no on bar exam questions. University Oct 2023 Not in cards
IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. School Oct 2023 35.8% Dropped
AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. University Sep 2023 72.4% Dropped
CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. Expert Sep 2023 83.3 Not in cards
Fin-Eva Models answer 13,106 Chinese multiple-choice questions about financial knowledge, calculations, analysis, and compliance. The evaluator checks the chosen option against the answer key. Expert Sep 2023 Not in cards
Fin-Fact Models judge whether evidence supports a financial claim, contradicts it, or leaves too little information. The original evaluation also tests models with images. University Sep 2023 61% Not in cards
LawBench Models answer Chinese legal questions across 20 tasks, including statute recall and case prediction. Each task has its own scoring rule. University Sep 2023 44.5 Not in cards
FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. University Aug 2023 69.4% Not in cards
FinEval 6.0 Models solve financial evaluation tasks. The score is the official 6.0 total score. University Aug 2023 84.5% Not in cards
LegalBench Models do 162 legal reasoning tasks written by legal professionals, covering six types of legal reasoning such as issue spotting and rule application. Expert Aug 2023 88.6% Not in cards
News With GPT Instructions Models classify financial news by sentiment. ChatGPT supplied seven sentiment labels and explanations. The public test contains 4,047 news items. Everyday Jul 2023 63.9 Not in cards
Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. University Jul 2023 80% Dropped
CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. University Jun 2023 90.8% Fading
SuperCLUE The CLUE team tests Chinese and foreign language models on periodic sets of Chinese questions covering language, professional knowledge, reasoning, agents and safety. Open answers are graded by a strong LLM judge, and results feed an overall ranking. University May 2023 73.6 Not in cards
TheoremQA Models solve 800 university-level questions in math, physics, EE and CS, and finance that each need a specific theorem. Scored by answer accuracy. University May 2023 44.4% Dropped
AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. School Apr 2023 89.2% Fading
FiQA SA (BloombergGPT protocol) Models classify sentiment toward a named financial target as negative, neutral, or positive. Bloomberg uses five examples and reports weighted F1. Everyday Mar 2023 0.63 Not in cards
MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. University Mar 2023 0.07 System cards only
LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. University Jan 2023 Not in cards
MAUD Models classify provisions in merger agreements into expert-defined legal categories. The benchmark compares predictions with lawyer annotations and reports precision-recall area and micro and macro F1. Expert Jan 2023 Not in cards
ConvFinQA Models answer a conversation of related numerical questions about a company's financial report, with tables and text, where later questions build on earlier answers. The dataset has about 3,900 conversations with 14,000 questions, built from FinQA. Answers are checked by execution accuracy against the gold reasoning program. University Oct 2022 58.7% Not in cards
Demosthenes Argument Classification Models identify premises and conclusions in European Union court decisions on fiscal state aid. They also classify premise types and legal reasoning schemes. Expert Oct 2022 Not in cards
EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. University Oct 2022 Not in cards
KPI-EDGAR Models identify key performance indicators and related values in financial reports. The original annotations also link entities within the text. University Oct 2022 Not in cards
Sayfullina Soft Skill Span Extraction Models mark the words that express a soft skill in English job ads. The test scores exact span matches. Everyday Sep 2022 Not in cards
SkillSpan-ESCO Models link skills mentioned in job advertisements to entries in the European skills classification. Human annotators assigned the reference labels. University Sep 2022 Not in cards
Twitter Financial News Sentiment Models classify financial news posts as bearish, bullish, or neutral. The released validation file contains 2,388 posts. Everyday Sep 2022 Not in cards
RMU:ECHR Models mark argument spans in European Court of Human Rights rulings and identify the type of argument and the party making it. University Aug 2022 Not in cards
TAT-DQA Models answer questions about financial report pages that mix tables and text, often needing arithmetic. Scored by exact match and F1. University Jul 2022 37.8% Dropped
Multi-LexSum Models summarize documents from civil rights lawsuits at different lengths. Legal experts wrote the reference summaries. Expert Jun 2022 Not in cards
French Insurance Job Offer Models mark and classify soft skills in French job ads from Canadian insurers. A domain expert annotated 47 ads. School Apr 2022 Not in cards
ContractNLI Models decide whether a contract supports, contradicts, or omits a claim. The original benchmark also asks them to identify the passages that justify their decision. Expert Oct 2021 Not in cards
GerDaLIR Models find the German court decisions cited by a passage after the authors mask its references. The original test split has 12,298 queries. University Oct 2021 Not in cards
LexGLUE Models solve a suite of English legal text tasks drawn from court cases, EU law, contracts and terms of service. Most tasks are classification, and one asks for the holding of a US case. Each task reports micro and macro F1. Expert Oct 2021 Not in cards
Swiss-Judgment-Prediction Models predict whether the Swiss Federal Supreme Court accepts or dismisses a case. Expert Oct 2021 Not in cards
FinQA Models answer numerical questions about company earnings reports that mix tables and text. Financial experts wrote 8,281 question and answer pairs, each with a gold reasoning program. Answers are checked by execution accuracy and program accuracy. University Sep 2021 Not in cards
Greek Legal Code Models classify Greek legal texts by subject using their place in the Greek Legal Code hierarchy. University Sep 2021 56.5% Not in cards
JobBERT Job Title Normalization Models match English vacancy titles from Malaysia’s public job board to standardized occupation labels. The authors report reciprocal rank and recall at several cutoffs. University Sep 2021 Not in cards
MultiEURLEX Models assign subject labels to European Union legislation written in different languages. Expert Sep 2021 0.05 Not in cards
RAFT (Real-world Annotated Few-shot Tasks) Models learn text classification tasks from small labeled samples. The tasks use real documents, including customer messages, court decisions, and scientific abstracts. University Sep 2021 62.7% Not in cards
LegalQuAD Models find the German legal document that answers each of 200 questions. The original data also mark the answer text and its starting position. University Aug 2021 0.70 Not in cards
BVA Case Citation Dataset Models recommend the next legal citation in a veterans’ appeal decision. The authors measure how often the recorded citation appears among the top recommendations. Expert May 2021 Not in cards
TAT-QA Models answer 16,552 questions about 2,757 passages from real financial reports that mix a table with text. Many answers need arithmetic, counting or comparison. Scored by exact match and F1. University May 2021 Not in cards
BEIR FiQA-2018 Retrieval Models rank 57,638 answer posts for 648 investment questions. BEIR sampled its test questions from the original FiQA training split. Everyday Apr 2021 0.65 Not in cards
CaseHOLD Models read a passage from a US court decision with the holding of a cited case masked out and pick the correct holding from five choices. The dataset has more than 53,000 questions drawn from US case law. Expert Apr 2021 Not in cards
Earnings-21 Models transcribe long earnings calls. The benchmark tests recognition of financial terminology in real conversations. Everyday Apr 2021 Not in cards
SPGISpeech Models transcribe English financial recordings. The references include formatting such as capitalization and punctuation. Everyday Apr 2021 1.84% Not in cards
CUAD Models review commercial contracts and highlight the clauses a lawyer would flag, across 41 categories such as governing law, non-compete and change of control. Legal experts from The Atticus Project wrote more than 13,000 labels over 510 contracts. Scores measure how well the extracted spans match the expert labels. Expert Mar 2021 Not in cards
ECHR Legal Corpus for Argument Mining Models identify argumentative clauses, link related clauses, and distinguish premises from conclusions in European Court of Human Rights rulings. University Dec 2020 Not in cards
AILA Statute Retrieval Models retrieve the Indian statutes that apply to a described legal situation. University Oct 2020 0.85 Not in cards
Legal Opinion Sentiment Classification Models classify the sentiment expressed in passages from legal opinions. Expert annotations provide the reference labels. University Oct 2020 Not in cards
Gold Commodity News Models classify news headlines about gold. Labels distinguish price direction, timing, and comparisons with other assets. School Sep 2020 Not in cards
MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. University Sep 2020 92.3% In use
German Argument Mining Models label sentences from Bavarian court judgments as conclusions, definitions, applications of law to facts, or other text. The authors report F1 and accuracy. University Jul 2020 Not in cards
RCTs to Scale nudge forecasts Models predict how each of 14 government interventions changes participation or compliance relative to a control. Researchers correlate predictions with observed changes in percentage points. University Jul 2020 Not in cards
BillSum Models summarize legislative bills. The dataset pairs bill text with summaries written for the US Congress and the California legislature. University Nov 2019 Not in cards
ECHR Binary Violation Classification Models read the facts of a European Court of Human Rights case and predict whether the court found a violation. Expert Jun 2019 Not in cards
Plain English Summarization of Contracts Models summarize contract clauses in plain English. Human summaries provide references for evaluating how well each response conveys the legal language. University Jun 2019 Not in cards
chABSA Models identify the target and sentiment of opinions in the business results sections of Japanese annual reports. University Dec 2017 Not in cards
Expert Forecasts of Real-Effort Experiment Models predict how incentives change the number of alternating key presses workers make in ten minutes. Researchers compare predictions with observed effort across 15 conditions. University Jul 2015 Not in cards
Financial PhraseBank Models classify the financial sentiment of a news sentence as positive, negative, or neutral. University Jul 2013 95.2% Not in cards
AP Macroeconomics Models take a simulated AP Macroeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. School 5.00 System cards only
AP Microeconomics Models take a simulated AP Microeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. School 5.00 System cards only
CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. Expert 44.4% Not in cards
Contract Workflows (Applications) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. Expert Not in cards
Contract Workflows (Models) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. Expert 48.4% Not in cards
CorpFin v2 Models answer expert-written questions about credit agreements that often exceed 200 pages. An LLM judge grades each answer for accuracy. Expert 73.2% In use
CortexRetrievalBench Retrieval systems rank financial documents for a practitioner’s question. The benchmark checks whether an authoritative primary source appears at the top. Expert Not in cards
CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Expert 75.9% In use
Data Extraction (Applications) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. Expert Not in cards
Data Extraction (Models) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. Expert 76.7% Not in cards
DesignArena Spreadsheet Agents build spreadsheets such as financial models, budgets and trackers as Excel or CSV workbooks. Users compare accuracy, structure and formatting and vote. Everyday 1,429 Not in cards
EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. Expert 74.4% Not in cards
FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. Everyday $215,319 Not in cards
Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. Not in cards
Harvey LAB-AA v1.0 Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed. Expert 94.6% In use
Italian-Administrative-CaseHold Models classify Italian administrative judgments paired with their official holdings by legal subject. University 68.9% Not in cards
LegalSupport Models read a legal claim and two court case summaries. They choose the case that most directly supports the claim. University Not in cards
MAPA EUR-Lex Models mark sensitive entities in EU legal texts. MAPA annotates general and fine-grained categories. The multilingual conversion covers 21 languages. University Not in cards
MBE (Multistate Bar Examination) Models answer the 200 multiple choice questions of an official Multistate Bar Examination practice exam. Scored by accuracy. Expert 85% System cards only
Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. Expert 5.74% System cards only
MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. Everyday 72.1% Not in cards
Open Portuguese LLM Leaderboard Models take Portuguese exams and complete language tasks for entailment, similarity, sentiment, and hate speech detection. The leaderboard aggregates task scores. University 88.4 Not in cards
RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. Expert 94.2% Not in cards
Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. Expert 1,637 System cards only
Realm Financial Reasoning Agents analyze financial documents and spreadsheets. A model grades their answers against practitioner rubrics. Expert 56.2% Not in cards
SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. Expert 64.7% Not in cards
SuperCLUE-Law Models respond to Chinese legal questions. Expert 92.1 Not in cards
SuperCLUE-Mkt Models complete tasks in advertising and marketing. University 85.7 Not in cards
TaxEval v2 Models answer expert-written tax questions. A judge grades answer correctness and stepwise reasoning against expert solutions. Expert 80.4% Not in cards
Uniform Bar Exam (MBE+MEE+MPT) Models take a simulated Uniform Bar Exam with multiple-choice, essay and performance test parts. Scored on the 400-point scale. Expert 298.0 System cards only