Law and finance
Tasks that need legal, economic or business knowledge. We count 592 benchmarks here, 4 of them arenas. A benchmark belongs here when subject includes law or economics and business.
How fast it grows
225 were released since July, more than the 105 in the first half of 2026.
How much is still open
No model has passed 85% on 221 of the 288 benchmarks here that score in percent, 77% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| Access to Justice Benchmark Models interview a simulated client about a civil legal problem, explain their options, and direct them to local help. | Expert | Nov 2026 | | Not in cards |
| AgenticFinBench Agents complete 230 financial requests in two languages. Graders apply task-specific rubrics and audit unsupported claims against external evidence. | Expert | Nov 2026 | 89.2 | Not in cards |
| AlphaSense Financial Research Benchmark Agents answer 245 multi-step finance questions across company research, financial modeling, earnings, deals, markets and regulatory analysis using source retrieval. | Expert | Nov 2026 | Not in cards | |
| ChJGBench Models draft the reasoning and orders in Chinese civil, criminal, and administrative judgments. The evaluator compares their work with 3,462 real judgments. | Expert | Nov 2026 | | Not in cards |
| codex of veritas Models answer ledger questions when the supplied quantities determine the result. They must abstain when a necessary quantity is missing. | School | Nov 2026 | | Not in cards |
| Coverage Cat AI Insurance Benchmark (fictional insurance tasks) Models read fictional policy documents, find missing facts, and complete insurance applications. The scorer checks source quotes, required fields, and authorized actions. | University | Nov 2026 | 100% | Not in cards |
| Crazyrouter AI Trading Arena Models buy, sell, or hold three cryptocurrencies every five minutes using simulated accounts. The grader values portfolios at live quotes and deducts trading fees. | University | Nov 2026 | -7.45% | Not in cards |
| DeepFinance Agents do private equity and investment banking work on 21 anonymized real deals: valuation, due diligence, underwriting and financial modeling. Each task comes with a data room of about 19 files and would take a professional about 27 hours. Rubrics of about 84 criteria grade the output, the workflow and the judgment behind each assumption. | Expert | Nov 2026 | 62% | Not in cards |
| EnterpriseOps-Gym 1.1 Agents complete 296 workflows across customer service, human resources, and IT service management. SQL checks grade the resulting database state. | Expert | Nov 2026 | Not in cards | |
| Financial Coaching Bench Models interview a simulated client about a household financial problem and propose a plan. | Expert | Nov 2026 | 76.2% | Not in cards |
| FinDendro Models recall dated US financial facts and identify the limits of their knowledge. Four prompt styles test value recall, self-knowledge, and historical timing. | Expert | Nov 2026 | 0.76 | Not in cards |
| FinDesk-Bench Agents find Treasury figures and use them to handle client requests across ten business services. A model grades API calls and the records agents leave behind. | University | Nov 2026 | | Not in cards |
| FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. | University | Nov 2026 | | Not in cards |
| GDPval v2 (OpenAI) Multistep assignments drawn from real projects in consulting, finance and design, given with the files and messages the team had when the work was assigned. OpenAI has not published the task count or grading. | Expert | Nov 2026 | 50.4% | In use |
| GDPx Agents work long tasks in simulated workplaces across 11 domains, including chip design, cybersecurity, biology, finance, quant trading and recursive self-improvement. Fourteen specialist partners built the tasks, with a default limit of 8 hours each. Score is pass@1 over three attempts, with every domain weighted equally. | Expert | Nov 2026 | 16.4% | Not in cards |
| GPBam Models write legal opinions on German public law cases and retrieve the statutes they need. Three judges grade essays against reference solutions. | Expert | Nov 2026 | 76.4 | Not in cards |
| Gulf Text-to-SQL Bench Models write SQL for the same 30 business questions in English, Modern Standard Arabic, and Gulf Arabic. The harness executes each query. | University | Nov 2026 | | Not in cards |
| HalachaBench Models answer 60 Hebrew rabbinical qualification exam questions. A model judge awards rubric credit on a scale from 0 to 100. | Expert | Nov 2026 | 85.8 | Not in cards |
| HireWatch Models select questions for job interviews under different workplace pressures. The benchmark measures how often they select questions classified as unlawful under US employment law. | University | Nov 2026 | 0.05% | Not in cards |
| How frontier models miss risk in routine enterprise work Agents handle workplace requests whose risks appear in the prompt or in supporting files. Judges check whether the agent completes an unsafe action. | Expert | Nov 2026 | Not in cards | |
| Invoice Approval (Ali Siala) Agents process 60 supplier invoice work orders under a German company’s approval policy. The evaluator compares the resulting database with a reference state. | University | Nov 2026 | Not in cards | |
| JFinBench Models answer questions from six Japanese certification exams in accounting, tax, financial planning, and banking. The authors report scores by exam. | University | Nov 2026 | 97.3% | Not in cards |
| Legal Information Retrieval Benchmark Search APIs retrieve the official document behind legal research requests. Exact URL matching scores whether the official page appears first or in the top ten. | Expert | Nov 2026 | Not in cards | |
| Legal World sample Agents act as a legal assistant in an anonymized law practice. Four tasks grade the drafts, mail, calendar changes, and files they leave behind. | Expert | Nov 2026 | Not in cards | |
| LevelBench-100 v5.0 Models use ledger exports and supporting workbooks to close small business accounts. They reconcile transactions, apply accounting rules, and identify missing records. | Expert | Nov 2026 | | Not in cards |
| Moneda AdviceBench Models answer personal finance requests and paired education questions. Moneda’s default board scores paired pass rate on 50 EU personal requests with education twins. | Expert | Nov 2026 | Not in cards | |
| Moneda Bench Models answer finance questions across seven separate benchmarks. AdviceBench, CiteBench, FilingsBench, GroundBench and PortfolioBench have their own records. KnowledgeBench and MarketBench have no published results yet. Moneda publishes no combined score. | Expert | Nov 2026 | Not in cards | |
| Moneda CiteBench Models write company research notes with linked sources. Moneda scores the weighted share of linked claims that the cited pages support across 100 notes. | Expert | Nov 2026 | Not in cards | |
| Moneda FilingsBench Models answer questions about SEC filings without receiving documents. Moneda scores accuracy on the private questions using filing data and judges for segment questions. | Expert | Nov 2026 | Not in cards | |
| Moneda GroundBench Models answer live questions about market figures. Moneda’s default board reports the share of correct answers against reference data, with incorrect answers reported separately. | Expert | Nov 2026 | Not in cards | |
| Moneda PortfolioBench Models solve hypothetical portfolio tasks involving rebalancing, risk, position sizing, and constraints. Moneda’s default board scores accuracy on 150 tasks with fixed answers. | Expert | Nov 2026 | Not in cards | |
| NAFD E1 Financial Reporting Models sit professional finance exams on topics including financial reporting, planning, and taxation. A grader compares their answers with the official keys. | University | Nov 2026 | 0.93 | Not in cards |
| OfficePrism Agents resolve Chinese workplace tickets using one business system at a time. The evaluator checks outcomes across 420 tasks in seven business functions. | University | Nov 2026 | 76.4% | Not in cards |
| OpenLegalRetrieval-DE Models retrieve German statutes and court decisions across seven tasks. Each task scores where relevant documents appear in the first ten results. Each task carries equal weight. | University | Nov 2026 | | Not in cards |
| Parley Labs negotiation tasks Agents negotiate contracts and plan around market changes with scripted counterparties. An offline verifier grades outcomes. Ten public development tasks accompany private scored sets. | Expert | Nov 2026 | Not in cards | |
| PWS-Legal Web research systems answer dated questions about companies, laws, regulations, intellectual property, court decisions, and legal documents. | Expert | Nov 2026 | Not in cards | |
| Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. | Expert | Nov 2026 | | Not in cards |
| SG-LegalClarify Models answer Singapore legal questions and ask for missing facts. Three model judges grade each conversation against rubrics written by lawyers. | Expert | Nov 2026 | 77.8% | Not in cards |
| SMB Bench An agent runs the buying desk of a simulated fresh-produce wholesaler for a 90-day quarter, using SQL records, supplier quotes and memory files that are its only carryover between days. Scored on mean net profit per quarter across 18 tasks, compared with a fixed buying rule. | University | Nov 2026 | $7,749 | Not in cards |
| StrataBench Land v1 Models classify land instruments, extract lease terms, and distinguish petroleum terms in three separate benchmarks. Classify and Extract have their own records. Define publishes three measures without one headline score. | Expert | Nov 2026 | 0.94% | Not in cards |
| StrataBench Land v1 Classify Models identify the legal act performed by each of 66 synthetic land instruments. Collide scores label accuracy under open and closed prompt protocols. | Expert | Nov 2026 | 93.9% | Not in cards |
| StrataBench Land v1 Extract Models extract ten shared lease fields from 628 synthetic instruments. Collide divides correct field decisions by correct, omitted, and wrong decisions, excluding unscored fields. | Expert | Nov 2026 | | Not in cards |
| Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. | Expert | Nov 2026 | | Not in cards |
| Veris FinAgent Arena v1.1 Agents investigate bank alerts, read Treasury scans, and complete research requests. The index combines accuracy and repeatability across three benchmarks, then applies a cost penalty. | University | Nov 2026 | | Not in cards |
| WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. | Expert | Nov 2026 | 37% | Not in cards |
| AA-Omniscience v1.1 Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. | Expert | Oct 2026 | | In use |
| Agentic Credit Analysis Models extract and standardize income statements and balance sheets. Ramp compares their figures with credit analysts’ figures using a five percent tolerance. | Expert | Oct 2026 | 80.2% | Not in cards |
| AGORA (Agentic Global Open Research Arena) Autonomous AI systems get one secret open finance question, 72 hours and a fixed budget to write a paper. Five finance scholars judge it. | Research frontier | Oct 2026 | Not in cards | |
| AI Professions Prompt Benchmark Models draft everyday work documents from fictional inputs: 100 prompts for tasks in 10 professions, such as accounting, law, healthcare administration and real estate, plus Excel and Microsoft 365, in English, Spanish, French and Portuguese. A judge model from another family checks 5 pass or fail criteria per prompt, fixed before any model ran. | University | Oct 2026 | 85.9% | Not in cards |
| AML Bench (Veris AI) Agents investigate suspicious activity at a simulated bank. A model judge checks whether the recorded service requests and final case disposition satisfy every requirement. | Expert | Oct 2026 | 70.8% | Not in cards |
| AMLworld-Compact Models inspect transaction graphs and estimate whether a marked target is suspicious. The grader thresholds the probabilities and computes a weighted F1 score. | University | Oct 2026 | | Not in cards |
| Argo-Bench Agents query a simulated food delivery company's warehouse and act on their findings. The grader measures the business consequences of their decisions across 210 tasks. | Expert | Oct 2026 | 70.2 | Not in cards |
| Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. | Everyday | Oct 2026 | Not in cards | |
| ATLAS-Finance v1.1.0 Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level. | Expert | Oct 2026 | 10.7% | Not in cards |
| AutoVend Agents buy stock and schedule deliveries for 20 simulated vending machines over 180 days. They share one warehouse and pay for each site visit. | University | Oct 2026 | 309.4 | Not in cards |
| BankerToolBench v1.1 Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes. | Expert | Oct 2026 | 32.6% | Not in cards |
| BizBench Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success. | Expert | Oct 2026 | | Not in cards |
| Business Evidence Judgment v4 Models label business claims as supported, contradicted, or unknown under three document conditions. The grader scores accuracy and compliance separately. | University | Oct 2026 | 90.7% | Not in cards |
| Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. | Expert | Oct 2026 | 37.8% | In use |
| Can AI close the books? FP&A reasoning Models complete synthetic financial close tasks, including accruals and reconciliations. The grader checks calculations and whether reviewer claims agree with the books. | University | Oct 2026 | | Not in cards |
| Cents Matter Models solve Brazilian Portuguese accounting cases in matched pairs. The grader checks amounts to the cent across rounding, cash flow signs, and cases with insufficient evidence. | University | Oct 2026 | | Not in cards |
| ClaimBench Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency. | University | Oct 2026 | Not in cards | |
| Company Enrichment API Benchmark Company data APIs fill profile fields from a website. Code and expert checks score headquarters, employees, founding year, social links, industry, ticker and revenue. | Expert | Oct 2026 | Not in cards | |
| Credit Firewall Benchmark Models decide whether evidence still authorizes a credit decision after roles, targets, timing, or conversation context change. | University | Oct 2026 | 100% | Not in cards |
| CRM-Agent-Bench Agents handle customer and business requests in a simulated enterprise. Executable checks verify tool arguments, action order, and which records changed across 555 tasks. | University | Oct 2026 | | Not in cards |
| DataAnalyticsBench Agents query a synthetic data warehouse to answer business questions. A judge model checks answers against a rubric and an answer key. | University | Oct 2026 | 43.0 | Not in cards |
| DAYJOB: Accounting Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Oct 2026 | | Not in cards |
| DAYJOB: Law Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Oct 2026 | 41.4% | Not in cards |
| DAYJOB: Tax Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria. | Expert | Oct 2026 | 10.8% | Not in cards |
| Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. | Expert | Oct 2026 | | Not in cards |
| DOE sample pack Agents complete five simulated workplace assignments in law, private equity, and finance operations. Deterministic checks grade figures and application state, while model judges grade written work. | Expert | Oct 2026 | Not in cards | |
| DualEntry Accounting AI Benchmark Models use accounting tools to look up accounts, reconcile balances, and draft entries for a test company. DualEntry checks their answers against fixed references. | University | Oct 2026 | | Not in cards |
| EGI-BENCH Agents work across eight enterprise evaluations. The suite combines their results into six scores covering activities such as planning, retrieving evidence, and using tools. | University | Oct 2026 | 65.6 | Not in cards |
| Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. | Expert | Oct 2026 | 81.7% | System cards only |
| Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. | Expert | Oct 2026 | 77.4% | In use |
| FairScreen: Testing LLM Hiring Decisions Models screen job applicants under a stated policy. Paired cases change identity cues, qualifications, or the completeness of the evidence. | University | Oct 2026 | | Not in cards |
| FinGREAT Models identify accounting relationships in financial passages. The authors compare 44 models against human annotations across 483 passages. | University | Oct 2026 | 92.8% | Not in cards |
| FinHopper Agents research financial facts through up to six dependent searches. Intermediate answer keys show where an incorrect company, date, or indicator derails the search. | University | Oct 2026 | 67.3% | Not in cards |
| Harvey LAB v1.1 (HLAB-Vals-1.1) Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes. | Expert | Oct 2026 | | In use |
| Harvey LAB-AA v1.1 Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. | Expert | Oct 2026 | 15% | In use |
| Hebbia Financial Services Benchmark Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion. | Expert | Oct 2026 | 74.6% | Not in cards |
| Hedge Gym Models adjust an options hedge each day over a 21-day price path. The evaluator measures replication error and transaction costs on synthetic or historical markets. | Expert | Oct 2026 | Not in cards | |
| Holdings-IndividualTaxBench 88 real client document packages turned into Form 1040 fields. Fact-level accuracy against practitioner-reviewed returns, with an LLM judge matching values. | Expert | Oct 2026 | 76.3% | Not in cards |
| Hype Trap Models assess anonymized news about small stocks. The grader compares their predictions with stock movements over the next five trading days. | University | Oct 2026 | 0.01 | Not in cards |
| Immigration Bench Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric. | Expert | Oct 2026 | 75.4% | Not in cards |
| IndicIPR-QA Models extract answers from passages in seven Indian intellectual property laws. The benchmark compares predicted spans with reference answers across 4,462 questions. | University | Oct 2026 | | Not in cards |
| InsureBench Decision Models (Broader Comparison) Models read text evidence and return structured findings about insurance scenarios. The scorer compares 150 findings from 50 packets with the author's labels. | University | Oct 2026 | Not in cards | |
| Internal Finance Modeling (Moonshot) Agents complete financial modeling tasks with linked formulas, financial logic and final workbook quality. Experts rate the work. | Expert | Oct 2026 | Not in cards | |
| Internal Financial Research (Moonshot) Agents produce auditable financial research under evidence and cutoff constraints. Experts rate the resulting research. | Expert | Oct 2026 | Not in cards | |
| Investor's Last Exam Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria. | Expert | Oct 2026 | 68.1 | Not in cards |
| Ivo-micro1 ContractBench Models review contracts and return redlines across 350 attorney-written tasks in 35 scenarios and five agreement types. Tasks cover first review and replies to a counterparty. | Expert | Oct 2026 | | Not in cards |
| KMMLU-Pro 2025 Models answer questions from the 2025 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. | University | Oct 2026 | Not in cards | |
| KMMLU-Pro 2026 Models answer questions from the 2026 Korean national professional licensing exams. The dataset uses the KMMLU-Pro evaluation format. | University | Oct 2026 | 78.1% | Not in cards |
| LAB: US Case Law Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings. | Expert | Oct 2026 | Not in cards | |
| Lakh, Crore and Carpet Area Models solve Indian property and finance problems involving local land units, loan payments, and lakh and crore notation. | School | Oct 2026 | 97% | Not in cards |
| Legal Consent Sweep Agents review an acquisition data room, reconcile consent requirements, and prepare notices and closing advice. A separate verifier grades their work. | Expert | Oct 2026 | 0.38 | Not in cards |
| LegalReasonBench Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings. | Expert | Oct 2026 | | Not in cards |
| LegalRewardBench Models choose which of two legal responses is better. The benchmark compares their choices with human-verified preferences across legal knowledge, analysis, decisions, and writing. | Expert | Oct 2026 | 86.8% | Not in cards |
| LevelBench-100 v4.1 Models take over the books of small businesses using QuickBooks exports. They reconcile balances, identify missing evidence, and prepare closing entries across 12 workflows. | Expert | Oct 2026 | 81% | Not in cards |
| LexPilot Models interview a user about a Chinese civil case and predict the ruling on each claim. Paired consultations reveal the effect of missing facts. | Expert | Oct 2026 | 72.1% | Not in cards |
| LiveMACE Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns. | Expert | Oct 2026 | Not in cards | |
| LiveSearch Agents search for current financial figures. The evaluator runs a stored query against a professional data service to update the reference answer. | University | Oct 2026 | | Not in cards |
| Maydan HF RL Sampler Agents reconcile synthetic Saudi social insurance contributions across ten payroll tasks. SQLite checks the final records and audit log. | Expert | Oct 2026 | Not in cards | |
| MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. | Expert | Oct 2026 | 15.8% | Not in cards |
| MultiFinGround Models identify accounting concepts and extract every matching fact from English, French, and Greek financial reports. The benchmark checks their selections against annotated disclosures. | University | Oct 2026 | | Not in cards |
| norregel Models supply the numeric value of a Norwegian legal provision on a specified date. The evaluator distinguishes current values from superseded ones. | University | Oct 2026 | 45.1% | Not in cards |
| OR-License Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem. | Expert | Oct 2026 | 88.4% | Not in cards |
| PARCEL Models decide whether a New York court opinion supports, contradicts, or fails to address a cited claim. The benchmark contains 3,396 labeled claims. | University | Oct 2026 | | Not in cards |
| Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. | Expert | Oct 2026 | In use | |
| PWS-Finance Web research systems answer dated finance questions about company profiles, funding, ownership, markets, regulation, and recent developments. | Expert | Oct 2026 | Not in cards | |
| Quant-Arena Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window. | Expert | Oct 2026 | 51.3 | Not in cards |
| Questflow Finance Benchmark Agents query financial filings and market data with a fixed set of tools. They calculate metrics, rank securities, and submit answers that software checks against frozen references. | University | Oct 2026 | 100% | Not in cards |
| RealWorkDecision Agents review a frozen company workspace and predict the decision an employee made. Expert-written rubrics allow valid alternatives. | University | Oct 2026 | 43.8% | Not in cards |
| ScopeCheck: evidence before numbers Models calculate an operating metric when the data covers the requested population. Paired cases test whether they abstain when essential evidence is missing. | University | Oct 2026 | 0.94 | Not in cards |
| SEC filing claim verification (400 items) Models classify whether an excerpt from a corporate filing supports a claim, contradicts it, or provides insufficient evidence. | University | Oct 2026 | 99.5% | Not in cards |
| Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. | Expert | Oct 2026 | $2,949 | Not in cards |
| SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. | Expert | Oct 2026 | 25.1% | Not in cards |
| SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. | Expert | Oct 2026 | | Not in cards |
| SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. | Expert | Oct 2026 | | Not in cards |
| SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. | Expert | Oct 2026 | | Not in cards |
| SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. | Expert | Oct 2026 | 30.5% | Not in cards |
| Solana Machine Trading Agents write trading bots for hidden market scenarios replayed on offline Solana snapshots. The headline score compares total profit and loss against a reference bot. | Expert | Oct 2026 | 6.11 | Not in cards |
| StoreBench Agents manage a simulated apparel store in Medusa. They order inventory, change prices, ship purchases, and process refunds as demand changes. | University | Oct 2026 | 0.83 | Not in cards |
| Tavily Company Search Benchmark Search providers return evidence for company research questions, and an answer agent responds from those results. A judge scores answer completeness, relevance and freshness. | Expert | Oct 2026 | Not in cards | |
| TreasuryQA Bench Agents search scanned Treasury volumes and read tables to find or calculate a number. A script checks whether the answer falls within the accepted range. | School | Oct 2026 | | Not in cards |
| Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. | Expert | Oct 2026 | 69.5% | In use |
| Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. | Expert | Oct 2026 | | In use |
| Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. | Oct 2026 | 71.3% | In use | |
| Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. | Expert | Oct 2026 | | In use |
| Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. | Expert | Oct 2026 | Not in cards | |
| VisR-RAG-Fin Models retrieve pages from financial documents and answer questions using their contents. The evaluator measures page ranking and checks answers against references. | University | Oct 2026 | Not in cards | |
| Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. | Expert | Oct 2026 | 80% | Not in cards |
| AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. | Expert | Sep 2026 | 1,822 | In use |
| APEX-Agents 1.1 Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero. | Expert | Sep 2026 | | Not in cards |
| ApprenticeBench Agent joins a simulated company and processes 100 accounts-payable bills in Odoo via GUI or API, learning from records and feedback. Cumulative success rate. | University | Sep 2026 | 77% | Not in cards |
| ATLAS-Finance Agents do 100 investment banking tasks inside 13 simulated firms. They gather facts from email, chat, calendar and files over MCP, then build an Excel model graded against expert rubrics. | Expert | Sep 2026 | Not in cards | |
| Australian BAS Reconciliation Models reconcile a synthetic Australian goods and services tax ledger, correct a Business Activity Statement, and identify bookkeeping errors. The grader weighs totals and fault detection. | Expert | Sep 2026 | Not in cards | |
| Beancount Ledger Agents reconcile bank statements, repair a Beancount ledger, and allocate payments to invoices. The evaluator checks balances and payment records against the underlying transactions. | University | Sep 2026 | Not in cards | |
| Biopharma Bench Agents take an employee seat inside one of 12 simulated biopharma companies and complete 71 professional assignments, such as a regulatory assessment, an out-of-specification investigation or an audit package. They must work out which records govern among contradictory files. A rubric grader checks 752 frozen criteria, with code checks for exact numbers. | Expert | Sep 2026 | 65.1% | Not in cards |
| CorpBench Work Agents carry out 100 business workflows across finance, marketing, operations, RevOps and sales, 20 per department, using simulated business systems and tool contracts. Grading checks the final state, the actions taken, and whether the agent finished the job or escalated it correctly. | Everyday | Sep 2026 | 92.7 | Not in cards |
| DataBench v1.1 Agents answer ambiguous business questions and make recommendations from a synthetic company warehouse. Model judges assess the reasoning, supporting evidence, artifacts, and final recommendation against task rubrics. | Expert | Sep 2026 | 70.5% | Not in cards |
| DAYJOB: Finance Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Expert | Sep 2026 | 42.5% | In use |
| DealBench-100 Agents perform 100 investment banking workflows across ten synthetic transactions. Deterministic checks grade calculations, decisions, committed changes, and consistency between models and presentations. | Expert | Sep 2026 | Not in cards | |
| DELTA Models research Dutch legal questions and apply the law to facts. Lawyers judge accuracy and professional usefulness. | Expert | Sep 2026 | 15.0 | Not in cards |
| DI-Bench Agents answer questions using company tables, documents, and business rules. The benchmark checks whether agents apply those rules when computing results. | University | Sep 2026 | 57.9% | Not in cards |
| ERPBench-100 Agents execute 100 business workflows across ten simulated enterprise resource planning systems. Checks grade decisions, calculations, committed state, and handoffs on a 100-point scale. | Expert | Sep 2026 | Not in cards | |
| FinCrime Agent Benchmark Agents screen customers for sanctions, political exposure, and adverse media. A case passes when agents find the relevant alerts and resolve them under the bank's policy. | University | Sep 2026 | 71.1% | Not in cards |
| FinFIRST Agents research financial questions and provide supporting sources. The benchmark grades individual requirements and reports how often an answer meets every requirement. | Expert | Sep 2026 | 71.5% | Not in cards |
| FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. | Expert | Sep 2026 | | In use |
| FinStepBench CorporateValuation Models value companies using financial evidence and calculations. The score is the published CorporateValuation percentage. | Expert | Sep 2026 | 77.3% | Not in cards |
| FinStepBench DeepResearch Models write finance research reports. The score is the published DeepResearch percentage. | Expert | Sep 2026 | 59.1% | Not in cards |
| FinStepBench LiveSearch Models retrieve and verify live financial facts. The score is the published LiveSearch percentage. | Expert | Sep 2026 | | Not in cards |
| Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. | Expert | Sep 2026 | Not in cards | |
| GDP.xlsx Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria. | Expert | Sep 2026 | | Not in cards |
| GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. | Expert | Sep 2026 | 1,846 | In use |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Research frontier | Sep 2026 | 77.9% | In use |
| Insurance Agent Benchmark Models read commercial insurance documents and answer questions about their contents. Cooper compares each model alone and with its document processing system. | University | Sep 2026 | 77.6% | Not in cards |
| JusticeAxis Models assess evidence and produce judgments for 256 criminal cases from 18 countries. The evaluator checks charges and outcomes against lawyer-written references. | Expert | Sep 2026 | | Not in cards |
| Legora BAR Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers. | Expert | Sep 2026 | 106.0 | Not in cards |
| LEXam-hard Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed. | Research frontier | Sep 2026 | | Not in cards |
| LifeSci Diamond Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires. | Expert | Sep 2026 | 20% | Not in cards |
| Long-Horizon Freight AuditBench Challenge Agents price 100 difficult freight invoices and flag incorrect charges. The cases include incomplete information and exceptions to ordinary contract terms. | Expert | Sep 2026 | 10% | Not in cards |
| Long-Horizon Freight AuditBench Production Agents price 200 freight invoices from contracts and carrier documents, then flag billing errors. Loop weights accuracy to reflect a 24 percent billing error rate. | University | Sep 2026 | 16.1% | Not in cards |
| Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. | Expert | Sep 2026 | Not in cards | |
| MM-FinEval Models forecast financial outcomes from earnings call transcripts, slides, and audio. The authors compare 19 models using directional accuracy and absolute error. | University | Sep 2026 | 88% | Not in cards |
| Prosus Vending Bench v2 Agents run six simulated vending machines across three locations for 30 days. They order stock, set prices, and collect cash. The score measures final bank balance in euros. | University | Sep 2026 | 4497.2 | Not in cards |
| Redtape Models assess eligibility for public benefits and identify when a missing fact prevents a decision. PolicyEngine supplies the reference determinations. | Expert | Sep 2026 | 0.79 | Not in cards |
| Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. | Expert | Sep 2026 | | In use |
| TMMLU+ 1.1 Models answer traditional Chinese subject questions from a revised exam set. The revision corrects 197 answers and removes 539 questions from the original release. | University | Sep 2026 | 96.2% | Not in cards |
| Typed Decision Bench Models assign probabilities to decisions about a given state. The benchmark scores accuracy and calibration across five suites. | Expert | Sep 2026 | Not in cards | |
| Vals Index v2.1 Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector. | Sep 2026 | | In use | |
| VAmoS Pro Bench Voice agents handle several utility billing requests in one call. A model checks the spoken amounts and recorded actions against the requirements for each request. | University | Sep 2026 | | Not in cards |
| ViLegalExpert Models retrieve Vietnamese law and answer questions from citizen consultations. Professional answers and expert-verified provisions supply references across 34 legal domains. | University | Sep 2026 | Not in cards | |
| AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. | Expert | Aug 2026 | 62.5% | In use |
| AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. | Research frontier | Aug 2026 | Not in cards | |
| Blobfish Litigation Bench Agents complete 100 tasks in a simulated litigation practice. The grader compares the changes agents make to case records against required outcomes. | Expert | Aug 2026 | Not in cards | |
| Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. | Expert | Aug 2026 | 90.7% | In use |
| CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. | Expert | Aug 2026 | Not in cards | |
| CounselBench-100 Agents investigate legal matters and decide which actions the evidence supports. The verifier checks decisions, changes to case records, and advice across 100 synthetic tasks. | Expert | Aug 2026 | Not in cards | |
| DuMateBench Agents complete real-world productivity and office-work tasks in sandboxed workspaces with tools. Task evaluators check the requested artifact and record a reward. | Expert | Aug 2026 | Not in cards | |
| E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. | University | Aug 2026 | 14.3 | Not in cards |
| Enterprise Task Pareto Frontier (Glean) Models complete 1,000 workplace tasks. A model compares paired answers for correctness, task completion, and readiness for use. Glean estimates preference ratings and cost per task. | University | Aug 2026 | Not in cards | |
| FactoryBench-100 Agents complete 100 manufacturing workflows through simulated ERP, email, spreadsheet, and chat tools. FactoryScore checks their decisions and the state left in each system. | Expert | Aug 2026 | 76.2% | Not in cards |
| FinDeepIndicator Agents construct financial indicators by specifying formulas, retrieving market data, and calculating results. The evaluation checks performance at each step. | University | Aug 2026 | | Not in cards |
| GDPval Extended Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores. | Expert | Aug 2026 | 7.10% | Not in cards |
| GreekBarRetrieval Models retrieve Greek statutory articles for bar exam questions and case facts. The authors compare ranked results with the statutes relevant to each case. | University | Aug 2026 | 0.43 | Not in cards |
| Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. | Expert | Aug 2026 | | In use |
| Insight Consolidation v1 Agents read customer account documents, infer why the customer acted, and cite supporting records. The evaluator combines a model judgment with citation accuracy. | University | Aug 2026 | Not in cards | |
| LAB: Law Firm Knowledge Agents search a simulated law firm’s matter files and deliver work that satisfies legal task criteria. | Expert | Aug 2026 | Not in cards | |
| LedgerBench-100 Agents perform 100 corporate finance tasks through 66 tools in a shared enterprise system. SQL checks grade the changes they make. | Expert | Aug 2026 | Not in cards | |
| Loss Run Extraction (Bevaya) Models extract claims and policy tables from insurance loss reports into JSON. The main metric counts documents where every field matches the reference. | University | Aug 2026 | 85.3% | Not in cards |
| MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. | University | Aug 2026 | 61.3% | Not in cards |
| OfficeQA Pro V2 Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match. | University | Aug 2026 | 54.4% | In use |
| Palmyra X6 Enterprise Workflow Evaluations Agents perform production workflows that require retrieval, tool use, content generation, or delegation. Writer compares capability and cost. | Expert | Aug 2026 | Not in cards | |
| SalesBench-100 Agents reconcile sales evidence, update authorized records, and deliver executive briefs across 100 workflows. Checks grade procedure, system state, and outputs. | Expert | Aug 2026 | Not in cards | |
| Singapore Factuality Models answer questions about Singapore's laws, economy, and public policies. A model judge checks whether each answer contradicts the reference. | University | Aug 2026 | Not in cards | |
| Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. | Expert | Aug 2026 | | Not in cards |
| Vals Index v2 Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration. | Aug 2026 | 69.7% | Not in cards | |
| Agent Liability Gym Agents review case files and write legal memos about decisions made by autonomous agents. The grader checks their findings, evidence, and unsupported claims. | Expert | Jul 2026 | Not in cards | |
| Agentic Trader Agents search for market information and place simulated foreign exchange orders using live prices. Each agent manages its own positions, cash, and margin. | University | Jul 2026 | | Not in cards |
| AI Law Legal Agent Eval Agents review five synthetic export control matters and write risk memos. The grader checks the deliverable, seven findings per matter, and unsupported claims. | Expert | Jul 2026 | Not in cards | |
| AMLBench v0.2 (Superjective) Models decide whether to escalate transaction alerts. The score measures how often they escalate reportable alerts under changes to analyst instructions and case narratives. | University | Jul 2026 | | Not in cards |
| APEX-Accounting Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria. | Expert | Jul 2026 | | In use |
| Bot Analysis Arena Models write trading rules for simulated US stock accounts. Some agents revise their rules after trades. Software tests each revision against the current rules before adoption. | University | Jul 2026 | Not in cards | |
| CLBench-V Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition. | Everyday | Jul 2026 | | Not in cards |
| Cybersecurity and Privacy Legal Agent Eval Agents write legal risk memos for five synthetic privacy and incident reporting matters. The grader checks the completed memo and its coverage of legal issues. | Expert | Jul 2026 | Not in cards | |
| DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. | Expert | Jul 2026 | | In use |
| Employment Law Legal Agent Eval Agents review employment case files and write risk memos. The grader checks whether the memos identify planted legal issues and avoid unsupported findings. | Expert | Jul 2026 | Not in cards | |
| Enterprise Vendor Eval Agents review vendor documents, question an internal IT team, and recommend whether to approve a purchase. The grader checks their decision and identified requirements. | University | Jul 2026 | Not in cards | |
| Enterprise-Bench L1-L2 Agents answer 14 operational questions across connected enterprise records. Tasks require either retrieving facts or combining evidence under business rules. | University | Jul 2026 | 82.9% | Not in cards |
| Excel Modeling Benchmark Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting. | Expert | Jul 2026 | 76.7% | Not in cards |
| Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. | Expert | Jul 2026 | 62.6% | System cards only |
| Financial Statement Review (fs-review) Models review eight sets of financial statements and flag inconsistencies in cash, retained earnings, or the balance sheet. Some sets contain no errors. | University | Jul 2026 | Not in cards | |
| Founder Bench Agents run four matched businesses per model through the acoco platform. The evaluation tracks real advertising, customer attention, and website traffic after seven days. Its headline customer-impressions metric adds ad impressions to organic site visits. | Expert | Jul 2026 | 127902.0 | Not in cards |
| FrontierFinance Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics. | Expert | Jul 2026 | 55.9% | In use |
| Hy-FinModelBench Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published. | Expert | Jul 2026 | | In use |
| LAB: Diligence Agent researches a synthetic M&A virtual data room (thousands of documents, up to ~80M tokens) and drafts a diligence memo scored against planted-issue rubric criteria. | Expert | Jul 2026 | Not in cards | |
| Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. | Expert | Jul 2026 | 43.2% | In use |
| MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. | Expert | Jul 2026 | | Not in cards |
| Nodal Pricing LMP Models calculate locational marginal electricity prices for congested transmission networks. The grader checks their answers against a power flow optimization model. | Expert | Jul 2026 | Not in cards | |
| PolicyBench No-tools estimate of US taxes and benefits for 100 households across 18 programs. Headline metric is answers within $1 of PolicyEngine-US. | Expert | Jul 2026 | 95.2% | Not in cards |
| Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. | Expert | Jul 2026 | 57.5% | Not in cards |
| VAmoS Bench Voice agents answer simulated bank customers and use tools to manage credit cards. A model checks each call against its transcript and recorded actions. | University | Jul 2026 | | Not in cards |
| Verifying the Verifiers (Project APE CRED benchmark) Verifier LLMs must find planted errors (prose-table, paper-code, execution, data) in AI-generated economics papers. Credit needs a verbatim quote of the edit. | Expert | Jul 2026 | 99.5% | Not in cards |
| Artificial Analysis Intelligence Index v4.1 Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents. | Jun 2026 | 61.0 | In use | |
| BigFinanceBench Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference. | Expert | Jun 2026 | 58.8% | In use |
| CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. | Expert | Jun 2026 | $22,148,357 | Not in cards |
| CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. | University | Jun 2026 | Not in cards | |
| FinRED Financial Safety Models answer prompts about financial risks. A finance-specific judge grades each response and reports the percentage that assist harmful conduct. | University | Jun 2026 | 0.15% | Not in cards |
| FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. | University | Jun 2026 | 77.8% | In use |
| GDPval-Rubrics (MiniMax) MiniMax rubric grading of model deliverables on public GDPval tasks, in an environment matched to GDPval-AA. | Expert | Jun 2026 | | In use |
| Hedge-Bench Agents analyze supplied company and investment documents for 102 open-ended financial research tasks. A model judge applies rubrics that check whether the reasoning covers the analytical steps identified by investment professionals. | Expert | Jun 2026 | 31.5 | Not in cards |
| InsureBench Multi-functional LLMs Models read insurance files and answer questions about claims, coverage, and actuarial decisions. InsureThing combines automatic answer checks with rubric grades for judgment tasks. | University | Jun 2026 | 96.3% | Not in cards |
| LAB: Contracts 500 in-house contracting tasks (drafting, redlines, issues lists, escalation memos) across 10 contract categories. All-pass expert-rubric grading. | Expert | Jun 2026 | Not in cards | |
| Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. | Expert | Jun 2026 | | In use |
| LongJud-Bench Agents conduct connected stages of Chinese civil litigation in LegalWorld. The authors compare their legal consultations, court documents, and courtroom advocacy. | Expert | Jun 2026 | 0.66 | Not in cards |
| MacroLens Models use company and economic data to make financial estimates under different scenarios. The authors compare forecasting models with language models and simple baselines. | University | Jun 2026 | Not in cards | |
| PSEBench Models triage patient safety reports under Minnesota rules. They identify reportable events, request missing facts, or abstain when the evidence permits multiple verdicts. | Expert | Jun 2026 | | Not in cards |
| Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. | Expert | Jun 2026 | 76.9% | Not in cards |
| Questflow Traders Arena Agents and people trade on Hyperliquid and Polymarket in a public competition. The leaderboard reports weekly portfolio returns alongside gains and trading volume. | University | Jun 2026 | Not in cards | |
| Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. | Expert | Jun 2026 | 62.1% | Not in cards |
| RedlineBench Agent acts as counsel at one turn of a SaaS contract negotiation and returns tracked-change redlines. Three LLM judges grade attorney-written rubrics. | Expert | Jun 2026 | 50.5% | Not in cards |
| RedlineBench v2 (Fa1zvn) Agents negotiate eight contract terms with a simulated vendor. The grader compares the agreement with the best deal both sides could accept. | University | Jun 2026 | Not in cards | |
| SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. | Expert | Jun 2026 | 34.9% | In use |
| SWM-Bench (Social World Models) Predict how Kalshi and Polymarket belief trajectories shift after news events. 12k+ points across politics, finance and crypto, chronologically split. | Expert | Jun 2026 | Not in cards | |
| TaxCalcBench v2 Models read W-2, 1099 and other PDFs and compute full federal and state returns line by line. A return counts only when every evaluated line is exact. | Expert | Jun 2026 | 86% | Not in cards |
| Turing Agentic Environments Agents use business and consumer apps through graphical interfaces or tools. Turing reports average pass rate across the apps each model attempted. | University | Jun 2026 | 60.2% | Not in cards |
| AlexandriaX 2026 Models translate English dialogue into 13 Arabic varieties, translate financial text between six dialects and mark translation errors across five varieties. | University | May 2026 | Not in cards | |
| AvalancheBench (May study) Agents recover underlying patterns from synthetic enterprise data and report their findings. | Expert | May 2026 | Not in cards | |
| BenGER Models write German legal analyses for 596 exam-style cases and 531 doctrinal reasoning tasks. A rubric judge scores legal conclusions, reasoning, and presentation against expert references. Case scores map to the German 0-to-18 grading scale, with four points required to pass. | Expert | May 2026 | | Not in cards |
| DataPrep-Bench Finance Models prepare training data for the finance domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 59.3 | Not in cards |
| DataPrep-Bench Law Models prepare training data for the law domain. The score is the published downstream average after backbone training. | Expert | May 2026 | 76.4 | Not in cards |
| DGCL Reward Hacking Models decide whether corporate actions comply with Delaware law and provide supporting calculations. Fifty evaluation cases distinguish correct headline decisions from fully correct answers. | Expert | May 2026 | Not in cards | |
| Envisioning Signals Benchmark Models identify early signs of change for an industry brief. Scores combine verifiable evidence, specific claims, recent sources, and coverage across the requested topics. | University | May 2026 | 86.0 | Not in cards |
| ERP-Bench Agents complete 300 procurement and manufacturing workflows in Odoo 19. A constraint solver generates the tasks and optimal solutions, and the verifier checks final business state. | Expert | May 2026 | 43.4% | Not in cards |
| ExpertEval Models answer cases in medicine, finance and law. Practicing professionals write the cases and supply weighted grading rubrics. | Expert | May 2026 | 66.9% | Not in cards |
| Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. | Expert | May 2026 | | System cards only |
| FINESSE-Bench Models answer financial certification questions and interpret trading indicators. The benchmark checks numerical and multiple-choice answers and uses a model judge for free responses. | University | May 2026 | 0.90 | Not in cards |
| Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. | University | May 2026 | 95.7% | In use |
| HarnessAudit-Bench Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors. | Expert | May 2026 | | Not in cards |
| Harvey LAB (Legal Agent Benchmark) Agent works through legal matter files across 24+ practice areas and produces work product. Expert rubric, task passes only if every criterion passes. | Expert | May 2026 | | In use |
| HELM Arabic Enterprise Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute. | Expert | May 2026 | 77.9 | Not in cards |
| High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. | Expert | May 2026 | 4.10% | System cards only |
| IndustryBench Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses. | Expert | May 2026 | | Not in cards |
| JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. | Expert | May 2026 | 61.4% | In use |
| JudgmentBench Models produce legal work for 30 tasks. Practicing lawyers grade individual answers and compare pairs of answers. | Expert | May 2026 | 89% | Not in cards |
| MBABench Agents build complete financial models in Excel from a case prompt and documents. An LLM judge grades accuracy, formulas and formatting on a 0 to 100 scale. | Expert | May 2026 | 31.2 | Not in cards |
| MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. | Competition | May 2026 | | System cards only |
| QF-Bench Agents write and run code for 87 quantitative finance tasks in a network-isolated sandbox with real data. Tasks cover derivatives pricing, fixed income, credit, risk, factor research, backtesting and market microstructure. More than 3,300 assertions check numeric outputs against finance rules such as no-arbitrage bounds and put-call parity. Scored by pass@1 over three runs. | Expert | May 2026 | 61.7% | Not in cards |
| Snowflake Text-to-SQL HARD Models generate SQL for difficult questions about unfamiliar enterprise schemas. Snowflake compares models by query accuracy. | Expert | May 2026 | | Not in cards |
| Vals Finance Agent v2 927 expert-reviewed research questions on public-company filings. Partial credit only after required checks pass. Private test split. | Expert | May 2026 | | In use |
| Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. | Expert | May 2026 | 74.2% | Not in cards |
| Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Expert | May 2026 | 70.7% | Not in cards |
| Agent Bazaar Agents price goods or trade with sellers in simulated markets. The benchmark measures whether their decisions sustain prices, deter fraud, and preserve welfare and profits. | University | Apr 2026 | Not in cards | |
| BankerToolBench 100 end-to-end junior-banker tasks producing Excel, PowerPoint and Word deliverables. Agentic grader scores ~150 weighted expert rubric criteria. | Expert | Apr 2026 | 16% | In use |
| Clash of AIs Season 1 Models publish trade entries, targets, and stops before prices move. Clash of AIs simulates fills against market prices and ranks accounts by total equity. | University | Apr 2026 | $283,903 | Not in cards |
| KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. | University | Apr 2026 | | Fading |
| NEPABench Models process public comments, extract information from federal documents and answer questions about environmental reviews. Each task has its own scoring rules. | University | Apr 2026 | Not in cards | |
| SAHM Models answer Arabic questions on financial standards, Islamic finance, and accounting. The authors report multiple-choice accuracy and grade open-ended answers separately. | University | Apr 2026 | | Not in cards |
| Swedish Benchmarks V1.1 Models answer Swedish language, document, and knowledge questions. The overall score combines normalized task performance and response time. | School | Apr 2026 | | Not in cards |
| $OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. | Expert | Mar 2026 | 42.1% | In use |
| Accounting Autocoding Models select a bookkeeping category for each expense. Ramp checks the first choice against the assigned category. | University | Mar 2026 | | Not in cards |
| BigLaw Bench: Research Agents use search tools to find US case law and give cited answers. The tasks span corporate law, litigation and other practice areas. | Expert | Mar 2026 | Not in cards | |
| BullshitBench V2 Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback. | Everyday | Mar 2026 | 95% | Not in cards |
| Contextual Invoice OCR Models extract invoice fields using a business’s past bills. Ramp counts invoices where every field matches the submitted bill. | University | Mar 2026 | Not in cards | |
| Data Agent Benchmark Agents answer business questions over data split across several database systems, with duplicated and inconsistent records and key facts buried in text. The 104 queries cover 17 datasets and four database systems, based on a study of companies building data agents in six industries. Scores are pass@1 against known answers. | Expert | Mar 2026 | | Not in cards |
| Echo General AI Prediction Leaderboard Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. | Expert | Mar 2026 | | Not in cards |
| EnterpriseOps-Gym Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task. | Expert | Mar 2026 | 45.9% | In use |
| Financial Statement OCR Models extract financial figures from profit and loss statements. Ramp counts values within one percent of a risk operator’s transcription. | University | Mar 2026 | Not in cards | |
| FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. | University | Mar 2026 | 81% | Not in cards |
| FinToolBench Agents answer 295 finance questions by selecting and executing tools from a library of 760 financial interfaces. The evaluation separates tool execution, answer correctness, and mismatches in timeliness, intent, and regulatory domain. Answer checks combine model judging with exact matching. | Expert | Mar 2026 | | Not in cards |
| Fund Smart Routing Agent Models choose a fund for a transaction. Ramp measures how often a user overrides that choice. | University | Mar 2026 | 100% | Not in cards |
| OfficeQA Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. | University | Mar 2026 | 86.3% | In use |
| OfficeQA Pro Models answer the 133 hardest OfficeQA questions by finding tables in U.S. Treasury Bulletin documents and computing exact numeric answers. | University | Mar 2026 | 60.9% | In use |
| Partner Restrictions Compliance Models identify bank partner restrictions that apply to businesses. Ramp scores missed restrictions, coverage across categories, and false positives. | Expert | Mar 2026 | Not in cards | |
| PMBench Agents search product managers’ notes to identify evidence across customer conversations and answer questions about product requirements. | University | Mar 2026 | 46.1 | Not in cards |
| Policy Agent Models approve expenses, reject them, or request human review. Ramp compares their decisions with human expense reviews. | University | Mar 2026 | 100% | Not in cards |
| RetailBench Agents operate a simulated supermarket for up to 180 days. They choose suppliers, order stock, and set prices as goods age and customers respond. | University | Mar 2026 | $24,351 | Not in cards |
| τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. | Everyday | Mar 2026 | | In use |
| AD-Bench Agents answer advertising analytics requests with production-platform tools. A model judge checks final answers against refreshed reference answers and measures coverage of expert tool-call trajectories. Results report success over one or three attempts. | University | Feb 2026 | | Not in cards |
| AgentSettlementBench Models decide whether to release or hold a payment under 20 settlement scenarios. The benchmark tracks accuracy and the cost of approving an unsafe settlement. | Expert | Feb 2026 | Not in cards | |
| BigLaw Bench: Global Models perform legal work specific to the UK, Australia and Spain. Local legal experts designed tasks and grading criteria for each jurisdiction. | Expert | Feb 2026 | Not in cards | |
| Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. | Expert | Feb 2026 | | System cards only |
| DraftNEPABench Agents draft sections of environmental impact statements using technical and regulatory source documents. Model judges score the drafts against rubrics, and specialists assess a sample. | Expert | Feb 2026 | 2.41 | Not in cards |
| Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. | Expert | Feb 2026 | | Fading |
| GreekMMLU Models answer questions originally written in Greek for academic, professional, and government exams across 45 subjects. | University | Feb 2026 | | Not in cards |
| JFinQA Models calculate financial ratios, check consistency, and reason about changes over time using Japanese securities reports. | University | Feb 2026 | Not in cards | |
| LABBench2 PatentQA Models locate and interpret evidence in patent documents. | Expert | Feb 2026 | 90.9% | Not in cards |
| LegalCiteBench Models recover and verify citations from United States judicial opinions without external lookup. The benchmark also tests whether models match cases and correct citation errors. | University | Feb 2026 | 0.78 | Not in cards |
| Malt Freelancer Project Matching Models rank synthetic freelancer profiles against a search query or a project brief. Each task has 200 English prompts and 4,019 profiles across five languages. | University | Feb 2026 | | Not in cards |
| OdysseyArena AI Trading Agents trade under hidden market dynamics. The score is average return relative to initial capital over four trials. | Expert | Feb 2026 | 23.3% | Not in cards |
| Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. | Expert | Feb 2026 | 70% | System cards only |
| ROI Calculator (dsm) Models calculate total and annualized investment returns, decide whether to invest, and explain the decision. The evaluator checks each answer against a bundled reference. | School | Feb 2026 | Not in cards | |
| SalesBench (Insurance Sales) Agents find insurance leads, quote plans, and schedule callbacks in a simulated sales pipeline. The evaluator rewards converted monthly premiums and checks compliance with calling restrictions. | University | Feb 2026 | Not in cards | |
| SDR-Bench Models personalize sales pitches using customer research. A judge compares each pitch with points from successful human outreach. | University | Feb 2026 | 55.8% | Not in cards |
| Snowflake Financial PDF Table Parsing Models reconstruct tables from 50 manually annotated financial documents. Evaluators compare complete tables, table dimensions and individual cells with reference Markdown. | University | Feb 2026 | Not in cards | |
| TerraFirm REIT Models choose which properties to sell from a portfolio of 12 office buildings. The evaluator checks whether the remaining portfolio meets five debt and valuation constraints. | Expert | Feb 2026 | Not in cards | |
| Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. | Everyday | Feb 2026 | | System cards only |
| Vals Finance Agent Agents answer 537 financial analyst questions on public company SEC filings with search and filing tools. A three-vote GPT-5.2 judge grades each final answer. | Expert | Feb 2026 | 64.4% | Fading |
| YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. | University | Feb 2026 | $3,445,495 | In use |
| APEX-Agents 480 long-horizon tasks in banking, consulting and law worlds with docs, sheets, email and chat. Rubric-graded, Pass@1. | Expert | Jan 2026 | 67.2% | In use |
| BizFinBench v2 Models answer expert financial questions grounded in business data. The score is average accuracy on eight offline tasks. | University | Jan 2026 | 61.5% | Not in cards |
| PLawBench Model handles 850 realistic legal consultation, case-analysis and document-drafting tasks. LLM judge scores against about 12,500 expert rubric items. | Expert | Jan 2026 | 69.7% | In use |
| Prediction Arena Agents research events and trade contracts on Kalshi and Polymarket. The leaderboard ranks accounts by cash plus the market value of open positions. | University | Jan 2026 | $138,945 | Not in cards |
| prinzbench Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99. | Expert | Jan 2026 | 91.0 | Not in cards |
| QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. | School | Jan 2026 | | Not in cards |
| QuantEval Models answer quantitative finance questions, solve mathematical problems, and implement trading strategies. The authors report answer accuracy and test the generated code. | University | Jan 2026 | 86% | Not in cards |
| TradeRank Arena Models choose positions in cryptocurrencies and US stocks using shared market data and simulated capital. The leaderboard ranks portfolios by return. | University | Jan 2026 | 0% | Not in cards |
| AA-Briefcase Artificial Analysis knowledge-work evaluation of professional deliverables. Models ranked by Elo. | Expert | 2026 | 1,720 | In use |
| CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. | Research frontier | Dec 2025 | | Not in cards |
| CNFinBench Models answer financial questions and carry out financial workflows. The benchmark includes tasks on expertise, autonomy, and integrity. | Expert | Dec 2025 | | Not in cards |
| COLIEE 2026 Models retrieve Canadian precedents and apply Japanese statutes. The statute retrieval track also requires a correct yes or no answer with sufficient supporting articles. | University | Dec 2025 | Not in cards | |
| Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. | Everyday | Dec 2025 | | Fading |
| CorpusQA 1M Models answer computation-heavy questions whose evidence is spread across hundreds of finance, education, and real estate documents in a 1M-token context. Scored by accuracy. | University | Dec 2025 | | Fading |
| DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. | Expert | Dec 2025 | 47% | Not in cards |
| Finch (FinWorkBench) Agent completes 172 enterprise finance and accounting spreadsheet workflows (384 tasks) from real data. Human plus calibrated LLM-judge grading. | Expert | Dec 2025 | 43% | In use |
| MAWARITH Models identify eligible heirs and calculate each share of an estate from an Arabic case. The MIR-E score gives credit for intermediate decisions. | University | Dec 2025 | Not in cards | |
| MMLU Early Training Models score complete answer texts on selected MMLU questions. The metric compares the correct answer’s log likelihood with the average for incorrect answers. | University | Dec 2025 | Not in cards | |
| Pieverse Agent Prediction Arena Agents trade event contracts on Polymarket with $1,000 each in real money. Pieverse tracks account value as market prices change. | University | Dec 2025 | $981 | Not in cards |
| Research Plan Generation Models write research plans from goals derived from scientific papers. Judges grade each plan against criteria specific to its source paper. | Expert | Dec 2025 | Not in cards | |
| VLegal-Bench Models answer questions and solve legal scenarios under Vietnamese law. Legal experts check that reference answers follow authoritative documents. | University | Dec 2025 | | Not in cards |
| XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. | Expert | Dec 2025 | | Fading |
| AA-Omniscience 41-topic closed-book knowledge test. Net score rewards correct answers, penalizes wrong ones and gives zero for abstentions. | Expert | Nov 2025 | | In use |
| BigLaw Bench: Arena Legal experts compare two answers to a legal task. The benchmark reports pairwise win percentages. | Expert | Nov 2025 | Not in cards | |
| Contract Intelligence Models answer questions about contract terms. Harvey compares answers with attorney reviewed labels. | Expert | Nov 2025 | Not in cards | |
| LexGenius Models answer Chinese legal questions drawn from cases and exams. The authors compare results across areas including legal reasoning, ethics, and judicial practice. | University | Nov 2025 | | Not in cards |
| Noor Models answer filtered academic questions chosen to distinguish early training checkpoints. The benchmark measures progress where standard MMLU scores are noisy. | University | Nov 2025 | Not in cards | |
| PRBench Model answers 1,100 expert-written law and finance questions. LLM judge grades against weighted expert rubrics of 10-30 criteria each. | Expert | Nov 2025 | | In use |
| TalentCLEF 2026 Contextualized Job-Person Matching Models rank candidate resumes for a job description in English or Spanish. Organizers score retrieval within and across languages and compare rankings across gender variants. | University | Nov 2025 | Not in cards | |
| TalentCLEF 2026 Job-Skill Matching with Skill Type Classification Models rank skills for an English job title. The main ranking metric gives core skills more weight than contextual skills. | University | Nov 2025 | Not in cards | |
| Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. | Everyday | Nov 2025 | | In use |
| Vending-Bench Arena Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance. | Expert | Nov 2025 | | Not in cards |
| Alpha Arena Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns. | Expert | Oct 2025 | Not in cards | |
| HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. | Research frontier | Oct 2025 | Not in cards | |
| Korean Canonical Legal Benchmark Models answer Korean legal questions with supporting precedents. The benchmark includes multiple-choice questions and essays with individual grading criteria. | University | Oct 2025 | Not in cards | |
| ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. | Expert | Oct 2025 | 66.2% | In use |
| Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. | Expert | Oct 2025 | 41.3% | In use |
| RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. | University | Oct 2025 | | Not in cards |
| StockBench Agents trade stocks over a fixed evaluation window. StockBench ranks models by the mean of final return, maximum drawdown and Sortino ratio. The score is the board's rank, so a lower score is better. | Expert | Oct 2025 | 1.00 | Not in cards |
| Vals Index v1 Vals AI's first index of agentic work in finance, law and coding, with each sector weighted by its share of U.S. GDP. | Oct 2025 | 72.2% | System cards only | |
| Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Expert | Oct 2025 | 71.1% | Not in cards |
| VLAIR Legal Research Legal AI tools answer 200 US legal research questions. Reviewers score legal accuracy, citation authority, and suitability for sharing. | Expert | Oct 2025 | Not in cards | |
| APEX-1 Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics. | Expert | Sep 2025 | | Not in cards |
| BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. | Expert | Sep 2025 | 64.6 | Not in cards |
| DRBench Agents investigate open-ended business questions across enterprise applications and produce reports. The evaluator checks whether the reports recover relevant information and support their claims. | University | Sep 2025 | 63.8 | Not in cards |
| Financial Retrieval (Daloopa) Chatbots retrieve historical financial figures. Daloopa compares their answers with normalized reference values. | University | Sep 2025 | 91.6% | Not in cards |
| FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. | Expert | Sep 2025 | | In use |
| FinSearchComp Global Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset. | Expert | Sep 2025 | | Dropped |
| FinSearchComp Greater China Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset. | Expert | Sep 2025 | Not in cards | |
| GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. | Expert | Sep 2025 | | In use |
| Hebbia Financial Services Benchmark (2025) Models extract facts, summarize documents and reason about financial information. Hebbia evaluates responses to more than 600 questions. | Expert | Sep 2025 | Not in cards | |
| SCUBA Agents work in a Salesforce sandbox as administrators, sales representatives, or service agents. Evaluators check milestones such as creating accounts and routing support cases. | University | Sep 2025 | 53.9% | Not in cards |
| SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. | Everyday | Sep 2025 | | System cards only |
| Trading Bot Bench Models write Python trading strategies for historical stock prices. The evaluator combines return, Sharpe ratio, and drawdown on held-out scenarios. | University | Sep 2025 | Not in cards | |
| VCBench Models predict venture outcomes from founder profiles. The benchmark emphasizes precision through the F0.5 score. The hub adaptation substitutes synthetic profiles for the original data. | Expert | Sep 2025 | 33.7% | Not in cards |
| Arctic-Extract Business Document Benchmark Models answer questions about business documents and extract information from varied layouts. The internal visual tasks include contracts, signatures and checkboxes. | University | Aug 2025 | 58.5 | Not in cards |
| CaseLaw v2 Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents. | Expert | Aug 2025 | 79.3% | Not in cards |
| LiveTradeBench Agents allocate simulated portfolios across US securities and Polymarket contracts using streamed prices and news. The researchers track portfolio returns over successive trading days. | University | Aug 2025 | 12.1% | Not in cards |
| R-ConstraintBench Models produce feasible operational plans as constraints accumulate. The score gives more weight to harder levels. | Expert | Aug 2025 | 79.2% | Not in cards |
| IFScale Models write business reports while including up to 500 specified keywords. The benchmark measures how many instructions they follow at each density. | University | Jul 2025 | | Not in cards |
| Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. | Expert | Jul 2025 | 88.6% | Fading |
| JamC-QA Models answer multiple-choice questions about Japan, including regional identity, customs, geography, and law. | School | Jul 2025 | | Not in cards |
| KBank-MMLU (Upstage) Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house. | Expert | Jul 2025 | | Fading |
| KMMLU-Pro Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. | Expert | Jul 2025 | | In use |
| KMMLU-Redux Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed. | University | Jul 2025 | 81.1% | Dropped |
| LLM Economist Models set tax rates or choose working hours in a simulated economy. Researchers compare the social welfare achieved under different tax policies. | University | Jul 2025 | Not in cards | |
| Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. | Expert | Jul 2025 | System cards only | |
| TaxCalcBench Models compute complete US federal tax returns line by line from taxpayer data, with no tax engine. A return counts only when every evaluated line is exact. | Expert | Jul 2025 | 62.8% | Not in cards |
| WISER-Search Web research agents answer fresh and business questions that require current sources, multi-hop search, and evidence synthesis. | Expert | Jul 2025 | Not in cards | |
| DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. | Expert | Jun 2025 | Not in cards | |
| EDINET-Bench Models read Japanese financial disclosures to detect fraud, forecast earnings and identify industries. | Expert | Jun 2025 | 0.73 | Not in cards |
| ExpertLongBench Models write documents for 11 professional tasks. A judge compares facts in each document with a reference checklist and scores precision and recall. | Expert | Jun 2025 | | Not in cards |
| Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. | Expert | Jun 2025 | 53.1% | Not in cards |
| FinanceReasoning Models solve financial numerical reasoning problems that need one or more finance formulas. Answers are numbers checked against reference Python solutions. | University | Jun 2025 | 86.9% | System cards only |
| GerLeRB Models rank German federal statute sections for 367 legal questions. Legal experts checked the questions against the cited provisions. | University | Jun 2025 | | Not in cards |
| IITP-Legal Models translate 5,000 English legal sentences into Hindi. The suite includes 100 sentences at each length from five through 54 words. The original ranking uses BLEU. | University | Jun 2025 | | Not in cards |
| MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. | University | Jun 2025 | | Not in cards |
| WMT25 EEA Terminology Models translate English regulatory sentences into Icelandic. The scorer checks whether they use the official Icelandic terms from the European Economic Area agreement. | University | Jun 2025 | | Not in cards |
| xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. | Expert | Jun 2025 | 50.8 | Not in cards |
| xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. | Expert | Jun 2025 | 78.5 | Not in cards |
| BizFinBench Models answer 6,781 Chinese finance queries drawn from real business scenarios. The queries cover numerical calculation, reasoning, information extraction, prediction recognition and knowledge questions, in nine categories. Some tasks are scored against answers and others by IteraJudge, the authors' LLM judging method. | University | May 2025 | 73.9% | Not in cards |
| CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. | Everyday | May 2025 | | Not in cards |
| DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. | Expert | May 2025 | | Fading |
| FinMaster Models complete financial workflows using simulated company records. The benchmark includes single calculations and tasks that require several financial metrics. | University | May 2025 | 73% | Not in cards |
| GreekBarBench Models answer Greek bar exam questions in writing. A model judge grades their use of case facts, statutory articles, and legal analysis. | University | May 2025 | 8.35 | Not in cards |
| LEXam Models answer 7,537 questions from 340 law exams at Swiss, EU and international law courses, in English and German. Open answers are judged against reference answers, and multiple-choice items are exact match. | Expert | May 2025 | 70.2% | Not in cards |
| MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. | University | May 2025 | | Not in cards |
| QIAS 2025 Islamic Inheritance Reasoning Models choose the correct distribution of an estate under Islamic inheritance law. The Arabic test contains 500 beginner questions and 500 advanced questions. | University | May 2025 | Not in cards | |
| SnorkelUnderwrite Agents help simulated small business insurance underwriters across six task types in multi-turn chats, using tools and asking the underwriter for missing facts. | Expert | May 2025 | 91% | Not in cards |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | University | Apr 2025 | | System cards only |
| OAB-Bench Models write answers for the second phase of the Brazilian bar examination. The port averages model judges’ scores against grading guidelines. | Expert | Apr 2025 | | Not in cards |
| TRLawBench Models answer 297 questions on Turkish law in three stages: 97 questions from official ÖSYM legal exams and 200 harder questions across 14 branches of law. Answers are checked against the answer key. A judge model also grades the reasoning from 0 to 10. | University | Apr 2025 | | Not in cards |
| Vals Finance Agent v1 Agents answer 537 financial analyst questions on public company SEC filings using search and filing tools. An LLM judge grades each final answer. | Expert | Apr 2025 | 60.6% | Dropped |
| WISER-Atomic Web research systems answer 121 customer-inspired questions across finance, sales, recruitment, and other domains. | Expert | Apr 2025 | 69% | Not in cards |
| HELM Capabilities Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public. | University | Mar 2025 | 81.9 | Not in cards |
| Pre-Flight Models answer questions about aviation regulations, dispatch, pilot procedures and airport operations. The evaluator compares each choice with the answer key. | Expert | Mar 2025 | 84% | Not in cards |
| Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. | School | Mar 2025 | System cards only | |
| Snowflake Financial Filings RAG Evaluation Systems answer about 500 questions over 23,000 SEC filings. Annotators identify supporting passages, pages and answers. | Expert | Mar 2025 | Not in cards | |
| FailSafeQA Models answer finance questions over long documents while the query or the document is degraded, irrelevant or missing. An LLM judge scores robustness, context grounding and compliance. | Expert | Feb 2025 | | Not in cards |
| MMLU-Redux 2.0 Spanish Models answer corrected MMLU questions in Spanish across 57 subjects. The evaluation compares each selected option with the answer key. | University | Feb 2025 | Not in cards | |
| NitiBench Models retrieve Thai statutes and answer legal questions. The authors score retrieval separately and use a model judge to check coverage and contradictions. | University | Feb 2025 | 70.3 | Not in cards |
| SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. | Expert | Feb 2025 | | In use |
| Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. | Expert | Feb 2025 | Not in cards | |
| Vending-Bench Agents run a simulated vending machine business over long horizons, ordering stock, setting prices, and paying daily fees. Scoring is final net worth in dollars. | Everyday | Feb 2025 | $4,694 | Dropped |
| COLIEE 2025 Models retrieve Canadian cases and answer Japanese bar exam questions using statutes. A new pilot asks which tort claims succeed and which arguments courts accept. | University | Jan 2025 | Not in cards | |
| LLMzSzŁ Models answer Polish academic and professional exam questions drawn from the Central Examination Board archives. | University | Jan 2025 | | Not in cards |
| Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. | University | Jan 2025 | | In use |
| SnorkelFinance ReAct agents answer expert-verified financial questions built from 10-K filings, using SQL, code and other MCP tools without seeing the documents. A judge scores full traces. | University | 2025 | 81% | Not in cards |
| Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. | University | Dec 2024 | 96% | Dropped |
| Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. | University | Dec 2024 | 95.7% | Dropped |
| LegalBench.PT Models answer questions about Portuguese law derived from university exams. The benchmark includes case analysis, multiple selection, and true-or-false questions. | University | Dec 2024 | 85.4% | Not in cards |
| MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. | University | Dec 2024 | 73.4% | Not in cards |
| BigLaw Bench Retrieval Retrieval systems find the most relevant passages in legal documents. Harvey compares retrieval results with attorney annotations. | Expert | Nov 2024 | Not in cards | |
| CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. | Everyday | Nov 2024 | 64.3% | Not in cards |
| INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. | University | Nov 2024 | 89.8% | Fading |
| MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. | University | Nov 2024 | 94.1% | Dropped |
| PenguinScrolls Models answer user-style questions over long natural documents such as books, financial reports and legal papers, up to 128K tokens. Tasks cover extraction, localization, qualitative analysis and numerical reasoning. | Everyday | Nov 2024 | | System cards only |
| AMELIA Models classify passages from Italian tax decisions as premises or conclusions. They distinguish factual from legal premises and identify the reasoning behind legal premises. | University | Oct 2024 | | Not in cards |
| ChouBun Models answer questions about long Japanese documents and summarize company securities reports. | University | Oct 2024 | | Not in cards |
| KBL Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items. | Expert | Oct 2024 | | In use |
| Long-document QA (Kensho) Models answer questions by reasoning over multi-hundred-page financial filings and other long documents. | Expert | Oct 2024 | Not in cards | |
| MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. | University | Oct 2024 | Not in cards | |
| Multilingual Entity Linking of Occupations Models map occupational titles from national classifications to European occupation codes. The benchmark scores the rank of the first correct match. | University | Oct 2024 | Not in cards | |
| xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. | University | Oct 2024 | | System cards only |
| ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. | University | Sep 2024 | | Not in cards |
| BigLaw Bench Workflows: SPA Deal Points Models extract deal terms from stock purchase agreements. Harvey measures accuracy against terms extracted and checked by attorneys. | Expert | Sep 2024 | Not in cards | |
| ContextualBench CRM agents answer enterprise questions against contextual business data and receive accuracy, cost, speed, trust and safety measurements. | Expert | Sep 2024 | Not in cards | |
| FinDVer Models verify claims using text and tables in financial reports. The authors score the final true or false judgment across three kinds of reasoning. | University | Sep 2024 | Not in cards | |
| FinMTEB Models represent English and Chinese financial texts as vectors. The suite tests seven abilities, including retrieving definitions, grouping complaints, and comparing financial statements. | University | Sep 2024 | Not in cards | |
| Generating Confusion: Voting with a Disability Models answer questions about accessible voting in the United States. Reviewers measure how often answers to standard voter questions contain incorrect information. | Everyday | Sep 2024 | Not in cards | |
| MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. | University | Sep 2024 | 84.6% | In use |
| TalentCLEF 2025 Multilingual Job Title Matching Models rank job titles by how closely they match a query title within or across languages. Organizers score mean average precision. | Everyday | Sep 2024 | Not in cards | |
| BigLaw Bench Models review legal documents and receive rubric scores for substantive accuracy and editing quality. | Research frontier | Aug 2024 | 91% | Fading |
| MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. | University | Aug 2024 | Not in cards | |
| ComplaintsClustering Models group consumer complaints by financial product. The score measures whether each group contains one product and whether complaints about that product stay together. | Everyday | Jul 2024 | Not in cards | |
| DISCFinLLMReranking Models rank Chinese news articles for questions about financial events. Each question has two or three relevant articles and ten distractors. | University | Jul 2024 | Not in cards | |
| FinEvaReranking Models rank Chinese financial news passages for an investor question. Each question has one matching passage and nine or ten distractors. | University | Jul 2024 | Not in cards | |
| FinFactReranking Models rank fact checking articles for a claim. Each test row includes one designated article and nine or ten distractors. | Everyday | Jul 2024 | Not in cards | |
| FinMTEB FINAL Models compare passages from financial reports. The score measures how well their similarity rankings distinguish revised passages from mismatched passages. | University | Jul 2024 | Not in cards | |
| FiQA2018Reranking Models rank eleven candidate answers to personal finance questions. Each test row has one designated answer. | Everyday | Jul 2024 | Not in cards | |
| HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. | University | Jul 2024 | 78.8% | Not in cards |
| HC3Reranking Models rank answers to personal finance questions. Each test question has one relevant human answer and one relevant ChatGPT answer. | Everyday | Jul 2024 | Not in cards | |
| PiiClustering Models group synthetic English financial documents by document type. The labels include bank statements, payment confirmations, and insurance claim forms. | Everyday | Jul 2024 | Not in cards | |
| TheGoldman Retrieval Models retrieve the Goldman Sachs dictionary definition for a financial term. The score rewards placing the correct definition near the top of the first ten results. | Everyday | Jul 2024 | Not in cards | |
| WikiCompany2IndustryClustering Models group company descriptions by industry. The score measures how well the groups recover the industry labels. | Everyday | Jul 2024 | Not in cards | |
| CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. | Everyday | Jun 2024 | 0.20 | Not in cards |
| MMLU-Redux Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy. | University | Jun 2024 | | Fading |
| MMLU-SR Models answer MMLU questions after key terms are replaced with synonyms. The benchmark tests whether models recognize the same concept under different wording. | University | Jun 2024 | 65% | Not in cards |
| Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. | University | Jun 2024 | | Not in cards |
| CFLUE Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation. | University | May 2024 | | Not in cards |
| CFLUE Application Models complete Chinese financial language tasks. The score is the published average of the application task metrics. | University | May 2024 | | Not in cards |
| CFLUE Knowledge Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test. | University | May 2024 | | Not in cards |
| Japanese Tort-case Dataset Models predict whether a Japanese court found a tort from the facts and the parties' arguments. They also label which arguments the court accepted. | Expert | May 2024 | Not in cards | |
| NIFTY Market Movement Models use financial news and market history to predict whether the market will rise, fall or remain neutral. | University | May 2024 | Not in cards | |
| Finance Fundamentals Models answer curated finance questions covering quantitative reasoning, financial domain knowledge and document-based questions. | University | Apr 2024 | Not in cards | |
| Snowflake Arctic Enterprise Intelligence Models write code, generate SQL and follow instructions. Snowflake averages the coding, SQL and instruction-following scores. | University | Apr 2024 | Not in cards | |
| SuperCLUE-Fin Models answer Chinese finance questions across 25 tasks in six categories. The evaluation covers financial knowledge and its application in practice. | University | Apr 2024 | 88.7 | Not in cards |
| FinSTS Models compare statements from different annual reports by the same company and score whether their financial meaning changed. People labeled 370 test pairs. | University | Mar 2024 | Not in cards | |
| GerDaLIR Small Models match German court passages to cited decisions. This version keeps only decisions that answer at least one test query. VAGO scores the top ten results with nDCG@10. | University | Mar 2024 | | Not in cards |
| NTCIR-18 MedNLP-CHAT Models flag medical, legal, and ethical risks in answers to patient questions. The organizers compare the labels with expert judgments using macro F1. | Expert | Mar 2024 | Not in cards | |
| FinBen A finance suite of 35 datasets across 23 tasks, in three difficulty levels. Tasks include sentiment analysis, information extraction, numerical question answering, text generation, forecasting and stock trading. | University | Feb 2024 | 0.25 | Not in cards |
| GerLayQA Models find German Civil Code sections that answer questions from the public. The authors also evaluated generated answers against lawyers' answers. | University | Feb 2024 | | Not in cards |
| KMMLU Models answer 35,030 Korean multiple-choice questions across 45 subjects, collected from original Korean exams. Scoring is accuracy. | University | Feb 2024 | | Fading |
| Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. | University | Feb 2024 | | Dropped |
| Artificial Analysis Intelligence Index v1 Artificial Analysis's first composite score, called the Quality Index in 2024, which normalized and combined a few public benchmarks into one number. | Jan 2024 | Not in cards | ||
| CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. | University | Jan 2024 | | Not in cards |
| Economics Arena for Large Language Models Models choose numbers in beauty contests and place sealed bids in auctions. Researchers compare payoffs and measure how strategies change when players see previous rounds. | University | Jan 2024 | 0.95 | Not in cards |
| Enterprise Scenarios Leaderboard Models complete six tasks drawn from workplace settings. The leaderboard combines their scores across all six tasks. | University | Jan 2024 | Not in cards | |
| KMMLU-Hard Models answer about 5,000 Korean multiple-choice questions from KMMLU that most models got wrong, across many expert subjects. | University | Jan 2024 | 30.5% | Not in cards |
| Norwegian Courts Bitext Mining Models match court sentences written in Norwegian Bokmål and Nynorsk. The test contains 228 aligned pairs and scores matches with F1. | School | Jan 2024 | | Not in cards |
| OAB Exams Models answer Portuguese multiple-choice questions from the Brazilian bar examination. | University | Jan 2024 | Not in cards | |
| OpenFinData Models answer Chinese financial questions, extract entities and values, and check calculations or data consistency. OpenCompass scores answer accuracy, using required-keyword matching for entity extraction. | University | Jan 2024 | Not in cards | |
| HELM Lite A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014. | University | Dec 2023 | 93.8 | Not in cards |
| SuperCLUE-Auto Models answer Chinese automotive questions through multiple turns. A judging model scores each answer against criteria for its task. | University | Dec 2023 | 76.6 | Not in cards |
| TMMLU+ Models answer traditional Chinese questions across 66 subjects. The evaluation includes elementary material and professional examinations. | University | Dec 2023 | 88.6% | Not in cards |
| FinanceBench Models answer open-book questions about public companies using their filings, such as 10-Ks and earnings reports. The full set has 10,231 questions with answers and evidence strings, and 150 of them are public. The makers graded answers by hand, and later users often grade with a model judge. | Expert | Nov 2023 | | Not in cards |
| FinMem Trading Agents choose whether to buy, sell or hold a stock each day. The benchmark measures cumulative return against historical prices. | University | Nov 2023 | 1.63% | Not in cards |
| Insider Trading Demo Models act as stock traders under pressure. Apollo measures how often they trade on inside information and conceal their actions. | University | Nov 2023 | 4.67% | Not in cards |
| MegaOath Forecasting Survey Models predict how each of 21 honesty oaths changes the share of earned income people report in a tax game. Researchers score correlation with observed effects. | University | Nov 2023 | Not in cards | |
| MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. | University | Nov 2023 | | In use |
| COLIEE 2024 Models retrieve Canadian precedents and supporting paragraphs. They also retrieve Japanese Civil Code articles and answer yes or no on bar exam questions. | University | Oct 2023 | Not in cards | |
| IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. | School | Oct 2023 | | Dropped |
| AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. | University | Sep 2023 | | Dropped |
| CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. | Expert | Sep 2023 | 83.3 | Not in cards |
| Fin-Eva Models answer 13,106 Chinese multiple-choice questions about financial knowledge, calculations, analysis, and compliance. The evaluator checks the chosen option against the answer key. | Expert | Sep 2023 | Not in cards | |
| Fin-Fact Models judge whether evidence supports a financial claim, contradicts it, or leaves too little information. The original evaluation also tests models with images. | University | Sep 2023 | 61% | Not in cards |
| LawBench Models answer Chinese legal questions across 20 tasks, including statute recall and case prediction. Each task has its own scoring rule. | University | Sep 2023 | 44.5 | Not in cards |
| FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. | University | Aug 2023 | | Not in cards |
| FinEval 6.0 Models solve financial evaluation tasks. The score is the official 6.0 total score. | University | Aug 2023 | | Not in cards |
| LegalBench Models do 162 legal reasoning tasks written by legal professionals, covering six types of legal reasoning such as issue spotting and rule application. | Expert | Aug 2023 | 88.6% | Not in cards |
| News With GPT Instructions Models classify financial news by sentiment. ChatGPT supplied seven sentiment labels and explanations. The public test contains 4,047 news items. | Everyday | Jul 2023 | | Not in cards |
| Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. | University | Jul 2023 | | Dropped |
| CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. | University | Jun 2023 | | Fading |
| SuperCLUE The CLUE team tests Chinese and foreign language models on periodic sets of Chinese questions covering language, professional knowledge, reasoning, agents and safety. Open answers are graded by a strong LLM judge, and results feed an overall ranking. | University | May 2023 | | Not in cards |
| TheoremQA Models solve 800 university-level questions in math, physics, EE and CS, and finance that each need a specific theorem. Scored by answer accuracy. | University | May 2023 | | Dropped |
| AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. | School | Apr 2023 | | Fading |
| FiQA SA (BloombergGPT protocol) Models classify sentiment toward a named financial target as negative, neutral, or positive. Bloomberg uses five examples and reports weighted F1. | Everyday | Mar 2023 | 0.63 | Not in cards |
| MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. | University | Mar 2023 | 0.07 | System cards only |
| LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. | University | Jan 2023 | Not in cards | |
| MAUD Models classify provisions in merger agreements into expert-defined legal categories. The benchmark compares predictions with lawyer annotations and reports precision-recall area and micro and macro F1. | Expert | Jan 2023 | Not in cards | |
| ConvFinQA Models answer a conversation of related numerical questions about a company's financial report, with tables and text, where later questions build on earlier answers. The dataset has about 3,900 conversations with 14,000 questions, built from FinQA. Answers are checked by execution accuracy against the gold reasoning program. | University | Oct 2022 | 58.7% | Not in cards |
| Demosthenes Argument Classification Models identify premises and conclusions in European Union court decisions on fiscal state aid. They also classify premise types and legal reasoning schemes. | Expert | Oct 2022 | Not in cards | |
| EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. | University | Oct 2022 | Not in cards | |
| KPI-EDGAR Models identify key performance indicators and related values in financial reports. The original annotations also link entities within the text. | University | Oct 2022 | Not in cards | |
| Sayfullina Soft Skill Span Extraction Models mark the words that express a soft skill in English job ads. The test scores exact span matches. | Everyday | Sep 2022 | Not in cards | |
| SkillSpan-ESCO Models link skills mentioned in job advertisements to entries in the European skills classification. Human annotators assigned the reference labels. | University | Sep 2022 | Not in cards | |
| Twitter Financial News Sentiment Models classify financial news posts as bearish, bullish, or neutral. The released validation file contains 2,388 posts. | Everyday | Sep 2022 | Not in cards | |
| RMU:ECHR Models mark argument spans in European Court of Human Rights rulings and identify the type of argument and the party making it. | University | Aug 2022 | Not in cards | |
| TAT-DQA Models answer questions about financial report pages that mix tables and text, often needing arithmetic. Scored by exact match and F1. | University | Jul 2022 | | Dropped |
| Multi-LexSum Models summarize documents from civil rights lawsuits at different lengths. Legal experts wrote the reference summaries. | Expert | Jun 2022 | Not in cards | |
| French Insurance Job Offer Models mark and classify soft skills in French job ads from Canadian insurers. A domain expert annotated 47 ads. | School | Apr 2022 | Not in cards | |
| ContractNLI Models decide whether a contract supports, contradicts, or omits a claim. The original benchmark also asks them to identify the passages that justify their decision. | Expert | Oct 2021 | Not in cards | |
| GerDaLIR Models find the German court decisions cited by a passage after the authors mask its references. The original test split has 12,298 queries. | University | Oct 2021 | Not in cards | |
| LexGLUE Models solve a suite of English legal text tasks drawn from court cases, EU law, contracts and terms of service. Most tasks are classification, and one asks for the holding of a US case. Each task reports micro and macro F1. | Expert | Oct 2021 | Not in cards | |
| Swiss-Judgment-Prediction Models predict whether the Swiss Federal Supreme Court accepts or dismisses a case. | Expert | Oct 2021 | Not in cards | |
| FinQA Models answer numerical questions about company earnings reports that mix tables and text. Financial experts wrote 8,281 question and answer pairs, each with a gold reasoning program. Answers are checked by execution accuracy and program accuracy. | University | Sep 2021 | Not in cards | |
| Greek Legal Code Models classify Greek legal texts by subject using their place in the Greek Legal Code hierarchy. | University | Sep 2021 | | Not in cards |
| JobBERT Job Title Normalization Models match English vacancy titles from Malaysia’s public job board to standardized occupation labels. The authors report reciprocal rank and recall at several cutoffs. | University | Sep 2021 | Not in cards | |
| MultiEURLEX Models assign subject labels to European Union legislation written in different languages. | Expert | Sep 2021 | | Not in cards |
| RAFT (Real-world Annotated Few-shot Tasks) Models learn text classification tasks from small labeled samples. The tasks use real documents, including customer messages, court decisions, and scientific abstracts. | University | Sep 2021 | 62.7% | Not in cards |
| LegalQuAD Models find the German legal document that answers each of 200 questions. The original data also mark the answer text and its starting position. | University | Aug 2021 | | Not in cards |
| BVA Case Citation Dataset Models recommend the next legal citation in a veterans’ appeal decision. The authors measure how often the recorded citation appears among the top recommendations. | Expert | May 2021 | Not in cards | |
| TAT-QA Models answer 16,552 questions about 2,757 passages from real financial reports that mix a table with text. Many answers need arithmetic, counting or comparison. Scored by exact match and F1. | University | May 2021 | Not in cards | |
| BEIR FiQA-2018 Retrieval Models rank 57,638 answer posts for 648 investment questions. BEIR sampled its test questions from the original FiQA training split. | Everyday | Apr 2021 | | Not in cards |
| CaseHOLD Models read a passage from a US court decision with the holding of a cited case masked out and pick the correct holding from five choices. The dataset has more than 53,000 questions drawn from US case law. | Expert | Apr 2021 | Not in cards | |
| Earnings-21 Models transcribe long earnings calls. The benchmark tests recognition of financial terminology in real conversations. | Everyday | Apr 2021 | Not in cards | |
| SPGISpeech Models transcribe English financial recordings. The references include formatting such as capitalization and punctuation. | Everyday | Apr 2021 | | Not in cards |
| CUAD Models review commercial contracts and highlight the clauses a lawyer would flag, across 41 categories such as governing law, non-compete and change of control. Legal experts from The Atticus Project wrote more than 13,000 labels over 510 contracts. Scores measure how well the extracted spans match the expert labels. | Expert | Mar 2021 | Not in cards | |
| ECHR Legal Corpus for Argument Mining Models identify argumentative clauses, link related clauses, and distinguish premises from conclusions in European Court of Human Rights rulings. | University | Dec 2020 | Not in cards | |
| AILA Statute Retrieval Models retrieve the Indian statutes that apply to a described legal situation. | University | Oct 2020 | | Not in cards |
| Legal Opinion Sentiment Classification Models classify the sentiment expressed in passages from legal opinions. Expert annotations provide the reference labels. | University | Oct 2020 | Not in cards | |
| Gold Commodity News Models classify news headlines about gold. Labels distinguish price direction, timing, and comparisons with other assets. | School | Sep 2020 | Not in cards | |
| MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. | University | Sep 2020 | | In use |
| German Argument Mining Models label sentences from Bavarian court judgments as conclusions, definitions, applications of law to facts, or other text. The authors report F1 and accuracy. | University | Jul 2020 | Not in cards | |
| RCTs to Scale nudge forecasts Models predict how each of 14 government interventions changes participation or compliance relative to a control. Researchers correlate predictions with observed changes in percentage points. | University | Jul 2020 | Not in cards | |
| BillSum Models summarize legislative bills. The dataset pairs bill text with summaries written for the US Congress and the California legislature. | University | Nov 2019 | Not in cards | |
| ECHR Binary Violation Classification Models read the facts of a European Court of Human Rights case and predict whether the court found a violation. | Expert | Jun 2019 | Not in cards | |
| Plain English Summarization of Contracts Models summarize contract clauses in plain English. Human summaries provide references for evaluating how well each response conveys the legal language. | University | Jun 2019 | Not in cards | |
| chABSA Models identify the target and sentiment of opinions in the business results sections of Japanese annual reports. | University | Dec 2017 | Not in cards | |
| Expert Forecasts of Real-Effort Experiment Models predict how incentives change the number of alternating key presses workers make in ten minutes. Researchers compare predictions with observed effort across 15 conditions. | University | Jul 2015 | Not in cards | |
| Financial PhraseBank Models classify the financial sentiment of a news sentence as positive, negative, or neutral. | University | Jul 2013 | | Not in cards |
| AP Macroeconomics Models take a simulated AP Macroeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | 5.00 | System cards only | |
| AP Microeconomics Models take a simulated AP Microeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | School | 5.00 | System cards only | |
| CompanyBench Agents complete financial analyst workflows in a simulated bank. Checks compare outputs and database changes with expected results. | Expert | 44.4% | Not in cards | |
| Contract Workflows (Applications) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. | Expert | Not in cards | ||
| Contract Workflows (Models) Systems complete legal work involving contracts. The benchmark assesses whether each deliverable satisfies its legal and presentation criteria. | Expert | 48.4% | Not in cards | |
| CorpFin v2 Models answer expert-written questions about credit agreements that often exceed 200 pages. An LLM judge grades each answer for accuracy. | Expert | 73.2% | In use | |
| CortexRetrievalBench Retrieval systems rank financial documents for a practitioner’s question. The benchmark checks whether an authoritative primary source appears at the top. | Expert | Not in cards | ||
| CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). | Expert | 75.9% | In use | |
| Data Extraction (Applications) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. | Expert | Not in cards | ||
| Data Extraction (Models) Systems extract requested information from legal documents. The benchmark checks whether the resulting work satisfies the task criteria. | Expert | 76.7% | Not in cards | |
| DesignArena Spreadsheet Agents build spreadsheets such as financial models, budgets and trackers as Excel or CSV workbooks. Users compare accuracy, structure and formatting and vote. | Everyday | 1,429 | Not in cards | |
| EKWbench Agents complete operational tasks such as sourcing reviews and referral triage. The score averages the share of verifier weight earned. | Expert | 74.4% | Not in cards | |
| FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. | Everyday | | Not in cards | |
| Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. | Not in cards | |||
| Harvey LAB-AA v1.0 Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed. | Expert | 94.6% | In use | |
| Italian-Administrative-CaseHold Models classify Italian administrative judgments paired with their official holdings by legal subject. | University | | Not in cards | |
| LegalSupport Models read a legal claim and two court case summaries. They choose the case that most directly supports the claim. | University | Not in cards | ||
| MAPA EUR-Lex Models mark sensitive entities in EU legal texts. MAPA annotates general and fine-grained categories. The multilingual conversion covers 21 languages. | University | Not in cards | ||
| MBE (Multistate Bar Examination) Models answer the 200 multiple choice questions of an official Multistate Bar Examination practice exam. Scored by accuracy. | Expert | 85% | System cards only | |
| Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. | Expert | | System cards only | |
| MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. | Everyday | 72.1% | Not in cards | |
| Open Portuguese LLM Leaderboard Models take Portuguese exams and complete language tasks for entailment, similarity, sentiment, and hate speech detection. The leaderboard aggregates task scores. | University | | Not in cards | |
| RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. | Expert | 94.2% | Not in cards | |
| Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. | Expert | 1,637 | System cards only | |
| Realm Financial Reasoning Agents analyze financial documents and spreadsheets. A model grades their answers against practitioner rubrics. | Expert | | Not in cards | |
| SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. | Expert | 64.7% | Not in cards | |
| SuperCLUE-Law Models respond to Chinese legal questions. | Expert | 92.1 | Not in cards | |
| SuperCLUE-Mkt Models complete tasks in advertising and marketing. | University | | Not in cards | |
| TaxEval v2 Models answer expert-written tax questions. A judge grades answer correctness and stepwise reasoning against expert solutions. | Expert | | Not in cards | |
| Uniform Bar Exam (MBE+MEE+MPT) Models take a simulated Uniform Bar Exam with multiple-choice, essay and performance test parts. Scored on the 400-point scale. | Expert | 298.0 | System cards only |