| Vals Finance Agent v2 927 expert-reviewed research questions on public-company filings. Partial credit only after required checks pass. Private test split. | Finance | May 2026 | 65.4% | In use |
| Vals Finance Agent Agents answer 537 financial analyst questions on public company SEC filings with search and filing tools. A three-vote GPT-5.2 judge grades each final answer. | Finance | Feb 2026 | 64.4% | Fading |
| Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. | Composite work indices | Oct 2026 | 71.3% | In use |
| Vibe Code Bench End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. | Building whole projects | Mar 2026 | 93.8% | In use |
| Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. | Law | Jun 2026 | 56.3% | In use |
| Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. | Finance | Sep 2026 | 52.3% | In use |
| Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. | Composite work indices | Oct 2026 | 68.3% | In use |
| SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. | Capture-the-flag challenges | Aug 2026 | 56.9% | In use |
| CorpFin v2 Models answer expert-written questions about credit agreements that often exceed 200 pages. An LLM judge grades each answer for accuracy. | Finance | | 73.2% | In use |
| Vals Index v2.1 Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector. | Composite work indices | Sep 2026 | 68.9% | In use |
| Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. | Finance | Oct 2026 | 77.4% | In use |
| Vals Finance Agent v1 Agents answer 537 financial analyst questions on public company SEC filings using search and filing tools. An LLM judge grades each final answer. | Finance | Apr 2025 | 60.6% | Dropped |
| Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. | Finance | Oct 2026 | 69.5% | In use |
| SAGE v1.1 (Vals) Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. | Other | Oct 2026 | 91.3% | In use |
| Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. | Composite indices | Oct 2026 | 68.4% | In use |
| ProofBench v1.1 Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts. | Formal proofs | Aug 2026 | 100% | System cards only |
| CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Finding and exploiting real bugs | Oct 2026 | 80.4% | In use |
| Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. | Composite work indices | Oct 2026 | 78.4% | In use |
| MysteryMechanism Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold. | Abstract puzzles and logic | Sep 2026 | 53.1% | System cards only |
| Vals Index v1 Vals AI's first index of agentic work in finance, law and coding, with each sector weighted by its share of U.S. GDP. | Composite work indices | Oct 2025 | 72.2% | System cards only |
| Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Fixing real repositories | Oct 2026 | 86.9% | System cards only |
| TaxEval v2 Models answer expert-written tax questions. A judge grades answer correctness and stepwise reasoning against expert solutions. | Finance | | 80.4% | Not in cards |
| MedScribe Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure. | Medicine | Feb 2026 | 91.4% | Not in cards |
| MedCode Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders. | Medicine | Feb 2026 | 63.6% | Not in cards |
| MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. | Finance | | 72.1% | Not in cards |
| MedQA Bias Evaluation Models answer medical exam questions before and after Vals adds racial information. Vals compares accuracy across patient groups. | Medicine | Jan 2025 | 96.5% | Not in cards |
| SAGE (Vals) Models grade handwritten undergraduate work in PDEs, linear algebra and computer science against rubrics. Scores compare their deductions with expert graders. | Other | Oct 2025 | 56.1% | Not in cards |
| Code Migration Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Fixing real repositories | Jun 2026 | 69.8% | Not in cards |
| Excel Modeling Benchmark Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting. | Finance | Jul 2026 | 76.7% | Not in cards |
| Vals Index v2 Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration. | Composite work indices | Aug 2026 | 69.7% | Not in cards |
| ProofBench Models write Lean 4 proofs of graduate-level theorems, with Mathlib search and code tools for up to 40 turns. Scored by the share that compile. | Formal proofs | Jan 2026 | 78% | Not in cards |
| IOI (Vals) Agents solve International Olympiad in Informatics problems from 2024 and 2025 in C++. A submission tool modeled on the contest grades up to 50 tries per problem by subtask. | Competitive programming | Aug 2025 | 91.7% | Not in cards |
| CaseLaw v2 Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents. | Law | Aug 2025 | 79.3% | Not in cards |
| Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. | Other professions | Jun 2026 | 76.9% | Not in cards |
| IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. | Competitive programming | Oct 2026 | 100% | Not in cards |
| Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. | Composite indices | May 2026 | 74.2% | Not in cards |
| Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Composite work indices | Oct 2025 | 71.1% | Not in cards |
| Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. | AI research tasks | Aug 2026 | 0.38 | Not in cards |
| Financial Coaching Bench Models interview a simulated client about a household financial problem and propose a plan. | Finance | Nov 2026 | 76.2% | Not in cards |
| CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Finding and exploiting real bugs | Jun 2026 | 88.1% | Not in cards |
| Vibe Code Bench 1-100 Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it. | Building whole projects | Sep 2026 | 34% | Not in cards |
| Immigration Bench Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric. | Law | Oct 2026 | 75.4% | Not in cards |
| Access to Justice Benchmark Models interview a simulated client about a civil legal problem, explain their options, and direct them to local help. | Law | Nov 2026 | 76.4% | Not in cards |
| SAFE-Teen Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate. | Wellbeing and sycophancy | Oct 2026 | 6.35% | Not in cards |
| SAFE-Teen Health advice Models respond to simulated teens in health advice scenarios. The score is the published conversation failure rate. | Wellbeing and sycophancy | Oct 2026 | 9.00% | Not in cards |
| SAFE-Teen Relationship boundaries Models respond to simulated teens in relationship boundary scenarios. The score is the published conversation failure rate. | Wellbeing and sycophancy | Oct 2026 | 19% | Not in cards |
| SAFE-Teen Urgent safety Models respond to simulated teens in urgent safety scenarios. The score is the published conversation failure rate. | Wellbeing and sycophancy | Oct 2026 | 15% | Not in cards |
| Vibe Code Bench v1 Agents build web applications from scratch from a natural-language spec in OpenHands. The built apps are tested against functional requirements. | Building whole projects | Nov 2025 | 41.3% | Not in cards |
| Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Composite work indices | May 2026 | 70.7% | Not in cards |
| Vals Smith: torvalds/linux Agents modify torvalds/linux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 74.1% | Not in cards |
| Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. | Video games | Jul 2026 | 91.3% | Not in cards |
| Vals Smith: apache/spark Agents modify apache/spark to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 84.5% | Not in cards |
| Vals Smith: astral-sh/uv Agents modify astral-sh/uv to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 66% | Not in cards |
| Vals Smith: jax-ml/jax Agents modify jax-ml/jax to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 79.6% | Not in cards |
| Vals Smith: oven-sh/bun Agents modify oven-sh/bun to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 54.2% | Not in cards |
| Vals Smith: palantir/blueprint Agents modify palantir/blueprint to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 93.2% | Not in cards |
| Vals Smith: sgl-project/sglang Agents modify sgl-project/sglang to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 78% | Not in cards |
| Vals Smith: vercel/next.js Agents modify vercel/next.js to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 74.3% | Not in cards |
| Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. | Simulated robots | Oct 2026 | 32.7% | Not in cards |
| SEC filing claim verification (400 items) Models classify whether an excerpt from a corporate filing supports a claim, contradicts it, or provides insufficient evidence. | Faithful to sources | Oct 2026 | 99.5% | Not in cards |
| Vibe Code Bench 1-100 v1.1 Agents extend a working web application across successive requests while preserving earlier features. | Building whole projects | Nov 2026 | 55.2% | Not in cards |
| CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. | Video games | Sep 2026 | 19.2% | Not in cards |
| Autoform-Hard Agents complete Lean 4 proofs on 56 formalization tasks that earlier models failed. | Formal proofs | Nov 2026 | 87.5% | Not in cards |
| Evaluating AI Safety in Teen Conversations Chatbots respond to simulated teenagers across ten replies. Judges measure the share of conversations that fail a critical safety check. | Wellbeing and sycophancy | Sep 2026 | 9.70% | Not in cards |
| CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. | Capture-the-flag challenges | Oct 2026 | 87.7% | Not in cards |
| Vals Smith: schubydoo/clauster Agents modify schubydoo/clauster to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 96.7% | Not in cards |
| Vals Smith: sashabaranov/go-openai Agents modify sashabaranov/go-openai to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 96.7% | Not in cards |
| Vals Smith: tensorflow/tensorflow Agents modify tensorflow/tensorflow to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 90% | Not in cards |
| Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. | Deep research | Jul 2026 | 57.5% | Not in cards |
| Vals Smith: arandomhooman/hoomans-morphe-patches Agents modify arandomhooman/hoomans-morphe-patches to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 20% | Not in cards |
| Vals Smith: chrisanderton/tyk-gateway Agents modify chrisanderton/tyk-gateway to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 76.7% | Not in cards |
| Vals Smith: Expertium/fsrs-autoresearch Agents modify Expertium/fsrs-autoresearch to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 96.7% | Not in cards |
| Vals Smith: geopandas/geopandas Agents modify geopandas/geopandas to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 66.7% | Not in cards |
| Vals Smith: goharbor/harbor Agents modify goharbor/harbor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 60% | Not in cards |
| Vals Smith: ladydascalie/core Agents modify ladydascalie/core to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 76.7% | Not in cards |
| Vals Smith: leanprover-community/mathlib4 Agents modify leanprover-community/mathlib4 to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 76.7% | Not in cards |
| Vals Smith: skypilot-org/skypilot Agents modify skypilot-org/skypilot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 30% | Not in cards |
| Vals Smith: suitedaces/dorabot Agents modify suitedaces/dorabot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 66.7% | Not in cards |
| AI Detection Benchmark Systems label private pre-2022 human documents and AI rewrites of them as human or AI written. Both LLMs and commercial detectors are scored by balanced accuracy. | Taste and style | Sep 2026 | 98.4% | Not in cards |
| Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. | Composite work indices | Nov 2026 | 71.2% | Not in cards |
| Vals Smith: arduino/arduino-ide Agents modify arduino/arduino-ide to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 43.3% | Not in cards |
| Vals Smith: AumeshMisra/tensorzero Agents modify AumeshMisra/tensorzero to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Oct 2026 | 86.7% | Not in cards |
| Vals Smith: conjfrnk/blackbird Agents modify conjfrnk/blackbird to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 56.7% | Not in cards |
| Vals Smith: dwojtaszek/zipper Agents modify dwojtaszek/zipper to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 93.3% | Not in cards |
| Vals Smith: fmtlib/fmt Agents modify fmtlib/fmt to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 83.3% | Not in cards |
| Vals Smith: NVIDIA/nccl Agents modify NVIDIA/nccl to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 46.7% | Not in cards |
| Vals Smith: OpenTTD/OpenTTD Agents modify OpenTTD/OpenTTD to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 100% | Not in cards |
| Vals Smith: punt-labs/lux Agents modify punt-labs/lux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 73.3% | Not in cards |
| Vals Smith: vastsa/PI-Desktop Agents modify vastsa/PI-Desktop to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 40% | Not in cards |
| Vals Smith: abhishek203/SfAgent Agents modify abhishek203/SfAgent to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 80% | Not in cards |
| Vals Smith: benredmond/apex Agents modify benredmond/apex to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 63.3% | Not in cards |
| Vals Smith: BerriAI/litellm Agents modify BerriAI/litellm to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 70% | Not in cards |
| Vals Smith: conjfrnk/memovox Agents modify conjfrnk/memovox to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 90% | Not in cards |
| Vals Smith: Effect-TS/effect Agents modify Effect-TS/effect to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 93.3% | Not in cards |
| Vals Smith: flutter/flutter Agents modify flutter/flutter to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 50% | Not in cards |
| Vals Smith: Lemonawa/music163bot-rust Agents modify Lemonawa/music163bot-rust to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 93.3% | Not in cards |
| Vals Smith: mirumee/ariadne Agents modify mirumee/ariadne to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 83.3% | Not in cards |
| Vals Smith: nationalsecurityagency/ghidra Agents modify nationalsecurityagency/ghidra to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 73.3% | Not in cards |
| Vals Smith: pallets-eco/flask-wtf Agents modify pallets-eco/flask-wtf to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Oct 2026 | 100% | Not in cards |
| Vals Smith: poalrom/sharepaste Agents modify poalrom/sharepaste to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 50% | Not in cards |
| Vals Smith: tock/tock Agents modify tock/tock to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Aug 2026 | 86.7% | Not in cards |
| Vals Smith: vllm-project/llm-compressor Agents modify vllm-project/llm-compressor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Sep 2026 | 96.7% | Not in cards |
| ClaimBench Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency. | Finance | Oct 2026 | | Not in cards |
| RollerCoasterTycoonBench Models manage a theme park by choosing weekly actions in RollerCoaster Tycoon 2. | Video games | Nov 2026 | | Not in cards |
| The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. | Deception and alignment | Oct 2026 | | Not in cards |
| Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. | Law | Feb 2025 | | Not in cards |
| Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. | Composite indices | Oct 2026 | | Not in cards |
| Vals Smith: pingdotgg/t3code Agents modify pingdotgg/t3code to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. | Fixing real repositories | Nov 2026 | | Not in cards |
| VLAIR Legal Research Legal AI tools answer 200 US legal research questions. Reviewers score legal accuracy, citation authority, and suitability for sharing. | Law | Oct 2025 | | Not in cards |