Benchmark Almanac

Vals AI

We track 109 benchmarks and 1 arena that Vals AI made or helped make.

15 in use 9 with public code 90 with a leaderboard here

Arena leaderboards

Top five in each of its 1 arena with three or more rated models, the most rated first.

Benchmarks

Name Kind Released Best score Status
Vals Finance Agent v2 927 expert-reviewed research questions on public-company filings. Partial credit only after required checks pass. Private test split. Finance May 2026 65.4% In use
Vals Finance Agent Agents answer 537 financial analyst questions on public company SEC filings with search and filing tools. A three-vote GPT-5.2 judge grades each final answer. Finance Feb 2026 64.4% Fading
Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. Composite work indices Oct 2026 71.3% In use
Vibe Code Bench End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. Building whole projects Mar 2026 93.8% In use
Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. Law Jun 2026 56.3% In use
Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. Finance Sep 2026 52.3% In use
Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. Composite work indices Oct 2026 68.3% In use
SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. Capture-the-flag challenges Aug 2026 56.9% In use
CorpFin v2 Models answer expert-written questions about credit agreements that often exceed 200 pages. An LLM judge grades each answer for accuracy. Finance 73.2% In use
Vals Index v2.1 Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector. Composite work indices Sep 2026 68.9% In use
Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. Finance Oct 2026 77.4% In use
Vals Finance Agent v1 Agents answer 537 financial analyst questions on public company SEC filings using search and filing tools. An LLM judge grades each final answer. Finance Apr 2025 60.6% Dropped
Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. Finance Oct 2026 69.5% In use
SAGE v1.1 (Vals) Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. Other Oct 2026 91.3% In use
Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. Composite indices Oct 2026 68.4% In use
ProofBench v1.1 Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts. Formal proofs Aug 2026 100% System cards only
CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Finding and exploiting real bugs Oct 2026 80.4% In use
Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. Composite work indices Oct 2026 78.4% In use
MysteryMechanism Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold. Abstract puzzles and logic Sep 2026 53.1% System cards only
Vals Index v1 Vals AI's first index of agentic work in finance, law and coding, with each sector weighted by its share of U.S. GDP. Composite work indices Oct 2025 72.2% System cards only
Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. Fixing real repositories Oct 2026 86.9% System cards only
TaxEval v2 Models answer expert-written tax questions. A judge grades answer correctness and stepwise reasoning against expert solutions. Finance 80.4% Not in cards
MedScribe Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure. Medicine Feb 2026 91.4% Not in cards
MedCode Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders. Medicine Feb 2026 63.6% Not in cards
MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. Finance 72.1% Not in cards
MedQA Bias Evaluation Models answer medical exam questions before and after Vals adds racial information. Vals compares accuracy across patient groups. Medicine Jan 2025 96.5% Not in cards
SAGE (Vals) Models grade handwritten undergraduate work in PDEs, linear algebra and computer science against rubrics. Scores compare their deductions with expert graders. Other Oct 2025 56.1% Not in cards
Code Migration Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. Fixing real repositories Jun 2026 69.8% Not in cards
Excel Modeling Benchmark Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting. Finance Jul 2026 76.7% Not in cards
Vals Index v2 Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration. Composite work indices Aug 2026 69.7% Not in cards
ProofBench Models write Lean 4 proofs of graduate-level theorems, with Mathlib search and code tools for up to 40 turns. Scored by the share that compile. Formal proofs Jan 2026 78% Not in cards
IOI (Vals) Agents solve International Olympiad in Informatics problems from 2024 and 2025 in C++. A submission tool modeled on the contest grades up to 50 tries per problem by subtask. Competitive programming Aug 2025 91.7% Not in cards
CaseLaw v2 Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents. Law Aug 2025 79.3% Not in cards
Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. Other professions Jun 2026 76.9% Not in cards
IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. Competitive programming Oct 2026 100% Not in cards
Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. Composite indices May 2026 74.2% Not in cards
Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. Composite work indices Oct 2025 71.1% Not in cards
Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. AI research tasks Aug 2026 0.38 Not in cards
Financial Coaching Bench Models interview a simulated client about a household financial problem and propose a plan. Finance Nov 2026 76.2% Not in cards
CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. Finding and exploiting real bugs Jun 2026 88.1% Not in cards
Vibe Code Bench 1-100 Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it. Building whole projects Sep 2026 34% Not in cards
Immigration Bench Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric. Law Oct 2026 75.4% Not in cards
Access to Justice Benchmark Models interview a simulated client about a civil legal problem, explain their options, and direct them to local help. Law Nov 2026 76.4% Not in cards
SAFE-Teen Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate. Wellbeing and sycophancy Oct 2026 6.35% Not in cards
SAFE-Teen Health advice Models respond to simulated teens in health advice scenarios. The score is the published conversation failure rate. Wellbeing and sycophancy Oct 2026 9.00% Not in cards
SAFE-Teen Relationship boundaries Models respond to simulated teens in relationship boundary scenarios. The score is the published conversation failure rate. Wellbeing and sycophancy Oct 2026 19% Not in cards
SAFE-Teen Urgent safety Models respond to simulated teens in urgent safety scenarios. The score is the published conversation failure rate. Wellbeing and sycophancy Oct 2026 15% Not in cards
Vibe Code Bench v1 Agents build web applications from scratch from a natural-language spec in OpenHands. The built apps are tested against functional requirements. Building whole projects Nov 2025 41.3% Not in cards
Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. Composite work indices May 2026 70.7% Not in cards
Vals Smith: torvalds/linux Agents modify torvalds/linux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 74.1% Not in cards
Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. Video games Jul 2026 91.3% Not in cards
Vals Smith: apache/spark Agents modify apache/spark to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 84.5% Not in cards
Vals Smith: astral-sh/uv Agents modify astral-sh/uv to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 66% Not in cards
Vals Smith: jax-ml/jax Agents modify jax-ml/jax to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 79.6% Not in cards
Vals Smith: oven-sh/bun Agents modify oven-sh/bun to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 54.2% Not in cards
Vals Smith: palantir/blueprint Agents modify palantir/blueprint to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 93.2% Not in cards
Vals Smith: sgl-project/sglang Agents modify sgl-project/sglang to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 78% Not in cards
Vals Smith: vercel/next.js Agents modify vercel/next.js to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 74.3% Not in cards
Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. Simulated robots Oct 2026 32.7% Not in cards
SEC filing claim verification (400 items) Models classify whether an excerpt from a corporate filing supports a claim, contradicts it, or provides insufficient evidence. Faithful to sources Oct 2026 99.5% Not in cards
Vibe Code Bench 1-100 v1.1 Agents extend a working web application across successive requests while preserving earlier features. Building whole projects Nov 2026 55.2% Not in cards
CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. Video games Sep 2026 19.2% Not in cards
Autoform-Hard Agents complete Lean 4 proofs on 56 formalization tasks that earlier models failed. Formal proofs Nov 2026 87.5% Not in cards
Evaluating AI Safety in Teen Conversations Chatbots respond to simulated teenagers across ten replies. Judges measure the share of conversations that fail a critical safety check. Wellbeing and sycophancy Sep 2026 9.70% Not in cards
CyberOffense Agents attack the Nova and Lighthouse networks without human help. The score averages the percentage of objectives completed on each range, weighting both equally. Capture-the-flag challenges Oct 2026 87.7% Not in cards
Vals Smith: schubydoo/clauster Agents modify schubydoo/clauster to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 96.7% Not in cards
Vals Smith: sashabaranov/go-openai Agents modify sashabaranov/go-openai to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 96.7% Not in cards
Vals Smith: tensorflow/tensorflow Agents modify tensorflow/tensorflow to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 90% Not in cards
Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. Deep research Jul 2026 57.5% Not in cards
Vals Smith: arandomhooman/hoomans-morphe-patches Agents modify arandomhooman/hoomans-morphe-patches to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 20% Not in cards
Vals Smith: chrisanderton/tyk-gateway Agents modify chrisanderton/tyk-gateway to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 76.7% Not in cards
Vals Smith: Expertium/fsrs-autoresearch Agents modify Expertium/fsrs-autoresearch to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 96.7% Not in cards
Vals Smith: geopandas/geopandas Agents modify geopandas/geopandas to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 66.7% Not in cards
Vals Smith: goharbor/harbor Agents modify goharbor/harbor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 60% Not in cards
Vals Smith: ladydascalie/core Agents modify ladydascalie/core to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 76.7% Not in cards
Vals Smith: leanprover-community/mathlib4 Agents modify leanprover-community/mathlib4 to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 76.7% Not in cards
Vals Smith: skypilot-org/skypilot Agents modify skypilot-org/skypilot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 30% Not in cards
Vals Smith: suitedaces/dorabot Agents modify suitedaces/dorabot to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 66.7% Not in cards
AI Detection Benchmark Systems label private pre-2022 human documents and AI rewrites of them as human or AI written. Both LLMs and commercial detectors are scored by balanced accuracy. Taste and style Sep 2026 98.4% Not in cards
Vals Inference Provider Index Agents use different inference providers to complete the same finance, coding, and legal tasks. Composite work indices Nov 2026 71.2% Not in cards
Vals Smith: arduino/arduino-ide Agents modify arduino/arduino-ide to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 43.3% Not in cards
Vals Smith: AumeshMisra/tensorzero Agents modify AumeshMisra/tensorzero to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Oct 2026 86.7% Not in cards
Vals Smith: conjfrnk/blackbird Agents modify conjfrnk/blackbird to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 56.7% Not in cards
Vals Smith: dwojtaszek/zipper Agents modify dwojtaszek/zipper to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 93.3% Not in cards
Vals Smith: fmtlib/fmt Agents modify fmtlib/fmt to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 83.3% Not in cards
Vals Smith: NVIDIA/nccl Agents modify NVIDIA/nccl to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 46.7% Not in cards
Vals Smith: OpenTTD/OpenTTD Agents modify OpenTTD/OpenTTD to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 100% Not in cards
Vals Smith: punt-labs/lux Agents modify punt-labs/lux to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 73.3% Not in cards
Vals Smith: vastsa/PI-Desktop Agents modify vastsa/PI-Desktop to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 40% Not in cards
Vals Smith: abhishek203/SfAgent Agents modify abhishek203/SfAgent to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 80% Not in cards
Vals Smith: benredmond/apex Agents modify benredmond/apex to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 63.3% Not in cards
Vals Smith: BerriAI/litellm Agents modify BerriAI/litellm to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 70% Not in cards
Vals Smith: conjfrnk/memovox Agents modify conjfrnk/memovox to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 90% Not in cards
Vals Smith: Effect-TS/effect Agents modify Effect-TS/effect to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 93.3% Not in cards
Vals Smith: flutter/flutter Agents modify flutter/flutter to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 50% Not in cards
Vals Smith: Lemonawa/music163bot-rust Agents modify Lemonawa/music163bot-rust to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 93.3% Not in cards
Vals Smith: mirumee/ariadne Agents modify mirumee/ariadne to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 83.3% Not in cards
Vals Smith: nationalsecurityagency/ghidra Agents modify nationalsecurityagency/ghidra to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 73.3% Not in cards
Vals Smith: pallets-eco/flask-wtf Agents modify pallets-eco/flask-wtf to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Oct 2026 100% Not in cards
Vals Smith: poalrom/sharepaste Agents modify poalrom/sharepaste to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 50% Not in cards
Vals Smith: tock/tock Agents modify tock/tock to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Aug 2026 86.7% Not in cards
Vals Smith: vllm-project/llm-compressor Agents modify vllm-project/llm-compressor to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Sep 2026 96.7% Not in cards
ClaimBench Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency. Finance Oct 2026 Not in cards
RollerCoasterTycoonBench Models manage a theme park by choosing weekly actions in RollerCoaster Tycoon 2. Video games Nov 2026 Not in cards
The Impossible Refund Assistants pursue a late train refund in a simulated email environment. Separate judges rate deception and other behaviors from the messages. Deception and alignment Oct 2026 Not in cards
Vals Legal AI Report Legal AI tools perform seven kinds of legal work. Vals compares their task scores with scores from practicing lawyers. Law Feb 2025 Not in cards
Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. Composite indices Oct 2026 Not in cards
Vals Smith: pingdotgg/t3code Agents modify pingdotgg/t3code to complete software changes from merged pull requests. Hidden tests check the changes and test for regressions. Fixing real repositories Nov 2026 Not in cards
VLAIR Legal Research Legal AI tools answer 200 US legal research questions. Reviewers score legal accuracy, citation authority, and suitability for sharing. Law Oct 2025 Not in cards