Benchmark Almanac

How are evals changing?

Agent tasks made up 6% of the new benchmarks in 2023 and 48% in 2026. Multiple-choice tests fell from 24% to 5%. We count every benchmark in our catalog by the year it came out.

Agent tasks made up 6% of the new benchmarks labs cite in 2023 and 47% in 2026. Multiple-choice tests fell from 34% to 3%. We count every benchmark in our catalog that a lab has named in a launch post by the year it came out.

How are new benchmarks built?

Benchmarks scored by another model went from 10% of the year's new benchmarks in 2023 to 28% in 2026. Those that keep some test items private went from 15% to 35%.

Agent tasks

The model acts over many steps in an environment, a browser or a terminal

0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 3% 48% 0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 3% 48%

Multiple choice

The model picks an answer from a list

0% 10% 20% 30% 2020 2021 2022 2023 2024 2025 2026 27% 5% 0% 10% 20% 30% 2020 2021 2022 2023 2024 2025 2026 27% 5%

Scored by a model

Another model grades the answers, often against a rubric

0% 10% 20% 30% 2020 2021 2022 2023 2024 2025 2026 1% 28% 0% 10% 20% 30% 2020 2021 2022 2023 2024 2025 2026 1% 28%

Private test items

Some or all of the test is kept out of public view, to stop leaks into training data

0% 10% 20% 30% 40% 2020 2021 2022 2023 2024 2025 2026 6% 35% 0% 10% 20% 30% 40% 2020 2021 2022 2023 2024 2025 2026 6% 35%

Source: The Benchmark Almanac catalog

Who makes new benchmarks?

Universities and institutes helped make 37% of the new benchmarks in 2020 and 18% in 2026. Eval orgs went from 0% to 11%, and data vendors from 0% to 4%.

AI labs

Companies and groups that train and release their own models

0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 20% 27% 0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 20% 27%

Universities and institutes

Universities, their labs, and public research institutes

0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 37% 18% 0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 37% 18%

Eval orgs

Groups whose main work is evaluating or ranking AI, such as Vals AI, Epoch AI and METR

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 11% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 11%

Data vendors

Companies that sell training data, expert labeling or raters to labs, such as Scale AI and Mercor

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 4% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 4%

Other companies

AI products and startups that release no models of their own, and companies outside AI

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 5% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 5%

Government and nonprofits

Government institutes, nonprofits, exam boards and open communities

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 2% 2% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 2% 2%

Source: The Benchmark Almanac catalog

How are new benchmarks built?

Benchmarks scored by another model went from 8% of the year's new benchmarks labs cite in 2023 to 36% in 2026. Those that keep some test items private went from 11% to 57%.

Agent tasks

The model acts over many steps in an environment, a browser or a terminal

0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 0% 47% 0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 0% 47%

Multiple choice

The model picks an answer from a list

0% 10% 20% 30% 40% 2020 2021 2022 2023 2024 2025 2026 38% 3% 0% 10% 20% 30% 40% 2020 2021 2022 2023 2024 2025 2026 38% 3%

Scored by a model

Another model grades the answers, often against a rubric

0% 10% 20% 30% 40% 2020 2021 2022 2023 2024 2025 2026 0% 36% 0% 10% 20% 30% 40% 2020 2021 2022 2023 2024 2025 2026 0% 36%

Private test items

Some or all of the test is kept out of public view, to stop leaks into training data

0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 8% 57% 0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 8% 57%

Source: The Benchmark Almanac catalog

Who makes new benchmarks?

Universities and institutes helped make 38% of the new benchmarks labs cite in 2020 and 13% in 2026. Eval orgs went from 0% to 19%, and data vendors from 0% to 8%.

AI labs

Companies and groups that train and release their own models

0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 46% 53% 0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 46% 53%

Universities and institutes

Universities, their labs, and public research institutes

0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 38% 13% 0% 20% 40% 60% 2020 2021 2022 2023 2024 2025 2026 38% 13%

Eval orgs

Groups whose main work is evaluating or ranking AI, such as Vals AI, Epoch AI and METR

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 19% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 19%

Data vendors

Companies that sell training data, expert labeling or raters to labs, such as Scale AI and Mercor

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 8% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 8%

Other companies

AI products and startups that release no models of their own, and companies outside AI

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 6% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 0% 6%

Government and nonprofits

Government institutes, nonprofits, exam boards and open communities

0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 8% 2% 0% 5% 10% 15% 20% 2020 2021 2022 2023 2024 2025 2026 8% 2%

Source: The Benchmark Almanac catalog

What do new benchmarks test?

Coding tests rose from 6% of new benchmarks in 2023 to 15% in 2026, as knowledge tests fell from 38% to 12%.

All benchmarks
EverydaySchoolUniversityCompetitionExpertResearch frontier

Select an area to break it down.

0%25%50%75%100%Knowledge and languageMath and scienceCodingAgents and toolsProfessional workVision, audio and roboticsSafety and security20232024202520260%25%50%75%100%2023202420252026

Source: The Benchmark Almanac catalog

How fast do benchmarks saturate?

Ten months after release, 13% of the benchmarks from 2023 had a score past 85%. Of those from 2026, 26% did. Each line follows the benchmarks that came out in one year, from release to the first score past 85%.

0% 20% 40% 60% 0 6 12 18 24 30 36 Months since release 2023 2024 2026 2025 0% 20% 40% 60% 0 6 12 18 24 30 36 Months since release 2023 2024 2026 2025

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

How much of the catalog is past 85%?

Today 26% of the benchmarks with scores have a best score past 85%, and 32% of the ones labs featured in launch posts. A year ago those shares were 21% and 33%.

0% 20% 40% 60% 2023 2024 2025 2026 Featured in launch posts Every benchmark with scores 0% 20% 40% 60% 2023 2024 2025 2026 Featured in launch posts Every benchmark with scores

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

Inspired by Ethan Mollick’s SatBench. Pick a day on the Time machine to see the leaders then.

Which areas saturate first?

Math and science benchmarks saturate first: 42% of them have a score past 85%. Coding trails at 20%. Each chart is the share of an area's benchmarks with a score past 85%, at the end of each half year.

Knowledge and language

0% 20% 40% 60% 2024 2025 2026 15% 42% 0% 20% 40% 60% 2024 2025 2026 15% 42%

Math and science

0% 20% 40% 60% 2024 2025 2026 7% 42% 0% 20% 40% 60% 2024 2025 2026 7% 42%

Coding

0% 20% 40% 60% 2024 2025 2026 8% 20% 0% 20% 40% 60% 2024 2025 2026 8% 20%

Agents and tools

0% 20% 40% 60% 2024 2025 2026 11% 22% 0% 20% 40% 60% 2024 2025 2026 11% 22%

Professional work

0% 20% 40% 60% 2024 2025 2026 3% 23% 0% 20% 40% 60% 2024 2025 2026 3% 23%

Vision, audio and robotics

0% 20% 40% 60% 2024 2025 2026 3% 22% 0% 20% 40% 60% 2024 2025 2026 3% 22%

Safety and security

0% 20% 40% 60% 2024 2025 2026 15% 41% 0% 20% 40% 60% 2024 2025 2026 15% 41%

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

How fast do benchmarks saturate?

Nine months after release, 9% of the benchmarks from 2023 had a score past 85%. Of those from 2026, 23% did. Each line follows the benchmarks labs cite that came out in one year, from release to the first score past 85%.

0% 20% 40% 60% 80% 0 6 12 18 24 30 36 Months since release 2023 2024 2025 2026 0% 20% 40% 60% 80% 0 6 12 18 24 30 36 Months since release 2023 2024 2025 2026

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

How much of the catalog is past 85%?

Today 32% of the benchmarks labs have featured in launch posts have a best score past 85%, against 33% a year ago.

0% 20% 40% 60% 2023 2024 2025 2026 48% 32% 0% 20% 40% 60% 2023 2024 2025 2026 48% 32%

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

Inspired by Ethan Mollick’s SatBench. Pick a day on the Time machine to see the leaders then.

Which areas saturate first?

Math and science benchmarks saturate first: 55% of them have a score past 85%. Professional work trails at 14%. Each chart is the share of an area's benchmarks labs cite with a score past 85%, at the end of each half year.

Knowledge and language

0% 20% 40% 60% 2024 2025 2026 34% 54% 0% 20% 40% 60% 2024 2025 2026 34% 54%

Math and science

0% 20% 40% 60% 2024 2025 2026 12% 55% 0% 20% 40% 60% 2024 2025 2026 12% 55%

Coding

0% 20% 40% 60% 2024 2025 2026 13% 27% 0% 20% 40% 60% 2024 2025 2026 13% 27%

Agents and tools

0% 20% 40% 60% 2024 2025 2026 21% 28% 0% 20% 40% 60% 2024 2025 2026 21% 28%

Professional work

0% 20% 40% 60% 2024 2025 2026 14% 14% 0% 20% 40% 60% 2024 2025 2026 14% 14%

Vision, audio and robotics

0% 20% 40% 60% 2024 2025 2026 4% 30% 0% 20% 40% 60% 2024 2025 2026 4% 30%

Safety and security

0% 20% 40% 60% 2024 2025 2026 11% 48% 0% 20% 40% 60% 2024 2025 2026 11% 48%

Sources: Official leaderboards, Epoch AI (CC BY 4.0) and labs’ model cards

How this is counted

The catalog holds 6,423 benchmarks with a release month. We find recent benchmarks more completely than old ones, so the number per year rises faster than the field did. The shares are less affected, and each point's tooltip shows how many benchmarks it rests on. The current year isn't over, so its point is hollow.

"Only benchmarks labs cite" keeps the 1,104 benchmarks that at least one lab named in a launch post or the documents published with it. It applies to every chart but "What do new benchmarks test?", which counts the whole catalog.

Format, scoring and test access come from each benchmark's entry, the same fields the benchmarks table filters on. A benchmark with no value for a field is left out of that chart.

Makers come from each benchmark's credit. AI labs are the labs in our lab list that release models. Universities and institutes are named as such. Eval orgs, data vendors, other companies, government and nonprofits are sorted by hand for the makers with the most benchmarks. A benchmark can count under two kinds. Independent researchers made too few to chart. 39% of the new benchmarks from 2026 have only makers we haven't sorted yet.

A benchmark saturates here when any score we know of passes 85%, from a leaderboard, Epoch AI or a lab's own report. A model's score counts from the day the model came out, so a benchmark that older models already passed counts as saturated on day one. Benchmarks reach different ages, so each line is a survival estimate: every month, of the benchmarks old enough and not yet past 85%, the share that passed. A line stops when fewer than 15 benchmarks are left to follow. The comparison above uses month 10, the latest month every line reaches.

More insights