Insights
How AI evaluation has changed since 2023. Every chart comes from the same data as the rest of the almanac, and updates with it.
-
What labs report
In the first half of 2024, 34% of the benchmarks labs named had come out in the year before the release. In the second half of 2026 so far, 63% had.
-
How are evals changing?
Agent tasks made up 6% of the new benchmarks in 2023 and 48% in 2026. Multiple-choice tests fell from 24% to 5%.
-
How fast is the frontier moving?
The best open model today, Kimi K3.1 Pro, scores what the best closed model scored seven months ago. Since 2024 the median gap has been six months.
-
How do new models get tested?
Of the benchmarks labs named in launch posts since 2024, an official leaderboard or Epoch AI has scored the same model on 14%.
In numbers
- 20% of benchmark mentions in launch posts went to the ten most named in the second half of 2026 so far, down from 35% in the first half of 2024
- 33% of the benchmarks labs first named in 2025 appeared in only one release
- 35% of the benchmarks from 2026 keep some test items private, against 15% of those from 2023
- 26% of the benchmarks from 2026 had a score past 85% ten months after release, against 13% of those from 2023
- 18% of the new benchmarks from 2026 were made with a university or institute, against 37% in 2020. Eval orgs made 11%
- 28 days is how long Arena's number one has lasted in 2026 so far, at the median, against 40 days in 2025
- 42% of contested benchmark-days at number one went to Anthropic in the second half of 2026 so far
- 2.1× cheaper to match the best score from a year ago, on the median leaderboard that prices each task
- 14% of the benchmarks labs named in launch posts since 2024 have an independent score of the same model
Elsewhere in the almanac
- Progress over time Plot the best score so far on up to eight benchmarks at once, by the release date of the model that set each record.
- Time machine The almanac as it stood on any day since 2023, with who led the capabilities index and Arena.
- Leaders Who held number one in each category, for how long, and who took benchmarks past 85%.
- How do claims and official scores differ? How the scores labs publish compare with independent runs of the same models.
- Model cards Every launch post and system card we read, and what labs feature by area each half year.
- Query with SQL Every score and date behind these charts, to query in the browser or download.