Submit a benchmark

What gets cataloged

We track 3,322 benchmarks so far. A new one gets in if it passes at least one of these four tests.

1. A major lab reports it A lab that ships frontier or widely used models reports it in a launch post, model card, system card or technical report.
2. An independent leaderboard runs it Its makers or a third party keep a leaderboard with scores for five or more current models.
3. An established evaluation group made it For example Epoch AI, METR, Scale, Vals, ARC Prize, Apollo Research, UK AISI, US CAISI, SecureBio, Arena, HELM or Kaggle.
4. People use it About 100 GitHub stars or 30 Hugging Face upvotes, coverage in a widely read newsletter, or other papers and model cards that run it.

What stays out: test sets made only to evaluate one paper’s own method, training datasets, surveys, renamed copies of a test we already have, and narrow tasks outside general AI ability, such as crowd counting or defect detection. A new version that changes the tasks or the scoring gets its own record. One that only fixes broken tasks doesn’t.

What to include

The form asks for all of these. You only have to give the name, what it tests, its area, a paper or report, and the evidence, but the rest makes our review faster.

The name

And the version, when there is more than one.

What it tests

One or two sentences about the tasks and how a model is judged on them. Leave the motivation for the paper.

The paper or report

arXiv, OpenReview, a technical report or a launch post.

The code and the data

Links, with their licenses. Say whether the test set is public, held out or partly held out.

The scores

A leaderboard, if there is one, and a file of scores we can read, such as a CSV, JSON or a Hugging Face dataset. If each score has a date, we can chart who held number one.

The evidence

Links that show which of the four tests it passes, such as a model card that reports it, a leaderboard that runs it, or papers that use it.

What it replaces

If it’s a new version of a benchmark we already list, say which one and what changed in the tasks or the scoring.

What happens next

1. Within a minute, you get an automatic reply on your submission. It tells you whether we already have the benchmark or left it out before, with the reason. It also lists any items that are missing.
2. Within a few days, we review it against the four tests. If it passes, we add a record for it, and it shows up on the New page after the next daily update.
3. If it doesn’t pass yet, we say which test it missed. You can add evidence to the same submission at any time, for example once a lab reports it.

To report a mistake in one of our records, use the correction form. Include a link to the page and a source for the right value.

Want to help without a benchmark of your own? The Contribute page lists open gaps in our data, such as missing release dates and leaderboards we don't read yet, each with a form that's already filled in.

Other forms cover the rest. Use Add or fix a score for a result a lab or leaderboard published that we lack, and Log a known critique for a published problem with a benchmark, such as a broken grader or leaked answers. If you make or maintain a benchmark, tell us where to read your results or how your benchmark should be described.