Benchmark Almanac

Audits

These are published audits that found problems in benchmarks we track, like graders that reject right answers or answers that leak online. Each finding also shows up on its benchmark's page. Know of an audit we're missing? Let us know here.

5 audits 7 findings on 7 benchmarks 6 not fixed in the benchmark itself What we include

Unclear tasks show up in 4 of 7 findings, more than any other problem

Pick a kind of problem to see only those findings.

  1. Nov 3, 2026

    Learnings from Auditing Public Evaluations

    Prime Intellect, a write-up, by Florian Brand

    • AutomationBench Fixed in a copy

      Unclear tasks and broken grader

      Prime Intellect reports that AutomationBench describes API request bodies in prose instead of JSON Schema, so models had to guess parameters. It also reports graders that enforced requirements the tasks did not state.

    • OfficeQA Pro V2 Fixed in a copy

      Broken grader, leaked answers, unclear tasks and trips content filters

      Prime Intellect reports that the OfficeQA Pro v2 regex grader rejects many valid answers and that agents with web access retrieved solutions from Hugging Face. It also reports a default prompt that trips Anthropic content filters and some ambiguous tasks.

    • HLE-Diamond Still open

      Leaked answers and reward hacking

      Prime Intellect reports that HLE-Diamond answers leak through mirrors and public rollout logs, so domain blocklists do not stop agents from finding them. In one trace, DeepSeek V4.1-Flash 1017 read a graded answer through OpenReward's public API and submitted it.

  2. Jul 23, 2025

    About 30% of Humanity’s Last Exam Chemistry/Biology Answers are Likely Wrong

    FutureHouse, a write-up

    • Humanity's Last Exam Fixed in a copy

      Wrong labels

      FutureHouse reports that about 29% of the text-only chemistry and biology answers in Humanity's Last Exam conflict with peer-reviewed literature. FutureHouse released HLE Bio/Chem Gold, a subset its system graded as supported by research.

  3. Aug 13, 2024

    Introducing SWE-bench Verified

    OpenAI, a write-up

    • SWE-bench Replaced by a new version

      Unclear tasks and broken grader

      OpenAI reports that its annotators flagged many SWE-bench test samples for underspecified problem statements and for unit tests that may reject valid solutions. OpenAI and the SWE-bench authors released the 500 samples that passed review as SWE-bench Verified.

  4. Jun 6, 2024

    Are We Done with MMLU?

    Gema et al., a paper

    • MMLU Fixed in a copy

      Wrong labels and unclear tasks

      Gema et al. report numerous ground-truth errors in MMLU, with Virology the worst subject they analyzed. They re-annotated 5,700 questions across all 57 subjects as MMLU-Redux and estimate the error rate for all of MMLU.

  5. May 1, 2024

    A Careful Examination of Large Language Model Performance on Grade School Arithmetic

    Scale AI, a paper

    • GSM8K Still open

      Contamination

      Scale AI reports that many models score lower on GSM1k, a new set written to mirror GSM8K, than on GSM8K itself. It reads the gap as evidence that some models partially memorized GSM8K, while frontier models show minimal signs of overfitting.

What we include

We list an audit once its author publishes how they did it, in a paper, a write-up or a repo with the fixes. A post or a newsletter line on its own isn't enough.

We name who found each problem and quote their numbers as printed. If the benchmark's makers reply or fix it, we link that too.

We don't change leaderboard scores because of an audit. Re-scored results often use a different harness or only the public tasks, so they stay in the finding's text.

Updated November 3rd, 2026. To suggest an audit, send a correction.