| WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. | Web browsing | Jul 2023 | 68% | Fading |
| RACE-H Models answer multiple-choice reading questions from English exams for Chinese high school students. The test split has 3,498 questions, scored by accuracy. | Reading comprehension | Apr 2017 | 92.9% | Dropped |
| RACE Models answer reading comprehension questions from Chinese middle and high school English exams. Reported here without a subset split. | Reading comprehension | Apr 2017 | 94.2% | System cards only |
| VisualPuzzles Models solve visual puzzles that need algorithmic, analogical, deductive, inductive and spatial reasoning but little domain knowledge. Scored by accuracy. | Spatial reasoning and counting | Apr 2025 | 57% | Fading |
| PubMedQA Models answer biomedical research questions with yes, no, or maybe, given a PubMed abstract without its conclusion. Scored by accuracy. | Medicine | Sep 2019 | 78.3% | Dropped |
| OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. | General visual questions | May 2019 | 60.2% | Dropped |
| RACE-M Models read passages from English exams for Chinese middle school students and answer multiple-choice questions. Scored by accuracy. | Reading comprehension | Apr 2017 | 77% | Dropped |
| MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. | Video understanding | Mar 2025 | 64.1% | Fading |
| xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. | General visual questions | Oct 2024 | 41.4% | System cards only |
| miniCTX-v2 Provers prove Lean theorems from real projects using long repository context, with theorems taken after a November 28, 2024 cutoff. | Formal proofs | Mar 2025 | 81.8% | Dropped |
| HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. | Reading comprehension | Sep 2018 | 92.1% | System cards only |
| TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. | Simulated companies | Dec 2024 | | Dropped |
| xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. | General visual questions | Oct 2024 | 70.1 | System cards only |
| FinGREAT Models identify accounting relationships in financial passages. The authors compare 44 models against human annotations across 483 passages. | Finance | Oct 2026 | 92.8% | Not in cards |
| CMIMC 2025 Models answer 40 final-answer problems from the 2025 Carnegie Mellon Informatics and Mathematics Competition. MathArena scores the average accuracy over four attempts. | Competition problems with answers | May 2025 | 95.6% | Not in cards |
| GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. | Building whole projects | Feb 2026 | 70.6% | Not in cards |
| PianoBench Models answer questions about piano performances from recordings or symbolic transcriptions. The authors compare accuracy with and without access to the music. | Audio understanding and voice | Nov 2026 | 93.2% | Not in cards |
| GSM-Infinite Models solve generated arithmetic problems across different context lengths and numbers of reasoning steps. The benchmark varies how many irrelevant details appear alongside each problem. | Reasoning over long documents | Feb 2025 | 8534.9 | Not in cards |
| SusVibes Coding agents rebuild a feature in a real open-source Python project from a plain-language request. Each task comes from a historical vulnerability fix, so the original human version was insecure. Hidden functional tests and security tests check the result. Scores report the share of solutions that work and the share that are also secure. | Secure code and review | Dec 2025 | 43.5% | Not in cards |
| CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. | Simulated robots | Mar 2026 | 32.3% | Not in cards |
| MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. | Desktop apps | Jun 2026 | 62% | Not in cards |
| IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. | Research science | Jul 2026 | 27.3% | Not in cards |
| K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. | Knowledge in many languages | Jun 2026 | 45.7% | Not in cards |
| Kinayat Models interpret Egyptian Arabic idioms, select appropriate contexts and identify connotations. The evaluation includes separate generation and multiple-choice tasks. | Reasoning in many languages | Oct 2025 | 95.3% | Not in cards |
| EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. | Research science | Sep 2026 | 29.8 | Not in cards |
| NaturalBench Models answer 10,000 human-checked questions on natural images. Each question pairs with two images that need different answers, so a model cannot pass without looking. | General visual questions | Oct 2024 | 55.1% | Not in cards |
| SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. | Genomics and life sciences | May 2026 | 40.2% | Not in cards |
| VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. | Web browsing | Jan 2024 | 16.4% | Not in cards |
| iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. | Phones | Jun 2026 | 51.9% | Not in cards |
| Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. | Video games | Oct 2026 | 9.00% | Not in cards |
| AnyAppBench Agents repeat the same task in different Android apps, with and without suggested steps. Each app has a verifier that checks the same success condition. | Phones | Sep 2026 | 67.8% | Not in cards |
| CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. | Desktop apps | Apr 2026 | 27.5% | Not in cards |
| DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer. | Data science and SQL | Nov 2022 | 53.9% | Not in cards |
| FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. | Tasks from real jobs | Jun 2025 | 52% | Not in cards |
| Per-DOC Models use a reader’s previous comparison to predict which of two new plots the reader prefers on each of five criteria. | Taste and style | Oct 2023 | 0.21% | Not in cards |
| Per-MPST Models use a reviewer’s earlier movie ratings and reviews to predict their rating of a new plot. The authors measure correlation with human ratings. | Taste and style | Oct 2023 | 18.1 | Not in cards |
| RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. | Simulated robots | Oct 2026 | 0.06 | Not in cards |
| TurnBench Turn-taking systems listen to two-person conversations and must detect when a speaker's turn ends and when the other speaker interrupts. The test set has 116 English conversations in six interaction styles, each labeled by three annotators. Systems rank by recall, and a system whose false-positive rate exceeds 0.15 ranks below every qualifying system. Latency after each event is also reported. | Speech recognition and translation | Aug 2026 | 65.7% | Not in cards |
| LLM-SRBench Models infer scientific equations from observations on 239 problems across four scientific domains. Symbolic accuracy checks whether the discovered equation recovers the underlying relationship. | Research science | Apr 2025 | 15.1% | Not in cards |
| MobileAgentBench Agents complete 100 tasks across ten Android apps. The grader checks app state and accessibility events, so agents can take different paths. | Phones | Jun 2024 | 8.00% | Not in cards |
| AdvBench Models get harmful requests, such as a request for malware, and should refuse them. The set has 520 harmful behaviors and 574 harmful strings. Most papers add an attack, such as an adversarial suffix, and report the attack success rate. | Jailbreaks and prompt injection | Jul 2023 | | Not in cards |
| Alharthi LibriTTS Naturalness Listening Test Models estimate how natural synthesized speech sounds. Ten listeners rated each sample on a five-point scale for comparison. | Audio understanding and voice | Oct 2023 | | Not in cards |
| IdeaTaste Agents propose research ideas from literature frozen at 2025. Evaluation compares their citations and ideas with 2026 publications and measures diversity. | AI research tasks | Nov 2026 | | Not in cards |
| PAM-Audio Models judge the quality of generated environmental sounds and how closely each recording matches its caption. The listening study used 100 AudioCaps captions. | Audio understanding and voice | Feb 2024 | | Not in cards |
| RefAV Models find driving scenarios described in natural language. They identify when each scenario occurs and track the objects involved. | Detection and captioning | Mar 2025 | | Not in cards |
| SingMOS Models listen to Chinese and Japanese singing and predict how listeners rate its quality. The recordings include human voices and synthesized voices. | Audio understanding and voice | Jun 2024 | | Not in cards |
| Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. | Research mathematics | May 2026 | | Not in cards |
| XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. | Reading in many languages | Mar 2020 | | Not in cards |