SciArena
Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings.
- o3 1,151
- Claude Opus 4.1 1,126
- GPT-5 1,123
-
Gemini 3 Pro 1,086
- GPT-5.1 1,080
Official leaderboard
We track 61 benchmarks and 2 arenas that Ai2 made or helped make. Its models are on its lab page.
Top five in each of its 2 arenas with three or more rated models, the most rated first.
Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings.
Official leaderboard
Users upload an image with a question, compare answers from two anonymous vision-language models, and vote. Votes give Elo ratings.
Official leaderboard
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. | Visual math and diagrams | Mar 2016 | | Fading |
| ARC-Challenge Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy. | Exam questions | Mar 2018 | | Dropped |
| ZebraLogic Models solve logic grid puzzles of controlled size, derived from constraint satisfaction problems. Scored by the share of puzzles solved fully. | Abstract puzzles and logic | Feb 2025 | | Fading |
| OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. | Exam questions | Sep 2018 | 96.6% | Fading |
| SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. | Commonsense | Apr 2019 | | Dropped |
| ARC-Easy Models answer grade-school science multiple-choice questions from the ARC Easy Set, which has 5,197 questions. Scoring is accuracy. | Exam questions | Mar 2018 | | Dropped |
| CommonSenseQA Models answer five-way multiple-choice questions built from ConceptNet relations that need everyday background knowledge. Scored by accuracy. | Commonsense | Nov 2018 | 90.4% | Dropped |
| QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. | Reading comprehension | Aug 2018 | | Dropped |
| Tulu 3 Safety Eval Models answer harmful, jailbreak and benign prompts from several safety benchmarks in Ai2's safety-eval suite. A WildGuard classifier judges the outputs, and the score is an average. | Jailbreaks and prompt injection | Nov 2024 | | Fading |
| TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. | Video understanding | Oct 2024 | | Fading |
| Qasper Models answer 5,049 questions about 1,585 NLP research papers from the full paper text. Answers are scored with token F1. | Reasoning over long documents | May 2021 | | Dropped |
| RealToxicityPrompts Models continue 100,000 web text prompts. Scored on the rate of toxic generations judged by a toxicity classifier. | Refusing harmful requests | Sep 2020 | | System cards only |
| WildBench 1,024 challenging real-user tasks with checklist-guided LLM judging. | Overall chat quality | Jun 2024 | | Dropped |
| WildGuardTest Models classify whether user prompts are harmful, whether responses are harmful, and whether responses refuse, across 13 risk categories. Scored by F1. | Refusing harmful requests | Jun 2024 | | System cards only |
| WildJailbreak Models face adversarial jailbreak prompts built from in-the-wild tactics, plus benign prompts that look harmful. Evaluations score refusals of harmful requests and compliance with benign ones. | Jailbreaks and prompt injection | Jun 2024 | | Fading |
| WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. | Refusing harmful requests | May 2024 | 98% | System cards only |
| OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. | General visual questions | May 2019 | | Dropped |
| OMEGA Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically. | Competition problems with answers | Jun 2025 | | Fading |
| RewardBench Models act as judges and pick the preferred response in chosen and rejected pairs across chat, hard chat, safety and reasoning. Scored by accuracy. | Overall chat quality | Mar 2024 | | Dropped |
| PixMo-Count Models count objects in natural images and give the exact number. Scored by accuracy on the test split. | Detection and captioning | Sep 2024 | | Fading |
| ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. | General visual questions | Sep 2022 | | Dropped |
| WildVision Vision-language models answer 500 real user image prompts from the WildVision Arena. A GPT-4 judge compares each answer with a fixed baseline model. | General visual questions | Jun 2024 | 74.1 | Dropped |
| olmOCR-Bench OCR systems convert 1,403 PDFs to markdown. 7,010 unit tests check text presence and absence, reading order, tables and math. | Documents, charts and text in images | Feb 2025 | 87.8% | In use |
| MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. | Simulated robots | Feb 2026 | | Fading |
| CulturalBench Models answer 1,696 human-written and human-verified questions on cultural knowledge across 45 global regions and 17 topics. Scored by accuracy. | Knowledge in many languages | Oct 2024 | 62% | System cards only |
| IFEval-OOD Models answer unseen WildChat prompts with added verifiable constraints drawn from 52 types outside IFEval. Scored by prompt-level loose accuracy. | Following precise instructions | Nov 2024 | | System cards only |
| SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. | Spatial reasoning and counting | Dec 2024 | | System cards only |
| MathQA Models answer multiple-choice math word problems drawn from AQuA. Each problem is annotated with an operation program. Scored by accuracy. | Word problems | May 2019 | | System cards only |
| RewardBench 2 Reward models must score one correct answer above three wrong ones on 1,865 new human prompts. Subsets cover factuality, precise instruction following, math, safety, focus and ties, and the score averages them. | Overall chat quality | Jun 2025 | | Not in cards |
| Open CoT Leaderboard Models answer logical reasoning questions with and without writing out their reasoning. | Language reasoning | Jan 2024 | | Not in cards |
| AstaBench Agents work through more than 2,400 scientific research problems in 11 benchmarks across four areas: literature understanding, code and execution, data analysis, and end-to-end discovery. Many problems come from real requests to Ai2's deployed Asta agents. Agents use a fixed set of production-grade search tools, and the leaderboard plots score against cost per problem. | Research science | Aug 2025 | 58% | Not in cards |
| AppWorld Test-Challenge Agents complete 417 app tasks that require Amazon or Gmail APIs held out from training. The score is the percentage passing every task test. | Apps and MCP servers | Jul 2024 | 92.8% | Not in cards |
| AppWorld Test-Normal Agents complete 168 everyday app tasks from familiar applications by calling APIs through code. The score is the percentage passing every task test. | Apps and MCP servers | Jul 2024 | 94% | Not in cards |
| CoCoNot Models get 1,000 requests they should not fully comply with, such as incomplete, unsupported or humanizing requests. A judge model rates whether each response complies. | Deception and alignment | Jul 2024 | 5.34% | Not in cards |
| CommonsenseQA 2.0 Models answer 14,343 yes or no commonsense questions that players wrote to fool an AI in a game. Scored by accuracy. | Commonsense | Jan 2022 | | Not in cards |
| Infrathon Agents optimize AI infrastructure under correctness and fidelity constraints. Hidden workloads measure performance gains, with additional checks for rule compliance. | Speed and optimization | Nov 2026 | 91.2 | Not in cards |
| XYBench Models answer requests that pursue a mistaken solution to an underlying problem. A model judge checks the useful solution, its emphasis, and identification of the misconception. | Overall chat quality | Sep 2026 | 31% | Not in cards |
| New Yorker Caption Contest Humor Understanding Models match a cartoon to its caption, choose the funnier of two captions, and explain a joke. They receive the cartoon or its description. | General visual questions | Sep 2022 | | Not in cards |
| ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. | Lab and data work | Mar 2022 | 0.15 | Not in cards |
| A-OKVQA Models answer about 25,000 visual questions that need world knowledge beyond the image, in multiple-choice and direct-answer forms. | General visual questions | Jun 2022 | Not in cards | |
| ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. | Simulated robots | Dec 2019 | Not in cards | |
| AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. | Apps and MCP servers | Jul 2024 | Not in cards | |
| AssistantBench Web agents do 214 realistic, time-consuming tasks on the open web, such as monitoring real-estate markets. Answers are checked automatically against gold answers. | Web browsing | Jul 2024 | Not in cards | |
| DiscoveryBench Agents analyze datasets and produce data-driven hypotheses that evaluators score against structured evidence and domain criteria. | AI research tasks | Jul 2024 | Not in cards | |
| ExFunTube Models explain why a moment in a short video is funny. The authors compare explanations with human references and ask viewers to rate them. | Video captioning | Jul 2023 | Not in cards | |
| FairStress Models choose between candidates as demographic cues and persuasive statements change. The benchmark measures how often models change a correct decision. | Bias, fairness and politics | Oct 2026 | Not in cards | |
| KiloGram Models choose which of ten tangram pictures matches a description. The descriptions can name the whole shape or explain what its pieces represent. | Spatial reasoning and counting | Nov 2022 | Not in cards | |
| MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. | Documents, charts and text in images | Oct 2020 | Not in cards | |
| MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. | Simulated robots | Jun 2026 | Not in cards | |
| MS² Models synthesize the findings of multiple medical studies into a short answer to the research question of a systematic review. | Medicine | Apr 2021 | Not in cards | |
| Multi-LexSum Models summarize documents from civil rights lawsuits at different lengths. Legal experts wrote the reference summaries. | Law | Jun 2022 | Not in cards | |
| NaturalProofs Models retrieve or generate references needed for a mathematical theorem's proof. Ranking and sequence metrics compare predicted references with those used in the source proof. | Written proofs | Apr 2021 | Not in cards | |
| NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. | Research science | Nov 2025 | Not in cards | |
| Novelty Judge Bench Models judge whether generated and human-authored research ideas are novel using pairwise and pointwise comparisons. | AI research tasks | Oct 2026 | Not in cards | |
| OCNLI Models decide whether a Chinese hypothesis follows from, contradicts, or remains neutral toward a premise. Accuracy measures agreement with the majority annotation. | Reasoning in many languages | Oct 2020 | Not in cards | |
| Paloma Language models are measured by perplexity on held-out text from 585 domains, from news sites to individual subreddits. | Commonsense | Dec 2023 | Not in cards | |
| PreScience Models forecast future scientific contributions through collaborator, prior-work, contribution and impact prediction tasks. | Real-world events | Feb 2026 | Not in cards | |
| SciDocs Models compare representations of scientific papers on citation prediction, recommendation, and classification tasks. | Research science | Apr 2020 | Not in cards | |
| SciFact Models retrieve research abstracts and determine whether they support or refute a scientific claim. | Faithful to sources | Apr 2020 | Not in cards | |
| SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. | Science questions | Jul 2017 | Not in cards | |
| StateBench Models follow variables through long runs of assignments and report a final value. Four tiers share one source trace and differ only in the update: fixed values, reassignment, self-referential updates, and assignment by reference. Problems come in several formats, such as code, spreadsheet edits and people moving between rooms, at lengths from under 2K to over 2M tokens. | Reasoning over long documents | Nov 2026 | Not in cards |