Benchmark Almanac

Microsoft

We track 96 benchmarks and 2 arenas that Microsoft made or helped make. Its models are on its lab page.

3 in use 65 with public code 46 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
MathVista Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. Visual math and diagrams Oct 2023 90.7% In use
AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. Exam questions Apr 2023 89.2% Fading
ODinW-13 Models detect objects in 13 varied real-world datasets, such as aquarium and aerial images, from text category names. Scored by mean average precision. Detection and captioning Dec 2021 51.8% Fading
MM-Vet Models answer open questions about images that combine recognition, OCR, knowledge, spatial reasoning, language generation and math. GPT-4 grades each answer against a reference. General visual questions Aug 2023 82.8% Dropped
ToxiGen Models continue ToxiGen prompts about 13 minority groups. Scored by the average toxicity of the generations. Bias, fairness and politics Mar 2022 29.8 System cards only
WebTailBench Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals. Web browsing Nov 2025 57.4% Fading
Windows Agent Arena Agents complete more than 150 tasks in a real Windows OS with its apps, tools and browsers. Scripts adapted from OSWorld check the final state. Desktop apps Sep 2024 50.6% Dropped
PhiBench Models answer original prompts that Microsoft's Phi team wrote to test a wide range of skills and reasoning. Other Dec 2024 74.2% Dropped
Microsoft RAI benchmark: Harmful Content Continuation (DR-3) Models continue conversations about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Refusing harmful requests Apr 2024 0.30% System cards only
Microsoft RAI benchmark: Harmful Content Summarization (DR-3) Models summarize content about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Refusing harmful requests Apr 2024 10% System cards only
Microsoft RAI benchmark: Jailbreak (DR-1) Models face simulated jailbreak conversations on topics such as child grooming, illegal advice and leaking guidelines. Scored as the defect rate at severity 1. Jailbreaks and prompt injection Apr 2024 7.30% System cards only
Microsoft RAI benchmark: Third Party Harm (DR-1) Models face simulated conversations that try to extract third-party content such as books, news, recipes and songs. Scored as the defect rate at severity 1. Refusing harmful requests Apr 2024 10.5% System cards only
Microsoft RAI benchmark: Ungroundedness Models answer in GPT-4 simulated multi-turn conversations. A judge rates how ungrounded each answer is in the given prompt, from 0 to 4. Faithful to sources Apr 2024 0.21 System cards only
COCO Captions Models write a caption for each image, and outputs are compared with five human captions per image using BLEU, METEOR, ROUGE, and CIDEr. Detection and captioning Apr 2015 116.0 Dropped
Kitab Models list books by a given author that meet extra constraints, such as publication dates or title words. Scored by precision and constraint satisfaction. Long-form factuality Oct 2023 Dropped
Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. Refusing harmful requests Mar 2025 1.40% System cards only
SpatialEval Maze-Nav Models answer questions about routes and turns in mazes given as text or images. 1,500 multiple-choice questions scored by accuracy. Spatial reasoning and counting Jun 2024 41% System cards only
SpatialEval Spatial-Map Models answer questions about relations and counts among objects placed on a map given as text or images. 1,500 questions scored by accuracy. Spatial reasoning and counting Jun 2024 64.3% System cards only
3SAT-Search Models find satisfying assignments for 960 3-SAT formulas across difficulty levels. A checker verifies each assignment. Abstract puzzles and logic Mar 2025 96.2% System cards only
TSP-Opt Models find optimal tours for 960 traveling salesman instances across 8 difficulty levels. Answers are checked for optimality. Abstract puzzles and logic Mar 2025 56.4% System cards only
WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. Refusing harmful requests Nov 2025 100% System cards only
AIME 1983-2024 Models solve 949 past AIME problems from 1983 to 2024. Integer answers are checked exactly. Competition problems with answers Mar 2025 89.4% System cards only
BA-Calendar Models find a common meeting slot for several people under constraints such as time zones, buffer time and priority. 2,000 problems checked by code. Abstract puzzles and logic Oct 2024 67.7% System cards only
Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. Refusing harmful requests Aug 2024 8.16 System cards only
OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. Data science and SQL 34.9% System cards only
ToxiGen (discriminative) Models label statements about 13 minority groups as toxic or neutral. Scored by accuracy, often split into toxic and neutral labels. Bias, fairness and politics Mar 2022 85.5% System cards only
MSR-VTT retrieval Models retrieve videos from text queries in the MSR-VTT collection. Video understanding Jun 2016 68.0 Fading
Robust IF Bench Models follow a diverse set of hard instructions from Microsoft AI's internal benchmark. Following precise instructions Jun 2026 61.2% In use
SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch. Fixing real repositories May 2025 71% Dropped
AMC 10/12 (November 2024) Models take the four November 2024 AMC 10 and 12 contests, 78 questions in all. Scored in contest points out of 150, averaged over tests. Competition problems with answers Nov 2024 124.5 System cards only
Do You See Me Models answer 2,612 visual perception questions on 1,758 images across seven 2D and 3D subtasks drawn from human psychology. Questions are multiple choice or numeric. General visual questions May 2025 51.1% System cards only
SITE Models answer multiple-choice spatial questions over single images, multiple images and video, drawn from about 30 vision datasets. Tasks span figural to environmental scale. Spatial reasoning and counting May 2025 32.8% Fading
COCO Models detect and locate objects in everyday photos from the Microsoft COCO dataset. Scored by mean average precision over box overlap thresholds. Detection and captioning May 2014 78.7% In use
Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. Audio understanding and voice Mar 2025 6.28 System cards only
Microsoft RAI benchmark: Grounding Models answer in GPT-4o simulated multi-turn conversations. A judge scores from 0 to 5 how fully each answer is grounded in the given prompt. Faithful to sources Dec 2024 4.62 System cards only
NTREX-128 Models translate 1,997 English news sentences into 128 languages, with human reference translations. Scored with metrics such as COMET or chrF. Translation Oct 2022 68.8 System cards only
OSI-Bench Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates. Spatial reasoning and counting Dec 2025 30.6% System cards only
ReCoRD Models fill in a masked entity in a query about a news article, using the article as context. Scoring is exact match and token F1. Reading comprehension Oct 2018 93.8% Dropped
MAI internal jailbreak evaluation suite Models face about 9,500 jailbreak prompts built from 2,500 seed scenarios, grouped as foundational, compositional and adaptive. Scored as attack success rate. Jailbreaks and prompt injection Jun 2026 System cards only
MAI internal safety and over-refusal evaluation Models answer harmful and benign prompts in eight harm categories. Judges score safety pass rate on sensitive prompts and helpfulness as one minus over-refusal. Refusing harmful requests Jun 2026 System cards only
MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. Exam questions Dec 2024 73.4% Not in cards
ConStory-Bench Models write long stories, up to tens of thousands of words, from 2,000 prompts in four task types. An automated checker, ConStory-Checker, finds contradictions in facts, timelines, characters and world rules and cites the passages behind each one. Scores count consistency errors, using a taxonomy of 5 categories and 19 subtypes. Creative writing Mar 2026 0.11 Not in cards
SimulatorArena Models act as students or writing clients. Researchers compare their messages and assistant ratings with those from people who used the same assistants. Multi-turn conversation 1.01 Not in cards
Thinkingbox-bench Agents complete 507 business workflows through tools and simulated user conversations. Executable checks require correct backend updates without forbidden side effects. Customer service Aug 2026 66.5% Not in cards
Taste-Bench Models choose the better next step at 502 decision points in software engineering and research trajectories. A question scores only when both option orders receive the correct answer. The overall score equally weights engineering and research. Machine learning engineering Sep 2026 59.7% Not in cards
ISA-Bench Models write programs for five games with restricted instruction sets. A verifier checks each program and gives feedback before the next attempt. Puzzles and chess Sep 2026 60.9% Not in cards
MMTU Models answer about 30,000 questions over real tables across 25 tasks, such as data cleaning, table joins, column and table transformation, table question answering and turning language into code. The tasks come from decades of computer science research on tabular data. Answers are checked automatically against references. Data science and SQL Jun 2025 69.6% Not in cards
WeaveBench Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts. Desktop apps Jun 2026 41.2% Not in cards
HealthAgentBench Agents use terminal tools to complete 54 clinical and biomedical tasks across seven categories. Task-specific verifiers measure success, including a model judge for radiology report correction. Medicine Jun 2026 55% Not in cards
LoopsBench Agents complete 112 software tasks with dependencies among smaller development units. Executable tests measure resolved tasks and the fraction of units that pass. Long terminal sessions Jul 2026 25% Not in cards
ProgramDistill Agents inspect working web applications and restore missing features in editable copies. Replayed browser traces check whether patches recover the reference behavior. The paper evaluates partial repairs and full application reconstruction. Web and app front ends Sep 2026 84.3% Not in cards
AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. Simulated robots Mar 2026 76.5% Not in cards
EmailBench Agents handle email and productivity requests in a synthetic inbox. The evaluator combines executable checks and model-graded rubrics across 206 scenarios. Documents and spreadsheets Sep 2026 33.5% Not in cards
HarnessAudit-Bench Agents run inside execution harnesses such as Claude Code, Codex and OpenClaw on 210 tasks across eight domains, from finance to software engineering, with safety rules built in. The audit reads the full trajectory through channels the agent cannot change. It scores whether the agent stays inside its permission and information-flow policies, completes checkpoints, and keeps both under prompt injection, vague goals and tool errors. Refusing harmful requests May 2026 0.41 Not in cards
OdysseyBench Agents recover requirements from conversation histories and complete workflows across documents, spreadsheets, PDFs, email, and calendars. Task-specific checks inspect the resulting file system and report the percentage of completed tasks. Documents and spreadsheets Aug 2025 58.7% Not in cards
HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. Medicine Oct 2026 57.3% Not in cards
AVGen-Bench Models generate videos with audio from prompts across eleven real-world categories. Specialist models and multimodal judges measure perceptual quality, synchronization, and adherence to detailed visual, speech, music, and physical requirements. Video generation Apr 2026 67.9 Not in cards
Delulu Models fill missing code in repository contexts across seven programming languages. Per-sample containers check compilation or type checking, while separate metrics compare completions with valid and hallucinated references. Function-level problems May 2026 84.5% Not in cards
PazaBench v2 Models transcribe speech in 61 African languages. The leaderboard measures character error rate, word error rate and transcription speed. Speech recognition and translation Jul 2026 73.3% Not in cards
ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1. Fixing real repositories Oct 2026 73.3% Not in cards
SmartPlay Language-model agents play a collection of games designed to test planning, reasoning, memory and interaction abilities. Puzzles and chess Oct 2023 0.80 Not in cards
STATE-Bench Main 0.7.0 Agents complete enterprise tasks in travel, customer support, and shopping. The score is overall pass@1 for this track and version. Customer service May 2026 55.7% Not in cards
TaskBench Models decompose requests, choose tools and dependencies, and predict tool parameters. Text metrics score decomposition, while graph and parameter F1 scores measure the remaining stages. Function calling Nov 2023 93.5 Not in cards
AndroidArena Agents use apps such as Gmail and Google Maps, sometimes switching apps or following extra constraints. GPT-4 judges success from the recorded actions and observations. Phones Feb 2024 12.7% Not in cards
STATE-Bench Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. Customer service Jun 2026 58.9% Not in cards
SWE-bench-Live Verified Task success on the official Verified split. The source reports each submission denominator separately. Other language and platform splits remain separate. Fixing real repositories May 2025 40% Not in cards
Chinese Essay Fluency Evaluation 2023 Models identify grammar errors in Chinese school essays, correct individual characters, and rewrite flawed sentences. Judges also review the highest ranked rewrites. Reading in many languages Mar 2023 51.6 Not in cards
GeoMap-Bench Models answer 3,864 questions about 124 geologic maps. They locate map features, interpret legends, and use geological evidence to explain conditions at a site. Documents, charts and text in images Jan 2025 0.37 Not in cards
LiveDRBench Agents perform 100 deep research tasks and return key claims discovered during search. Scores use claim-level F1. Deep research Aug 2025 55.5% Not in cards
Chinese Essay Discourse Coherence Corpus Models grade coherence in Chinese school essays, identify topic sentences, and classify relationships between passages. The authors hold out the final 100 essays. Reading in many languages Dec 2023 Not in cards
Chinese Essay Fluency Evaluation 2024 Models identify errors and rewrite sentences from Chinese school essays. A new track asks them to rate the fluency of a whole essay. Reading in many languages Jan 2024 Not in cards
Chinese Essay Rhetoric Recognition and Understanding 2024 Models classify figures of speech in sentences from Chinese student essays. They identify the words that form each figure. Reading in many languages Jan 2024 Not in cards
CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. Spatial reasoning and counting Oct 2026 Not in cards
CodeXGLUE A suite of 10 code tasks across 14 datasets, grouped as code to code, text to code, code to text and text to text. Tasks include clone detection, defect detection, code completion, code repair, code search, code generation and code summarization. Each task has its own metric, such as accuracy, BLEU or MRR. Function-level problems Feb 2021 Not in cards
CompactionBench Agents continue multi-turn coding sessions while context compaction occurs during software evolution tasks. Scores measure task quality and compaction cost. Long terminal sessions Oct 2026 Not in cards
DELEGATE-52 Models perform long delegated document-editing workflows across 52 professional domains. Scores measure artifact fidelity after repeated transformations. Documents and spreadsheets Apr 2026 Not in cards
FAD Music Listening Test Models estimate acoustic and musical quality for ten-second music excerpts. The test compares predictions with human ratings from ten listeners. Audio understanding and voice Nov 2023 Not in cards
How2QA Models answer multiple-choice questions about instructional videos. The task tests understanding of the actions shown in each clip. Video understanding May 2020 Not in cards
LLaVA-Bench (In-the-Wild) Multimodal assistants answer open questions about a small set of diverse real images. A GPT-4 judge scores each answer relative to a reference answer. General visual questions Apr 2023 Not in cards
Magentic Marketplace Agents search for sellers, exchange proposals and make purchases. The simulation measures how much value customers receive after paying for their purchases. Running a business Oct 2025 Not in cards
MMIE Models interpret and generate interleaved text and images. MMIE-Score, a fine-tuned InternVL-2-4B model, grades the resulting multimodal responses. General visual questions Oct 2024 Not in cards
MS MARCO Question Answering Models answer real Bing queries using retrieved web passages. BLEU and ROUGE-L compare generated answers with human-written references. Reading comprehension Nov 2016 Not in cards
MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. Medicine Jan 2023 Not in cards
NLPCC 2023 Chinese Essay Discourse Coherence Evaluation Models rate coherence in Chinese school essays, extract topic sentences, and label relationships between paragraphs or sentences. Reading in many languages Apr 2023 Not in cards
NLPCC 2024 Chinese Essay Discourse Logic Evaluation and Integration Models locate logical errors, identify paragraph topics and their relationships, and write feedback on Chinese school essays. The organizers average three track scores. Reading in many languages Mar 2024 Not in cards
NOTSOFAR-1 Models transcribe distant meetings and assign speech to speakers. NVIDIA also measures diarization error on the evaluation recordings. Audio understanding and voice Jan 2024 Not in cards
PAM-Audio Models judge the quality of generated environmental sounds and how closely each recording matches its caption. The listening study used 100 AudioCaps captions. Audio understanding and voice Feb 2024 Not in cards
PazaBench Models transcribe speech in 39 African languages. The leaderboard measures character error rate, word error rate and transcription speed. Speech recognition and translation Feb 2026 Not in cards
RTP-LX Models continue or judge toxic prompts hand-translated from RealToxicityPrompts into dozens of languages, with native-speaker labels. Measures toxic output and toxicity detection. Bias, fairness and politics Apr 2024 Not in cards
SDBench Diagnostic agents order tests, ask questions and commit diagnoses in sequential clinical cases. Scores combine diagnostic correctness with test cost. Medicine Oct 2026 Not in cards
SentinelBench Agents monitor changing web environments and respond when scripted events make progress possible. Scores measure completion, reaction time and resource use. Web browsing Jun 2026 Not in cards
Sequential Question Answering Models answer sequences of related questions about Wikipedia tables. Answers identify table cells, so later questions can depend on earlier selections. Evaluation checks predicted answers against the annotated cells. Reading comprehension Nov 2016 Not in cards
SVAMP Models solve 1,000 grade-school arithmetic word problems. The problems are variations of problems from earlier datasets, made to catch solvers that ignore the question or the word order. The final numeric answer is scored by exact match. Word problems Mar 2021 Not in cards
UW Clinical Record Section Chunker i2b2 Discharge Summary Corpus Models locate section boundaries in discharge summaries and label each section. Researchers compare the predicted labels and boundaries with human annotations. Medicine May 2012 Not in cards
Verus-Bench Agents synthesize Verus proof annotations for 150 Rust verification tasks from algorithms, arrays and loop programs. Function-level problems Oct 2024 Not in cards
VeruSAGE-Bench Agents synthesize Verus proofs for 849 tasks extracted from eight real-world Rust systems projects. Fixing real repositories Dec 2025 Not in cards

Arenas

Name Kind Released Best score Status
MAI human side-by-side evaluation Paid raters compare two models' answers to 1,276 English prompts from experts and Copilot logs on a 7-point scale from -1.5 to 1.5. Overall chat quality Jun 2026 0.07 System cards only
MAI Thinking 1 blind human preference evaluation Professional raters compare anonymous assistant responses to single-turn and multi-turn prompts and score their overall preference. Overall chat quality Jun 2026 Not in cards