Amazon
We track 21 benchmarks that Amazon made or helped make. Its models are on its lab page.
1 in use 19 with public code 9 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| BOLD Models continue 23,679 English Wikipedia prompts about professions, gender, race, religion and politics. Classifiers score sentiment, toxicity and polarity per group. | Bias, fairness and politics | Jan 2021 | | System cards only |
| tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. | Customer service | Dec 2025 | 82.7% | Fading |
| RealKIE-FCC Verified Model extracts structured fields from scanned FCC invoices into a JSON schema. Amazon human-verified labels. Field-level similarity (ALNS). | Documents, charts and text in images | Dec 2025 | | Fading |
| MBXP Models solve MBPP problems transpiled with their tests into more than 10 programming languages, such as Java, C++ and Go. Unit tests check each solution. | Function-level problems | Oct 2022 | | Dropped |
| GaRAGe Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages. | Faithful to sources | Jun 2025 | | In use |
| MIRAGE-MMST Models answer growers’ questions using photographs and written descriptions. They identify the cause of a problem and recommend how to manage it. | General visual questions | Jun 2025 | 68.0 | Not in cards |
| Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. | Running a business | Oct 2024 | 72.6 | Not in cards |
| PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. | Medicine | Jul 2026 | 4.25 | Not in cards |
| MIRAGE-MMMT Models decide whether to ask for more information or answer a grower during an agricultural consultation. They then write the question or answer. | Multi-turn conversation | Jun 2025 | 65.5% | Not in cards |
| SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. | Fixing real repositories | Apr 2025 | 14.1% | Not in cards |
| Chronos Benchmark II Models forecast future values across 27 time series datasets. The test compares point and probabilistic forecasts against a seasonal baseline. | Real-world events | Mar 2024 | Not in cards | |
| CiteBench Models answer information-seeking questions with citations from retrieved passages. Human annotations score citation quality across relevance, support and completeness. | Faithful to sources | Jun 2025 | Not in cards | |
| CSR-Bench Agents deploy and run computer-science research repositories with shell, retrieval and web-search tools. Task success measures whether the requested repository workflow completes. | Fixing real repositories | Jan 2025 | Not in cards | |
| ESCI Shopping Queries Dataset Models rank products for a shopping query and distinguish exact matches, substitutes, complements, and irrelevant items. | Questions that need search | Jun 2022 | Not in cards | |
| ExPUNations Models explain English puns and write new ones from keywords. Human raters check whether a generated pun supports both meanings. | Creative writing | Oct 2022 | Not in cards | |
| FEVER Systems check 185,445 claims against Wikipedia and label each Supported, Refuted or NotEnoughInfo. Annotators wrote the claims by altering Wikipedia sentences. The full FEVER score also requires the right evidence sentences. | Faithful to sources | Mar 2018 | Not in cards | |
| LongLeader Models answer a unified suite of long-context tasks across multiple context lengths and abilities. Accuracy tables compare language models and context-extension methods. | Long context | Jan 2025 | Not in cards | |
| MASSIVE Models identify an utterance’s intent and label its argument slots across 51 languages. The held-out test reports intent accuracy, slot-filling F1, and exact match accuracy for the complete prediction. | Reading in many languages | Apr 2022 | Not in cards | |
| RefChecker Models generate answers that are checked for fine-grained claims against references. Claim-level labels measure factual consistency and hallucination detection quality. | Hallucination rates | Jan 2024 | Not in cards | |
| SOP-Bench Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points. | Running a business | Feb 2026 | Not in cards | |
| TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. | Simulated robots | Oct 2021 | Not in cards |