Benchmark Almanac

Amazon

We track 21 benchmarks that Amazon made or helped make. Its models are on its lab page.

1 in use 19 with public code 9 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
BOLD Models continue 23,679 English Wikipedia prompts about professions, gender, race, religion and politics. Classifiers score sentiment, toxicity and polarity per group. Bias, fairness and politics Jan 2021 49.1 System cards only
tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. Customer service Dec 2025 82.7% Fading
RealKIE-FCC Verified Model extracts structured fields from scanned FCC invoices into a JSON schema. Amazon human-verified labels. Field-level similarity (ALNS). Documents, charts and text in images Dec 2025 67% Fading
MBXP Models solve MBPP problems transpiled with their tests into more than 10 programming languages, such as Java, C++ and Go. Unit tests check each solution. Function-level problems Oct 2022 78.4% Dropped
GaRAGe Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages. Faithful to sources Jun 2025 60.7% In use
MIRAGE-MMST Models answer growers’ questions using photographs and written descriptions. They identify the cause of a problem and recommend how to manage it. General visual questions Jun 2025 68.0 Not in cards
Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. Running a business Oct 2024 72.6 Not in cards
PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. Medicine Jul 2026 4.25 Not in cards
MIRAGE-MMMT Models decide whether to ask for more information or answer a grower during an agricultural consultation. They then write the question or answer. Multi-turn conversation Jun 2025 65.5% Not in cards
SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. Fixing real repositories Apr 2025 14.1% Not in cards
Chronos Benchmark II Models forecast future values across 27 time series datasets. The test compares point and probabilistic forecasts against a seasonal baseline. Real-world events Mar 2024 Not in cards
CiteBench Models answer information-seeking questions with citations from retrieved passages. Human annotations score citation quality across relevance, support and completeness. Faithful to sources Jun 2025 Not in cards
CSR-Bench Agents deploy and run computer-science research repositories with shell, retrieval and web-search tools. Task success measures whether the requested repository workflow completes. Fixing real repositories Jan 2025 Not in cards
ESCI Shopping Queries Dataset Models rank products for a shopping query and distinguish exact matches, substitutes, complements, and irrelevant items. Questions that need search Jun 2022 Not in cards
ExPUNations Models explain English puns and write new ones from keywords. Human raters check whether a generated pun supports both meanings. Creative writing Oct 2022 Not in cards
FEVER Systems check 185,445 claims against Wikipedia and label each Supported, Refuted or NotEnoughInfo. Annotators wrote the claims by altering Wikipedia sentences. The full FEVER score also requires the right evidence sentences. Faithful to sources Mar 2018 Not in cards
LongLeader Models answer a unified suite of long-context tasks across multiple context lengths and abilities. Accuracy tables compare language models and context-extension methods. Long context Jan 2025 Not in cards
MASSIVE Models identify an utterance’s intent and label its argument slots across 51 languages. The held-out test reports intent accuracy, slot-filling F1, and exact match accuracy for the complete prediction. Reading in many languages Apr 2022 Not in cards
RefChecker Models generate answers that are checked for fine-grained claims against references. Claim-level labels measure factual consistency and hallucination detection quality. Hallucination rates Jan 2024 Not in cards
SOP-Bench Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points. Running a business Feb 2026 Not in cards
TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. Simulated robots Oct 2021 Not in cards