Benchmark Almanac

Apple

We track 29 benchmarks and 1 arena that Apple made or helped make. Its models are on its lab page.

1 in use 20 with public code 6 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. Detection and captioning Nov 2020 13.5 Fading
ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. General assistants Aug 2024 80.7% In use
MIA-Bench Models answer 400 image prompts that stack several strict instructions on format, length and content. An LLM judge scores how well each answer follows every instruction. General visual questions Jul 2024 91% Dropped
GSM-Symbolic Models solve grade-school word problems generated from symbolic templates of GSM8K questions, with names and numbers changed. P1 and P2 add one or two extra clauses. Word problems Oct 2024 94.9% Fading
ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. Detection and captioning Nov 2021 53.7% Dropped
RSI-Inference Bench Coding agents add model support to inference engines, improve serving speed and fix performance regressions. The harness builds and runs each submission. Machine learning engineering Oct 2026 2,044 Not in cards
KnowU-Bench Agents use Android apps such as Mattermost and Taodian. They infer preferences, ask for missing details, and decide when to act or stay silent. Phones Apr 2026 60.4% Not in cards
AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. Audio understanding and voice Dec 2025 Not in cards
Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. General assistants Jul 2025 Not in cards
ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. General assistants Oct 2026 Not in cards
CapQuiz Models are evaluated through human-verified questions answered from generated video captions, measuring factuality and coverage of salient content. Video captioning Sep 2026 Not in cards
COMPASS Agents plan travel across transportation, lodging and tickets while satisfying hard constraints and optimizing stated preferences over multiple turns. Apps and MCP servers Oct 2025 Not in cards
DeepAmbigQA Search-augmented models answer multi-hop questions that combine entity-name ambiguity with complete answer-set retrieval. Questions that need search Nov 2025 Not in cards
EncQA Vision-language models answer chart questions that isolate visual encoding channels and analytic operations. Documents, charts and text in images Aug 2025 Not in cards
Entity-Deduction Arena Models ask questions to identify a hidden entity in multi-turn games. Success, question count and a combined score measure clarification, state tracking and planning. General assistants Feb 2024 Not in cards
ETVA Text-to-video systems are scored on fine-grained prompt alignment through generated atomic questions and video-language answers. Video generation Mar 2025 Not in cards
GIE-Bench Image-editing models follow text instructions while preserving unedited regions. GPT-4o question answers and masked image metrics score functional correctness and preservation. General visual questions May 2025 Not in cards
LVSum Models summarize long videos with timestamped segments, relevance scores and descriptions using video frames, transcripts or both. Long videos Apr 2026 Not in cards
MAD-Bench Multimodal models answer image questions containing deceptive prompt information about objects, counts and spatial relationships. General visual questions Feb 2024 Not in cards
MKQA Models answer 10,000 open-domain questions translated into 26 languages. Exact match and token F1 measure agreement with human answers, including unanswerable questions. Knowledge in many languages Jul 2020 Not in cards
MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. General assistants Jul 2026 Not in cards
NarrativeTrack Video-language models answer entity-centric questions about existence, changes and ambiguity as narratives unfold across clips. Video understanding Jan 2026 Not in cards
OverSearchQA Search-augmented models decide when to abstain from retrieval on unknown, false-premise and underspecified questions. Questions that need search Jan 2026 Not in cards
Pare-Bench Proactive assistants infer user goals and act across stateful digital applications with simulated users and multi-app task environments. Apps and MCP servers Apr 2026 Not in cards
REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. Simulated robots Sep 2026 Not in cards
SFI-Bench Multimodal models answer video questions about spatial layouts, object affordances and functional reasoning in indoor scenes. Spatial reasoning and counting May 2026 Not in cards
SO-Bench Multimodal models produce schema-compliant structured outputs grounded in images from four visual domains. Static checks and rubric-based grading score semantic correctness and schema compliance. Documents, charts and text in images Dec 2025 Not in cards
SPACE Models solve spatial cognition tasks involving maps, object layouts, spatial attention and memory in text and image presentations. Accuracy measures task performance. Spatial reasoning and counting Apr 2025 Not in cards
VSAS-Bench Streaming vision-language models answer tasks over online video frames while evaluation measures accuracy, response timing and consistency. Video understanding Apr 2026 Not in cards

Arenas

Name Kind Released Best score Status
Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. Overall chat quality Jun 2025 64.7% Fading