Apple
We track 29 benchmarks and 1 arena that Apple made or helped make. Its models are on its lab page.
1 in use 20 with public code 6 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. | Detection and captioning | Nov 2020 | | Fading |
| ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. | General assistants | Aug 2024 | 80.7% | In use |
| MIA-Bench Models answer 400 image prompts that stack several strict instructions on format, length and content. An LLM judge scores how well each answer follows every instruction. | General visual questions | Jul 2024 | 91% | Dropped |
| GSM-Symbolic Models solve grade-school word problems generated from symbolic templates of GSM8K questions, with names and numbers changed. P1 and P2 add one or two extra clauses. | Word problems | Oct 2024 | 94.9% | Fading |
| ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. | Detection and captioning | Nov 2021 | | Dropped |
| RSI-Inference Bench Coding agents add model support to inference engines, improve serving speed and fix performance regressions. The harness builds and runs each submission. | Machine learning engineering | Oct 2026 | 2,044 | Not in cards |
| KnowU-Bench Agents use Android apps such as Mattermost and Taodian. They infer preferences, ask for missing details, and decide when to act or stay silent. | Phones | Apr 2026 | 60.4% | Not in cards |
| AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. | Audio understanding and voice | Dec 2025 | Not in cards | |
| Asper-Bench Agents turn complex user requests into programs that execute multi-step actions in a simulated digital assistant environment. Automated checks compare the resulting state with the requested goal. | General assistants | Jul 2025 | Not in cards | |
| ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. | General assistants | Oct 2026 | Not in cards | |
| CapQuiz Models are evaluated through human-verified questions answered from generated video captions, measuring factuality and coverage of salient content. | Video captioning | Sep 2026 | Not in cards | |
| COMPASS Agents plan travel across transportation, lodging and tickets while satisfying hard constraints and optimizing stated preferences over multiple turns. | Apps and MCP servers | Oct 2025 | Not in cards | |
| DeepAmbigQA Search-augmented models answer multi-hop questions that combine entity-name ambiguity with complete answer-set retrieval. | Questions that need search | Nov 2025 | Not in cards | |
| EncQA Vision-language models answer chart questions that isolate visual encoding channels and analytic operations. | Documents, charts and text in images | Aug 2025 | Not in cards | |
| Entity-Deduction Arena Models ask questions to identify a hidden entity in multi-turn games. Success, question count and a combined score measure clarification, state tracking and planning. | General assistants | Feb 2024 | Not in cards | |
| ETVA Text-to-video systems are scored on fine-grained prompt alignment through generated atomic questions and video-language answers. | Video generation | Mar 2025 | Not in cards | |
| GIE-Bench Image-editing models follow text instructions while preserving unedited regions. GPT-4o question answers and masked image metrics score functional correctness and preservation. | General visual questions | May 2025 | Not in cards | |
| LVSum Models summarize long videos with timestamped segments, relevance scores and descriptions using video frames, transcripts or both. | Long videos | Apr 2026 | Not in cards | |
| MAD-Bench Multimodal models answer image questions containing deceptive prompt information about objects, counts and spatial relationships. | General visual questions | Feb 2024 | Not in cards | |
| MKQA Models answer 10,000 open-domain questions translated into 26 languages. Exact match and token F1 measure agreement with human answers, including unanswerable questions. | Knowledge in many languages | Jul 2020 | Not in cards | |
| MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. | General assistants | Jul 2026 | Not in cards | |
| NarrativeTrack Video-language models answer entity-centric questions about existence, changes and ambiguity as narratives unfold across clips. | Video understanding | Jan 2026 | Not in cards | |
| OverSearchQA Search-augmented models decide when to abstain from retrieval on unknown, false-premise and underspecified questions. | Questions that need search | Jan 2026 | Not in cards | |
| Pare-Bench Proactive assistants infer user goals and act across stateful digital applications with simulated users and multi-app task environments. | Apps and MCP servers | Apr 2026 | Not in cards | |
| REVERSAL-BENCH Reset-free reinforcement learning agents learn manipulation policies across environments with controlled reversibility and recoverability checks. | Simulated robots | Sep 2026 | Not in cards | |
| SFI-Bench Multimodal models answer video questions about spatial layouts, object affordances and functional reasoning in indoor scenes. | Spatial reasoning and counting | May 2026 | Not in cards | |
| SO-Bench Multimodal models produce schema-compliant structured outputs grounded in images from four visual domains. Static checks and rubric-based grading score semantic correctness and schema compliance. | Documents, charts and text in images | Dec 2025 | Not in cards | |
| SPACE Models solve spatial cognition tasks involving maps, object layouts, spatial attention and memory in text and image presentations. Accuracy measures task performance. | Spatial reasoning and counting | Apr 2025 | Not in cards | |
| VSAS-Bench Streaming vision-language models answer tasks over online video frames while evaluation measures accuracy, response timing and consistency. | Video understanding | Apr 2026 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. | Overall chat quality | Jun 2025 | 64.7% | Fading |