| PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. | Medicine | May 2023 | 66.2% | Fading |
| SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. | Web and app front ends | Aug 2025 | 0.92 | Fading |
| R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. | General visual questions | Oct 2024 | 75.4% | System cards only |
| SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. | Fixing real repositories | May 2025 | 66.6% | Dropped |
| MemGUI-Bench Agents remember details while working across Android apps. Tasks include recording audio and creating calendar events. A model checks the actions and results across repeated attempts. | Phones | Feb 2026 | 73.4% | Not in cards |
| OBI-Bench Models locate inscriptions, rejoin bone fragments, classify characters, retrieve matches, and explain glyphs. The authors publish separate scores for each task. | Documents, charts and text in images | Oct 2024 | 0.61 | Not in cards |
| CaLM Models answer causal discovery, intervention, and counterfactual reasoning tasks in English and Chinese. Evaluation compares answers with reference labels to measure accuracy and robustness across causal targets and prompting methods. | Language reasoning | May 2024 | 45.3% | Not in cards |
| IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. | Research science | Jul 2026 | 27.3% | Not in cards |
| OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. | CAD models | Aug 2026 | 100% | Not in cards |
| MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. | General visual questions | Feb 2025 | 82.5% | Not in cards |
| ConceptEdit-Bench Models edit images according to 1,000 instructions across six categories. A vision-language judge scores instruction following, visual quality, and consistency in unedited regions from 0 to 10. The per-case score averages those three dimensions. | Image generation and editing | Aug 2026 | 66.2 | Not in cards |
| HABIT-Bench Agent memory systems must infer user habits from weak evidence spread across many past conversations and apply them correctly later. | Learning from the context | Oct 2026 | 42% | Not in cards |
| SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. | Fixing real repositories | Jun 2026 | 21.5% | Not in cards |
| SWE-Game Agents build, complete, repair, or port games across 247 tasks derived from 41 reference games. Runtime probes and input replays check gameplay mechanics and preserved behavior. Vision-language rubrics separately grade presentation in construction tasks. | Building whole projects | Sep 2026 | 64.0 | Not in cards |
| PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. | Image generation and editing | Oct 2025 | 61.9% | Not in cards |
| VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. | Video generation | Sep 2026 | 79.0 | Not in cards |
| OS-Kairos Agents predict Android actions for tasks such as finding a product in Amazon. The task succeeds when every action matches the recorded trajectory. | Phones | Feb 2025 | 29.8% | Not in cards |
| APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. | Long videos | Sep 2026 | 60.2% | Not in cards |
| CMD-AD-Eval Models describe what happens on screen during gaps in movie dialogue. The authors compare the descriptions with timed human narration using text overlap and model judgments. | Video captioning | Apr 2024 | | Not in cards |
| doteBench Models edit English and Chinese recordings using a transcript that marks where each change belongs. The evaluator checks edit success and preservation outside those regions. | Audio understanding and voice | Aug 2026 | | Not in cards |
| M³AV Speech Recognition Models transcribe academic lectures. The authors measure word errors across the transcript and separately for rare words. Slide text can provide context. | Speech recognition and translation | Mar 2024 | | Not in cards |
| MAD-Eval Models write descriptions for scenes from ten movies. One set of references names characters. Another replaces their names with the word “SOMEONE.” | Video captioning | Mar 2023 | | Not in cards |
| META-GUI Models predict the next Android action and reply from screenshots and dialogue history. A turn succeeds when every predicted action matches the recorded interaction. | Phones | May 2022 | | Not in cards |
| TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. | Simulated robots | Oct 2026 | | Not in cards |
| TV-AD Models describe what viewers can see during clips from television episodes. The test pairs each clip with a timed reference description. | Video captioning | Jul 2024 | | Not in cards |
| WebSRC Models answer questions about a web page from its screenshot, HTML or both. The set has 400K question and answer pairs from 6.4K web pages. Scored by exact match and F1. | Documents, charts and text in images | Jan 2021 | | Not in cards |