Vision
Models work from images or video. We count 1392 benchmarks here, 50 of them arenas. A benchmark belongs here when input includes images or video.
How fast it grows
289 were released since July, more than the 201 in the first half of 2026.
How much is still open
No model has passed 85% on 543 of the 650 benchmarks here that score in percent, 84% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| 3D GameDev Arena Agents build playable 3D browser games from specifications and reference images. Players compare anonymous game pairs. The authors also test functionality and playability. | Expert | Nov 2026 | 99.8% | Not in cards |
| 3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. | Expert | Nov 2026 | 59.5 | Not in cards |
| ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. | Everyday | Nov 2026 | | Not in cards |
| ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. | Everyday | Nov 2026 | Not in cards | |
| AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. | University | Nov 2026 | Not in cards | |
| AVCC Next-Speaker Prediction Models predict who will speak after a pause in a conversation. The authors compare two models with human listeners on the same 1,730 events. | Everyday | Nov 2026 | Not in cards | |
| BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. | Expert | Nov 2026 | 66.7% | Not in cards |
| CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. | Everyday | Nov 2026 | | Not in cards |
| DashBench Models explain rare events in driving videos, including near misses and collisions. The authors compare 25 video models on 1,709 clips. | Everyday | Nov 2026 | 34.8% | Not in cards |
| DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. | Everyday | Nov 2026 | | Not in cards |
| EarthRecord Agents analyze multidecade satellite observations to answer land-change questions. Reference checks score answers to 238 questions across 56 sites. | Expert | Nov 2026 | 85.4% | Not in cards |
| Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. | University | Nov 2026 | Not in cards | |
| EgoHome Models annotate 137 egocentric clips of household chores, about two hours in all, with subtasks, hand actions, hand captions and hand boxes. Segments are as short as 0.1 seconds. A model judge matches segment labels, and boxes are scored by centroid hits. | Everyday | Nov 2026 | 0.80 | Not in cards |
| EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. | University | Nov 2026 | Not in cards | |
| Encord Egocentric VQA Benchmark Models answer 1,500 multiple-choice questions about actions, objects, space and time in private first-person videos. | Everyday | Nov 2026 | 76.5% | Not in cards |
| EVF-Bench Models generate product videos with visual effects. The evaluator checks motion, product appearance, text stability, and whether the effects fit the request. | University | Nov 2026 | | Not in cards |
| ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. | Everyday | Nov 2026 | Not in cards | |
| FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. | University | Nov 2026 | | Not in cards |
| FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. | Expert | Nov 2026 | 36.7% | Not in cards |
| GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. | Everyday | Nov 2026 | | Not in cards |
| GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. | Everyday | Nov 2026 | | Not in cards |
| GradeBench Models read a student's handwritten answer to a university STEM exam question, with the rubric and a reference solution, and pick the rubric items to award. Scored by exact match with the items the instructor chose, averaged over five runs. | University | Nov 2026 | 57.3% | Not in cards |
| HandWorldBench Models generate videos of human or robotic hands moving objects. The benchmark combines 17 measures of hand motion, object state, and video quality. | Everyday | Nov 2026 | | Not in cards |
| HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. | Everyday | Nov 2026 | 18.8% | Not in cards |
| HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. | Everyday | Nov 2026 | 61.9% | Not in cards |
| Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. | Expert | Nov 2026 | 83.9% | Not in cards |
| LongBanana Models synthesize images from many reference images and textual constraints. Atomic rubric checks score content fidelity, relationships, and global constraints across 90 tasks. | Everyday | Nov 2026 | | Not in cards |
| ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. | Everyday | Nov 2026 | Not in cards | |
| MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. | University | Nov 2026 | | Not in cards |
| MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. | Everyday | Nov 2026 | 50.4% | Not in cards |
| MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. | Expert | Nov 2026 | | In use |
| MotoVerse Models use a rider's camera to anticipate maneuvers and predict nearby road users' trajectories. The authors compare vision, language, and conventional prediction models. | Everyday | Nov 2026 | | Not in cards |
| MVDriveBench Models answer driving questions using synchronized videos from cameras facing different directions. The authors compare surround-view results with front-camera results. | Everyday | Nov 2026 | 80.1% | Not in cards |
| NEST Models interpret children's gaze, gestures, and physical contact in social videos. The benchmark checks classification and choice questions and separately grades written descriptions. | Everyday | Nov 2026 | | Not in cards |
| NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. | Expert | Nov 2026 | 23% | Not in cards |
| OmniScore-Bench Models read sheet music, locate symbols, and transcribe passages. The evaluator checks their responses against annotations for the relevant part of the score. | University | Nov 2026 | 91.0 | Not in cards |
| PhysCapBench Models caption videos of physical interactions. The benchmark checks which motions, contacts, and changes of state the captions describe across 1,500 clips. | Everyday | Nov 2026 | Not in cards | |
| Pixel2Perspective Models locate and segment objects from aerial and ground views. Tests vary the camera angle, weather, and use of visible or infrared images. | Everyday | Nov 2026 | Not in cards | |
| PReMind-Bench Models inspect a scene and decide whether to retrieve a personal memory before answering. Judges check whether the answer addresses the hazard and uses the right person’s information. | Everyday | Nov 2026 | 87.3% | Not in cards |
| Rapidata Multi-Step Image Editing (Identity Drift) Image editors apply five sequential changes to 199 photographs, and people compare the final faces for identity drift. | Expert | Nov 2026 | 1,866 | Not in cards |
| Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. | Expert | Nov 2026 | 0.60 | Not in cards |
| ReSpaceBench Models recognize rooms across separate visits and track objects that move between visits. The authors test 32 models on synthetic and real homes. | Everyday | Nov 2026 | 50.5% | Not in cards |
| RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. | Everyday | Nov 2026 | | Not in cards |
| RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. | Everyday | Nov 2026 | 16% | Not in cards |
| RoboColosseum Physical robots execute fresh instructions in paired blind trials. Human evaluators choose the better run or a tie, and comparisons determine ratings for each embodiment. | Everyday | Nov 2026 | Not in cards | |
| RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. | University | Nov 2026 | 80.6% | Not in cards |
| RSLiveBench-UAV Models answer questions about vehicles in drone images. The benchmark checks perception and spatial reasoning across different views and adds newly captured scenes. | Everyday | Nov 2026 | 59.9 | Not in cards |
| Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. | Research frontier | Nov 2026 | 44.7 | Not in cards |
| Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. | Everyday | Nov 2026 | 42.9% | Not in cards |
| ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. | Everyday | Nov 2026 | 48.8 | Not in cards |
| Snowflake AI_EXTRACT Document Extraction Evaluation Models extract fields, lists and tables across 2,789 annotated tasks from 11 datasets. Missing items, extra items and incorrect cells reduce the score. | University | Nov 2026 | Not in cards | |
| Social Signal Bench Models identify overlapping social signals in conversations from audio or video. The score weights each label by annotator agreement and gives credit for related predictions. | Everyday | Nov 2026 | | Not in cards |
| SolidWorks Bench Agents model the custom parts of 73 real machines in SolidWorks and assemble them from raster drawing packages, through screenshots and the SolidWorks API. A grader checks the geometry against the reference and reads mates, motion and feature trees from the native files. | Expert | Nov 2026 | 0.57 | Not in cards |
| Spatial Intelligence Index v2 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. | University | Nov 2026 | 88.7 | Not in cards |
| Terminal-Bench Math Modeling Agents use a terminal to solve 18 mathematical modeling problems and submit numerical answers. Feasibility checks and numerical scoring compare answers with reference solutions. | Competition | Nov 2026 | 64.3 | Not in cards |
| TriPhysBench Models answer quantitative questions about physical events in videos. Some questions require measuring an object or looking up its specifications. | School | Nov 2026 | | Not in cards |
| VibePrevisBench Coding agents write Blender Python that turns a script, photos, an opening frame or reference footage into a graybox 4D previs scene and render it. Rule checks score the exported scene against hidden facts, and a VLM judge checks that those facts are visible in the render. | Expert | Nov 2026 | 77.7 | Not in cards |
| VideoACE Models answer questions about fast actions in short videos. They can inspect selected frames more closely and request hints from a simulated viewer. | Everyday | Nov 2026 | | Not in cards |
| WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. | University | Nov 2026 | Not in cards | |
| 4DCodeBench Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion. | Expert | Oct 2026 | 0.79 | Not in cards |
| AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. | Everyday | Oct 2026 | Not in cards | |
| ABACUS Models add, remove, or change a specified number of objects in an image. The evaluator checks the count and preservation of the surrounding scene. | School | Oct 2026 | | Not in cards |
| AD4Edu-Bench Models write audio descriptions for university lectures. A rubric checks accessibility rules, and the authors compare the rankings with blind and low-vision listeners' preferences. | University | Oct 2026 | | Not in cards |
| AdSpark-Bench Models generate advertisement videos from product images and prompts. The benchmark checks whether products remain consistent and whether each shot conveys the requested selling point. | University | Oct 2026 | | Not in cards |
| ANCHOR Models explain answers to audiovisual questions and locate the supporting evidence. The benchmark includes time intervals, image regions, and relevant sounds. | Everyday | Oct 2026 | | Not in cards |
| AVPS-360 Models locate and track the person making a sound in panoramic video. The annotations connect each sound source to a person across frames. | Everyday | Oct 2026 | Not in cards | |
| BenchCAD 2.0 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. | Expert | Oct 2026 | 59% | Not in cards |
| BenchCAD 2.1 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. | Expert | Oct 2026 | 83.5% | In use |
| BenchCAD 2.1-beta Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. | Expert | Oct 2026 | Not in cards | |
| BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. | Expert | Oct 2026 | 79.9 | Not in cards |
| BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. | Everyday | Oct 2026 | 0.56 | Not in cards |
| ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. | Competition | Oct 2026 | 64.9% | Not in cards |
| CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. | Expert | Oct 2026 | 76.6 | Not in cards |
| CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. | Expert | Oct 2026 | Not in cards | |
| Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. | Expert | Oct 2026 | $9,514 | Not in cards |
| ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. | Everyday | Oct 2026 | 44.7% | Not in cards |
| Co-Design Bench Agents revise existing applications from multimodal user requests. A judge checks browser and repository evidence against 2,331 expert criteria across 273 tasks. | Expert | Oct 2026 | 69.9% | Not in cards |
| Code2Video-Compass Multimodal models write programs that render complete videos from text, an image or a video. The default Overall board ranks models by their mean rank across quality metrics, so a lower score is better. | Everyday | Oct 2026 | 1.07 | Not in cards |
| CodeCap-Bench Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail. | Everyday | Oct 2026 | 73.2 | Not in cards |
| CREATE Models find a word that connects three photographs. A judge grades each answer against the intended connection. | Everyday | Oct 2026 | | Not in cards |
| DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. | Everyday | Oct 2026 | 40% | Not in cards |
| Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. | University | Oct 2026 | 957 | In use |
| DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. | Everyday | Oct 2026 | 52.1% | Not in cards |
| Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. | Everyday | Oct 2026 | Not in cards | |
| Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. | Expert | Oct 2026 | 59.8 | Not in cards |
| Earth-Bench V2 Agents analyze prepared satellite observations. The corrected release measures final answer accuracy, tool selection, argument accuracy, and execution efficiency. | University | Oct 2026 | 80% | Not in cards |
| EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. | University | Oct 2026 | Not in cards | |
| Ego2Act Models generate first-person video showing how to complete a task from an initial image. A judge checks the steps and their physical plausibility. | Everyday | Oct 2026 | | Not in cards |
| Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. | Everyday | Oct 2026 | Not in cards | |
| EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. | Everyday | Oct 2026 | 52% | Not in cards |
| EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. | University | Oct 2026 | 41.4% | Not in cards |
| EvaOmni Models answer questions about videos and identify the audio and visual evidence behind their answers. The score accounts for both correctness and evidence timing. | Everyday | Oct 2026 | | Not in cards |
| FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. | Expert | Oct 2026 | 0.67 | Not in cards |
| FrontierCAD Parts Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score. | Expert | Oct 2026 | 0.63 | Not in cards |
| GeoFlip Models generate videos from paired scenes where a geometry change reverses the expected physical outcome. Model judges check the event and the scene structure. | School | Oct 2026 | | Not in cards |
| GeoGuessBench (Nutlope) Models place a pin after seeing one street photo. Each game has five rounds, and closer guesses earn more points. | Everyday | Oct 2026 | 23412.0 | Not in cards |
| GhostInk: Honest OCR on Bill Photos Models read text from phone photos and mask unreadable characters. The grader penalizes guessed text, even when the guess happens to match. | Everyday | Oct 2026 | 0% | Not in cards |
| HealBench-Eval Models generate images intended to provide psychological comfort. The authors compare 14 image generators and prompt strategies on 1,095 cases. | Everyday | Oct 2026 | | Not in cards |
| HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. | Expert | Oct 2026 | 57.3% | Not in cards |
| HearingBench Models identify sounds from spectrogram images without hearing the audio. The test covers everyday sounds, instruments, vocals, words and sentences. | University | Oct 2026 | | Not in cards |
| HMU-Bench Models answer Chinese and English questions about musical notation and recordings. The authors score responses against reference answers. | University | Oct 2026 | | Not in cards |
| Human Verification Gap Models respond to paired requests about the same synthetic image. The evaluation measures how the stated purpose changes their refusal rate. | Everyday | Oct 2026 | 1.00 | Not in cards |
| Humanity's Sixth Sense Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric. | Everyday | Oct 2026 | | In use |
| HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. | Everyday | Oct 2026 | | Not in cards |
| Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. | Expert | Oct 2026 | Not in cards | |
| ImageJevBench v0.1.5 Models answer decisions about images and assign probabilities to the answers. The maker ranks capability using accuracy and calibration within cost and latency limits. | Everyday | Oct 2026 | 16.1 | Not in cards |
| ImageJevBench v0.3.0 Models make decisions about images and report probabilities. This version draws 300 public and 1,200 sealed items from a new pool. | Everyday | Oct 2026 | 17.3 | Not in cards |
| ImgMT Models translate text embedded in diagrams, maps, and infographics. Evaluation measures text recognition, region matching, and translation with CER, F1, chrF, and COMET. | Everyday | Oct 2026 | 61.3 | Not in cards |
| InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. | Everyday | Oct 2026 | 41.7% | Not in cards |
| IntPhys2-Timing Models identify when a physical rule breaks in a video. The authors add violation intervals to IntPhys2 and score temporal localization. | Everyday | Oct 2026 | | Not in cards |
| LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. | Everyday | Oct 2026 | 63.3% | Not in cards |
| LightGenBench Models predict which parts of a 3D surface emit light and what color they emit. The authors compare generated emission with reference assets. | University | Oct 2026 | Not in cards | |
| Mind the Gap Models describe visible events during natural pauses in a video's soundtrack. The evaluator renders the narration and checks its timing and coverage. | University | Oct 2026 | | Not in cards |
| Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. | Expert | Oct 2026 | | Not in cards |
| MMEnterprise-Bench Enterprise agents answer 1,613 questions over workplace documents, chats and figures that change over time. They must resolve conflicting versions and say when evidence is not enough. | Expert | Oct 2026 | 68% | Not in cards |
| MoI-Bench Models generate and edit images for mobile users. The authors compare image quality across 64 subtasks and measure the cost of running models on phones. | Everyday | Oct 2026 | 0.97 | Not in cards |
| OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. | Everyday | Oct 2026 | Not in cards | |
| Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. | Everyday | Oct 2026 | | Not in cards |
| Omni-InteractScope Explicit Request Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7. | Everyday | Oct 2026 | | Not in cards |
| Omni-InteractScope Proactive Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall. | Everyday | Oct 2026 | | Not in cards |
| Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. | Everyday | Oct 2026 | | Not in cards |
| PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. | Everyday | Oct 2026 | | Not in cards |
| ReactionBench Models rank a viewer’s reactions, match a reaction to one of four video clips, and explain which cue prompted it. | Everyday | Oct 2026 | 0.51 | Not in cards |
| Read My Aura Models respond to requests for aura readings. The tasks probe invented perceptions, flattery, and whether a change in a name or image tint changes the reading. | Everyday | Oct 2026 | | Not in cards |
| RGraph-Eval Models generate robot manipulation videos. The evaluator compares objects, changing relationships, and three-dimensional motion paths with a reference demonstration. | Everyday | Oct 2026 | | Not in cards |
| RISEBench++ Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types. | University | Oct 2026 | 56.6% | Not in cards |
| Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. | Expert | Oct 2026 | 32.7% | Not in cards |
| Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. | Everyday | Oct 2026 | Not in cards | |
| RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. | Everyday | Oct 2026 | 19.1% | Not in cards |
| RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. | Expert | Oct 2026 | 0.06 | Not in cards |
| RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. | Everyday | Oct 2026 | 17% | Not in cards |
| RSNA Knee AUC10 Models label twelve knee findings from images, radiology reports, or both. The grader measures how well their predictions distinguish findings on ten reference studies. | Expert | Oct 2026 | 0.92 | Not in cards |
| RVG-Bench Models locate the video interval that answers a question about sounds and images. The evaluator measures overlap with the annotated interval. | Everyday | Oct 2026 | | Not in cards |
| Safety-Scene Vision Benchmark Models identify visible hazards in everyday photos. The evaluation compares answers under a neutral prompt and a prompt claiming an inspector found danger. | Everyday | Oct 2026 | | Not in cards |
| SAVU-Bench Models answer spatial questions about real scenes using sound and video. The authors report accuracy across seven tasks. | Everyday | Oct 2026 | | Not in cards |
| ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. | Expert | Oct 2026 | 0.53 | Not in cards |
| ScribbleEdit Models edit images using a scribble as the only instruction. The evaluator checks whether the change matches the intended operation and location. | Everyday | Oct 2026 | | Not in cards |
| Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. | Expert | Oct 2026 | $2,949 | Not in cards |
| SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. | Expert | Oct 2026 | 100% | Not in cards |
| SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. | Expert | Oct 2026 | Not in cards | |
| Spatial Agent Index v1 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. | University | Oct 2026 | Not in cards | |
| Spatial Intelligence Index v1 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. | University | Oct 2026 | Not in cards | |
| SplatBench A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100. | University | Oct 2026 | 49.6 | Not in cards |
| StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. | Everyday | Oct 2026 | Not in cards | |
| TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. | Everyday | Oct 2026 | Not in cards | |
| TextVBench Models render requested text in video or preserve text from a reference image. The score tracks spelling, placement, and consistency across frames. | Everyday | Oct 2026 | | Not in cards |
| TireSidewall-Bench: Multimodal Zero-Shot Tire OCR Models read the size markings on tire sidewalls from photographs. The grader compares extracted specifications with reference annotations. | Everyday | Oct 2026 | | Not in cards |
| VCIS Models inspect a grid and decide whether a proposed movement plan reaches its goal. Separate tasks check object coordinates and the step where a plan fails. | School | Oct 2026 | 100% | Not in cards |
| VIALS 2 Models interpret 161 professional life-science visual artifacts. The score is judged accuracy averaged across three rollouts. | Expert | Oct 2026 | 49.9% | Not in cards |
| Video-Index Models answer 840 multiple-choice questions about videos, the 210 hardest in each of four groups: perception, time, space and reasoning. Questions that shortcut attacks could answer without the video were left out. | Everyday | Oct 2026 | | In use |
| VisR-RAG-Fin Models retrieve pages from financial documents and answer questions using their contents. The evaluator measures page ranking and checks answers against references. | University | Oct 2026 | Not in cards | |
| VISTA (Visually Impaired Scene and Task Annotations) Models describe scenes, answer questions, and give guidance to blind or low-vision users from glasses recordings. A judge scores responses from 0 to 5. | Everyday | Oct 2026 | | Not in cards |
| WeGenEditBench Models edit images from instructions that can include text or multiple references. A shared checklist scores each requested change and the details that should stay intact. | Everyday | Oct 2026 | | Not in cards |
| WeRef-Bench Models locate objects described in English or Chinese and return no boxes when the image contains no match. | Everyday | Oct 2026 | | Not in cards |
| WorldState-Bench Video models generate two clips from the same image and action. A hidden state changes which ending the rule requires. A vision model checks both clips. | Everyday | Oct 2026 | | Not in cards |
| Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. | Expert | Sep 2026 | 40% | Not in cards |
| APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. | Everyday | Sep 2026 | | Not in cards |
| Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. | Expert | Sep 2026 | 72.2% | System cards only |
| BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. | Expert | Sep 2026 | 76.4 | In use |
| CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. | Expert | Sep 2026 | | Not in cards |
| Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. | University | Sep 2026 | 0.62 | Not in cards |
| Decision Index Vision Models make decisions over images and screens. The score is the published equal-weight mean of public and private vision scores. | Everyday | Sep 2026 | | Not in cards |
| DrivingBench Vision-language models drive a real Toyota Corolla through a cone course with up to three attempts in one conversation. Scored on the furthest progress along the course. | Everyday | Sep 2026 | 100% | Not in cards |
| DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. | Expert | Sep 2026 | 40.0 | Not in cards |
| Earth-Bench-Pro Agents complete Earth observation workflows under three conditions. They follow instructions, plan from supplied observations, or obtain observations at runtime. | University | Sep 2026 | Not in cards | |
| Furniture Assembly Benchmark Model gets an IKEA manual and a photo of a partly built item, then names the faulty step or next step. Accuracy on 60 photos. | Everyday | Sep 2026 | 90% | Not in cards |
| GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. | Everyday | Sep 2026 | 80.2 | Not in cards |
| GenPuzzle Models generate images that solve visual puzzles such as mazes, tangrams, and Sudoku. The authors select an automated judge using human reference scores. | School | Sep 2026 | | Not in cards |
| GenScale Models generate or edit images containing objects with known relative sizes. A judge calibrated against human ratings scores the relative scale of object pairs. | Everyday | Sep 2026 | Not in cards | |
| GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. | Everyday | Sep 2026 | 0.64 | Not in cards |
| HappyWorld Embodied The HappyWorld arena publishes a separate embodied vote pool and Elo ranking. | Everyday | Sep 2026 | | Not in cards |
| HappyWorld Spatial Reconstruction The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking. | Everyday | Sep 2026 | 1,252 | Not in cards |
| HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. | Everyday | Sep 2026 | Not in cards | |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Research frontier | Sep 2026 | 77.9% | In use |
| JusticeAxis Models assess evidence and produce judgments for 256 criminal cases from 18 countries. The evaluator checks charges and outcomes against lawyer-written references. | Expert | Sep 2026 | | Not in cards |
| Last Translation Benchmark Models translate inputs that other systems struggled with. A model checks each translation against human-written rules. A translation passes only if it meets every rule. | Research frontier | Sep 2026 | 45.3% | Not in cards |
| metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. | Everyday | Sep 2026 | Not in cards | |
| MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. | Expert | Sep 2026 | 82.2% | In use |
| MindWorldBench Models animate a scene to show how a person would act given their beliefs or intentions. Paired prompts change mental states while holding the scene fixed. | Everyday | Sep 2026 | Not in cards | |
| Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. | Expert | Sep 2026 | 34% | System cards only |
| MUSE Models answer questions about artworks used in language education. The tasks cover visual details, emotions, cultural context, and relationships within the image. | School | Sep 2026 | 56.7% | Not in cards |
| OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. | Everyday | Sep 2026 | | Not in cards |
| OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. | Everyday | Sep 2026 | | Not in cards |
| OpenRSI Index Molmo2 Video-Pointing Inference Strategy Agents tune video inputs and inference for a frozen Molmo2 model. The score is held-out spatiotemporal pointing soft F1. | Research frontier | Sep 2026 | 0.46 | Not in cards |
| OpenScore String Quartets Models transcribe images of string quartet sheet music into symbolic notation. Scored as 1 minus normalized edit distance from the reference encoding. | University | Sep 2026 | 0.84 | In use |
| ORAV Bench Models combine reference images, sounds, and videos into a requested audiovisual scene. A judge compares pairs of outputs against each reference's assigned role. | Everyday | Sep 2026 | | Not in cards |
| Panda-CVL Models accept correct image-grounded responses or locate and replace their first erroneous text span. The test expands 652 conversations into 2,126 instances. Its overall F1 combines correct acceptance with exact correction of faulty responses. | University | Sep 2026 | 17.1% | Not in cards |
| Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. | Everyday | Sep 2026 | 36.3% | Not in cards |
| Parametric CAD Bench Agents create and edit parametric FreeCAD models from text and engineering drawings. Programmatic checks award continuous task rewards for geometry and specification consistency. | Expert | Sep 2026 | 61.0 | Not in cards |
| Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. | Expert | Sep 2026 | | Not in cards |
| PhysVista Vision-language models perceive, reason about and judge whether videos follow real-world physics. The benchmark also tests if they can judge the physical realism of generated videos. | Everyday | Sep 2026 | 85% | Not in cards |
| ReactHuman Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate. | Everyday | Sep 2026 | 63.7% | Not in cards |
| Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. | Everyday | Sep 2026 | 52.5% | Not in cards |
| RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. | Everyday | Sep 2026 | 65.7% | Not in cards |
| Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. | Everyday | Sep 2026 | 20% | System cards only |
| Runway Model Router Benchmark Video generation systems produce clips for 250 image-to-video prompts across ten production categories and are compared by usable rate and cost. | Expert | Sep 2026 | Not in cards | |
| ScreenASR-Bench Models transcribe speech using on-screen text, visible objects, or clues from the scene. The authors report word error rate and recall. | Everyday | Sep 2026 | | Not in cards |
| Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. | Expert | Sep 2026 | Not in cards | |
| SolveEdit Models edit an image to solve a visual problem. The evaluator checks whether the goal holds and whether unrelated scene details survive the edit. | School | Sep 2026 | 57% | Not in cards |
| StateBench (video continuation) Models continue a video while tracking what happened in earlier segments. A judge checks whether visible objects, hidden processes, and later state changes remain consistent. | Everyday | Sep 2026 | | Not in cards |
| StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. | Everyday | Sep 2026 | 46.0 | Not in cards |
| SWE-bench Multimodal v2 Agents patch JavaScript repositories using issue reports that include images. Automated tests check whether each patch fixes the issue. | Expert | Sep 2026 | 70.2% | System cards only |
| TimelineBench Agents turn raw footage and a client brief into a finished video. The edit must pass delivery checks, meet the brief, and clear a quality threshold. | University | Sep 2026 | 32.1% | Not in cards |
| TrafficImag Models edit roadside traffic videos to show a different sequence of events. The evaluator checks routes, interactions, and preservation of the surrounding scene. | Everyday | Sep 2026 | 90.0 | Not in cards |
| TriWorldBench World models generate synchronized robot-view videos from head and wrist cameras. Nineteen signals measure cross-view agreement, task alignment, physical coherence, motion, temporal stability, and visual quality. | Everyday | Sep 2026 | Not in cards | |
| UFO-Bench Models generate customized images from text and visual references. The evaluator breaks each request into individual requirements and checks how they interact. | Everyday | Sep 2026 | | Not in cards |
| VANTAGE-BENCH Models answer questions, describe events, and locate objects or actions in infrastructure-camera footage. Task-specific metrics measure answer accuracy, caption quality, localization, and tracking. | Everyday | Sep 2026 | 75.3 | Not in cards |
| VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. | Expert | Sep 2026 | 60.5% | System cards only |
| VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. | Everyday | Sep 2026 | | Not in cards |
| VIF-Bench Models generate images from multiple references and visual instructions. Judges check whether the result follows controls such as pose or lighting without copying unwanted features. | Everyday | Sep 2026 | 7.60 | Not in cards |
| VisEditBench (TikZ diagram editing) Models edit TikZ code to follow instructions drawn on a diagram. A judge scores adherence, preservation of other content, and readability. | University | Sep 2026 | | Not in cards |
| WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. | Expert | Sep 2026 | 46.5% | Not in cards |
| WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. | Everyday | Sep 2026 | 42.3% | Not in cards |
| ADVANCE audio and image retrieval (MTEB) Models match field recordings and aerial images from 5,075 locations. The correct recording and image come from the same place. | Everyday | Aug 2026 | Not in cards | |
| AnomalyXL Models locate and describe anomalies in long synthetic time series. The benchmark checks multiple-choice answers and scores precise answers using the known anomaly properties. | Expert | Aug 2026 | 0.33 | Not in cards |
| Bio-VQA Models identify organisms, describe visible traits, judge spatial relationships, and recognize individual animals across photographs. The benchmark has 24 tasks. | University | Aug 2026 | | Not in cards |
| Brennan Center election image identification Chatbots inspect 14 synthetic election images in English and Spanish. Researchers count how many images each chatbot identifies as synthetic in each language. | Everyday | Aug 2026 | Not in cards | |
| Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. | Expert | Aug 2026 | 90.7% | In use |
| CharXiv Extended Models answer descriptive and reasoning questions about scientific charts. Mercor adds 150 new expert-written tasks to the 5000 public ones and reports both scores. | University | Aug 2026 | | Not in cards |
| ConceptEdit-Bench Models edit images according to 1,000 instructions across six categories. A vision-language judge scores instruction following, visual quality, and consistency in unedited regions from 0 to 10. The per-case score averages those three dimensions. | Everyday | Aug 2026 | | Not in cards |
| Cyc3D Models generate 3D assets from images. The evaluator checks mesh quality and measures how much shape and identity drift across repeated rendering and reconstruction. | University | Aug 2026 | Not in cards | |
| Effective FPS Models decide whether video frames are consecutive or duplicated to simulate a higher capture rate. The evaluator checks each prediction against the construction of the clip. | University | Aug 2026 | Not in cards | |
| EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. | Everyday | Aug 2026 | Not in cards | |
| Escalation Router Models predict whether Qwen2.5-VL 7B correctly identifies visible hands or grippers in a clip. The reference label comes from two observed runs per clip. | University | Aug 2026 | Not in cards | |
| H2R-Bench Models turn first-person demonstrations into videos of a specified robot performing the task. The benchmark checks contacts, object responses, and completion of the goal. | Everyday | Aug 2026 | | Not in cards |
| Hands Visible Models inspect 12 frames per clip and determine whether a human hand or robot gripper appears. The 45-clip evaluation uses provisional machine-generated labels. | Everyday | Aug 2026 | Not in cards | |
| HarnessEval-W Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning. | Expert | Aug 2026 | | Not in cards |
| HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. | Expert | Aug 2026 | 77.3% | Not in cards |
| HM3D-MFMON Agents navigate across floors and find a sequence of objects. | Everyday | Aug 2026 | Not in cards | |
| LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. | Everyday | Aug 2026 | | Not in cards |
| LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. | Everyday | Aug 2026 | | In use |
| MedXpertQA MM Extended Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores. | Expert | Aug 2026 | 71.2% | Not in cards |
| MineAmongUs Vision-language agents play an embodied Among Us game with verbal and nonverbal deception. | Everyday | Aug 2026 | | Not in cards |
| MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. | University | Aug 2026 | 61.3% | Not in cards |
| OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. | University | Aug 2026 | | Not in cards |
| OmniMech Models write CAD programs to reconstruct parts from dimensioned drawings. The evaluator compares the reconstructed geometry with reference parts and measures relative volume error. | University | Aug 2026 | 15.9% | Not in cards |
| OmniPhys Models solve Chinese physics problems from middle school through university and draw diagrams. The evaluator grades answers, reasoning, and generated figures. | University | Aug 2026 | | Not in cards |
| PatternEval Models answer 2,415 image prompts in thinking and non-thinking modes. A model judge flags exposed draft reasoning, repetition, contradictions, and unsupported reasoning. The Trigger rate measures responses with any flagged failure. | Everyday | Aug 2026 | 0.78% | Not in cards |
| PAWBench Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior. | Everyday | Aug 2026 | Not in cards | |
| PlayWorld An agent player pursues objectives inside video world models. Gemini 3.1 Pro applies weighted rubrics for geometry, interaction, and state evolution across the resulting videos. | Everyday | Aug 2026 | Not in cards | |
| QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. | University | Aug 2026 | | In use |
| SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. | Everyday | Aug 2026 | | Not in cards |
| VGI-Bench Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame. | Everyday | Aug 2026 | | Not in cards |
| VIALS Vision models answer 161 questions about lab images such as gel blots, plasmid maps, microscopy and flow cytometry plots. An LLM judge grades each answer against a key. | Expert | Aug 2026 | 65.8% | Not in cards |
| VideoGAIA Agents inspect selected video segments, search the web, and read pages before answering questions. Model judges check answers, and the harness reports accuracy by category and difficulty. | Everyday | Aug 2026 | | Not in cards |
| VWE-Bench Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics. | Expert | Aug 2026 | 57.3% | Not in cards |
| WildArtifactBench Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo. | Expert | Aug 2026 | Not in cards | |
| WorldExam Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states. | Everyday | Aug 2026 | | Not in cards |
| ActiveVision Models inspect images repeatedly to count regions, trace structures, and compare visual attributes. The released split has five examples for each of 17 tasks. Exact match checks the final answer after normalizing case, spacing, and separators. | Everyday | Jul 2026 | 50.6% | Not in cards |
| Agent Memory Leaderboard Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework. | Everyday | Jul 2026 | Not in cards | |
| Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. | Expert | Jul 2026 | 78.3% | System cards only |
| ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. | Everyday | Jul 2026 | 47.6% | Not in cards |
| ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. | Everyday | Jul 2026 | System cards only | |
| AstroCLIMB Models compare pairs of astronomy figures or captions. They determine whether the pair shares a figure, shares a paper, comes from citation-linked papers, or comes from unrelated papers. | University | Jul 2026 | Not in cards | |
| AutoCAD Bench Computer-use agents turn a dimensioned reference drawing into a correct, editable DWG in AutoCAD through the screen alone, with mouse and keyboard. 50 tasks span 2D drafting and 3D modeling at four difficulty levels. | Expert | Jul 2026 | 46% | Not in cards |
| BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. | Everyday | Jul 2026 | Not in cards | |
| Blind-Spots-Bench Models answer 235 text and visual problems chosen to expose failures on tasks that people find simple. A Gemini 3 Flash grader compares each response with a structured reference solution and gives binary credit. The grader can execute code to check exact constraints. | Everyday | Jul 2026 | Not in cards | |
| CHAOS-Bench Models read deliberately corrupted words in 500 academic page images. Scores measure faithful recognition of the visible strings instead of correction to plausible words. | Expert | Jul 2026 | | In use |
| CLBench-V Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition. | Everyday | Jul 2026 | | Not in cards |
| DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. | Everyday | Jul 2026 | 34% | Not in cards |
| Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. | Expert | Jul 2026 | 97.3% | Not in cards |
| EduArt Models answer art history questions from Italian secondary schools and US Advanced Placement exams. The authors compare 12 models across seven question formats. | School | Jul 2026 | | Not in cards |
| EgoProceVQA Models watch first-person clips and answer questions about the steps in a procedure. Some questions ask when a step occurred or which step is missing. | Everyday | Jul 2026 | 51.6% | Not in cards |
| EmbodiedCLUE-World 2.0 World models predict outcomes in embodied settings under the second evaluation protocol. | Everyday | Jul 2026 | | Not in cards |
| Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. | Everyday | Jul 2026 | Not in cards | |
| GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. | Everyday | Jul 2026 | Not in cards | |
| Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. | Everyday | Jul 2026 | Not in cards | |
| Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. | Everyday | Jul 2026 | | In use |
| HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. | Everyday | Jul 2026 | | Not in cards |
| KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. | Everyday | Jul 2026 | | Not in cards |
| KWV Bench Models answer atomic visual questions extracted from tasks distilled from real knowledge-work scenarios. | Expert | Jul 2026 | 64.7% | System cards only |
| MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. | Expert | Jul 2026 | Not in cards | |
| Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. | Everyday | Jul 2026 | | In use |
| MPIE-Bench Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other. | Everyday | Jul 2026 | | Not in cards |
| MultiRef-Compass Models generate audio-video content from several visual references and text instructions across 350 samples. Fourteen metrics assess quality, reference fidelity, audio-visual consistency, and instruction following. The evaluator combines automatic measurements with multimodal model judgments. | Everyday | Jul 2026 | Not in cards | |
| OCT-Bench Models answer 10,076 questions about retinal scans, from anatomy to treatment and prognosis. The evaluator measures answer accuracy and counts invalid responses as incorrect. | Expert | Jul 2026 | 62% | Not in cards |
| OmniRouting Models route copper traces between pads on printed circuit boards. The evaluator counts required nets that connect without design rule violations. | University | Jul 2026 | 28% | Not in cards |
| Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. | Expert | Jul 2026 | 69.2% | System cards only |
| PerceptionBench (Moonshot) Moonshot in-house benchmark of atomic visual perception skills. Scoring details unpublished. | Everyday | Jul 2026 | 59.7% | In use |
| Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. | Everyday | Jul 2026 | | In use |
| RadLE 2.0 Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors. | Expert | Jul 2026 | 758.0 | Not in cards |
| RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. | Everyday | Jul 2026 | 27.6% | In use |
| RoughCut-Bench Models select and order video clips or mark speech to remove from a transcript. The evaluation compares cut plans with professional editors’ annotations. | University | Jul 2026 | | Not in cards |
| S-EMBER Models answer memory questions from first-person video using only frames available before the query time. The benchmark scores multiple-choice accuracy and temporal overlap between predicted evidence intervals and annotated intervals. | Everyday | Jul 2026 | | Not in cards |
| SoundingEarth audio and image retrieval (MTEB) Models match a field recording to an aerial image of the same location, or retrieve the recording from the image. Each direction has 1,000 queries and 2,048 candidates. | Everyday | Jul 2026 | Not in cards | |
| Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. | Expert | Jul 2026 | 81.6% | Not in cards |
| UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. | Everyday | Jul 2026 | 47.5% | In use |
| WorldModelGym World models predict returns for five candidate action sequences across 100+ Atari, Meta-World, DMC and classic-control tracks. Scored by normalized regret. | Jul 2026 | Not in cards | ||
| ZendoWorld Models infer hidden rules in the board game Zendo from labeled examples. | Everyday | Jul 2026 | Not in cards | |
| Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. | Everyday | Jun 2026 | | Not in cards |
| ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. | Everyday | Jun 2026 | 48.2% | Not in cards |
| AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. | Expert | Jun 2026 | 66.5 | Not in cards |
| ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. | Expert | Jun 2026 | | Not in cards |
| CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. | Everyday | Jun 2026 | 55.2 | Not in cards |
| DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. | Everyday | Jun 2026 | 10.6% | Not in cards |
| EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. | Everyday | Jun 2026 | Not in cards | |
| Ego-MC-Bench Models watch a cook follow spoken instructions and tell them when to correct a mistake. The evaluator checks intervention timing and recipe step completion. | Everyday | Jun 2026 | | Not in cards |
| FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. | University | Jun 2026 | 77.8% | In use |
| GeoDrive-Bench Models choose driving actions from road images and local traffic conventions. The benchmark covers six countries and omits explicit country labels. | Everyday | Jun 2026 | 91.9% | Not in cards |
| Harbor-Index Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. | Research frontier | Jun 2026 | 28.1% | In use |
| HealthAgentBench Agents use terminal tools to complete 54 clinical and biomedical tasks across seven categories. Task-specific verifiers measure success, including a model judge for radiology report correction. | Expert | Jun 2026 | 55% | Not in cards |
| Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. | Everyday | Jun 2026 | | In use |
| K9-Bench Models answer 4,744 questions about dog behavior in 907 videos. The evaluator compares each free-form answer with five options that the model never sees. | School | Jun 2026 | | Not in cards |
| KilometerVision Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures. | Everyday | Jun 2026 | Not in cards | |
| LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. | Expert | Jun 2026 | 61.3% | In use |
| LLM Agronomy Benchmark Image Identification (four choices) Models choose among four labels for each photo of a crop disease, insect pest, or weed. One accuracy score combines the 150 questions. | University | Jun 2026 | | Not in cards |
| MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. | University | Jun 2026 | | Not in cards |
| MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. | Expert | Jun 2026 | 69.2 | Not in cards |
| MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. | Everyday | Jun 2026 | Not in cards | |
| OmniCap-IF Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance. | Everyday | Jun 2026 | | In use |
| OmniLayout Models place components inside a board outline. The evaluator checks overlap, routing, and electrical functionality against the schematic and the reference layout. | University | Jun 2026 | | Not in cards |
| OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. | Everyday | Jun 2026 | | Not in cards |
| P3D-Bench Models generate parametric 3D objects from text, images, or assembly instructions. The evaluator checks whether outputs run and preserve geometry, topology, and part structure. | University | Jun 2026 | 69.4 | Not in cards |
| PhysicalRealismBench-U VLM watches 1,300 synthetic Isaac Sim videos, flags physics-law violations, names object and frames. Scored by joint realism F1. | Everyday | Jun 2026 | 60.8% | Not in cards |
| Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. | Everyday | Jun 2026 | 10.9 | Not in cards |
| RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. | Everyday | Jun 2026 | | Not in cards |
| Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. | Everyday | Jun 2026 | Not in cards | |
| Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. | Expert | Jun 2026 | Not in cards | |
| SLoMO Audio Description Competition 2026 Models write one sentence for each gap in a film’s dialogue. Judges score how well the descriptions convey visual information and match professional references. | University | Jun 2026 | Not in cards | |
| SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. | Everyday | Jun 2026 | 14.4% | Not in cards |
| SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. | Everyday | Jun 2026 | 14.4% | Not in cards |
| SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. | University | Jun 2026 | | In use |
| TEM-VQA Models give short written answers about structures, image features and experimental design in transmission electron micrographs. | Expert | Jun 2026 | 40.4 | Not in cards |
| UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. | Everyday | Jun 2026 | 55.8% | Not in cards |
| Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Jun 2026 | 74.6% | Not in cards |
| WorldBench Models answer hand-written reasoning questions about images drawn from a taxonomy of thousands of visual concepts. Questions were kept only if frontier models failed them. | Expert | Jun 2026 | | System cards only |
| WorldOlympiad Models generate long videos for gaming, robotics, and everyday scenes. The evaluator checks physical rules, reconstructed geometry, and instruction following across video segments. | Everyday | Jun 2026 | Not in cards | |
| X-Stream Models answer questions that combine concurrent video streams. A model judge grades answers, while forward-looking questions also require responses within the correct time window. | Everyday | Jun 2026 | | Not in cards |
| 3DCodeBench Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. | May 2026 | | In use | |
| ACII DaiKon Challenge Models predict interpersonal influence, turn taking and rapport trajectories from multimodal dyadic conversations across five languages. | Expert | May 2026 | Not in cards | |
| AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. | Expert | May 2026 | 73.2% | Not in cards |
| Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. | Everyday | May 2026 | 0.63 | Not in cards |
| AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. | Expert | May 2026 | 38.4% | In use |
| Animation2Code Models recreate a web animation from a video using HTML, CSS, and JavaScript. The evaluator renders their code and compares appearance and motion separately. | University | May 2026 | 0.84 | Not in cards |
| BenchCAD Programmatic CAD: 17,900 execution-verified CadQuery programs over 106 industrial part families. Four matched tasks scored on geometry. | University | May 2026 | 95.9% | In use |
| Blueprint-Bench 2 Agents turn about 20 interior photos of each of 50 apartments into 2D floor plans. Scores compare room connectivity with ground truth, normalized against random. | Everyday | May 2026 | | In use |
| CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. | University | May 2026 | 45.1% | System cards only |
| CC-OCR-Bench-V2 Models process 7,093 hard real-world document samples across five OCR tracks, including parsing and document QA. Each track uses its own metric. | Everyday | May 2026 | | In use |
| EmbodiedCLUE-World World models predict outcomes in embodied settings. | Everyday | May 2026 | | Not in cards |
| ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. | Everyday | May 2026 | 70% | Not in cards |
| EvalVerse Video models generate cinematic clips from prompts and reference material derived from professional videos. Film experts compare outputs across visual design, acting, cinematography, aesthetics, emotion, shot sequences, and sound. A calibrated vision-language evaluator also estimates these judgments. | Expert | May 2026 | 0.73 | Not in cards |
| HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. | Everyday | May 2026 | Not in cards | |
| LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. | Everyday | May 2026 | | Not in cards |
| MDMR-Bench Models generate one image from several references. A model judge checks whether the output preserves requested elements, follows instructions, and combines them correctly. | Everyday | May 2026 | 88.8% | Not in cards |
| MemEye Multimodal agent memory systems answer questions that need fine visual evidence kept from earlier sessions, which captions or text traces alone cannot supply. | Everyday | May 2026 | 49.4% | Not in cards |
| MemLens Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths. | Everyday | May 2026 | 53.6% | Not in cards |
| Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. | Expert | May 2026 | 37.6% | Not in cards |
| MM-OCEAN Models rate apparent personality traits in short videos, explain those ratings, and answer questions about observable cues. Evaluation reports rating accuracy, absolute error, cue-question accuracy, and model-judged explanation scores. | Everyday | May 2026 | Not in cards | |
| MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. | Expert | May 2026 | 16.2% | Not in cards |
| OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. | University | May 2026 | 0.37 | Not in cards |
| OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. | Expert | May 2026 | 46% | Not in cards |
| PaintBench Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks. | Everyday | May 2026 | | Not in cards |
| Polaris-Bench Models solve the same 53 visual reasoning tasks in Cartesian and polar layouts. The benchmark compares accuracy across the paired layouts. | School | May 2026 | 39.2% | Not in cards |
| ProactiveVideoQA Models answer spoken questions when a streamed video first reveals the answer. | Everyday | May 2026 | | Fading |
| QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. | University | May 2026 | 1,884 | In use |
| RepCount-A Models count repeated actions in streamed video and are scored by distance from the true count. | Everyday | May 2026 | | Fading |
| RoboChallenge Table30 v2 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. | Everyday | May 2026 | 0.73% | Not in cards |
| RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. | Everyday | May 2026 | 21.5% | Not in cards |
| SFI-Bench Multimodal models answer video questions about spatial layouts, object affordances and functional reasoning in indoor scenes. | University | May 2026 | Not in cards | |
| SpaceDG-Bench Models answer spatial questions under clean images and nine forms of visual degradation. Choice and binary answers use accuracy, while numerical answers use relative-error thresholds. The test has 1,102 verified questions from scenes excluded from the training set. | Everyday | May 2026 | | Not in cards |
| SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. | Expert | May 2026 | 40.5% | Not in cards |
| TeachObs Models identify teaching practices in classroom video and assess entire lessons. Human annotations cover 39 observable behaviors in 5,158 segments from eight countries. | University | May 2026 | 0.28 | Not in cards |
| Tricky Edits of Collected and Curated Images Models edit a source image to follow a written instruction. People and a model judge instruction following, preservation of unaffected content, and visual quality. | Everyday | May 2026 | | Not in cards |
| TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. | Everyday | May 2026 | | Not in cards |
| Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. | Expert | May 2026 | 74.2% | Not in cards |
| Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Expert | May 2026 | 70.7% | Not in cards |
| VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. | Everyday | May 2026 | | Not in cards |
| ViMU Models explain what a short video implies and select the cues that support their reading. They also identify rhetorical devices and social signals. | Everyday | May 2026 | 46.9% | Not in cards |
| Visual Aesthetic Benchmark Models see a set of images with the same subject and intent and must pick the best one, and in a harder setting both the best and the worst. The 400 sets hold 1,195 commissioned works across fine art, photography and illustration. Labels come from expert judgments, and only sets where experts largely agreed are kept. | Expert | May 2026 | 40.3% | Not in cards |
| WikiVQABench Vision-language models answer multiple-choice questions that require image evidence and linked Wikipedia and Wikidata knowledge. | University | May 2026 | Not in cards | |
| WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. | May 2026 | | Not in cards | |
| YoCausal Video generation models are tested for causal and arrow-of-time understanding by comparing real videos with their time-reversed versions, in two levels. | Everyday | May 2026 | | Not in cards |
| Aether-Bench Models retouch photographs automatically, match a requested style, or apply specified adjustments. The style test measures fidelity on 100 image pairs from unseen presets, including peak signal-to-noise ratio. | University | Apr 2026 | | Not in cards |
| AstroVLBench Models interpret galaxy images, light curves, and spectra across five observing tasks. The authors compare their classifications with reference labels and examine their explanations. | University | Apr 2026 | | Not in cards |
| Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. | University | Apr 2026 | | In use |
| Claw-Eval-MM Agents complete the 101 multimodal tasks of Claw-Eval, such as video questions, document extraction, and webpage generation. Rubric items graded from run traces set scores. | University | Apr 2026 | 54.7% | In use |
| CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. | Expert | Apr 2026 | 45.1% | Not in cards |
| DrawingVQA Models answer 92 multiple-choice questions about 33 construction drawings. They read symbols, cross-reference details, and calculate quantities. | University | Apr 2026 | | Not in cards |
| GraphicDesignBench Models analyze and produce graphic designs in 50 tasks spanning layout, typography, infographics, design semantics, and animation. The suite uses professional layered design templates. | Expert | Apr 2026 | 0.94 | Not in cards |
| GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. | Expert | Apr 2026 | 11.4% | Not in cards |
| Image-to-WebDev Arena Users upload a screenshot, wireframe or mockup, two anonymous models build the website in Code Arena, and users vote for the better result. Votes become ratings with bootstrapped confidence intervals. | Everyday | Apr 2026 | 1,756 | Not in cards |
| ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. | Expert | Apr 2026 | 30.7% | Fading |
| Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. | Everyday | Apr 2026 | | Fading |
| K-MetBench Models answer 1,774 questions from the Korean National Meteorological Engineer Examination under a zero-shot protocol. The benchmark reports answer accuracy across five subjects and separate visual and Korean-specific subsets. Expert rationales support an additional rubric score for reasoning. | Expert | Apr 2026 | | Not in cards |
| KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. | University | Apr 2026 | | Fading |
| LVSum Models summarize long videos with timestamped segments, relevance scores and descriptions using video frames, transcripts or both. | Expert | Apr 2026 | Not in cards | |
| MMEB-V3 Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval. | Everyday | Apr 2026 | | Not in cards |
| MMSearch Multimodal agents search and answer questions that require visual and textual evidence. | Expert | Apr 2026 | 72.9% | Fading |
| neuralCAD-Edit Agents edit existing CAD models from text or recorded instructions. CAD experts rate whether they would accept each edit. Software also compares the resulting geometry. | University | Apr 2026 | 0.25 | Not in cards |
| PulseBench-Tab Models convert 1,820 table images in nine languages into HTML tables. T-LAG represents adjacent cells as directed edges and matches their text and structure against human annotations. The reported score is F1 over matched edge weights. | Everyday | Apr 2026 | 0.89 | Not in cards |
| RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. | Everyday | Apr 2026 | 46.2% | In use |
| SHROOM-Visions 2026 Models inspect an image and a written response. They mark unsupported characters, estimate the probability of an error and classify each error in four languages. | Everyday | Apr 2026 | Not in cards | |
| TriScore Models identify composers and instruments from recordings, score images, and symbolic notation. They also retrieve excerpts from the same composition. | University | Apr 2026 | Not in cards | |
| VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. | Everyday | Apr 2026 | | Not in cards |
| VideoMME v2 (with subtitles) Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. | Everyday | Apr 2026 | | In use |
| VSAS-Bench Streaming vision-language models answer tasks over online video frames while evaluation measures accuracy, response timing and consistency. | Expert | Apr 2026 | Not in cards | |
| ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. | Everyday | Mar 2026 | 100% | In use |
| AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. | Expert | Mar 2026 | 76.5% | Not in cards |
| ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. | University | Mar 2026 | 86% | Not in cards |
| BigEarthNet.txt Models answer land cover questions, caption satellite images, and locate regions described in text. The authors checked annotations for 1,082 pairs of radar and optical images. | University | Mar 2026 | 0.79% | Not in cards |
| BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. | University | Mar 2026 | Not in cards | |
| CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. | University | Mar 2026 | | Not in cards |
| Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. | Everyday | Mar 2026 | 81.4% | In use |
| CoVR-R Models find the video that matches a reference clip after applying a change described in text. | School | Mar 2026 | | Not in cards |
| Crate Bench Models identify an album and its artist from the cover image. The benchmark provides 300 test covers and supports closed-book and web-search modes. | Everyday | Mar 2026 | Not in cards | |
| DesignJanus People rate generated objects from six viewpoints and mark inconsistencies in shape, materials, and identity. The harness compares human ratings with automatic measures. | Everyday | Mar 2026 | Not in cards | |
| HCSU-Bench Models match a Chinese calligraphy sample to one of eight styles. A second task asks models to describe the brushwork in fewer than 50 words. | University | Mar 2026 | 34.6% | Not in cards |
| LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. | Everyday | Mar 2026 | 71.4% | Not in cards |
| Live Gaming Benchmark Models watch game clips and decide when to speak. They narrate alone, join another commentator, or guide a player. A judge scores their commentary. | Everyday | Mar 2026 | 28.4% | Not in cards |
| LVOmniBench Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track. | Everyday | Mar 2026 | | In use |
| ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. | Everyday | Mar 2026 | Not in cards | |
| MDPBench Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage. | Everyday | Mar 2026 | | Not in cards |
| MediaPerf Models tag and summarize video advertisements. The benchmark measures tag accuracy, summary quality, cost and throughput. | University | Mar 2026 | 0.42 | Not in cards |
| MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. | Everyday | Mar 2026 | | System cards only |
| OmniGUI Models use screenshots, sound, and video to predict the next action in 29 phone apps. The grader compares each action with a human demonstration. | Everyday | Mar 2026 | | Not in cards |
| OmniSch Models convert circuit schematics into graphs of components, pins, and connections. A second task lets models search schematic images for individual features. | University | Mar 2026 | | Not in cards |
| PerceptionComp 1,114 five-choice questions on 279 complex videos. Each question needs evidence from several separate moments, so no single frame answers it. | Everyday | Mar 2026 | | System cards only |
| PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. | Everyday | Mar 2026 | 75.1% | Not in cards |
| Real5-OmniDocBench Models parse the same 1,355 document pages under five physical capture conditions. Text edit distance, formula matching, and table similarity determine the overall score. | Expert | Mar 2026 | | In use |
| RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. | Everyday | Mar 2026 | 10.4% | Not in cards |
| RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. | Everyday | Mar 2026 | | Not in cards |
| RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. | Everyday | Mar 2026 | | Not in cards |
| RVMS-Bench Agents search open-domain videos and localize moments using global impression, key moment, temporal context and auditory memory cues. | Expert | Mar 2026 | Not in cards | |
| TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. | Everyday | Mar 2026 | | Not in cards |
| TextEdit Real Models edit text in real images in the full TextEdit test set. The score is the published judged average. | Everyday | Mar 2026 | 0.90 | Not in cards |
| TextEdit Virtual Models edit text in virtual images in the full TextEdit test set. The score is the published judged average. | Everyday | Mar 2026 | 0.90 | Not in cards |
| VAREX-Bench Models extract structured fields from 1,777 government forms using a supplied JSON schema. Normalized exact match checks field values, while string similarity gives partial credit. Array entries use order-independent matching. | Everyday | Mar 2026 | | Not in cards |
| Video-Oasis A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions. | Everyday | Mar 2026 | | Not in cards |
| Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Mar 2026 | 82.3% | In use |
| Vision2Web S1-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Mar 2026 | 52.7% | Not in cards |
| Vision2Web S2-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Expert | Mar 2026 | 43.5% | Not in cards |
| WebVR Models recreate a webpage from a screen recording and supplied assets. A model judge scores the rendered page against visual and interaction criteria. | University | Mar 2026 | 79.1 | Not in cards |
| Wild-OmniDocBench Models parse 1,350 photographed document pages with geometric distortion, lighting changes, and other capture artifacts. OmniDocBench metrics score text, formulas, tables, and reading order. | Expert | Mar 2026 | | In use |
| Accessible Structured Subtitle Generation 2026 Systems produce timed Chinese subtitles for deaf and hard-of-hearing viewers. The structured track also scores speaker labels, key information, and readability. | University | Feb 2026 | Not in cards | |
| BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. | Expert | Feb 2026 | 52.3% | Fading |
| CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. | Everyday | Feb 2026 | Not in cards | |
| DETOUR Agents ask a simulated user for clues and identify a partially remembered item. | Everyday | Feb 2026 | Not in cards | |
| DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. | Everyday | Feb 2026 | 28.7% | Not in cards |
| HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. | Expert | Feb 2026 | | In use |
| Human SxS image editing Elo (Google) Google compares edited images side by side and converts human preferences into Elo ratings. | Everyday | Feb 2026 | | Fading |
| HY3D-Bench Models generate textured 3D meshes from text and images, with semantic and geometric quality evaluated across categories. | Expert | Feb 2026 | Not in cards | |
| Korean Dental Licensing Examination 2024 and 2025 Models answer 642 Korean dental licensing questions from two new exams. The authors score accuracy and break it down by subject and image type. | Expert | Feb 2026 | 81.5% | Not in cards |
| KorMedMCQA-Mixed Models take combined text and image exams. The 2022 set contains 281 questions and the 2023 set contains 307. Each year has a separate score. | Expert | Feb 2026 | | Not in cards |
| KorMedMCQA-V Models answer Korean medical licensing questions using clinical images. The full test contains 1,534 questions and 2,043 images from 2012 through 2023. | Expert | Feb 2026 | | Not in cards |
| LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. | Expert | Feb 2026 | 60.9% | In use |
| LABBench2 FigQA2 Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2. | Expert | Feb 2026 | 42.6% | Not in cards |
| LABBench2 FigQA2 (image) Models answer research questions about figures supplied as images. | Expert | Feb 2026 | 66.3% | Not in cards |
| LABBench2 TableQA2 (image) Models answer research questions about tables supplied as images. | Expert | Feb 2026 | | Not in cards |
| Life-Bench Models answer multimodal questions about synthetic personal lifelogs. Tasks cover concept identification, event understanding, and reasoning over multiple pieces of evidence. | Expert | Feb 2026 | | Not in cards |
| MarsRetrieval Models match Mars images to captions, retrieve examples of named landforms, and locate features in a global mosaic of the planet. | University | Feb 2026 | | Not in cards |
| MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. | Everyday | Feb 2026 | | Fading |
| OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. | University | Feb 2026 | | Fading |
| OpenEarthAgent Agents use tools such as QGIS to measure features in satellite images and maps. The authors grade tool decisions and final answers. | University | Feb 2026 | 44.5% | Not in cards |
| Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. | Expert | Feb 2026 | 86.2% | System cards only |
| Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. | Expert | Feb 2026 | 79.6% | System cards only |
| RefAdv-S Models locate the object a referring expression describes among hard distractors in real images. Scored by accuracy at 0.5 box overlap on 1,142 cases. | Everyday | Feb 2026 | | In use |
| SA-1B Hard human pose evaluation Models locate body and foot joints in photographs with unusual viewpoints, poses, and occlusions. Meta measures the fraction of correctly placed joints. | School | Feb 2026 | Not in cards | |
| SAM 3D Body synthetic pose evaluation Models estimate human pose and shape from synthetic images of individual people. Meta compares predicted joints and mesh vertices with the rendering parameters. | School | Feb 2026 | Not in cards | |
| Snowflake Financial PDF Table Parsing Models reconstruct tables from 50 manually annotated financial documents. Evaluators compare complete tables, table dimensions and individual cells with reference Markdown. | University | Feb 2026 | Not in cards | |
| SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. | Expert | Feb 2026 | 0.94 | Not in cards |
| VBVR-Bench Video generation models solve visual reasoning tasks by generating a video from a starting image and instruction. Rule-based scorers check the result instead of a judge model. Tasks cover abstraction, knowledge, perception, spatial reasoning and transformation, with in-domain and out-of-domain splits. | Everyday | Feb 2026 | | Not in cards |
| VBVR-Pro The current official results table evaluates the Pro release. Original VBVR results must remain separate. | Everyday | Feb 2026 | | Not in cards |
| VDR-Bench Models answer 2,000 visual questions that require finding entities in images and researching related facts. A Qwen3-VL judge checks final answers against references. Entity recall separately measures whether searches retrieve the relevant entities. | University | Feb 2026 | 27.2% | Not in cards |
| WorldArena Embodied world models generate RoboTwin 2.0 task videos from first frame plus text or actions. 15 video metrics (EWMScore) plus utility tracks. | Feb 2026 | | Not in cards | |
| WTP-Bench Models identify Pokémon from official artwork and silhouettes across 1,160 entries. The scorer checks exact forms, species, and evolutionary families. | School | Feb 2026 | 79% | Not in cards |
| Artificial Analysis Intelligence Index v4.0 Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests. | Jan 2026 | 60.2 | Fading | |
| AVMeme Exam Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge. | Everyday | Jan 2026 | | Fading |
| BabyVision Model solves 388 perception-heavy visual puzzles (discrimination, tracking, spatial, patterns) that young children solve. Accuracy versus 94% human baseline. | Everyday | Jan 2026 | | In use |
| CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. | Expert | Jan 2026 | Not in cards | |
| DatBench Models answer visual questions drawn from 33 datasets. The suite reports separate scores across nine visual capabilities. | University | Jan 2026 | | Not in cards |
| FutureOmni Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together. | Everyday | Jan 2026 | 66.7% | Fading |
| LLM Benchmark: Danske Idiomer Models match Danish idioms to pictures, complete expressions and write definitions. GPT-4.1 grades definitions, and answer keys cover the other tasks. | Everyday | Jan 2026 | 97.5% | Not in cards |
| LMArena Image-to-Video Arena Video generation models animate the same input image with the same prompt, anonymously and side by side. Votes give Elo-style ratings. | Everyday | Jan 2026 | | Not in cards |
| MedGenVidQA 2026 Visual Answer Localization Models mark the start and end of the video segment that answers a medical question. The organizers created 80 test examples from professional medical videos. | University | Jan 2026 | Not in cards | |
| Mem-Gallery Multimodal agents hold long conversations over many sessions that mix images and text, then answer questions that need what they remembered. Questions test memory extraction and test-time adaptation, reasoning over memory, and managing knowledge that changes over time. | Everyday | Jan 2026 | Not in cards | |
| MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. | Expert | Jan 2026 | 42.2% | Not in cards |
| MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. | Expert | Jan 2026 | Not in cards | |
| NarrativeTrack Video-language models answer entity-centric questions about existence, changes and ambiguity as narratives unfold across clips. | Expert | Jan 2026 | Not in cards | |
| OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. | Expert | Jan 2026 | 81% | Not in cards |
| PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. | Expert | Jan 2026 | 41.0 | Not in cards |
| Progress-Bench Vision-language models estimate how far a task has progressed from partial observations, such as frames of a robot or human doing a job. The benchmark varies how the demonstration is shown and the camera viewpoint, and it includes cases with no valid answer. | Everyday | Jan 2026 | 20.9 | Not in cards |
| RBench 650 image+text prompts for robot video generation (tasks and embodiments). MLLM judge scores task completion, physical plausibility, stability. | Jan 2026 | | System cards only | |
| RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. | Everyday | Jan 2026 | 76.3% | Not in cards |
| RoboRewardBench Vision-language models act as reward models on real robot videos from Open X-Embodiment and RoboArena, judging task success and partial progress. Counterfactual relabeling adds calibrated failures and near misses. | Everyday | Jan 2026 | 0.63 | Not in cards |
| VDEBench Image editing models make local text and table edits to 3,506 Chinese and English document images. Scores check the edit and that the rest of the page is preserved. | Everyday | Jan 2026 | 0.77 | Not in cards |
| Video Browser Showdown 2026 marine video search Participants use interactive retrieval systems to find five underwater clips after seeing each target. Each search has a five-minute limit. | Everyday | Jan 2026 | Not in cards | |
| VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. | University | Jan 2026 | | In use |
| WorldVQA Model names visual entities in 3,500 image-question pairs across nine categories, isolating memorized visual world knowledge. Exact-answer accuracy. | University | Jan 2026 | | In use |
| WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. | Expert | Jan 2026 | Not in cards | |
| Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. | Expert | 2026 | Not in cards | |
| AECV-Bench (2026 release) Models count four types of floor plan elements and answer questions about engineering drawings. Counting uses exact matches. A model grades the drawing answers. | University | Dec 2025 | | Not in cards |
| AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. | Expert | Dec 2025 | Not in cards | |
| AV-SpeakerBench Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together. | Everyday | Dec 2025 | | Fading |
| BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. | Competition | Dec 2025 | | Fading |
| CADExpert Models write CadQuery programs from dimensioned engineering drawings. The graders check whether each program runs and compare its 3D shape with the reference. | University | Dec 2025 | | Not in cards |
| CAIS Vision Capabilities Index Averages model scores on five vision tests run by the Center for AI Safety: ERQA, MindCube, SpatialViz-Bench, IntPhys 2 and EnigmaEval. | Research frontier | Dec 2025 | 84.0 | Not in cards |
| CrossPoint-Bench Models see two views of a scene and must find the point in the second view that matches a point in the first. Tasks go from perception and reasoning to exact coordinate prediction. | Everyday | Dec 2025 | | Not in cards |
| CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. | Expert | Dec 2025 | Not in cards | |
| FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. | University | Dec 2025 | | Fading |
| FACTS Multimodal Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions. | University | Dec 2025 | | Not in cards |
| JointAVBench Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans. | Everyday | Dec 2025 | | Fading |
| LongShOTBench Models answer single-turn and multi-turn questions about long videos using visual, spoken, and ambient audio evidence. Weighted rubrics award partial credit. | Everyday | Dec 2025 | | Not in cards |
| MMSI-Video-Bench Human-annotated questions on video-based spatial intelligence, organized in four levels from perception to planning and prediction. | Everyday | Dec 2025 | | System cards only |
| Multimodal RewardBench 2 Reward models compare two outputs containing text and images. Accuracy measures agreement with expert preferences across four tasks. | University | Dec 2025 | | Not in cards |
| OSI-Bench Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates. | Everyday | Dec 2025 | | System cards only |
| PE-AV internal video-to-audio retrieval Models match videos to music tracks. Meta compares retrieval accuracy on internal clips that last from five to thirty seconds. | Everyday | Dec 2025 | Not in cards | |
| RealKIE-FCC Verified Model extracts structured fields from scanned FCC invoices into a JSON schema. Amazon human-verified labels. Field-level similarity (ALNS). | Everyday | Dec 2025 | | Fading |
| REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. | Everyday | Dec 2025 | 53.1% | Not in cards |
| SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. | Everyday | Dec 2025 | 57.8% | Not in cards |
| SAM Audio-Bench Models isolate a requested sound from an audio mixture using text, visual, or temporal prompts. Listeners judge how well the output preserves the target. | Everyday | Dec 2025 | Not in cards | |
| SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. | Expert | Dec 2025 | | In use |
| SO-Bench Multimodal models produce schema-compliant structured outputs grounded in images from four visual domains. Static checks and rubric-based grading score semantic correctness and schema compliance. | University | Dec 2025 | Not in cards | |
| SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. | Everyday | Dec 2025 | | System cards only |
| SUPERChem Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. | Expert | Dec 2025 | | In use |
| TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. | Everyday | Dec 2025 | | Fading |
| TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. | Everyday | Dec 2025 | Not in cards | |
| Video Reality Test Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection. | Everyday | Dec 2025 | | Not in cards |
| VIST Image models generate a new image in a reference brand style. Human evaluators compare pairs of outputs. | University | Dec 2025 | Not in cards | |
| ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. | Expert | Dec 2025 | | System cards only |
| ArtEdit-Bench-Eval Models rate photo edits against English instructions. Spearman and Pearson correlations measure agreement with human ratings on 200 cases. | Everyday | Nov 2025 | | Not in cards |
| ArtEdit-Bench-Lr Models retouch photos from English or Chinese instructions. GPT-4o scores instruction following and visual quality. Pixel errors measure fidelity to reference edits. | University | Nov 2025 | | Not in cards |
| CrochetBench Models identify stitches, select instructions that match a photo, write crochet patterns, and translate patterns into programs for a crochet simulator. | University | Nov 2025 | 60.9 | Not in cards |
| CrossVid Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats. | Everyday | Nov 2025 | 45.2% | Fading |
| DravidianLangTech 2026 Multilevel Political Meme Classification in Tamil and Malayalam Models identify a political meme's stance and target from its image and text in Tamil or Malayalam. | Everyday | Nov 2025 | Not in cards | |
| EvaHan 2026 Models transcribe printed and handwritten Chinese books and locate page elements such as illustrations and annotations. | University | Nov 2025 | Not in cards | |
| GEO-Bench-2 Models interpret Earth imagery across 19 tasks, including field boundaries, forest biomass and tree species. The authors normalize task scores to compare model capabilities. | University | Nov 2025 | Not in cards | |
| MarineEval Models answer 2,000 questions about marine images and documents. The authors combine answer checks with model judgments. They report total accuracy and mean accuracy across seven tasks. | University | Nov 2025 | 50.4% | Not in cards |
| MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. | School | Nov 2025 | | System cards only |
| MME-CC Models answer 1,173 expert-annotated vision-grounded questions in 11 task types grouped as spatial, geometric and visual knowledge reasoning. | Everyday | Nov 2025 | 40.3% | Fading |
| Multi-Crit Multimodal judges compare responses under several criteria. Human judgments provide the reference for each criterion and reveal trade-offs between criteria. | University | Nov 2025 | 43% | Not in cards |
| NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. | Expert | Nov 2025 | Not in cards | |
| OCR Arena Users upload a document, compare two anonymous OCR or vision-language models side by side, and vote. Votes become Elo ratings. | Everyday | Nov 2025 | | Not in cards |
| Qualcomm Interactive Cooking Models watch a cook follow a recipe. They give timed instructions, recognize completed steps, and warn about mistakes. The benchmark scores completion accuracy and mistake detection. | Everyday | Nov 2025 | | Not in cards |
| SAM 3D Artist Objects Models reconstruct an object's shape and placement from a single photograph. The benchmark compares predictions with meshes created by 3D artists. | School | Nov 2025 | Not in cards | |
| SAM 3D Objects human preference evaluation Models reconstruct objects from images of real scenes. People compare the shape and texture of competing reconstructions, including objects seen from an egocentric viewpoint. | Everyday | Nov 2025 | Not in cards | |
| Segment Anything with Concepts Models find every instance of a concept named in a prompt. The benchmark compares image masks and video tracks with human annotations. | Everyday | Nov 2025 | Not in cards | |
| Table Judge Benchmark Vision language models judge whether rendered table images match candidate HTML. Controlled corruptions test false positives, true positives and injected-error detection. | Expert | Nov 2025 | 94.9% | Not in cards |
| TIR-Bench Models solve 13 kinds of image tasks that need tool use on images, such as rotation, jigsaw, and maze. Most items are multiple choice. | Everyday | Nov 2025 | | Fading |
| UniREditBench Image editing models carry out 2,700 edits that need reasoning, in real-world scenes and in game worlds with human-defined rules, across 8 primary and 18 secondary dimensions. A multimodal judge scores each edit against both a text reference and a ground-truth image. | Everyday | Nov 2025 | | Not in cards |
| VSI-Bench-Debiased Models answer the 2,362 VSI-Bench questions left after removing 2,768 that text-only models could solve from answer priors or world knowledge. | Everyday | Nov 2025 | | Not in cards |
| VSI-SUPER Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours. | Everyday | Nov 2025 | Not in cards | |
| ADQA Models describe film scenes. An evaluator answers questions from the descriptions and dialogue, then checks whether the context supports each correct answer. | Everyday | Oct 2025 | Not in cards | |
| AGI Score (A Definition of AGI) Experts test models on ten cognitive domains from Cattell-Horn-Carroll theory, such as reasoning, memory and perception, with adapted human psychometric tests. Each domain counts 10% toward a score out of 100%. | Expert | Oct 2025 | 57% | Not in cards |
| BlueprintSymVL Models use an example symbol to find matching components in piping and instrumentation diagrams. They must give both the correct count and the component labels. | School | Oct 2025 | Not in cards | |
| Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. | Everyday | Oct 2025 | 37% | Not in cards |
| CRAG-MM Multimodal retrieval agents answer single-turn and multi-turn questions grounded in images and changing web content. The benchmark includes wearable-style egocentric images. | Expert | Oct 2025 | Not in cards | |
| Dense 200 (bbox) Model detects and boxes every object in 200 densely annotated images (109 categories, about 91 boxes each). Box F1 at IoU thresholds. | Everyday | Oct 2025 | | In use |
| DSI-Bench Video questions about dynamic scenes where both the observer and objects move, testing whether models separate camera motion from object motion. | Everyday | Oct 2025 | | System cards only |
| FenBench Models identify chess pieces, locate them, count them or describe a board position in Forsyth-Edwards Notation. The test contains 200 examples. | Everyday | Oct 2025 | Not in cards | |
| HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. | Research frontier | Oct 2025 | Not in cards | |
| LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. | Everyday | Oct 2025 | | Not in cards |
| LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. | Everyday | Oct 2025 | 53% | Not in cards |
| Math Kangaroo 2025 (MathArena) Vision language models answer 168 multiple-choice problems from the 2025 Math Kangaroo, given as screenshots across six grade groups. Many problems need reading a diagram. | School | Oct 2025 | 94.9% | Fading |
| Math-VR Models answer 5,000 English and Chinese math questions whose solutions need visual reasoning, such as drawing auxiliary lines or plotting functions. Answers are checked against ground truth. | School | Oct 2025 | | System cards only |
| MathCanvas-Bench Models solve 3,000 math problems that call for interleaved visual and textual solutions, such as drawing auxiliary lines in geometry. | School | Oct 2025 | | Fading |
| MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. | University | Oct 2025 | 64% | Not in cards |
| MeasureBench Vision-language models read values from images of measuring instruments, such as gauges and rulers, in real and synthesized photos. Scored by matching the reading to the key. | Everyday | Oct 2025 | | In use |
| MeasureBench Real-world Models read values and units from real instrument images. The score is the published overall measurement accuracy. | Everyday | Oct 2025 | | Not in cards |
| MeasureBench Synthetic Models read values and units from generated instrument images. The score is the published overall measurement accuracy. | Everyday | Oct 2025 | | Not in cards |
| MM-HELIX Multimodal models solve 1,260 items from 42 synthetic tasks that need long reflective reasoning with backtracking. | University | Oct 2025 | 84.5% | Not in cards |
| NautData test set Models answer questions, describe underwater scenes, and locate organisms in images. The paper reports 7,916 test examples on 3,920 images. | University | Oct 2025 | | Not in cards |
| Omni-Cloze Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail. | Everyday | Oct 2025 | | Fading |
| OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. | University | Oct 2025 | | Fading |
| PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. | Everyday | Oct 2025 | | Not in cards |
| PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. | School | Oct 2025 | | Not in cards |
| Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. | Everyday | Oct 2025 | | System cards only |
| RefSpatial-Expand-Bench Models point to an object or free space described by multi-step spatial relations, in more indoor scenes plus new outdoor streets and parking lots. Scored by mask hit. | Everyday | Oct 2025 | Not in cards | |
| RefSpatial-Expand-Bench Location Models solve spatial location questions in the RefSpatial expansion set. The score is the published overall accuracy. | Everyday | Oct 2025 | Not in cards | |
| RefSpatial-Expand-Bench Placement Models solve spatial placement questions in the RefSpatial expansion set. The score is the published overall accuracy. | Everyday | Oct 2025 | Not in cards | |
| RoboChallenge Fine-tuned policies run 10 real rollouts on each of 30 tabletop tasks across four robot types. Scored by success rate and progress. | Everyday | Oct 2025 | 42.7% | Not in cards |
| SPBench Spatial perception and reasoning questions built from ScanNet scenes. A single-image set covers distance, size and direction, and a multi-view set adds counting across views. Answers are numbers or multiple choice. | Everyday | Oct 2025 | | System cards only |
| SuperCLUE-Edit Models edit an input image according to a Chinese instruction. The evaluation measures whether the requested change preserves relevant details. | Everyday | Oct 2025 | 82.5 | Not in cards |
| UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. | Everyday | Oct 2025 | | Dropped |
| USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. | Everyday | Oct 2025 | 37.6% | Not in cards |
| UWBench Models answer questions about underwater scenes, write captions, and locate objects from descriptions. Questions cover visibility, materials, and spatial relationships. | Everyday | Oct 2025 | Not in cards | |
| Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. | Expert | Oct 2025 | 71.1% | Not in cards |
| VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. | University | Oct 2025 | | Not in cards |
| ViVerBench Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX. | School | Oct 2025 | 74.4% | System cards only |
| World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. | Everyday | Oct 2025 | 64.8% | Not in cards |
| Arti4D Models watch first-person recordings of people opening furniture. They predict whether parts rotate or slide and estimate the axis of motion. | School | Sep 2025 | Not in cards | |
| ASIMOV Models spot physical hazards in text and generated videos from real injury reports, time safety interventions in video, and follow robot payload and handling constraints. | Sep 2025 | | Fading | |
| BEHAVIOR Challenge 2025 50 long-horizon household activities on a Galaxea R1 Pro in OmniGibson, trained from 10,000 demos. Scored by Q-score and success. | Everyday | Sep 2025 | Fading | |
| BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. | Expert | Sep 2025 | Not in cards | |
| Blueprint-Bench Models turn about 20 interior photos of each of 50 apartments into a 2D floor plan. Scored by similarity of room connectivity and size ranking to the true plan. | Everyday | Sep 2025 | 0.43 | Not in cards |
| CircuitSense Models read circuit schematics and block diagrams, derive equations, and select designs. SymPy checks symbolic answers. Other tasks use exact matches, a model judge, or simulation. | University | Sep 2025 | | Not in cards |
| ClockBench Models read analog clock faces and answer questions about the time shown. Most clocks stay private to avoid training leakage. | Everyday | Sep 2025 | 87.8% | Not in cards |
| CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. | Expert | Sep 2025 | | System cards only |
| Earth-Bench Agents analyze satellite observations with Earth science tools. The evaluator checks the final answer and compares tool calls with a reference sequence. | University | Sep 2025 | 66% | Not in cards |
| Ego3D-Bench Models answer spatial reasoning questions about ego-centric multi-view scenes, such as those from self-driving cars. Some items ask for distances, scored by error. | Everyday | Sep 2025 | | In use |
| GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. | Expert | Sep 2025 | | In use |
| GSM8K-V Vision-language models solve GSM8K word problems that are redrawn as multi-image visual scenes, so the facts must be read from pictures. | School | Sep 2025 | | Not in cards |
| HiPhO Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. | Competition | Sep 2025 | | In use |
| LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. | Competition | Sep 2025 | Not in cards | |
| MapAnything dense multi-view reconstruction Models reconstruct a scene from multiple images. The benchmark compares point geometry, metric scale, depth, and camera poses with reference measurements. | School | Sep 2025 | Not in cards | |
| MapAnything single-view calibration Models infer camera intrinsics from a single image. The benchmark measures angular error between predicted camera rays and reference rays. | School | Sep 2025 | Not in cards | |
| MatCha Models answer multiple-choice questions about material structure, shape and properties using images from materials characterization instruments. | Expert | Sep 2025 | | Not in cards |
| MulSeT Models answer spatial questions about direction, distance and masks across views. The paper studies single images, multiple views and video. | University | Sep 2025 | | Not in cards |
| OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. | University | Sep 2025 | 14.6 | Not in cards |
| ODV-Bench Streaming video models answer real-time questions about autonomous driving footage. | Everyday | Sep 2025 | | Fading |
| Radiology's Last Exam (RadLE) Multimodal models make a single spot diagnosis from images alone on 50 hard radiology cases. Board-certified radiologists scored about 83%. | Expert | Sep 2025 | 30% | Not in cards |
| ROME Models solve reasoning problems selected to limit contamination. | University | Sep 2025 | | Not in cards |
| Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. | Everyday | Sep 2025 | | Fading |
| Synthetic Sheet Music Reasoning Benchmark Models answer questions about rhythm, chords, intervals, and scales. Each test question appears in text notation and as an image. | School | Sep 2025 | 90.5% | Not in cards |
| VideoScore-Bench v2 Models score 500 generated videos on visual quality, agreement with the prompt, and physical consistency. The evaluation checks their ratings against human scores. | Everyday | Sep 2025 | 28.9% | Not in cards |
| Visual Reasoning Benchmark (Fab AI) Models choose which picture completes a pattern, matches another shape, or shows a folded figure. Questions come from school exams in Zambia and India. | School | Sep 2025 | | Not in cards |
| VisualProbe Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key. | Everyday | Sep 2025 | | Fading |
| VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. | Everyday | Sep 2025 | | Not in cards |
| VoxelBench Language models turn text or image prompts into voxel structures. Voters compare anonymous builds from the same prompt, and Glicko-2 ratings rank the models. | Everyday | Sep 2025 | 2,607 | Not in cards |
| VoxelBench Image Models turn image prompts into voxel structures. Human comparisons determine Glicko-2 ratings in a separate image-prompt vote pool. | Everyday | Sep 2025 | 2,215 | Not in cards |
| Arctic-Extract Business Document Benchmark Models answer questions about business documents and extract information from varied layouts. The internal visual tasks include contracts, signatures and checkboxes. | University | Aug 2025 | 58.5 | Not in cards |
| Artificial Analysis Intelligence Index v2.2 Version 2.2 of Artificial Analysis's index, which swapped the older math tests for AIME 2025 and added instruction following and long-context reasoning. | Aug 2025 | Not in cards | ||
| BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. | Expert | Aug 2025 | | Fading |
| CountQA Models count objects in over 1,500 real-world photos with dense clutter and occlusion. Scoring is accuracy on the count. | Everyday | Aug 2025 | | In use |
| DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. | Everyday | Aug 2025 | 39% | Not in cards |
| EgoCross Models answer questions about events in first-person videos. The challenge scores accuracy on 957 four-choice questions across four domains. | Everyday | Aug 2025 | Not in cards | |
| EncQA Vision-language models answer chart questions that isolate visual encoding channels and analytic operations. | University | Aug 2025 | Not in cards | |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | Expert | Aug 2025 | 16% | Fading |
| KRETA Models read and reason over Korean text in images across diverse visual contexts. It is a text-rich visual question answering test for Korean. | Everyday | Aug 2025 | | Fading |
| M3-Bench Multimodal agents answer questions about long videos while using long-term memory and multimodal retrieval. | Expert | Aug 2025 | Not in cards | |
| MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. | Expert | Aug 2025 | 24.3% | Fading |
| MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. | Expert | Aug 2025 | | Fading |
| MOSEv2 Models track and segment objects in videos with complex scenes. Objects may disappear, reappear, or blend into a crowd. | Everyday | Aug 2025 | Not in cards | |
| RQA-70K weather radar forecast evaluation Models compare forecast radar images or sequences with observations. They rate errors such as missed storms and false alarms and write assessments of forecast quality. | University | Aug 2025 | 52.7% | Not in cards |
| SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. | Everyday | Aug 2025 | 0.92 | Fading |
| SatLidar canopy height evaluation Models predict canopy height from satellite images. Meta compares height estimates with airborne lidar measurements and reports errors on held-out regions. | University | Aug 2025 | Not in cards | |
| WildScore Models read score excerpts from Reddit music theory discussions and choose among answers drawn from the comments. | University | Aug 2025 | 75% | Not in cards |
| AgroBench Models answer illustrated questions from agronomists about crops, diseases, pests, weeds, machinery and farming practices. The overall score averages accuracy across seven tasks. | University | Jul 2025 | 73.5% | Not in cards |
| EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. | Expert | Jul 2025 | Not in cards | |
| Fundus-MMBench Models identify retinal structures, classify diseases, and grade their severity in fundus photographs. Each of the 31 tasks contains 20 questions. | Expert | Jul 2025 | Not in cards | |
| MMCircuitEval Models answer questions about circuit diagrams, specifications, and design decisions. The authors combine text similarity with a language model’s judgment. | University | Jul 2025 | 69.4% | Not in cards |
| MWAHAHA Models write jokes from word pairs or news headlines in English, Spanish, and Chinese. They also caption animated GIFs in English. | Everyday | Jul 2025 | Not in cards | |
| OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. | Everyday | Jul 2025 | 53.4% | Not in cards |
| PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. | Expert | Jul 2025 | Not in cards | |
| RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. | Everyday | Jul 2025 | | Fading |
| SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. | University | Jul 2025 | Not in cards | |
| SF20K Competition 2025 Models answer story questions about short films. GPT-4.1-nano compares their answers with human references. Films released during the competition determine the final rankings. | Everyday | Jul 2025 | Not in cards | |
| SpatialViz-Bench Models answer 1,180 generated multiple choice puzzles on mental rotation, mental folding, visual penetration, and mental animation. Scoring is accuracy. | Everyday | Jul 2025 | 41.4% | System cards only |
| TimeScope Models answer questions about short clips inserted into videos lasting up to eight hours. Accuracy measures retrieval, synthesis and temporal perception. | Everyday | Jul 2025 | | Not in cards |
| TreeBench Models answer 405 visual questions about small targets in dense SA-1B images and give bounding boxes as evidence. Scored by answer accuracy, with box overlap as a second measure. | University | Jul 2025 | | Fading |
| US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. | Expert | Jul 2025 | Not in cards | |
| 34th Japanese National Examination for Dental Hygienists Models answer 213 Japanese dental hygiene questions, including 139 with visual material such as instrument photographs and radiographs. | University | Jun 2025 | Not in cards | |
| Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. | Everyday | Jun 2025 | 64.7% | Fading |
| Argus Inspection Models inspect images for small details that change the sensible response to a question. A judge scores whether the response identifies the relevant detail and chooses an appropriate action. The score combines those judgments and assigns zero when a separate description fails to recognize the detail. | Everyday | Jun 2025 | | Not in cards |
| BM-Bench Image editing models perform instruction-guided non-rigid image manipulations on the ByteMorph benchmark dataset. | Expert | Jun 2025 | Not in cards | |
| CausalVQA Models answer questions about real egocentric videos that probe physical cause and effect, in counterfactual, hypothetical, anticipation, planning and descriptive types. | Everyday | Jun 2025 | | System cards only |
| FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. | University | Jun 2025 | 52% | Not in cards |
| Fire360 Models interpret panoramic video from firefighter training drills. They answer questions, caption actions, assess procedures, locate equipment, and match damaged objects to clean examples. | Expert | Jun 2025 | 53.8% | Not in cards |
| IntPhys 2 Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy. | Everyday | Jun 2025 | | In use |
| K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. | School | Jun 2025 | | Not in cards |
| K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. | School | Jun 2025 | | Not in cards |
| Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. | Everyday | Jun 2025 | Not in cards | |
| LMArena Image Edit Arena Image editing models apply the same user edit request to the same image, anonymously and side by side. Community votes give Elo-style ratings. | Everyday | Jun 2025 | 1,519 | Not in cards |
| MindCube Models answer 21,154 multiple choice questions about scenes seen from a few views. Questions test positions, orientations, and what-if movements. | Everyday | Jun 2025 | | Fading |
| MIRAGE-MMMT Models decide whether to ask for more information or answer a grower during an agricultural consultation. They then write the question or answer. | University | Jun 2025 | 65.5% | Not in cards |
| MIRAGE-MMST Models answer growers’ questions using photographs and written descriptions. They identify the cause of a problem and recommend how to manage it. | University | Jun 2025 | 68.0 | Not in cards |
| MMArt-Bench Models retouch 200 photos from editing instructions. Gemini scores visual quality and instruction following, and image comparisons measure how closely edits match reference photos. | University | Jun 2025 | | Not in cards |
| MORSE-500 Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning. | Everyday | Jun 2025 | 23.6% | Fading |
| MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. | University | Jun 2025 | | Not in cards |
| MusiXQA Models read printed music and answer questions about notes, chords, text, and page layout. The authors measure answer accuracy and string similarity. | School | Jun 2025 | | Not in cards |
| MVPBench Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail. | Everyday | Jun 2025 | | Dropped |
| OmniSpatial Vision language models answer over 8,400 spatial reasoning questions in four groups: dynamic reasoning, complex spatial logic, spatial interaction and perspective taking. | Everyday | Jun 2025 | | Fading |
| OpenLMM Spatial Leaderboard Models answer spatial reasoning tasks involving images. OpenCompass compares performance across the component evaluations. | University | Jun 2025 | 62.5 | Not in cards |
| PST50 Models transfer a reference look to a photograph. The paired test compares 50 outputs with reference grades that professionals refined in DaVinci Resolve. | University | Jun 2025 | Not in cards | |
| PuzzleWorld Models solve 667 puzzle-hunt problems that require varied reasoning strategies. | Competition | Jun 2025 | | Not in cards |
| RefSpatial-Bench Real images where the model points to an object or free space defined by multi-step spatial relations. Scored by mask hit. | Everyday | Jun 2025 | | Fading |
| RoboArena Double-blind pairwise A/B trials of generalist DROID policies on real Franka arms, free-form tasks by volunteer evaluators. Elo-style rating. | Everyday | Jun 2025 | 1,608 | Not in cards |
| RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. | Everyday | Jun 2025 | 58.3% | Not in cards |
| Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. | Expert | Jun 2025 | | Fading |
| STARE Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation. | Everyday | Jun 2025 | | System cards only |
| V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. | Everyday | Jun 2025 | Not in cards | |
| Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. | Everyday | Jun 2025 | 84.9% | Not in cards |
| VRBench Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps. | Everyday | Jun 2025 | 68.2% | Not in cards |
| WildKhmerST Models find and read Khmer text in photographs of public places. Annotations mark text lines with polygons and transcriptions. | Everyday | Jun 2025 | Not in cards | |
| AgroMind (original) Models answer questions about farmland images across 13 tasks. They identify crops, assess crop health and interpret spatial relationships. | University | May 2025 | | Not in cards |
| ChartMuseum Models answer 1,162 expert-annotated questions about real-world charts from 184 sources. Many questions need visual reasoning that text cannot replace. | Expert | May 2025 | | Fading |
| CoralVQA Models answer questions about coral identity, growth, and health in reef photographs. GPT-4o checks answers against references across 16 question types. | University | May 2025 | 41.8% | Not in cards |
| CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. | School | May 2025 | 49.6% | System cards only |
| Daily-Omni Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy. | Everyday | May 2025 | | Fading |
| Do You See Me Models answer 2,612 visual perception questions on 1,758 images across seven 2D and 3D subtasks drawn from human psychology. Questions are multiple choice or numeric. | Everyday | May 2025 | 51.1% | System cards only |
| DreamGen Bench Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline. | May 2025 | Not in cards | ||
| EmotionTalk Models infer emotions and sentiment from Chinese conversations. | Everyday | May 2025 | 50.9 | Not in cards |
| General-Bench Models perform multimodal understanding and generation tasks. General-Level averages task scores and measures performance above specialist baselines at progressively broader levels. | University | May 2025 | | Not in cards |
| GIE-Bench Image-editing models follow text instructions while preserving unedited regions. GPT-4o question answers and masked image metrics score functional correctness and preservation. | University | May 2025 | Not in cards | |
| ImgEdit-Bench Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5. | Everyday | May 2025 | | System cards only |
| IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. | Everyday | May 2025 | | Dropped |
| KORGym Models play 51 text and visual games designed to test interactive reasoning. | Everyday | May 2025 | 0.82 | Not in cards |
| KRIS-Bench Image editing models follow 1,267 instructions that need factual, conceptual or procedural knowledge across 22 tasks. A GPT-4o judge scores the edits. | School | May 2025 | | System cards only |
| LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. | Everyday | May 2025 | 72.4% | Fading |
| LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. | Everyday | May 2025 | | Dropped |
| MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. | University | May 2025 | | Fading |
| MME-Reasoning Multimodal models answer logic questions grouped into inductive, deductive and abductive reasoning. | University | May 2025 | | Not in cards |
| MME-VideoOCR Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning. | Everyday | May 2025 | | Fading |
| MMEB-V2 Multimodal embedding models are scored on 78 tasks over images, videos and visual documents. V2 adds visual document retrieval, video retrieval, temporal grounding, video classification and video QA. | Everyday | May 2025 | | Fading |
| MMLongBench Vision-language models solve 13,331 long-context examples in five task types, such as visual RAG, needle search and many-shot in-context learning. Each example comes at five lengths from 8K to 128K tokens. | Everyday | May 2025 | | Fading |
| MMSI-Bench Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data. | Everyday | May 2025 | | Fading |
| MSEarth Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split. | Expert | May 2025 | | Fading |
| OCR-Reasoning Models answer 1,069 human-annotated questions that need reasoning over text-rich images. The questions cover 6 reasoning abilities and 18 practical tasks. | Everyday | May 2025 | 44.4% | Fading |
| OmniEarth-Bench Models interpret satellite images and observational data across Earth systems, answering questions about environmental processes and their interactions. Multiple-choice accuracy and model judgments score answers. Separate measures check localization, captions, and annotated reasoning steps. | Expert | May 2025 | | Not in cards |
| PhyX Models solve 3,000 image-based physics problems across six domains such as mechanics and optics. Versions include multiple choice and open-ended answers. | University | May 2025 | 45.8% | Fading |
| Point-Bench Models point to the image location a language request describes, in about 1,000 tasks across five reasoning categories. A point inside the target mask counts as correct. | Everyday | May 2025 | | Fading |
| RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. | Expert | May 2025 | 69.9% | Not in cards |
| Roboflow 100-VL Models locate objects named in natural language across image collections from different domains. The benchmark scores predicted bounding boxes using average precision. | Everyday | May 2025 | | Not in cards |
| Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. | University | May 2025 | | System cards only |
| SITE Models answer multiple-choice spatial questions over single images, multiple images and video, drawn from about 30 vision datasets. Tasks span figural to environmental scale. | Everyday | May 2025 | | Fading |
| ThinkGeo Agents use remote sensing tools to interpret aerial and satellite images. The authors compare individual tool calls and use GPT-4o mini to grade final answers. | University | May 2025 | 100% | Not in cards |
| Video-Holmes Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film. | Everyday | May 2025 | | Fading |
| Video-SafetyBench Video-language models receive 2,264 synthesized videos paired with a harmful or a benign-looking query across 13 unsafe categories and 48 subcategories. An LLM judge scores harm with a confidence-weighted metric called RJScore. | Everyday | May 2025 | 1.30 | Not in cards |
| VideoEval-Pro Video models give open-ended short answers to questions that need the whole long video, which removes the guessing that multiple choice allows. | Everyday | May 2025 | 40.8% | Fading |
| VideoReasonBench Video models watch a sequence of fine-grained operations on a partly hidden state and answer questions that recall, infer or predict that state. | Everyday | May 2025 | | Fading |
| ViewSpatial-Bench Models answer 5,712 questions that locate objects from the camera view and from the view of people in the scene. Five task types test egocentric and allocentric reasoning. | Everyday | May 2025 | | Fading |
| ViStoryBench Models turn stories and character references into coherent image sequences. Model judgments and image features measure prompt alignment, character consistency, style, and image quality. | University | May 2025 | | Not in cards |
| VLMsAreBiased Models count or identify features in altered images of famous subjects, such as a logo with an added stripe, across 7 domains. Scoring is accuracy. | Everyday | May 2025 | 84.1% | Fading |
| VSTI-Bench Video questions about how spatial relationships change over time in 3D scenes, over 138,000 pairs across five tasks. | Everyday | May 2025 | | System cards only |
| Word-Level BISINDO Models identify 32 signs from the Banten variety of Indonesian Sign Language in video clips. | Everyday | May 2025 | Not in cards | |
| 3D Animation Arena Models reconstruct a moving 3D human from a video. Voters compare motion fidelity, ground contact, smoothness, and overall appearance. | Everyday | Apr 2025 | Not in cards | |
| All-Angles Bench Models answer over 2,100 multi-view questions about 90 real scenes. Six tasks cover counting, attributes, relative distance and direction, object manipulation and camera pose. | Everyday | Apr 2025 | | Fading |
| AstroLLaVA image and text retrieval Models match astronomical images with descriptions. Google evaluates whether an image retrieves the matching description among its top five results. | University | Apr 2025 | Not in cards | |
| ChartQA Pro Model answers 1,948 diverse questions over 1,341 charts, infographics and dashboards, including conversational and unanswerable ones. Accuracy. | Everyday | Apr 2025 | 72.3% | Fading |
| ColorBench Models answer image questions about color perception, color reasoning, and robustness to color changes. The evaluator checks the selected answers against a single correct option. | Everyday | Apr 2025 | | Not in cards |
| FortisAVQA Models answer questions about sounds and visible events in music videos. Accuracy is reported separately for frequent and rare question distributions after the original questions are rephrased. | Everyday | Apr 2025 | 48.7% | Not in cards |
| GEdit-Bench Image editing models follow real-world user edit instructions. A model judge scores semantic consistency, perceptual quality and an overall score. | Everyday | Apr 2025 | | Dropped |
| Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. | Everyday | Apr 2025 | System cards only | |
| Japanese National Dental Examination 2025 Models answer 350 questions from Japan's 2025 dental licensing exam. The study reports accuracy separately for 202 text questions and 148 visual questions. | University | Apr 2025 | Not in cards | |
| Japanese National Examination for Occupational Therapists 2025 Models answer 199 Japanese occupational therapy licensing questions. The authors score answers against the official key and compare performance on text and visual questions. | University | Apr 2025 | 88.4% | Not in cards |
| JEEM Vision-language models caption images and answer visual questions in four Arabic dialects. The evaluation compares outputs with reference annotations. | Everyday | Apr 2025 | Not in cards | |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | University | Apr 2025 | | System cards only |
| LiveSports-3K Video models give real-time commentary and answer questions on sports clips. An LLM judge scores the free-form commentary. | Everyday | Apr 2025 | | Fading |
| MIEB Image and image-text embedding models are scored on 130 tasks in 38 languages, grouped into 8 task types such as retrieval, clustering and classification. | Everyday | Apr 2025 | Not in cards | |
| MM-IFEval Multimodal models follow image-grounded instructions with exact output constraints. Checks mix rule-based verification and a judge model. | Everyday | Apr 2025 | 64.6% | In use |
| MMK12 Models use images to answer school mathematics and science questions. The test split contains 2,000 problems with reference answers. | School | Apr 2025 | 73.9% | Not in cards |
| Open RAG Benchmark RAG systems answer 3,045 queries over 1,000 arXiv PDFs with text, tables, images, and hard negatives. | Expert | Apr 2025 | Not in cards | |
| OpenLMM Subjective Leaderboard Models generate open-ended answers to multimodal tasks. The board reports subjective evaluation results. | University | Apr 2025 | 76.6 | Not in cards |
| Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. | Everyday | Apr 2025 | System cards only | |
| PLM-VideoBench Models answer questions about activities, describe events involving a particular person or object, and locate those events in time. Meta also tests first-person video understanding. | Everyday | Apr 2025 | | Not in cards |
| PVD Bench Models match captions with short videos in both retrieval directions. Meta measures retrieval on held-out video and caption pairs. | Everyday | Apr 2025 | Not in cards | |
| RISEBench Image editing models follow 360 instructions that need temporal, causal, spatial or logical reasoning. An LMM judge scores instruction reasoning, appearance consistency and visual plausibility. | University | Apr 2025 | 51.4% | System cards only |
| RSCC Models describe disaster damage from satellite images taken before and after an event. The authors reserve 988 image pairs from 19 events for testing. | University | Apr 2025 | | Not in cards |
| Snowflake Multimodal PDF Retrieval Evaluation Retrievers find relevant pages in technical manuals, SEC filings and presentation slides. The tasks include 143 chart questions and 69 text questions about manuals. | University | Apr 2025 | Not in cards | |
| SPACE Models solve spatial cognition tasks involving maps, object layouts, spatial attention and memory in text and image presentations. Accuracy measures task performance. | University | Apr 2025 | Not in cards | |
| Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. | Everyday | Apr 2025 | System cards only | |
| VCR-Bench Video models answer 1,034 questions on 859 videos, and graders score both the final answer and each tagged chain-of-thought step against human rationales. | Everyday | Apr 2025 | 56.7% | Fading |
| VisualPuzzles Models solve visual puzzles that need algorithmic, analogical, deductive, inductive and spatial reasoning but little domain knowledge. Scored by accuracy. | Everyday | Apr 2025 | 57% | Fading |
| VisuLogic Models answer 1,000 visual reasoning puzzles in six categories that cannot be solved by describing the image in words. Scored by accuracy. | Everyday | Apr 2025 | | Fading |
| WorldScore 3,000 next-scene generation tasks with camera-trajectory layouts. Scores 3D/4D and video world generators on controllability, quality, dynamics. | Apr 2025 | Not in cards | ||
| 3DGen-Bench People compare generated 3D objects in pairs. The authors use their judgments to evaluate automated scoring methods for text-to-3D and image-to-3D generation. | Everyday | Mar 2025 | Not in cards | |
| ASIMOV v1 Models judge whether robot actions in scenes generated from real images and hospital injury reports are undesirable or unsafe. Scored by alignment with human labels. | Everyday | Mar 2025 | Not in cards | |
| AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. | Expert | Mar 2025 | 22% | Not in cards |
| AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. | Everyday | Mar 2025 | | Not in cards |
| AVUT Models answer questions about videos where the audio carries the key information. The benchmark removes questions that text alone can answer. | Everyday | Mar 2025 | | Fading |
| AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. | Everyday | Mar 2025 | | Dropped |
| BLUR Models identify a remembered item from incomplete clues in several languages and media. | Everyday | Mar 2025 | 54% | Not in cards |
| Creation-MMBench Models write creative pieces grounded in up to nine images plus a role, background and instructions, across 765 cases in 51 tasks. A judge scores quality and visual fact alignment against case-specific criteria. | Everyday | Mar 2025 | | Not in cards |
| CrossWordBench Models solve generated crossword grids from text or images. The tasks vary the grid size and the share of letters already filled in. | School | Mar 2025 | | Not in cards |
| EgoBlind Models answer questions from blind users using only video recorded before each question. GPT-4o mini checks each answer against human references. | Everyday | Mar 2025 | 59.3% | Not in cards |
| EgoLife (with memory) Models answer life-assistant questions about a week of egocentric video from six people, such as recalling past events and habits. | Everyday | Mar 2025 | | In use |
| EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. | Everyday | Mar 2025 | | Fading |
| EgoToM Video models infer a camera wearer’s goals, beliefs, and future actions from egocentric clips. Multiple-choice questions test theory of mind. | University | Mar 2025 | Not in cards | |
| ERQA 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. | Everyday | Mar 2025 | | In use |
| FiVE-Bench Models edit objects in videos from written instructions. A vision language model checks whether each edit succeeded, and other metrics measure changes to the background. | University | Mar 2025 | Not in cards | |
| Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. | Everyday | Mar 2025 | Not in cards | |
| GeoBench Models geolocate GeoGuessr photos from community maps by giving coordinates and a country. Scored by country accuracy and a distance-based score. | Everyday | Mar 2025 | 95% | Dropped |
| LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. | Everyday | Mar 2025 | 72% | Not in cards |
| m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. | Everyday | Mar 2025 | | Dropped |
| MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. | Everyday | Mar 2025 | 64.1% | Fading |
| MicroVQA Models answer expert-written multiple choice questions about biology microscopy images that test image understanding, hypothesis generation and experiment proposal. | Expert | Mar 2025 | 59.4% | Fading |
| OmniMMI Omni models answer 2,290 questions on 1,121 streaming videos, including proactive tasks where they must decide when to respond. | Everyday | Mar 2025 | | Fading |
| PHYSICS Models solve 1,297 PhD qualifying exam physics problems across six subfields, 298 of them with images. Answers are checked by symbolic equivalence. | Expert | Mar 2025 | 59.9% | Dropped |
| QIVD Models answer spoken questions about short phone videos in real time, as in a face to face talk. Each item has a short answer and a timestamp. | Everyday | Mar 2025 | | Fading |
| RefAV Models find driving scenarios described in natural language. They identify when each scenario occurs and track the objects involved. | University | Mar 2025 | Not in cards | |
| RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. | Everyday | Mar 2025 | | Fading |
| ROOMELSA Models use a marked region in a room panorama and a written description to retrieve the matching 3D object. They return their ten closest matches. | Everyday | Mar 2025 | Not in cards | |
| SPAR-Bench Spatial perception and reasoning questions over single and multiple views of indoor 3D scenes, from depth and distance estimates to object relations and spatial imagination. | Everyday | Mar 2025 | | System cards only |
| Video SimpleQA Video models answer multi-hop fact-seeking questions that need knowledge beyond the video and evidence from its time segments. Short definitive answers are graded for factual correctness. | University | Mar 2025 | 66.3% | Fading |
| ViSpeak-Bench Streaming video models must notice instructions given visually, such as a hand wave, and respond in seven subtasks of visual instruction feedback. | Everyday | Mar 2025 | 59.8% | Fading |
| VLADBench Models answer closed-form questions about driving scenes, from static traffic knowledge to reasoning about dynamic road situations. | University | Mar 2025 | | In use |
| XLRS-Bench Models answer perception and reasoning questions about extremely large, ultra-high-resolution remote sensing images. | Expert | Mar 2025 | 41.1% | Fading |
| AGI-Eval Community Image and Video Understanding Models interpret images and videos. The board reports visual question accuracy and separate comparison win rates. | Everyday | Feb 2025 | | Not in cards |
| Artificial Analysis Intelligence Index v2 Artificial Analysis's index of seven public reasoning, knowledge, math and coding benchmarks, each run by Artificial Analysis. | Feb 2025 | 67.7 | Dropped | |
| Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. | Expert | Feb 2025 | System cards only | |
| EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. | Everyday | Feb 2025 | 50.5% | Not in cards |
| EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. | Everyday | Feb 2025 | 67.7% | Not in cards |
| EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. | Everyday | Feb 2025 | 58.7% | Not in cards |
| EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. | Everyday | Feb 2025 | 28.5% | Not in cards |
| EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. | Everyday | Feb 2025 | | Not in cards |
| EmoBench-M Multimodal models are tested on emotional intelligence across 13 scenarios in three dimensions: emotion recognition, conversational emotion understanding and socially complex emotion analysis. | Everyday | Feb 2025 | | Not in cards |
| EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. | Competition | Feb 2025 | 39.3% | Not in cards |
| Flame-React-Eval Models write React components from UI design images. Rendered outputs are compared with gold components by image similarity and scored as pass@k. | University | Feb 2025 | 93.8% | Fading |
| HD-EPIC VQA Models watch first-person kitchen videos and choose among five answers. Questions ask what people cooked, where they moved objects, and where they looked. | Everyday | Feb 2025 | | Not in cards |
| Massive Visualized IR Benchmark Models retrieve screenshots that match a search query. The benchmark compares performance across four retrieval task groups. | Everyday | Feb 2025 | Not in cards | |
| MCS-Bench Models read seal inscriptions, identify calligraphy, and answer questions about Chinese paintings and artifacts. The overall score averages task scores. | Expert | Feb 2025 | | Not in cards |
| MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. | Everyday | Feb 2025 | 82.5% | Not in cards |
| MME-CoT Models reason step by step on image questions in math, science, OCR, logic, space-time and general scenes. A judge scores reasoning quality, robustness and efficiency. | University | Feb 2025 | 64.2% | Not in cards |
| Multimodal RewardBench Reward models choose between two answers to a visual prompt. Accuracy measures agreement with annotated preferences across six areas. | University | Feb 2025 | | Not in cards |
| Open FlagEval VLM Leaderboard Models answer questions about images. The leaderboard compares their scores across visual understanding tasks. | University | Feb 2025 | | Not in cards |
| RivaBench Models answer questions about stand-up routines and academic talks using speech and visual cues. They also classify clips as real or synthetic. | University | Feb 2025 | Not in cards | |
| SimpleVQA Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. | University | Feb 2025 | | In use |
| Virology Capabilities Test (VCT) Models answer 322 multimodal virology troubleshooting questions written by expert virologists. Multiple-select variant requires all correct options. | Expert | Feb 2025 | 67.4% | Fading |
| VisFactor Models take 20 vision-centric subtests adapted from a human cognitive psychology test battery, covering perception primitives and visual memory. Scored by accuracy. | Everyday | Feb 2025 | 55.9% | In use |
| WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. | Everyday | Feb 2025 | 45.8% | Not in cards |
| WorldSense (omnimodal) Models answer multiple choice questions about real-world videos that need both the visual and the audio track. Scored by accuracy. | Everyday | Feb 2025 | | Fading |
| ZeroBench 100 hand-made visual questions that all frontier models originally failed. Scored by exact answer, pass@1 and pass@5. | University | Feb 2025 | 54.6% | In use |
| DriveBench Vision-language models answer 20,498 questions about 19,200 driving frames, covering perception, prediction, planning and behavior. Inputs come in 17 settings: clean images, corrupted images and text only. The test checks whether answers rest on the image or on general knowledge. | Everyday | Jan 2025 | 48.0 | Not in cards |
| DriveBench Behavior Models answer driving behavior questions. The score is the published GPT evaluation score for this task family. | Everyday | Jan 2025 | | Not in cards |
| DriveBench Perception Models answer driving perception questions. The score is the published GPT evaluation score for this task family. | Everyday | Jan 2025 | 35.2 | Not in cards |
| DriveBench Planning Models answer driving planning questions. The score is the published GPT evaluation score for this task family. | Everyday | Jan 2025 | 57.0 | Not in cards |
| DriveBench Prediction Models answer driving prediction questions. The score is the published GPT evaluation score for this task family. | Everyday | Jan 2025 | | Not in cards |
| EMMA Models solve math, physics, chemistry and coding problems that need reasoning over text and images together, not each alone. Answers are checked against a key. | University | Jan 2025 | | Fading |
| GeoMap-Bench Models answer 3,864 questions about 124 geologic maps. They locate map features, interpret legends, and use geological evidence to explain conditions at a site. | Expert | Jan 2025 | 0.37 | Not in cards |
| HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. | Expert | Jan 2025 | | Fading |
| Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. | Expert | Jan 2025 | 54.8% | In use |
| Japanese Nephrology Board Renewal Questions 2014 to 2023 Models answer 209 Japanese nephrology board renewal questions from ten exam years, including questions with clinical images. Researchers compare answers with the society’s key. | Expert | Jan 2025 | 81.3% | Not in cards |
| MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. | Expert | Jan 2025 | | Fading |
| MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. | Expert | Jan 2025 | | In use |
| MER-UniBench Models describe emotions from audio, video, and text across nine datasets. Label normalization supports hit rate, weighted F1, and emotion-wheel F1. The reported mean averages the primary metric from each dataset. | Everyday | Jan 2025 | Not in cards | |
| MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. | University | Jan 2025 | | In use |
| MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. | Everyday | Jan 2025 | | Fading |
| OVO-Bench Video models answer questions asked at given timestamps of a video stream in 12 tasks over 644 videos. Tasks cover backward tracing, real-time understanding and forward active responding. | Everyday | Jan 2025 | | Fading |
| Physics-IQ Video models continue 396 filmed physical events from a frame or 3 s clip. Motion compared with the real continuation. | Jan 2025 | | System cards only | |
| StreamBench (StreamChat) Video models answer questions on streaming videos across media types, including multi-turn interaction and reasoning tasks. | Everyday | Jan 2025 | 59.1% | Dropped |
| Turkish Dental Specialty Examination 2020 and 2021 Models answer Turkish dental specialty entrance questions in basic and clinical sciences. The two official answer keys cover 238 scored questions, including questions with images. | Expert | Jan 2025 | Not in cards | |
| Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. | University | Jan 2025 | | In use |
| VideoGen-RewardBench Models rank two generated videos for the same prompt. The scorer compares their rankings with human preferences for visual quality, motion, prompt alignment, and overall quality. | Everyday | Jan 2025 | Not in cards | |
| VPCT Models view an image of a ball, ramps, and three buckets and predict which bucket the ball falls into. Scored by accuracy on 100 problems. | Everyday | Jan 2025 | | Fading |
| 3DSRBench Models answer questions about 3D spatial relations in images, such as height, location, orientation and relations between objects. | Everyday | Dec 2024 | | Fading |
| BdSLW401 Models identify one of 401 Bangla Sign Language words in a video. The collection records signs from the front and side. | Everyday | Dec 2024 | Not in cards | |
| CC-OCR Models read text in 7,058 images across four tracks: scene text, multilingual text, document parsing, and key information extraction. Each track has its own metric. | Everyday | Dec 2024 | | Fading |
| CG-Bench Video models answer 12,129 questions about 1,219 long videos and must also find the clue segments that support each answer. | Everyday | Dec 2024 | 33% | Fading |
| ICDAR 2025 MapText Competition Models detect and read words on historical maps and link them into phrases. This edition adds Taiwanese maps with Chinese labels and revises how links score. | School | Dec 2024 | Not in cards | |
| Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. | Everyday | Dec 2024 | 99% | System cards only |
| Labelbox Multimodal Reasoning Models interpret images, identify differences, write captions, and reason about spatial relationships. Human raters compare the responses. | Everyday | Dec 2024 | 1,220 | Not in cards |
| LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. | University | Dec 2024 | Not in cards | |
| Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. | Expert | Dec 2024 | 63.1% | System cards only |
| Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. | Everyday | Dec 2024 | | System cards only |
| OCRBench v2 Visual-text localization and reasoning tasks across 31 scenarios (documents, scenes, charts). Averaged task accuracy. | Everyday | Dec 2024 | | In use |
| Open LMM Reasoning Leaderboard Models solve multimodal reasoning tasks. OpenCompass ranks the models using the component results. | University | Dec 2024 | | Not in cards |
| OVBench Video models answer questions about past, current and future moments of an online video stream in 6 task types and 16 subtasks. | Everyday | Dec 2024 | | Fading |
| SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. | Everyday | Dec 2024 | | System cards only |
| ScanNet++ v2 Models reconstruct indoor scenes from images of real scans. The second release adds scene types and updates the annotations and camera poses. | School | Dec 2024 | Not in cards | |
| U-MATH Models solve 1,100 unpublished open-ended university math problems from teaching materials, balanced across six subjects. About 20% include images, and an LLM judge checks each solution. | University | Dec 2024 | 86.8% | Not in cards |
| Video-MonetBench Models rank two videos generated for the same prompt. The test compares each ranking with a human preference, including ties. | Everyday | Dec 2024 | Not in cards | |
| VisionArena-Bench Models answer visual questions from Arena users. A vision language model judges their answers to approximate human preference rankings. | Everyday | Dec 2024 | | Not in cards |
| VISTA Models answer 758 single-turn prompts about one image each, covering perception and reasoning. Three LLM judges grade answers with yes/no rubric questions. The main score is average task accuracy. | Expert | Dec 2024 | | Not in cards |
| VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. | Everyday | Dec 2024 | | Not in cards |
| VSI-Bench Models watch egocentric videos of 288 indoor scenes and answer over 5,000 questions on object counts, distances, sizes, room layout and route planning. | Everyday | Dec 2024 | | Fading |
| BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. | Everyday | Nov 2024 | 68.3% | Not in cards |
| GEOBench-VLM Models interpret satellite and aerial images, identify land use, and track changes across observations. Some tasks ask models to locate objects or outline regions. | University | Nov 2024 | 0.41 | Not in cards |
| HourVideo Multimodal models answer questions about first-person videos that run 20 to 120 minutes. The 500 videos come from Ego4D, and the 12,976 five-way multiple-choice questions cover summarization, perception, visual reasoning and navigation. Score is accuracy. | Everyday | Nov 2024 | Not in cards | |
| MaCBench Vision language models answer chemistry and materials questions from images. Tasks cover reading data, understanding experiments and interpreting results. | Expert | Nov 2024 | 72.6% | Not in cards |
| RD-TableBench Models extract 1,000 hand-labeled complex tables from document images, including scans, handwriting and merged cells. Scored by Levenshtein-based row and cell alignment. | Everyday | Nov 2024 | | System cards only |
| RoboSpatial-Home Models answer 350 spatial questions on home scenes for robots, in three types: object context, compatibility of placements and spatial configuration. | Everyday | Nov 2024 | | Dropped |
| StreamingBench Models answer questions while a video streams in, as a human viewer would. It tests real-time visual understanding, omni-source understanding and context understanding. | Everyday | Nov 2024 | | Fading |
| SURDS Models answer spatial questions about objects in nuScenes driving camera images across six single-object and multi-object tasks. Scoring is task accuracy. | Everyday | Nov 2024 | | In use |
| VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. | Everyday | Nov 2024 | | Not in cards |
| VL-RewardBench Vision-language reward models pick the better of two responses on general queries, hallucination detection and reasoning tasks. | Expert | Nov 2024 | | Not in cards |
| DynaMath Models answer visual math questions generated from 501 seed programs that vary values and graphs. Scored by average and worst-case accuracy across variants. | School | Oct 2024 | | Fading |
| Harmony4D Models recover the body shapes and poses of people interacting at close range. The benchmark includes sports where people occlude each other. | School | Oct 2024 | Not in cards | |
| HumanEval-V Models read a diagram and a function signature, then write Python code that must pass test cases. Tasks span six types of visual reasoning. | University | Oct 2024 | 40.6% | Fading |
| Image2Struct Vision-language models turn images of webpages, LaTeX and musical scores into code that renders them. The render is compared with the input image. | University | Oct 2024 | 94.7 | Not in cards |
| Khmer Handwritten Dataset (4.2k) Models transcribe line images from a handwriting collection that includes Khmer and English text. | Everyday | Oct 2024 | Not in cards | |
| LEVIR-CDC Models describe changes between aerial images and mark changed buildings. The authors paired LEVIR-CC captions with binary building masks. | Everyday | Oct 2024 | Not in cards | |
| LOKI Models distinguish synthetic content from real content across images, videos, audio, text, and 3D data. The benchmark checks real-or-synthetic labels and annotated abnormal details, with separate tasks for explaining anomalies. | Everyday | Oct 2024 | 60.8% | Not in cards |
| ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. | Everyday | Oct 2024 | Not in cards | |
| MEGA-Bench Models solve 505 real-world multimodal tasks with over 8,000 samples. Outputs include numbers, code, LaTeX, coordinates and JSON, scored by more than 40 task metrics. | University | Oct 2024 | 54.3% | System cards only |
| MM-MT-Bench Models hold multi-turn conversations about images such as charts, documents and photos. GPT-4o grades each answer against a reference on a 1 to 10 scale. | Everyday | Oct 2024 | | Dropped |
| MMEB Multimodal embedding models are scored on classification, visual question answering, multimodal retrieval and visual grounding, with 16 evaluation datasets in and out of distribution. | Everyday | Oct 2024 | | Not in cards |
| MMIE Models interpret and generate interleaved text and images. MMIE-Score, a fine-tuned InternVL-2-4B model, grades the resulting multimodal responses. | University | Oct 2024 | Not in cards | |
| NaturalBench Models answer 10,000 human-checked questions on natural images. Each question pairs with two images that need different answers, so a model cannot pass without looking. | Everyday | Oct 2024 | 55.1% | Not in cards |
| OBI-Bench Models locate inscriptions, rejoin bone fragments, classify characters, retrieve matches, and explain glyphs. The authors publish separate scores for each task. | Expert | Oct 2024 | 0.61 | Not in cards |
| QAG-360K Models compare two satellite images, answer questions about land cover, and mark the relevant pixels. The authors score answer accuracy and mask overlap separately. | School | Oct 2024 | Not in cards | |
| R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. | Everyday | Oct 2024 | | System cards only |
| SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests. | Expert | Oct 2024 | 74.1% | In use |
| TaxaBench-8k Models identify species or match observations across photographs, sounds, and other field data. The test contains 8,813 observations paired across six types of data. | Expert | Oct 2024 | Not in cards | |
| TemporalBench Video models answer about 10K questions on fine-grained temporal dynamics, built from about 2K detailed human annotations of video clips. | Everyday | Oct 2024 | 28.1% | Dropped |
| TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. | Everyday | Oct 2024 | | Fading |
| TVBench Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. | Everyday | Oct 2024 | | In use |
| VDC Benchmark Models describe the contents of a video in detail. The evaluation checks descriptions against human annotations using questions about the video. | Everyday | Oct 2024 | | Not in cards |
| VHELM Vision-language models are scored across nine aspects, including perception, reasoning, knowledge, bias, fairness, multilinguality, robustness, toxicity and safety, from many existing datasets. | University | Oct 2024 | | Not in cards |
| Vinoground Models match short videos with captions that distinguish events by their temporal order. | School | Oct 2024 | Not in cards | |
| WHU-CDC Models compare aerial photos of Christchurch, describe what changed, and mark changed pixels. Reference captions describe reconstruction after the earthquake. | Everyday | Oct 2024 | Not in cards | |
| WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. | Oct 2024 | Not in cards | ||
| xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. | Everyday | Oct 2024 | | System cards only |
| xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. | University | Oct 2024 | | System cards only |
| CHART-Info 2024 Models classify chart images, identify text and axes, and recover plotted values. The authors compare baselines on five of the seven tasks. | School | Sep 2024 | Not in cards | |
| DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. | University | Sep 2024 | Not in cards | |
| DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. | Expert | Sep 2024 | 89.9% | Dropped |
| EventHallusion Models answer questions about video events that can prompt false descriptions. The evaluation tests whether answers stay grounded in the clip. | Everyday | Sep 2024 | 76.5% | Not in cards |
| MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. | University | Sep 2024 | 84.6% | In use |
| OmniBench Models answer multiple choice questions that need an image, an audio clip and text together. Scored by accuracy. | Everyday | Sep 2024 | | Dropped |
| OpenVLM Video Leaderboard Models complete video understanding tasks. OpenCompass compares models across video evaluation datasets. | University | Sep 2024 | | Not in cards |
| PixMo-Count Models count objects in natural images and give the exact number. Scored by accuracy on the test split. | Everyday | Sep 2024 | | Fading |
| Q-Bench-Video Models judge the quality of natural, AI-generated and computer-graphics videos through yes-or-no, what-how, open-ended and pairwise comparison questions. | Everyday | Sep 2024 | 52.2% | Not in cards |
| RoboTwin 1.0 Dual-arm policies train on generated digital-twin demos and are tested on simulated single-arm and dual-arm tasks, with matching real-world data. Success rate. | Everyday | Sep 2024 | Not in cards | |
| Vistra Systems translate text in natural images across multiple language directions. | University | Sep 2024 | | Fading |
| GeoSound Models match environmental recordings to satellite images of their locations and retrieve recordings from images. The test holds out geographic areas and contains 9,931 examples. | Everyday | Aug 2024 | Not in cards | |
| GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. | Expert | Aug 2024 | | Not in cards |
| HR-Bench Models answer multiple-choice questions about fine details in 4K and 8K resolution images. Results are reported separately for the 4K and 8K sets. | Everyday | Aug 2024 | | Fading |
| Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. | Everyday | Aug 2024 | 8.16 | System cards only |
| MME-RealWorld Models answer human-annotated multiple choice questions about high-resolution real-world images in scenarios such as remote sensing, autonomous driving, monitoring and OCR. | Everyday | Aug 2024 | | Dropped |
| MMIU Vision-language models answer 11,000 multiple-choice questions over sets of images. 52 tasks cover 7 kinds of multi-image relationship. | Everyday | Aug 2024 | | System cards only |
| Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. | School | Aug 2024 | System cards only | |
| Baba Is AI Models see a puzzle and write a plan to reach the winning object. Some puzzles require them to push words together to change the rules. | Everyday | Jul 2024 | | Not in cards |
| ChipGPTV Models write Verilog from circuit diagrams and descriptions with different levels of detail. Testbenches check whether the generated circuits behave correctly. | University | Jul 2024 | 71.8% | Not in cards |
| CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. | Expert | Jul 2024 | Not in cards | |
| DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. | Everyday | Jul 2024 | 34.4% | Dropped |
| GTA Agents answer 229 human-written queries that need 1 to 4 real deployed tools for perception, operation, logic and creativity, often with image inputs. Queries leave the tools and steps implicit. | Everyday | Jul 2024 | 46.6% | Not in cards |
| LAB-Bench Multiple-choice tasks on practical biology research skills: literature, protocols, sequence and figure reasoning, database lookup. | Expert | Jul 2024 | 80.7% | System cards only |
| LAB-Bench FigQA Models interpret figures from biology papers and answer multiple-choice questions. FigQA subset of LAB-Bench. | Expert | Jul 2024 | 80.4% | Fading |
| LogicVista Models answer 448 multiple-choice questions on spatial, deductive, inductive, numeric, and mechanical reasoning set in images. Scored by accuracy. | School | Jul 2024 | | In use |
| LongVideoBench Models answer referring reasoning questions over 3,763 web videos up to an hour long, with interleaved subtitles. Scored by multiple-choice accuracy. | Everyday | Jul 2024 | | Fading |
| MIA-Bench Models answer 400 image prompts that stack several strict instructions on format, length and content. An LLM judge scores how well each answer follows every instruction. | Everyday | Jul 2024 | 91% | Dropped |
| MJ-Bench Models compare image pairs for text alignment, safety, visual quality, and bias. Scores measure preference accuracy, with separate dispersion and equality measures for bias. | Everyday | Jul 2024 | 44.5% | Not in cards |
| OpenCanopy Models estimate tree canopy height from satellite images. The evaluation compares predicted height maps with measurements from airborne laser scans. | University | Jul 2024 | Not in cards | |
| QEVD-FIT-COACH Models watch a workout and decide when to give feedback. The benchmark scores feedback timing and the content of the coaching advice. | Everyday | Jul 2024 | Not in cards | |
| ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. | Everyday | Jul 2024 | Not in cards | |
| SA-37 interactive segmentation Models refine an object mask after a user clicks on an image. The benchmark averages overlap across datasets at different numbers of clicks. | Everyday | Jul 2024 | Not in cards | |
| Segment Anything Video Models predict masks for prompted objects throughout a video. The evaluation compares predicted boundaries and regions with annotated object tracks. | Everyday | Jul 2024 | Not in cards | |
| TV-AD Models describe what viewers can see during clips from television episodes. The test pairs each clip with a timed reference description. | Everyday | Jul 2024 | Not in cards | |
| VLMs Are Blind Models answer seven simple visual tasks such as counting line crossings or overlapping circles, which young children solve easily. Scored by accuracy. | Everyday | Jul 2024 | | Fading |
| We-Math Models solve visual math problems decomposed into sub-problems by knowledge concept. Scoring compares sub-problem and full-problem results to diagnose knowledge and generalization gaps. | School | Jul 2024 | | Fading |
| 1X World Model Challenge Predict future egocentric frames of 1X's EVE humanoid from video and state. Compression scored by token cross-entropy, Sampling by PSNR on private tests. | Jun 2024 | Not in cards | ||
| A-Bench Models answer multiple-choice questions about AI-generated images, covering both meaning and low-level visual quality, to test them as judges of image generators. | Everyday | Jun 2024 | | Not in cards |
| BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. | School | Jun 2024 | | Dropped |
| ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original. | University | Jun 2024 | 60.3 | System cards only |
| CharXiv Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. | University | Jun 2024 | 93.5% | In use |
| CharXiv (DQ) Models answer descriptive questions about charts from arXiv papers, such as titles, axis labels and legend entries. It is the descriptive half of CharXiv. | University | Jun 2024 | 95% | System cards only |
| ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. | University | Jun 2024 | | Dropped |
| CVBench Models answer 2,638 manually checked 2D and 3D vision questions built from ADE20K, COCO, and Omni3D. Scoring is accuracy. | Everyday | Jun 2024 | | Fading |
| CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. | Everyday | Jun 2024 | | System cards only |
| EmbSpatialBench Models answer multiple-choice questions on six spatial relations between objects in egocentric embodied scenes. Scored by accuracy. | Everyday | Jun 2024 | | Fading |
| GenAI-Arena Users compare outputs from two anonymous models for image generation, image editing and video generation, and vote. Votes give Elo ratings. | Everyday | Jun 2024 | 1,133 | Not in cards |
| GenAI-Arena Image Editing Models edit images from user instructions. Anonymous human comparisons determine Elo in this independent vote pool. | Everyday | Jun 2024 | Not in cards | |
| Humor in AI: Cartoon Captioning Models write captions for New Yorker cartoons. Human judges and language models compare the captions with contest entries from different ranking groups. | Everyday | Jun 2024 | 73.9% | Not in cards |
| INQUIRE-Fullrank Models retrieve wildlife photographs from five million iNaturalist images for ecological queries. The benchmark measures average precision among the first 50 results. | University | Jun 2024 | Not in cards | |
| INQUIRE-Rerank Models reorder 100 photographs supplied for each ecological query. The benchmark measures average precision among the first 50 results. | University | Jun 2024 | 62.4% | Not in cards |
| K-Viscuit Models answer multiple-choice visual questions about Korean culture. Questions were written through human and VLM collaboration. | Everyday | Jun 2024 | 89.5% | Fading |
| LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. | Everyday | Jun 2024 | | In use |
| M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. | School | Jun 2024 | | System cards only |
| MIRB Models compare, analyze and reason across several images in four categories: perception, visual world knowledge, reasoning and multi-hop reasoning. | Everyday | Jun 2024 | 37.7% | System cards only |
| MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. | Everyday | Jun 2024 | | In use |
| MM-NIAH Vision-language models answer questions about long documents of interleaved images and text. Tasks are retrieval, counting and reasoning over needles hidden in the document. | Everyday | Jun 2024 | | Not in cards |
| MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. | Everyday | Jun 2024 | 2.15 | Dropped |
| MMDU Vision-language models hold long conversations about many images, up to 20 images and 27 turns per dialog. A GPT-4o judge scores the answers. | Everyday | Jun 2024 | 62.6 | Not in cards |
| MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. | School | Jun 2024 | 83% | In use |
| MuirBench Models answer multiple choice questions over sets of several images, including unanswerable variants. Scored by accuracy. | Everyday | Jun 2024 | | In use |
| Multi-Scenario Anomaly Detection Models score video frames for unusual events across 14 settings. The two protocols hold out different videos and score frame-level detection. | Everyday | Jun 2024 | Not in cards | |
| NOTA Models extract facts from sheet music, convert score images to ABC notation, and analyze musical structure. The authors report extraction rates and separate transcription distances. | University | Jun 2024 | | Not in cards |
| OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. | Competition | Jun 2024 | 40.5% | Dropped |
| Ref-L4 Models locate the object described by a long referring expression in an image. The set has 45,341 annotations over 365 object categories, scored by box accuracy. | Everyday | Jun 2024 | 89.5% | Fading |
| ReMI Models reason over several images per question in math, physics, logic, code, charts, and spatial or temporal tasks. Scored by accuracy. | School | Jun 2024 | | Fading |
| RoboCasa Simulated mobile manipulators and humanoids do 100 kitchen tasks, 25 atomic and 75 composite, across 120 scenes. Scored by success rate. | Everyday | Jun 2024 | | Fading |
| SF20K-Test Models answer questions about the story in a short film. The authors check multiple-choice answers against a key and grade free responses with GPT-3.5. | Everyday | Jun 2024 | 62.8% | Not in cards |
| SpatialEval Maze-Nav Models answer questions about routes and turns in mazes given as text or images. 1,500 multiple-choice questions scored by accuracy. | Everyday | Jun 2024 | 41% | System cards only |
| SpatialEval Spatial-Map Models answer questions about relations and counts among objects placed on a map given as text or images. 1,500 questions scored by accuracy. | Everyday | Jun 2024 | 64.3% | System cards only |
| VCG+ Diverse Models answer questions about videos drawn from varied settings. The evaluation grades their ability to describe and explain video content. | Everyday | Jun 2024 | 43.5 | Not in cards |
| VCR (Visual Caption Restoration) Models restore caption text that is partly covered in an image, using only the visible pixel fragments. Scored by exact match and Jaccard similarity of the words. | Everyday | Jun 2024 | 75.6% | System cards only |
| VELOCITI Models distinguish a video's correct description from a caption that assigns an action to the wrong person. | School | Jun 2024 | 43.1% | Not in cards |
| VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. | Everyday | Jun 2024 | 59.6% | Fading |
| VideoScore-Bench Models rate generated videos or choose which video better matches a human preference. The four test sets compare model judgments with human judgments. | Everyday | Jun 2024 | | Not in cards |
| Vision Arena People send an image with a prompt, compare answers from two anonymous vision-language models, and vote for the better one. Votes become Bradley-Terry ratings. | Everyday | Jun 2024 | 1,309 | Not in cards |
| VRSBench Models caption aerial images, locate objects described in words, and answer visual questions. The held-out files contain 9,350 captions, 16,159 object descriptions, and 37,409 questions. | Everyday | Jun 2024 | 65.6% | Not in cards |
| WeatherQA (severe weather) Models read weather maps to identify a threatened area and classify its severe weather potential. The second score includes only cases with the correct area. | Expert | Jun 2024 | | Not in cards |
| Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. | Everyday | Jun 2024 | | System cards only |
| WildVision Vision-language models answer 500 real user image prompts from the WildVision Arena. A GPT-4 judge compares each answer with a fixed baseline model. | Everyday | Jun 2024 | 74.1 | Dropped |
| AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. | Expert | May 2024 | 37.8% | Not in cards |
| CompassArena Users chat with two anonymous models side by side, mostly in Chinese, and vote for the better answer. A Bradley-Terry model turns the votes into Elo-style ratings. | Everyday | May 2024 | 1,062 | Not in cards |
| Mantis-Eval Models answer questions that need reasoning across several interleaved images. Scored by accuracy. | Everyday | May 2024 | | Dropped |
| MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. | Everyday | May 2024 | | Fading |
| PitVQA Models answer questions about a frame from pituitary surgery. Questions cover the surgical step, the instruments in view, and what the surgeon is doing. | Expert | May 2024 | Not in cards | |
| SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. | Everyday | May 2024 | 80% | In use |
| Vibe-Eval Models answer 269 image-based prompts, 100 of them hard, that have expert gold responses. A Reka Core judge rates each answer. | Everyday | May 2024 | | Dropped |
| Video-MME Models answer 2,700 multiple-choice questions about 900 videos from 11 seconds to 1 hour long, with and without subtitles. Scored by accuracy. | Everyday | May 2024 | | In use |
| Assembly101-O Models watch people assemble toy vehicles and flag the first procedural mistake. Training uses correct procedures, and evaluation stops at the first mistake in an incorrect sequence. | Everyday | Apr 2024 | Not in cards | |
| BLINK Models answer 3,807 multiple choice questions built from 14 classic computer vision tasks that use one or more images. Scoring is accuracy. | Everyday | Apr 2024 | | In use |
| CMD-AD-Eval Models describe what happens on screen during gaps in movie dialogue. The authors compare the descriptions with timed human narration using text overlap and model judgments. | Everyday | Apr 2024 | Not in cards | |
| DesignQA Models read Formula SAE rules and inspect CAD views and drawings of a student race car. They retrieve rules, identify parts, and check compliance. | University | Apr 2024 | 0.72 | Not in cards |
| Epic-tent-O Models watch someone pitch a tent and flag procedural mistakes using only the video so far. The test includes 15 videos. | Everyday | Apr 2024 | Not in cards | |
| Gecko People compare text to image outputs for prompt alignment. The benchmark studies model ordering, pairwise instance scoring, and pointwise instance scoring. | Everyday | Apr 2024 | Not in cards | |
| Japanese National Dental Examination 2024 Models select answers to 353 Japanese dental licensing questions, including 149 that use visual material. The exam covers 17 dental specialties. | University | Apr 2024 | 81.9% | Not in cards |
| Long-term Video Object Segmentation v2 Models track object masks through long videos. The evaluation tests whether they can recover an object's identity after an extended absence. | Everyday | Apr 2024 | Not in cards | |
| MMBench-EN v1.1 Models answer English multiple-choice image questions across 20 ability dimensions, using CircularEval with shuffled options. Scored by accuracy on the dev split. | Everyday | Apr 2024 | | In use |
| MMT-Bench Vision-language models answer 31,325 multiple-choice questions across many multimodal tasks such as recognition, localization, reasoning and planning. | Expert | Apr 2024 | 62.4% | In use |
| OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. | Everyday | Apr 2024 | | Dropped |
| RealWorldQA Models answer a question about each of 700+ real-world photos, many taken from vehicles, that test basic spatial understanding. Answers are easy to verify. | Everyday | Apr 2024 | | In use |
| SEED-Bench-2-Plus Multimodal models answer multiple-choice questions about text-rich images such as charts, maps and web pages. | Everyday | Apr 2024 | | System cards only |
| ArAIEval 2024 Models mark persuasive passages in Arabic text and name the technique. They also classify Arabic memes using the image, its extracted text, or both. | University | Mar 2024 | Not in cards | |
| BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. | Everyday | Mar 2024 | Not in cards | |
| CLIP representation bias evaluation Models associate images of everyday objects with gender terms. The evaluation measures how far the associations depart from an equal preference across groups. | Everyday | Mar 2024 | Not in cards | |
| Design2Code Models turn screenshots of 484 real webpages into HTML and CSS. Automatic metrics compare the rendered page with the original on layout, text, position, color and CLIP similarity. | Everyday | Mar 2024 | 87.2 | Fading |
| EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. | School | Mar 2024 | | Not in cards |
| HM3DSem-Walks Models use indoor color and depth scans to find objects described by room and floor. The original retrieval test covers eight scenes. | Everyday | Mar 2024 | Not in cards | |
| HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. | Everyday | Mar 2024 | Not in cards | |
| JA-VLM-Bench-In-the-Wild Models answer Japanese questions about images of places, objects and culture in Japan. | Everyday | Mar 2024 | 41.1 | Not in cards |
| LHRS-Bench Models answer 690 single-choice questions about 108 remote sensing images. The questions cover land use, object locations, image resolution, and other properties. | University | Mar 2024 | 39.1% | Not in cards |
| MathVerse Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram. | School | Mar 2024 | | Fading |
| MMStar Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. | School | Mar 2024 | | In use |
| Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. | Everyday | Mar 2024 | 99.2% | System cards only |
| TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. | Everyday | Mar 2024 | | Fading |
| TGVE+ Models follow video editing instructions on the original 76 TGVE videos. The extension adds object removal, object addition, and texture changes. | University | Mar 2024 | Not in cards | |
| 1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. | Everyday | Feb 2024 | | System cards only |
| ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. | Everyday | Feb 2024 | 20.9 | Not in cards |
| ContPhy Models infer physical properties such as mass and density of fluids and soft bodies from videos and predict their dynamics. | Everyday | Feb 2024 | | Fading |
| CRPE Models answer single-choice questions about objects and their relations in images, covering subject, predicate and object. Each question is asked again with options rotated (CircularEval). | Everyday | Feb 2024 | | Dropped |
| LoCoMo Models answer questions, summarize events and continue dialogue over very long multi-session chats of about 300 turns and 9K tokens, up to 35 sessions. Answers are scored against references. | Everyday | Feb 2024 | Not in cards | |
| MAD-Bench Multimodal models answer image questions containing deceptive prompt information about objects, counts and spatial relationships. | University | Feb 2024 | Not in cards | |
| MATH-Vision (MathVision) 3,040 competition math problems with visual context. Answer accuracy. | Competition | Feb 2024 | | In use |
| MedVidQA 2024 Query-Focused Instructional Step Captioning Models divide a medical demonstration into steps that answer a question. They give each step a caption and start and end times. | Everyday | Feb 2024 | Not in cards | |
| MedVidQA 2024 Video Corpus Visual Answer Localization Models search 48,605 medical videos for demonstrations that answer 52 new questions. They return ranked videos and the start and end times of each answer. | Everyday | Feb 2024 | Not in cards | |
| OlympiadBench Models solve bilingual olympiad-level math and physics problems in Chinese and English, many with images. Answers are checked against a key. | Competition | Feb 2024 | 18% | Dropped |
| OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. | Expert | Feb 2024 | | Not in cards |
| PAIRS (Parallel Images) Models answer questions about people in paired images. The evaluation tests whether changing demographic characteristics changes judgments about the subjects. | Everyday | Feb 2024 | 97.2% | Not in cards |
| PCA-Bench Models interpret visual scenes, explain their reasoning, and select actions for embodied tasks. GPT-4-0125 compares perception concepts, reasoning, and actions with annotated references. | Everyday | Feb 2024 | 0.70 | Not in cards |
| RRG24 Models write findings and impressions from one or more chest X-rays per study. Organizers score each section separately using five metrics. | Expert | Feb 2024 | Not in cards | |
| VLGuard Vision language models answer safe and unsafe image and text pairs from the VLGuard test set. Responses are judged for harmfulness and refusal. | Everyday | Feb 2024 | 9.10 | System cards only |
| WildVision Arena Users upload an image with a question, compare answers from two anonymous vision-language models, and vote. Votes give Elo ratings. | Everyday | Feb 2024 | 1,100 | Not in cards |
| CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. | University | Jan 2024 | | Not in cards |
| CMMU Models answer Chinese multimodal questions across seven school subjects. | School | Jan 2024 | 30.9 | Not in cards |
| ICDAR 2024 MapText Competition Models locate and transcribe words on historical maps and group them into place names. The tasks use David Rumsey maps and French land registers. | School | Jan 2024 | Not in cards | |
| Mementos Models describe what happens across a sequence of images. The evaluation checks whether their descriptions capture the objects and actions. | Everyday | Jan 2024 | 40.6% | Not in cards |
| MMVP Models answer 300 questions about 150 image pairs that CLIP embeds as near duplicates. A pair counts only when both answers are right. | Everyday | Jan 2024 | 87% | Fading |
| MultipanelVQA Models answer questions about images arranged into panels. The benchmark tests whether a model can locate the relevant panels and compare their contents. | Everyday | Jan 2024 | | Not in cards |
| OpenVLM Leaderboard OpenCompass runs vision-language models through VLMEvalKit on a fixed set of image benchmarks, such as MMBench, MMStar and MathVista. The headline is the average score across that set. | University | Jan 2024 | | Not in cards |
| RTVLM Vision language models answer red-teaming image prompts on faithfulness, privacy, safety and fairness. GPT-4V scores responses from 0 to 10. | Everyday | Jan 2024 | 7.92 | System cards only |
| SciMMIR Models match figures and tables from scientific papers with their descriptions. | Expert | Jan 2024 | | Not in cards |
| BdSLW60 Models identify an isolated Bangla Sign Language word in a video clip. Accuracy measures how often the predicted word matches its label. | Everyday | Dec 2023 | Not in cards | |
| Ego-Exo4D Models estimate human pose during skilled activities filmed from first-person and external viewpoints. Meta reports joint and mesh errors for physical and procedural activities. | School | Dec 2023 | Not in cards | |
| Ego4D Goal-Step Models find when a goal or step occurs in first-person video. They also identify the current step or locate a step described in words. | Everyday | Dec 2023 | Not in cards | |
| EgoPER Error Detection Models watch first-person cooking videos and flag incorrect steps. The evaluator checks accuracy by step and measures error detection by frame. | Everyday | Dec 2023 | Not in cards | |
| LingoQA Models answer free-form questions about short driving videos. A trained classifier, Lingo-Judge, scores whether each answer is truthful. | Everyday | Dec 2023 | | In use |
| SMILE Video Laugh Reasoning Models explain why people laugh during sitcoms and TED talks. The authors compare generated explanations with human references. | Everyday | Dec 2023 | Not in cards | |
| SVG-Bench (StarVector) Models generate SVG code from images or text. The RLRF port uses the SVG-Stack image-to-SVG task and rewards the quality of the rendering. | University | Dec 2023 | Not in cards | |
| V* Models answer 191 multiple choice questions about small details in high-resolution images, such as an object's color or relative position. | Everyday | Dec 2023 | | Fading |
| AMBER Multimodal models are checked for object, attribute and relation hallucinations in generative and discriminative tasks, scored without an LLM judge. | Everyday | Nov 2023 | | Not in cards |
| Bingo (Visual Hallucination) Models answer questions about images. The benchmark tests whether prior expectations or interference from other visual information cause false answers. | Everyday | Nov 2023 | Not in cards | |
| GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. | Everyday | Nov 2023 | 66.1% | Fading |
| MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. | Everyday | Nov 2023 | Not in cards | |
| MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. | University | Nov 2023 | | In use |
| MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. | Everyday | Nov 2023 | | Fading |
| SEED-Bench-2 Multimodal models answer multiple-choice questions that cover comprehension of interleaved image and text inputs and image generation, in hierarchical capability levels. | Everyday | Nov 2023 | | Not in cards |
| Unicorn (Vision Safety) Models answer questions about objects in unusual images and sketches. The benchmark tests how well visual judgments hold up when the images change. | Everyday | Nov 2023 | 91.2% | Not in cards |
| VideoCon Models distinguish descriptions supported by a video from captions that contradict it. | School | Nov 2023 | Not in cards | |
| CCBench (MMBench Chinese Culture) Vision language models answer 510 Chinese-language multiple-choice questions about images of Chinese culture, such as calligraphy, painting, food and historical figures. | Everyday | Oct 2023 | | Dropped |
| HallusionBench Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. | Everyday | Oct 2023 | | In use |
| MathVista Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. | School | Oct 2023 | | In use |
| Ref-ZOM Models mark the pixels described by a phrase in a COCO image. A phrase can refer to zero, one or several objects. | Everyday | Oct 2023 | Not in cards | |
| Fin-Fact Models judge whether evidence supports a financial claim, contradicts it, or leaves too little information. The original evaluation also tests models with images. | University | Sep 2023 | 61% | Not in cards |
| FORB Models retrieve other images of the same flat object despite changes in viewpoint or lighting. | School | Sep 2023 | Not in cards | |
| HoloAssist Mistake Detection Models watch a person perform a task and classify each annotated action as correct or mistaken. The challenge reports an F-score. | Everyday | Sep 2023 | Not in cards | |
| Object Detection Leaderboard Models locate and classify objects in images. The board compares average precision. | Everyday | Sep 2023 | Not in cards | |
| Q-Bench Multimodal models are tested on low-level vision: perceiving image distortions, describing them and predicting quality scores. | Everyday | Sep 2023 | 73.9% | Not in cards |
| Q-Bench Image-Pair Perception Models answer multiple-choice questions comparing low-level attributes across image pairs. The score is overall answer accuracy in the A1-Pair table. | Everyday | Sep 2023 | Not in cards | |
| Q-Bench Single-Image Perception Models answer multiple-choice questions about low-level attributes of individual images. The score is overall answer accuracy in the A1-Single table. | Everyday | Sep 2023 | Not in cards | |
| XTD-200 Models retrieve images and captions across languages using translations of the XTD test set. Results measure how often the matching item ranks first. | School | Sep 2023 | Not in cards | |
| EgoSchema Models answer five-way multiple-choice questions about three-minute egocentric video clips from Ego4D. Scored by accuracy on 5,031 questions. | Everyday | Aug 2023 | | Dropped |
| EMDB Models reconstruct human pose and body shape in everyday scenes. The benchmark compares joint positions with electromagnetic measurements. | School | Aug 2023 | Not in cards | |
| FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. | University | Aug 2023 | | Not in cards |
| MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. | Everyday | Aug 2023 | Not in cards | |
| MM-Vet Models answer open questions about images that combine recognition, OCR, knowledge, spatial reasoning, language generation and math. GPT-4 grades each answer against a reference. | Everyday | Aug 2023 | | Dropped |
| ReasonSeg Models infer which object an instruction describes and return its mask. Instructions can require reasoning about the image. | Everyday | Aug 2023 | Not in cards | |
| BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. | School | Jul 2023 | | Not in cards |
| COCO-O Models detect objects under natural changes in image appearance. The benchmark compares detection on drawings, cartoons, and other visual domains. | Everyday | Jul 2023 | Not in cards | |
| ExFunTube Models explain why a moment in a short video is funny. The authors compare explanations with human references and ask viewers to rate them. | Everyday | Jul 2023 | Not in cards | |
| ImageNet-UA Models classify images under distortions that differ from those used during training. The updated suite measures accuracy across 18 new distortions. | Everyday | Jul 2023 | Not in cards | |
| Malaysian Sign Language Medical Dataset Models identify Malaysian Sign Language terms used in health care. The training and test lists share all seven signer names. | Everyday | Jul 2023 | Not in cards | |
| MMBench Vision language models answer multiple-choice questions covering 20 perception and reasoning skills. CircularEval rotates the options. Scored by accuracy. | Everyday | Jul 2023 | | Fading |
| OPT-RSVG Models use a written description to locate an object in an aerial image. The test pairs 24,477 descriptions with reference boxes. | Everyday | Jul 2023 | Not in cards | |
| PUMaVOS Models track masks for object parts across video frames. A prompt can identify a small region, such as one cheek of a person's face. | Everyday | Jul 2023 | Not in cards | |
| RSIEval Models describe scenes in aerial photographs and answer questions about objects, positions, and road directions. Human graders score captions and answers separately. | Everyday | Jul 2023 | Not in cards | |
| SEED-Bench Models answer 19,000 human-annotated multiple-choice questions over 12 dimensions of image and video understanding. Scored by accuracy against the answer key. | Everyday | Jul 2023 | | Fading |
| Aria Digital Twin Models infer the layout of objects from first-person images. The SAM 3D Objects evaluation compares predicted positions and orientations with reference scene geometry. | School | Jun 2023 | Not in cards | |
| Babel-ImageNet Models classify ImageNet images using class descriptions in different languages. The benchmark measures how recognition accuracy changes with the prompt language. | Everyday | Jun 2023 | Not in cards | |
| Encyclopedic VQA Models answer questions about detailed properties of pictured objects and places. The Gemini Embedding 2 report measures whether they retrieve the correct answer. | Everyday | Jun 2023 | Not in cards | |
| LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. | Everyday | Jun 2023 | | Fading |
| M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. | School | Jun 2023 | Dropped | |
| MME Models answer yes or no questions about images across 14 perception and cognition subtasks with hand-written instructions. Labs usually report the perception sum, out of 2000. | Everyday | Jun 2023 | | Fading |
| NAVI Models match points on the same object across photographs taken from different viewpoints. Reference geometry supplies the correct correspondences. | School | Jun 2023 | Not in cards | |
| ReMuQ Models retrieve a knowledge passage that answers a question about an image. | School | Jun 2023 | Not in cards | |
| SugarCrepe Models choose between a caption that describes an image and a plausible caption with a changed detail. | School | Jun 2023 | | Not in cards |
| Video-ChatGPT video conversation benchmark Models answer questions about video content. The evaluation grades correctness, detail, context, timing, and consistency. | Everyday | Jun 2023 | Not in cards | |
| EDIS Models retrieve news images and their associated text using queries that refer to specific people, places, or events. | School | May 2023 | Not in cards | |
| LOVEU-TGVE-2023 Models edit 76 videos with four instructions per video. People compare each submission with Tune-A-Video on text alignment, structure, and visual quality. | University | May 2023 | Not in cards | |
| OCRBench Models answer 1,000 checked questions on text recognition, scene text, documents, key information extraction, and handwritten math. The score counts correct answers. | Everyday | May 2023 | 94% | In use |
| Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. | Everyday | May 2023 | | Dropped |
| PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. | University | May 2023 | | Fading |
| POPE Vision language models answer yes or no questions about whether objects appear in COCO images. Scored by accuracy or F1 on object hallucination. | Everyday | May 2023 | | Fading |
| DataComp image encoder evaluation Models trained on different image and text collections perform the same set of downstream tasks. The evaluation compares transfer performance across 37 tasks. | Everyday | Apr 2023 | Not in cards | |
| Large-Vocabulary Video Instance Segmentation Models detect, segment, and track object instances across video frames. The benchmark includes categories outside the training vocabulary. | Everyday | Apr 2023 | Not in cards | |
| LLaVA-Bench (In-the-Wild) Multimodal assistants answer open questions about a small set of diverse real images. A GPT-4 judge scores each answer relative to a reference answer. | Everyday | Apr 2023 | Not in cards | |
| OmniLabel Models locate objects described by category names and free-form phrases. The benchmark compares predicted boxes with annotated objects. | Everyday | Apr 2023 | Not in cards | |
| SoccerNet-Caption Models write timestamped commentary for soccer broadcasts. The set has almost 37k commentaries over 715.9 hours of video. Streaming reports also score when the model chooses to speak. | Everyday | Apr 2023 | Not in cards | |
| VALOR-32K Models match captions to video and audio clips. Meta reports separate retrieval results for text paired with audio and text paired with video. | Everyday | Apr 2023 | Not in cards | |
| Hi4D Models reconstruct the bodies of people interacting at close range. The SAM 3D Body evaluation compares joint and mesh errors with motion capture references. | School | Mar 2023 | Not in cards | |
| MAD-Eval Models write descriptions for scenes from ten movies. One set of references names characters. Another replaces their names with the word “SOMEONE.” | Everyday | Mar 2023 | Not in cards | |
| MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. | School | Mar 2023 | Not in cards | |
| CountBench Models count the objects in natural images from the CountBench image-text set. Lab cards often use a question-answer version, CountBenchQA, scored by accuracy. | Everyday | Feb 2023 | | In use |
| Event-based Segmentation Dataset Models segment objects in cluttered indoor scenes filmed from a moving robot arm. The benchmark provides reference masks for comparison. | Everyday | Feb 2023 | Not in cards | |
| MedVidQA 2023 Medical Instructional Question Generation Models write a question that a supplied medical video segment answers. They can use both the images and the subtitles. | Everyday | Feb 2023 | Not in cards | |
| MedVidQA 2023 Video Corpus Visual Answer Localization Models search 12,657 medical videos and return the times when a video shows the answer. Half of the 40 questions require visual evidence. | Everyday | Feb 2023 | Not in cards | |
| MOSE Models follow and segment objects in videos with crowds, occlusions, and objects leaving the screen. The benchmark compares masks against human annotations. | Everyday | Feb 2023 | Not in cards | |
| OVEN Models identify the Wikipedia entity that answers a question about an image. | University | Feb 2023 | Not in cards | |
| GeoDE Models recognize objects in images collected across different world regions. The evaluation compares accuracy across regions and object classes. | Everyday | Jan 2023 | Not in cards | |
| MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. | Expert | Jan 2023 | Not in cards | |
| RadSum23 Models write the impression section of radiology reports from findings. One track includes chest images. Organizers rank submissions by RadGraph F1. | Expert | Jan 2023 | 38.0 | Not in cards |
| Video Object Segmentation under Transformations Models follow objects whose appearance changes as people manipulate them. The benchmark compares predicted masks with annotations through these transformations. | Everyday | Dec 2022 | Not in cards | |
| ArtCap Models associate paintings with descriptions of their contents. Google compares image encoders by how often the matching caption appears in their top five. | Everyday | Nov 2022 | Not in cards | |
| CLIPSep Sound Separation Models isolate a requested sound from a mixture using a text or image query. The two tests mix music or everyday sounds with interfering audio. | University | Nov 2022 | Not in cards | |
| KiloGram Models choose which of ten tangram pictures matches a description. The descriptions can name the whole shape or explain what its pieces represent. | Everyday | Nov 2022 | Not in cards | |
| NEVIS’22 Models learn 106 image classification tasks in chronological order. The evaluation compares error rates against the computation each model uses. | Everyday | Nov 2022 | Not in cards | |
| TAP-Vid Models track individual points across frames and predict when each point disappears. The benchmark compares their trajectories and visibility predictions with reference tracks. | Everyday | Nov 2022 | Not in cards | |
| DIOR-RSVG Models draw a box around the object named in a description of an aerial image. The released test split contains 7,500 descriptions. | Everyday | Oct 2022 | Not in cards | |
| Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. | Everyday | Oct 2022 | Not in cards | |
| Video to Valence Models predict which of two videos viewers find more pleasant. Human preference annotations supply the reference judgments. | Everyday | Oct 2022 | Not in cards | |
| VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. | Everyday | Oct 2022 | Not in cards | |
| BURST Models recognize, segment, and track objects across video frames. The benchmark tests how well they associate the same object across time. | Everyday | Sep 2022 | Not in cards | |
| MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. | Everyday | Sep 2022 | | System cards only |
| New Yorker Caption Contest Humor Understanding Models match a cartoon to its caption, choose the funnier of two captions, and explain a joke. They receive the cartoon or its description. | Everyday | Sep 2022 | | Not in cards |
| Robust Multi-view Depth Models estimate depth from multiple photographs of a scene. The benchmark compares relative depth errors across datasets from different environments. | School | Sep 2022 | Not in cards | |
| ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. | School | Sep 2022 | | Dropped |
| EPIC-KITCHENS VISOR Hand Object Segmentation Models outline each hand and the object it touches in a kitchen image. They also identify which hand appears and whether it touches anything. | Everyday | Aug 2022 | Not in cards | |
| EPIC-KITCHENS VISOR Semi-Supervised Video Object Segmentation Models receive masks for hands and objects in the first frame of a kitchen video. They track those objects through later frames, including after occlusion. | Everyday | Aug 2022 | Not in cards | |
| Google Universal Image Embedding Challenge Models retrieve images of the same item across several visual domains using a single image representation. The challenge compares results against labeled matches. | Everyday | Aug 2022 | Not in cards | |
| EgoProceL Models find recurring steps in first-person videos and mark when each step occurs. The videos show 16 tasks, including cooking and assembling computers. | Everyday | Jul 2022 | Not in cards | |
| LineEX Models extract labels and numerical series from scientific line charts. The hub port uses a 20,000-image test split and grades the structured output. | University | Jul 2022 | Not in cards | |
| Omni3D A large benchmark for 3D object detection from a single image, built by unifying indoor and outdoor datasets such as SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI and nuScenes. Vision-language models answer its 3D grounding questions. | Everyday | Jul 2022 | | System cards only |
| A-OKVQA Models answer about 25,000 visual questions that need world knowledge beyond the image, in multiple-choice and direct-answer forms. | Everyday | Jun 2022 | Not in cards | |
| EndoVis18-VQA Models answer questions about organs, tools, tool positions, and interactions in images from robotic kidney surgery. The benchmark accepts class labels or sentence answers in separate tasks. | Expert | Jun 2022 | Not in cards | |
| Meta-Album Models classify unfamiliar images after seeing a few labeled examples. Tasks span ten domains, including microscopy, remote sensing and wildlife. | School | Jun 2022 | Not in cards | |
| MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. | Everyday | Jun 2022 | | Fading |
| RICH Models reconstruct people interacting with their surroundings. The evaluation compares body joints and mesh vertices with the reference geometry. | School | Jun 2022 | Not in cards | |
| SignBD-Word Models identify isolated Bangla signs from video or body pose. The authors score top one and top five accuracy. | Everyday | Jun 2022 | Not in cards | |
| Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. | School | May 2022 | | Not in cards |
| Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. | School | Apr 2022 | Not in cards | |
| Google Scanned Objects Models synthesize an object from a new viewpoint. Researchers compare the rendered result with reference views of scanned household items. | School | Apr 2022 | Not in cards | |
| LEVIR-CC Models compare aligned aerial images from Texas and describe what changed. Five annotators wrote reference captions for each of the 1,929 test pairs. | Everyday | Apr 2022 | Not in cards | |
| NWPU-Captions Models describe aerial scenes across 45 land use categories. The authors reserve 3,150 images for testing and provide five captions per image. | School | Apr 2022 | Not in cards | |
| VSR Models judge whether captions with 65 kinds of spatial relation, such as under or facing, are true of an image. Over 10,000 text and image pairs. | Everyday | Apr 2022 | Not in cards | |
| VSR Random Split Models judge whether image captions correctly describe spatial relations on the randomly partitioned test set. The score is accuracy over 2,195 test examples. | Everyday | Apr 2022 | Not in cards | |
| VSR Zero-Shot Split Models judge spatial captions on a test set whose object concepts do not appear in training. The score is accuracy over 616 test examples. | Everyday | Apr 2022 | Not in cards | |
| Winoground Models match two images to two captions that use the same words in a different order. | School | Apr 2022 | Not in cards | |
| AmsterTime Models match historic photographs of Amsterdam with contemporary street views. The benchmark measures recognition of the same place across time. | Everyday | Mar 2022 | Not in cards | |
| ChartQA Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. | School | Mar 2022 | | In use |
| HierText Models locate and recognize text arranged in lines and paragraphs in real scenes. Google also uses the annotations to compare text and image retrieval. | School | Mar 2022 | Not in cards | |
| ImageCoDe Models select the image that matches a description from a set of visually similar alternatives. | Everyday | Mar 2022 | Not in cards | |
| MET-Meme Models classify Chinese and English memes by metaphor, sentiment, intent, and degree of offensiveness. | Everyday | Feb 2022 | Not in cards | |
| MuMiN Models use social media context to classify claims as factual or misleading. A second task asks which verdict applies to the claim a post discusses. | University | Feb 2022 | Not in cards | |
| CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. | Everyday | Dec 2021 | Not in cards | |
| CDVQA Models compare two aerial images and answer questions about which land cover changed and by how much. The two tests use the same 968 image pairs with different answer distributions. They contain 39,686 and 31,036 questions, respectively. | School | Dec 2021 | Not in cards | |
| ODinW-13 Models detect objects in 13 varied real-world datasets, such as aquarium and aerial images, from text category names. Scored by mean average precision. | Everyday | Dec 2021 | | Fading |
| SLIP zero-shot transfer suite Models assign labels to images using text descriptions of the classes. The evaluation averages zero-shot accuracy across 26 recognition tasks. | Everyday | Dec 2021 | Not in cards | |
| ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. | Everyday | Nov 2021 | | Dropped |
| Clinical Laboratory Test Reports Models extract laboratory results from report images into structured JSON. Google compares predictions with reference fields using macro and micro F1. | School | Nov 2021 | Not in cards | |
| Natural Adversarial Objects Models identify objects in unmodified photographs that caused errors in earlier detectors. Human annotations identify the objects. | Everyday | Nov 2021 | Not in cards | |
| Ego4D State Change Object Detection Models locate the object that changes in a first-person video. The official test scores its bounding box in the frame where the change begins. | Everyday | Oct 2021 | Not in cards | |
| LoveDA Models label land cover in aerial images of urban and rural areas. The benchmark supports comparisons across these two settings. | University | Oct 2021 | Not in cards | |
| SciCap Models describe figures from scientific publications. Researchers compare generated captions with the text that accompanies each figure. | School | Oct 2021 | Not in cards | |
| TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. | Everyday | Oct 2021 | Not in cards | |
| GD-VCR Models answer questions about what people are doing and why. The test pairs 328 images, mainly from film and television, with 886 questions across four regions. | Everyday | Sep 2021 | Not in cards | |
| CIRR Models find an image that matches a reference photograph after applying a change described in text. | Everyday | Aug 2021 | | Not in cards |
| TREK-150 Models follow a marked object through first-person kitchen videos. In the one-pass test, they receive its bounding box in the first frame and track it to the end. | Everyday | Aug 2021 | Not in cards | |
| Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. | Expert | Jul 2021 | Not in cards | |
| QVHighlights Natural-language queries over YouTube videos. Model localizes relevant moments, scored by Recall@1 at IoU 0.5. | Everyday | Jul 2021 | | Fading |
| Inter-GPS (Geometry3K) Models solve high school geometry problems that combine a diagram and text from the Geometry3K test set. Scored by accuracy. | School | May 2021 | | Dropped |
| LAD2000 Models identify unusual events in videos and score when they happen. The test contains 280 videos with unusual events and 280 without them. | Everyday | May 2021 | Not in cards | |
| NExT-QA Models answer causal, temporal and descriptive questions about short videos of daily life. Labs usually report the multiple choice split. | Everyday | May 2021 | | Dropped |
| InfographicVQA Models answer about 30,000 questions on about 5,000 infographics that mix text, layout, charts, and graphics. Scoring is ANLS. | School | Apr 2021 | 92.6% | In use |
| Unidentified Video Objects Models segment and track objects in video without a fixed list of object categories. The evaluation compares predicted masks and tracks with annotated instances. | Everyday | Apr 2021 | Not in cards | |
| iNaturalist 2021 Models classify nature photographs by species. The 2021 competition uses a new collection of images and species labels. | University | Mar 2021 | Not in cards | |
| Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. | University | Mar 2021 | Not in cards | |
| PPR10K Models retouch groups of portraits against three expert retouchers’ Adobe Camera Raw edits. The test measures accuracy on people and color consistency across each group. | University | Mar 2021 | Not in cards | |
| Country211 Models predict the country where a photograph was taken. | University | Feb 2021 | Not in cards | |
| Occluded Video Instance Segmentation Models track and segment objects that overlap or disappear behind other objects. The evaluation compares predicted masks and identities with annotated video instances. | Everyday | Feb 2021 | Not in cards | |
| SLAKE Models answer English and Chinese questions about radiology images of the head, neck, chest, abdomen, and pelvis. Scoring is accuracy. | University | Feb 2021 | | Fading |
| Long Range Arena Transformers solve six long-sequence tasks covering list operations, text, retrieval, image, path and path crossing. | Expert | Jan 2021 | Not in cards | |
| WebSRC Models answer questions about a web page from its screenshot, HTML or both. The set has 400K question and answer pairs from 6.4K web pages. Scored by exact match and F1. | Everyday | Jan 2021 | Not in cards | |
| Cross-lingual Test Dataset Models retrieve images from captions written in different languages. The benchmark compares retrieval against paired photographs and descriptions. | School | Dec 2020 | Not in cards | |
| Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. | Everyday | Dec 2020 | Dropped | |
| Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. | Everyday | Nov 2020 | | Fading |
| INCLUDE Indian Sign Language Recognition Models identify isolated Indian Sign Language words in video. The authors provide separate test files for the full vocabulary and its smaller selection. | Everyday | Oct 2020 | Not in cards | |
| Kinetics-700-2020 Models recognize human actions in short videos from the 2020 edition of Kinetics-700. | Everyday | Oct 2020 | | Not in cards |
| MECCANO Action Recognition Models identify an assembly action from a first-person video clip of someone building a toy motorcycle. The answer key has 61 action classes. | Everyday | Oct 2020 | Not in cards | |
| MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. | University | Oct 2020 | Not in cards | |
| Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. | Everyday | Oct 2020 | Not in cards | |
| Widget Captioning Models describe an individual interface element in a mobile screenshot. The benchmark compares their captions with human descriptions of the same element. | School | Oct 2020 | Not in cards | |
| Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. | Everyday | Aug 2020 | Not in cards | |
| DocVQA Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers. | Everyday | Jul 2020 | | In use |
| EPIC-KITCHENS-100 action anticipation Models predict a person's next action from first-person kitchen video. The benchmark scores predicted verbs, nouns, and their combinations. | Everyday | Jun 2020 | Not in cards | |
| ImageNet ReaL Models classify ImageNet validation images against revised human annotations. An image can have more than one acceptable label. | Everyday | Jun 2020 | Not in cards | |
| ImageNet-R Models classify renditions of objects in drawings, sculptures, and other media. The benchmark measures recognition when appearance differs from ordinary photographs. | Everyday | Jun 2020 | Not in cards | |
| RP2K Models identify retail products from photographs taken in stores. The test distinguishes products with similar packaging. | Everyday | Jun 2020 | Not in cards | |
| Hateful Memes Challenge Models decide whether a meme is hateful by interpreting its image and caption together. Paired examples change one modality while preserving the other. | Everyday | May 2020 | Not in cards | |
| How2QA Models answer multiple-choice questions about instructional videos. The task tests understanding of the actions shown in each clip. | Everyday | May 2020 | Not in cards | |
| PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. | School | May 2020 | Not in cards | |
| Taiwan Mandarin Speech with Video: Speech Enhancement Models use lip video to remove noise from Mandarin sentences. Researchers compare the enhanced signal with clean speech for quality and intelligibility. | University | May 2020 | Not in cards | |
| Google Landmarks Dataset v2 Models recognize landmarks or retrieve photographs of the same landmark. The benchmark uses images from around the world. | Everyday | Apr 2020 | Not in cards | |
| VGGSound Models identify sound events in video clips where the sound source is visible. | Everyday | Apr 2020 | Not in cards | |
| Virtual KITTI 2 Models interpret simulated driving videos with reference geometry and pixel labels. Meta uses the sequences to compare video object segmentation models. | Everyday | Jan 2020 | Not in cards | |
| ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. | Everyday | Dec 2019 | Not in cards | |
| ARC-AGI Few-shot grid puzzles: infer a hidden transformation from about three examples and produce the output grid. Semi-private sets verified by ARC Prize. | Everyday | Nov 2019 | 98.5% | In use |
| ScanNet-1500 Models match features between pairs of indoor views and estimate relative camera poses. The evaluation measures pose accuracy on 1,500 image pairs. | School | Nov 2019 | Not in cards | |
| Jester gesture recognition Models classify hand gestures in short videos. The task distinguishes movements with similar appearance but different directions or timing. | Everyday | Oct 2019 | Not in cards | |
| FreiHAND Models recover a hand's joint positions and surface from a color image. The evaluation compares predictions with reference hand geometry. | School | Sep 2019 | Not in cards | |
| OCR-VQA Models answer questions about book covers by reading visible text. The benchmark compares answers with facts such as a book’s title or author. | School | Sep 2019 | 82% | Not in cards |
| RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. | Everyday | Sep 2019 | Not in cards | |
| DIODE Models estimate depth in indoor and outdoor scenes. The dataset provides depth measurements from a laser scanner for comparison. | School | Aug 2019 | Not in cards | |
| DIOR Models locate and classify objects in remotely sensed images. The benchmark measures detection accuracy across different object sizes and backgrounds. | University | Aug 2019 | Not in cards | |
| FairFace Models classify face images using the dataset’s annotated age, gender, and race categories. The benchmark compares recognition performance across demographic groups. | Everyday | Aug 2019 | Not in cards | |
| LVIS Models locate objects and outline each instance across a large vocabulary of categories. Meta reports box and mask average precision. | Everyday | Aug 2019 | Not in cards | |
| SPair-71k Models match corresponding parts of objects across image pairs. The benchmark includes changes in viewpoint, scale, and occlusion. | School | Aug 2019 | Not in cards | |
| SpatialSense VQA Models judge whether a stated spatial relation between two objects holds in an image. The relations were crowdsourced to defeat simple cues. | Everyday | Aug 2019 | | Dropped |
| ImageNet-A Models classify real photographs that commonly fool image classifiers. The test uses unmodified images from ImageNet categories. | Everyday | Jul 2019 | Not in cards | |
| Kinetics-700 Models classify short video clips into 700 human action categories. Meta compares encoders on this expanded Kinetics edition. | Everyday | Jul 2019 | Not in cards | |
| ActivityNet-QA Models answer open-ended questions about web videos of human activities, from 58,000 QA pairs on 5,800 videos. Scored by accuracy. | Everyday | Jun 2019 | | Dropped |
| Fashion IQ Models retrieve a garment using a reference photograph and text describing how the target garment differs. | Everyday | May 2019 | Not in cards | |
| ImageNet-Sketch Models classify black-and-white sketches using ImageNet category labels. The benchmark tests recognition when images lack photographic texture and color. | Everyday | May 2019 | Not in cards | |
| iSAID Models detect and segment individual objects in aerial images. The benchmark measures how well predicted masks match annotated object instances. | University | May 2019 | Not in cards | |
| OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. | Everyday | May 2019 | | Dropped |
| ST-VQA Models answer questions by reading text in a scene. The benchmark compares each answer with the text required by the question. | School | May 2019 | Not in cards | |
| YouTube-VIS Models detect, segment, and track object instances in YouTube videos. The SAM 3 report evaluates the 2021 edition. | Everyday | May 2019 | Not in cards | |
| MUSIC-21 Instrument Separation Models separate performances from a mixture using video of the musicians. When two musicians play the same instrument, their movements help identify which sound belongs to each. | University | Apr 2019 | Not in cards | |
| TextVQA Models answer 45,336 questions about 28,408 images that require reading text in the image. Scoring is VQA accuracy against human answers. | Everyday | Apr 2019 | | In use |
| VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. | Everyday | Apr 2019 | | Fading |
| VATEX ZH Models write Chinese captions for short videos of human activities from the VATEX dataset. Captions are scored with CIDEr against human references. | Everyday | Apr 2019 | | Dropped |
| COIN instructional video analysis Models recognize steps in videos of people completing everyday tasks. The benchmark supplies annotated actions and their time intervals. | Everyday | Mar 2019 | Not in cards | |
| ImageNet-C Models classify images under common corruptions at different severities. The benchmark measures errors under blur, noise, weather, and digital distortions. | Everyday | Mar 2019 | Not in cards | |
| Meta-Dataset Models learn from episodes sampled across diverse image datasets and classify novel classes from a few examples. | University | Mar 2019 | Not in cards | |
| Nuscene Models localize named objects as 3D boxes in nuScenes driving camera images. Scored by mAP. | Everyday | Mar 2019 | | Fading |
| nuScenes Detection Models detect 3D objects in nuScenes sensor data. The official test leaderboard ranks submissions by nuScenes Detection Score. | Everyday | Mar 2019 | Not in cards | |
| GQA Models answer compositional questions about real-world images, generated from scene graphs. Scored by exact-match accuracy, often on the balanced test-dev split. | Everyday | Feb 2019 | | Fading |
| ImageNetV2 Models classify a newly collected set of photographs using the original ImageNet labels. The test measures how accuracy changes on fresh images. | Everyday | Feb 2019 | Not in cards | |
| CheXpert Models read chest radiographs and label up to 14 observations. The test labels come from board-certified radiologists. | Expert | Jan 2019 | Not in cards | |
| MIMIC-CXR (classification) Models read chest X-rays from Beth Israel Deaconess and say whether five common conditions are present. Scored by average F1 over the conditions. | Expert | Jan 2019 | Not in cards | |
| MIMIC-CXR report generation Models write the findings of a radiology report for a MIMIC-CXR chest X-ray. Scored against the radiologist's report, most often by RadGraph F1. | Expert | Jan 2019 | Not in cards | |
| nocaps Models caption photographs that contain objects outside the caption training vocabulary. The benchmark compares generated captions with human references. | Everyday | Dec 2018 | Not in cards | |
| Dillen herbarium specimen label benchmark Models transcribe specimen labels from images of pressed plants. The collection pairs 1,800 images from nine European herbaria with transcribed records. | School | Nov 2018 | Not in cards | |
| NLVR2 Models judge whether a sentence is true of a pair of photographs. The benchmark measures agreement with annotated truth values. | Everyday | Nov 2018 | Not in cards | |
| VQA-RAD Vision-language models answer clinicians' questions about 315 radiology images, including head CTs, MRIs and chest X-rays. Questions are closed yes or no, or open free text. | Expert | Nov 2018 | Not in cards | |
| BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. | Everyday | Oct 2018 | 37.5% | Not in cards |
| MELD Models identify emotions in utterances from multiparty conversations. The dataset includes speech, video, and text. | Everyday | Oct 2018 | Not in cards | |
| TallyQA Models answer open-ended counting questions about images, including complex ones that need object relationships or attributes. Scored by exact count accuracy. | Everyday | Oct 2018 | | Dropped |
| 3DPW Models recover a person's three-dimensional pose in outdoor video. Reference poses come from wearable sensors and a moving camera. | School | Sep 2018 | Not in cards | |
| Diving48 Models classify competitive dives from videos by distinguishing their movement sequences. | Expert | Sep 2018 | Not in cards | |
| TVQA Models answer 152,545 multiple-choice questions about clips from 6 TV shows, using video frames and subtitles. Scoring is accuracy. | Everyday | Sep 2018 | 87.3% | Dropped |
| YouCook2 retrieval Models retrieve cooking videos from text queries in the YouCook2 collection. | University | Sep 2018 | | Fading |
| YouTube-VOS Models segment and track objects across frames in YouTube videos. The evaluation distinguishes categories seen during training from previously unseen categories. | Everyday | Sep 2018 | Not in cards | |
| Kinetics-600 Models recognize one of 600 human actions in short video clips. | Everyday | Aug 2018 | Not in cards | |
| PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. | Expert | Jun 2018 | | Not in cards |
| CrisisMMD Models classify disaster photographs and their accompanying posts for humanitarian use. | School | May 2018 | Not in cards | |
| RAVDESS Models identify an actor's expressed emotion from recorded speech or song. | Everyday | May 2018 | Not in cards | |
| MSD-I Multimodal Genre Classification Models predict a track’s genre from its audio and album cover. The published test contains 4,649 tracks across 15 genres. | Everyday | Apr 2018 | Not in cards | |
| MUSIC Instrument Separation Models use images of musicians to isolate an instrument from a mixture of performances. The authors measure separation on mixtures of solo recordings, where each instrument has a reference track. | University | Apr 2018 | Not in cards | |
| Places Audio Captions (Hindi) Models retrieve photographs from spoken Hindi descriptions or retrieve descriptions from photographs. The score counts how often the correct match appears among the top results. | Everyday | Apr 2018 | Not in cards | |
| Audio-Visual Event Localization Models identify an event that can be both seen and heard in a video and locate when it occurs. | Everyday | Mar 2018 | Not in cards | |
| Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. | Everyday | Mar 2018 | Not in cards | |
| Revisited Oxford and Paris Models retrieve photographs of the same landmark from Oxford and Paris image collections. The revised benchmark defines medium and hard evaluation protocols. | Everyday | Mar 2018 | Not in cards | |
| iNaturalist 2018 Models identify species in photographs submitted by naturalists. The 2018 competition uses an expanded set of plant and animal categories. | University | Feb 2018 | Not in cards | |
| VizWiz Models answer over 31,000 spoken questions that blind people asked about photos they took with phones. Some questions are unanswerable from the image. | Everyday | Feb 2018 | Not in cards | |
| MSRVTT-QA Models answer short open-ended questions about web video clips from the MSR-VTT captioning dataset. The question and answer pairs were generated automatically from the clip descriptions. Scored by answer accuracy. | Everyday | Oct 2017 | Not in cards | |
| RSICD Models describe aerial scenes in a sentence. The authors provide five reference captions per image and reserve 1,093 images for testing. | Everyday | Sep 2017 | Not in cards | |
| DiDeMo Models locate the time interval in a video that matches a natural language description. | Everyday | Aug 2017 | Not in cards | |
| EuroSAT Models classify land use and land cover in Sentinel-2 satellite images. | University | Aug 2017 | | Not in cards |
| Animals with Attributes 2 Models use shared attributes to recognize animal classes they did not see during training. The benchmark provides 37,322 photographs across 50 classes with 85 attributes. | School | Jun 2017 | Not in cards | |
| ActivityNet Captions Models match descriptions of events to segments of untrimmed videos. Meta evaluates retrieval in both directions. | Everyday | May 2017 | Not in cards | |
| Charades-STA Models read a sentence describing an action and return its start and end time in a Charades indoor video. Scored by temporal IoU with the labeled moment. | Everyday | May 2017 | | Fading |
| ChestX-ray14 Models read frontal chest X-rays from the NIH Clinical Center and say whether thoracic conditions are present. | Expert | May 2017 | Not in cards | |
| Kinetics-400 Models recognize one of 400 human actions in short video clips. | Everyday | May 2017 | | Not in cards |
| DAVIS 2017 Models track and segment individual objects throughout a video. The evaluation measures region overlap and boundary accuracy. | Everyday | Apr 2017 | Not in cards | |
| iNaturalist 2017 Models identify the species in photographs submitted by naturalists. Meta evaluates transfer to fine-grained species classification. | University | Apr 2017 | Not in cards | |
| TGIF-QA Models answer questions about actions, repetitions, and changes over time in animated clips. | Everyday | Apr 2017 | Not in cards | |
| NWPU-RESISC45 Models classify remote sensing images into 45 scene categories. | University | Mar 2017 | | Not in cards |
| YouCook2 Models describe the procedure steps in YouTube cooking videos. Scored by CIDEr against human step captions. | Everyday | Mar 2017 | | Dropped |
| YouCook2 Dense Video Description Models locate cooking steps and describe each step. The score is CIDEr for the complete dense video description task. | Everyday | Mar 2017 | Not in cards | |
| ScanNet Models infer indoor scene geometry from images. Meta evaluates depth predictions against measurements from scanned indoor environments. | School | Feb 2017 | Not in cards | |
| VQAv2 Open-ended questions about COCO images, balanced to reduce language priors. Scored by consensus accuracy against 10 human answers. | Everyday | Dec 2016 | 88.4% | Fading |
| LSMDC Models fill in a missing word in movie audio descriptions using the matching video clip. Scoring is accuracy. | Everyday | Nov 2016 | 45.7% | Dropped |
| ADE20K Models label the pixels in a scene by object or region. Meta measures overlap between predicted and reference segments. | Everyday | Aug 2016 | Not in cards | |
| RefCOCO / RefCOCO+ / RefCOCOg Model grounds a referring expression to a bounding box in a COCO image. Scored by accuracy at IoU 0.5. | Everyday | Jul 2016 | | In use |
| Sydney-Captions Models describe land use in aerial images of Sydney. The dataset pairs 613 images with five human captions each. | Everyday | Jul 2016 | Not in cards | |
| UCM-Captions Models describe aerial land use scenes from the UC Merced collection. The dataset pairs 2,100 images with five human captions each. | Everyday | Jul 2016 | Not in cards | |
| MSR-VTT Models write captions for short web videos. Human-written sentences provide multiple descriptions of each clip. | Everyday | Jun 2016 | Not in cards | |
| MSR-VTT retrieval Models retrieve videos from text queries in the MSR-VTT collection. | University | Jun 2016 | | Fading |
| Cityscapes Models assign a class to each pixel in street scenes filmed from a vehicle. The benchmark compares predicted segments with annotated road scenes. | Everyday | Apr 2016 | Not in cards | |
| AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. | School | Mar 2016 | | Fading |
| Stanford Online Products Models retrieve photographs of the same product. The test compares learned image features on products outside the training set. | Everyday | Nov 2015 | Not in cards | |
| SUNRGBD Models detect objects in about 10,000 indoor RGB-D images and predict their 3D bounding boxes. Scored by average precision. | Everyday | Jun 2015 | | Fading |
| COCO Captions Models write a caption for each image, and outputs are compared with five human captions per image using BLEU, METEOR, ROUGE, and CIDEr. | Everyday | Apr 2015 | | Dropped |
| Food-101 Models identify the dish in a food photograph from 101 categories. | Everyday | Sep 2014 | Not in cards | |
| TACoS Models find the time span in a cooking video that matches a sentence. Labs score temporal grounding by mean IoU with the annotated span. | Everyday | Sep 2014 | | Dropped |
| Birdsnap Models identify the North American bird species shown in a photograph. | University | Jun 2014 | Not in cards | |
| PASCAL-Context Models assign a semantic class to each pixel in a photograph. The benchmark includes objects and background regions. | Everyday | Jun 2014 | Not in cards | |
| COCO Models detect and locate objects in everyday photos from the Microsoft COCO dataset. Scored by mean average precision over box overlap thresholds. | Everyday | May 2014 | | In use |
| Stanford Cars Models identify a car's make, model, and year from a photograph. The benchmark measures fine-grained vehicle recognition. | University | Dec 2013 | Not in cards | |
| Describable Textures Dataset Models assign a texture description to a photograph based on how its surface looks. | Everyday | Nov 2013 | Not in cards | |
| EVVE Models retrieve other videos showing the same real-world event as a query video. | School | Jun 2013 | Not in cards | |
| FGVC-Aircraft Models identify aircraft variants from photographs. | University | Jun 2013 | | Not in cards |
| UCF101 Models recognize one of 101 human actions in video clips. | Everyday | Dec 2012 | Not in cards | |
| PASCAL VOC 2012 Models locate objects and label their pixels in photographs. Meta reports detection and segmentation results on the 2012 challenge. | Everyday | May 2012 | Not in cards | |
| HMDB51 Models identify human actions in short video clips across 51 categories. | Everyday | Nov 2011 | Not in cards | |
| Leeds Sports Pose Extended Models locate body joints in photographs of athletes. Meta compares the fraction of predicted joints within a distance threshold. | School | Jun 2011 | Not in cards | |
| MIT-Adobe FiveK Models adjust photographic tone to match human edits. Five photographers each edited 5,000 raw photos in Lightroom. The original evaluation compares luminance with each reference edit. | University | Jun 2011 | Not in cards | |
| STL-10 Models classify images into ten object categories using limited labeled training data. | Everyday | Jun 2011 | Not in cards | |
| German Traffic Sign Recognition Benchmark Models identify the class of a traffic sign in a cropped road image. | Everyday | Dec 2010 | Not in cards | |
| Caltech-UCSD Birds 200 Models identify bird species from photographs. The benchmark distinguishes species with similar appearance. | University | Sep 2010 | Not in cards | |
| SUN397 Models recognize the scene category in indoor and outdoor photographs. The classification task covers 397 categories. | Everyday | Jun 2010 | Not in cards | |
| CIFAR-10 Models classify small color images into ten object categories. | Everyday | Apr 2009 | Not in cards | |
| CIFAR-100 Models classify small color images into 100 object categories. | School | Apr 2009 | Not in cards | |
| Oxford Flowers 102 Models identify flower categories from photographs. The benchmark tests recognition across changes in shape, scale, and lighting. | University | Dec 2008 | Not in cards | |
| PASCAL VOC 2007 Models locate and classify objects in photographs. Meta evaluates object detection using the 2007 Visual Object Classes challenge images. | Everyday | Apr 2007 | Not in cards | |
| 3D Arena Users compare two 3D models generated from the same input and vote for the better one. A live leaderboard ranks the generators. | Everyday | Not in cards | ||
| 3D Worlds Arena Models generate explorable 3D scenes from an image and caption. Voters inspect two anonymous scenes from the same input and choose a favorite. | Everyday | Not in cards | ||
| AA-Image-Editing Validated private evaluators compare two image edits made from the same instruction and vote blind. Votes become Elo ratings. | Everyday | Not in cards | ||
| AA-Video-Editing Voters compare two edits of the same video made from one instruction and pick the better one. Votes become Elo ratings. | Everyday | Not in cards | ||
| Animal-Bench Models answer 41,839 questions about animal videos. The overall score averages accuracy across 13 tasks, including action prediction and recognizing predator and prey behavior. | School | Not in cards | ||
| Animation Bench Agents recreate web animations in HTML from reference frames and a network capture. The scorer compares appearance, motion and layout. | University | 0.61 | Not in cards | |
| Artificial Analysis Video Arena Voters compare videos from two text-to-video models for the same prompt and pick the better one. Votes become Elo ratings. | Everyday | Not in cards | ||
| AudioCaps Models match natural language descriptions to audio clips. Meta also compares how accurately encoders match audio, video, and text from the same clip. | Everyday | Not in cards | ||
| Caltech-101 Models identify object categories in photographs. Meta compares the classification accuracy of image encoders on this collection. | Everyday | Not in cards | ||
| Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. | Everyday | | System cards only | |
| COCO-Stuff 164K Models label each pixel with an object or background category. The benchmark adds annotations for regions such as sky, grass, and pavement. | Everyday | Not in cards | ||
| Common Objects in 3D v2 Models estimate camera poses from images of objects viewed from different angles. Meta compares predicted poses with the reference poses. | School | Not in cards | ||
| CompassArena Multimodal Users compare anonymous multimodal model answers and vote for the better response. A Bradley-Terry model estimates ratings from these votes. | Everyday | | Not in cards | |
| CURIE Biodiversity Georeferencing Models read a map and its caption from an ecology paper and return the geographic bounds. The score measures overlap with an expert's bounding box. | University | | Not in cards | |
| DeepSDO Description Models describe solar features visible in full-disk images from the Solar Dynamics Observatory. The scorer compares generated captions with human descriptions. | University | Not in cards | ||
| DesignArena Agentic Image to HTML Coding agents turn a user-uploaded image into a self-contained HTML website. Users compare how well each page reproduces the design and vote. | Everyday | 1,247 | Not in cards | |
| DesignArena Image Editing Image models edit a user image following a text instruction. Users compare the two edits and vote for the better one. | Everyday | 1,368 | Not in cards | |
| DesignArena Image to Frontend Coding agents turn a reference image into a multi-file frontend web app. Users compare the running apps and vote for the better one. | Everyday | | Not in cards | |
| DesignArena Image to HTML Models recreate a reference image or mockup as a working single-file HTML website. Users compare the rendered pages and vote for the closer, better one. | Everyday | 1,300 | Not in cards | |
| DesignArena Image to Video Video models animate a user image into a short video. Users compare two anonymous videos and vote for the better one. | Everyday | | Not in cards | |
| DesignArena Multi-Image to Video Video models compose one video from several reference images. Users compare two anonymous videos and vote for the better one. | Everyday | | Not in cards | |
| DesignArena Multimodal to Video Video models make a video from a mix of text, images and other inputs. Users compare two anonymous videos and vote for the better one. | Everyday | | Not in cards | |
| DesignArena Video Editing Video models edit a user video following a text instruction. Users compare the two results and vote for the better one. | Everyday | | Not in cards | |
| DesignArena Video to Website Models watch a reference video and recreate its content as a working single-file HTML website. Users compare the rendered pages and vote. | Everyday | | Not in cards | |
| Docci Models retrieve matching documents and images using the DOCCI corpus. | University | | Fading | |
| Dollar Street Models recognize household objects in photos from families at different income levels. The benchmark supports comparisons across regions and income groups. | Everyday | Not in cards | ||
| DTU Multi-View Stereo Models reconstruct objects from multiple camera views. The benchmark measures the accuracy and completeness of reconstructed surfaces against scanned reference geometry. | School | Not in cards | ||
| Dubai-CC Models describe land use changes between two satellite views of Dubai. The Change3D team used 150 of the 500 image pairs for testing. | School | Not in cards | ||
| EmbodiedCLUE-VLA (planning) Models reason about actions in physical environments. | Everyday | 86.8 | Not in cards | |
| EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. | Everyday | | Not in cards | |
| EmbodiedVerse-Bench Models answer questions about spatial relationships and actions from visual observations. The leaderboard compares models across embodied reasoning tasks. | Everyday | 50.7 | Not in cards | |
| EndoVis 2018 Robotic Scene Segmentation Models segment structures and surgical instruments in video from robot-assisted operations. The evaluation compares predicted masks with reference labels. | Everyday | Not in cards | ||
| ETH3D Models reconstruct scene geometry from images. Meta evaluates depth predictions against reference measurements on the ETH3D test scenes. | School | Not in cards | ||
| Flickr30k Models describe photographs of people, objects, and activities. Meta compares model captions with the human reference captions. | Everyday | Not in cards | ||
| Floorplan Arena Models furnish hand-drawn room sketches. People compare the layouts and vote for the better floorplan. | Everyday | | Not in cards | |
| Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. | Not in cards | |||
| GeoBiaset Models predict photo coordinates before and after researchers insert a person into the scene. The coordinates stay fixed across all edits of each background. | University | 69.2% | Not in cards | |
| GeoGuessBench (panoramic) Models see four Google Street View images taken at right angles. They predict coordinates and report their confidence. Closer guesses earn more points. | Everyday | | Not in cards | |
| Goliath human mesh recovery Models reconstruct a person's body from images of studio performances. Meta measures joint and surface errors against captured three-dimensional motion. | School | Not in cards | ||
| Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. | Expert | System cards only | ||
| Image Matching Challenge Models match photographs of landmarks taken from different viewpoints. The evaluation measures the accuracy of camera poses recovered from the matches. | School | Not in cards | ||
| ImageNet-1K Models assign one of a thousand object labels to each photograph. Meta reports classification accuracy on the validation images. | Everyday | Not in cards | ||
| KITTI depth prediction Models estimate scene depth from road images. The evaluation compares model predictions with depth measurements collected from a moving vehicle. | School | Not in cards | ||
| LiveOCRVQA Models read stylized text in images and answer questions about it. | Everyday | | Not in cards | |
| LMArena Video Edit Arena Video editing models apply the same edit request to the same clip, anonymously and side by side. Community votes give Elo-style ratings. | Everyday | | Not in cards | |
| MIEB Lite Embedding models complete the MIEB Lite task set. The official API reports the mean across task types. | | System cards only | ||
| MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. | Everyday | | Fading | |
| MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. | Everyday | 72.1% | Not in cards | |
| MSVD Models retrieve short videos from human descriptions and retrieve descriptions from videos. Meta reports retrieval accuracy in both directions. | Everyday | Not in cards | ||
| MuseBench (Music Score and Performance Understanding) Models answer music theory questions, read piano scores, and interpret recorded performances. The authors report accuracy separately for text, images, and audio. | University | | Not in cards | |
| NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. | Expert | 41% | Not in cards | |
| NYU Depth V2 Models estimate depth from indoor photographs. The benchmark compares predictions with depth measurements captured by a sensor. | School | Not in cards | ||
| ObjectNet Models identify familiar objects photographed with controlled changes in viewpoint, orientation, and background. Meta reports results on classes shared with ImageNet. | Everyday | Not in cards | ||
| OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. | System cards only | |||
| Oxford-IIIT Pet Dataset Models identify the breed of a cat or dog from a photograph. Meta measures breed classification across 37 categories. | School | Not in cards | ||
| PCB-Bench Models answer placement and routing questions and describe circuit boards from editor screenshots. The authors report choice accuracy and separate measures for free-form answers. | University | Not in cards | ||
| Personal Agent Bench Personal assistants complete everyday tasks using connected accounts. Experts check whether each assistant completes the work and respects the user’s instructions. | Everyday | Not in cards | ||
| Places205 Models classify photographs by scene type. Meta compares scene recognition on the 205-category edition of Places. | Everyday | Not in cards | ||
| Razavi-Bench Models explain what happens in analog circuits and why. A language model awards up to four points per answer against a reference and a rubric. | University | | Not in cards | |
| RealEstate10K Models estimate relative camera poses across frames from real estate videos. The benchmark supplies camera trajectories for comparison. | School | Not in cards | ||
| Recipe1M image and recipe retrieval Models match food photographs with recipes. Google evaluates image retrieval of the matching ingredients and instructions separately. | Everyday | Not in cards | ||
| Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. | Everyday | Not in cards | ||
| SHREC 2026 Human Perceived Visual Complexity of 3D Shapes Models rank CAD parts, primate teeth, and fractals by visual complexity. People compared pairs of CAD parts and fractals. A zoological measure ranks the teeth. | Everyday | Not in cards | ||
| Something-Something V2 Models recognize actions involving people manipulating everyday objects. The benchmark distinguishes actions that differ in direction, ordering, or outcome. | Everyday | | Not in cards | |
| STAR situated video reasoning Models answer multiple-choice questions about how people interact with objects in videos. The questions cover interactions, sequences, predictions, and feasibility. | Everyday | Not in cards | ||
| SuperCLUE Arena Image Editing Models edit images in response to instructions. The arena ranks systems from pairwise comparisons. | University | 1,308 | Not in cards | |
| SuperCLUE Arena Image to Video Models animate images. The arena ranks systems from pairwise comparisons. | University | Not in cards | ||
| SuperCLUE Arena Reference to Video Models generate videos from references. The arena ranks systems from pairwise comparisons. | University | Not in cards | ||
| SuperCLUE-AdsVU Models interpret video advertisements. | Everyday | | Not in cards | |
| SuperCLUE-I2V Models generate videos from still images. | University | | Not in cards | |
| SuperCLUE-R2V Models generate videos conditioned on reference material. | University | 64.9 | Not in cards | |
| SuperCLUE-V Models answer Chinese questions about visual inputs. | Everyday | 75.1 | Not in cards | |
| SuperCLUE-VideoEdit Models edit videos in response to Chinese instructions. | University | | Not in cards | |
| SuperCLUE-VLM Vision language models interpret images and answer questions in Chinese. | Everyday | | Not in cards | |
| SuperCLUE-VLR Models solve problems that require visual reasoning. | University | | Not in cards | |
| SuperCLUE-Voice (audiovisual interaction) Models respond to audio and video inputs in real time. | Everyday | Not in cards | ||
| SuperCLUE-World World models generate visual sequences. | University | | Not in cards | |
| TartanAir v2 Models reconstruct camera motion and scene geometry from simulated trajectories. The evaluation compares predictions with known poses and depths. | School | Not in cards | ||
| TextCaps Models retrieve matching text and images using the TextCaps corpus. | University | | Fading | |
| The Met Dataset Models match photographs of artworks to catalog images from the Metropolitan Museum of Art. The evaluation measures instance recognition across photographic conditions. | Everyday | Not in cards | ||
| TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. | Expert | 34% | Not in cards | |
| Toloka Visual Question Answering Benchmark Models answer questions about images by locating the object that answers each question. The benchmark compares predicted and reference bounding boxes. | Everyday | Not in cards | ||
| TRUE (Text Recognition and Understanding) Models recognize text in images and answer questions about its content. | Everyday | | Not in cards | |
| VIPSeg Models label video pixels and keep instance identities consistent across frames. The benchmark includes both countable objects and background regions. | Everyday | Not in cards | ||
| WMArena Video Arena Models generate a video from a starting image and an action. Viewers compare anonymous clips and vote for the more convincing result. | University | | Not in cards | |
| Wojak-Bench 3D Models turn the same drawing into a three-dimensional figure. Voters compare two anonymous prints or renders and choose which better matches the drawing. | University | 1,273 | Not in cards |