Benchmark Almanac

Vision

Models work from images or video. We count 1392 benchmarks here, 50 of them arenas. A benchmark belongs here when input includes images or video.

How fast it grows

289 were released since July, more than the 201 in the first half of 2026.

  1. Earlier 283
  2. 2024 H1 104
  3. 2024 H2 96
  4. 2025 H1 183
  5. 2025 H2 150
  6. 2026 H1 201
  7. 2026 H2 289

How much is still open

No model has passed 85% on 543 of the 650 benchmarks here that score in percent, 84% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
3D GameDev Arena Agents build playable 3D browser games from specifications and reference images. Players compare anonymous game pairs. The authors also test functionality and playability. Expert Nov 2026 99.8% Not in cards
3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. Expert Nov 2026 59.5 Not in cards
ALIVE-Bench Agents interpret social cues and act in simulated conversations. The evaluator tracks what agents perceive, what they infer, and how their actions affect other characters. Everyday Nov 2026 80.4% Not in cards
ASIMOV-Jailbreak Embodied agents handle adversarial instructions involving physical safety. Tool-call checks and model judges measure harmful compliance, refusals, and physical safety rubric adherence. Everyday Nov 2026 Not in cards
AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. University Nov 2026 Not in cards
AVCC Next-Speaker Prediction Models predict who will speak after a pause in a conversation. The authors compare two models with human listeners on the same 1,730 events. Everyday Nov 2026 Not in cards
BioBasicsBench Agents interpret raw experimental data in 100 wet-lab tasks, identify errors, and recognize when evidence cannot support a conclusion. Expert-written rubrics award partial credit for measurements, reasoning, and uncertainty. Scores report the mean percentage of available credit. Expert Nov 2026 66.7% Not in cards
CRONOS Robot policies manipulate objects in simulation without resetting the entire scene after each attempt. The benchmark compares performance as the scene accumulates changes. Everyday Nov 2026 0.56 Not in cards
DashBench Models explain rare events in driving videos, including near misses and collisions. The authors compare 25 video models on 1,709 clips. Everyday Nov 2026 34.8% Not in cards
DrivingBench v2 Vision-language models drive a real Tesla through two unseen cone courses with steering, accelerator and brake tools. The score is the furthest progress over three attempts in one conversation, averaged over six trials. Everyday Nov 2026 89.5% Not in cards
EarthRecord Agents analyze multidecade satellite observations to answer land-change questions. Reference checks score answers to 238 questions across 56 sites. Expert Nov 2026 85.4% Not in cards
Ego2Flow Driving policies operate on simulated city maps. The authors measure how each policy affects the surrounding traffic as other vehicles respond. University Nov 2026 Not in cards
EgoHome Models annotate 137 egocentric clips of household chores, about two hours in all, with subtasks, hand actions, hand captions and hand boxes. Segments are as short as 0.1 seconds. A model judge matches segment labels, and boxes are scored by centroid hits. Everyday Nov 2026 0.80 Not in cards
EmbodiedIntentBench Agents adapt perception during mobile robot missions as urgency, battery capacity, and connectivity change. The evaluator checks whether agents meet the mission objectives. University Nov 2026 Not in cards
Encord Egocentric VQA Benchmark Models answer 1,500 multiple-choice questions about actions, objects, space and time in private first-person videos. Everyday Nov 2026 76.5% Not in cards
EVF-Bench Models generate product videos with visual effects. The evaluator checks motion, product appearance, text stability, and whether the effects fit the request. University Nov 2026 51.6 Not in cards
ExtremeDrive Driving models plan trajectories from road recordings. A simulator executes the plans and measures success during emergency maneuvers and ordinary driving. Everyday Nov 2026 Not in cards
FORTE-MM Agents complete multimodal office tasks across 14 professions. A run passes only when the agent's output satisfies every task rubric. University Nov 2026 74.5% Not in cards
FrontierEE Bench Agents run electrical engineering experiments on real lab hardware through networked instruments, camera photos and tickets to a human operator. Deterministic verifiers check 10 tasks against protected bench records. Expert Nov 2026 36.7% Not in cards
GeoGuesser Bench Agents explore offline Street View panoramas to report coordinates, name places, and find objects. Coordinate guesses earn distance points. Other answers require an exact match. Everyday Nov 2026 22.8 Not in cards
GeoMind Robot agents manipulate tabletop objects using spatial relationships and remembered positions. The benchmark measures success across 54 task variants. Everyday Nov 2026 20.7% Not in cards
GradeBench Models read a student's handwritten answer to a university STEM exam question, with the rubric and a reference solution, and pick the rubric items to award. Scored by exact match with the items the instructor chose, averaged over five runs. University Nov 2026 57.3% Not in cards
HandWorldBench Models generate videos of human or robotic hands moving objects. The benchmark combines 17 measures of hand motion, object state, and video quality. Everyday Nov 2026 52.2 Not in cards
HARP-Bench Robot policies manipulate objects while people move through the workspace. The evaluator counts tasks the robot completes without a collision. Everyday Nov 2026 18.8% Not in cards
HomeLoop Agents coordinate robots and connected devices to investigate household alerts. A program checks their actions and whether they resolve the problem. Everyday Nov 2026 61.9% Not in cards
Hx2Dx ARENA Models interview a simulated patient, request examinations and tests, and give a diagnosis. A judge compares the diagnosis with the published case. Expert Nov 2026 83.9% Not in cards
LongBanana Models synthesize images from many reference images and textual constraints. Atomic rubric checks score content fidelity, relationships, and global constraints across 90 tasks. Everyday Nov 2026 74.6 Not in cards
ManiLadder Robot policies manipulate rigid and deformable objects in simulation. The authors report success across 112 tasks grouped into four difficulty levels. Everyday Nov 2026 Not in cards
MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. University Nov 2026 24.6 Not in cards
MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. Everyday Nov 2026 50.4% Not in cards
MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. Expert Nov 2026 56.5% In use
MotoVerse Models use a rider's camera to anticipate maneuvers and predict nearby road users' trajectories. The authors compare vision, language, and conventional prediction models. Everyday Nov 2026 46.3% Not in cards
MVDriveBench Models answer driving questions using synchronized videos from cameras facing different directions. The authors compare surround-view results with front-camera results. Everyday Nov 2026 80.1% Not in cards
NEST Models interpret children's gaze, gestures, and physical contact in social videos. The benchmark checks classification and choice questions and separately grades written descriptions. Everyday Nov 2026 76% Not in cards
NPCBench Models follow patients with nasopharyngeal carcinoma through diagnosis and care. The main patient-level measure requires correct decisions at successive stages. Expert Nov 2026 23% Not in cards
OmniScore-Bench Models read sheet music, locate symbols, and transcribe passages. The evaluator checks their responses against annotations for the relevant part of the score. University Nov 2026 91.0 Not in cards
PhysCapBench Models caption videos of physical interactions. The benchmark checks which motions, contacts, and changes of state the captions describe across 1,500 clips. Everyday Nov 2026 Not in cards
Pixel2Perspective Models locate and segment objects from aerial and ground views. Tests vary the camera angle, weather, and use of visible or infrared images. Everyday Nov 2026 Not in cards
PReMind-Bench Models inspect a scene and decide whether to retrieve a personal memory before answering. Judges check whether the answer addresses the hazard and uses the right person’s information. Everyday Nov 2026 87.3% Not in cards
Rapidata Multi-Step Image Editing (Identity Drift) Image editors apply five sequential changes to 199 photographs, and people compare the final faces for identity drift. Expert Nov 2026 1,866 Not in cards
Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. Expert Nov 2026 0.60 Not in cards
ReSpaceBench Models recognize rooms across separate visits and track objects that move between visits. The authors test 32 models on synthetic and real homes. Everyday Nov 2026 50.5% Not in cards
RoboARCade Agents control robots across three configurations to complete long manipulation tasks. The benchmark tests whether agents can use demonstrations and earlier attempts to improve. Everyday Nov 2026 51.4% Not in cards
RoboCasa-Mem Robots complete nine household tasks that require memory. Later steps depend on objects the robot saw or actions it took earlier. Everyday Nov 2026 16% Not in cards
RoboColosseum Physical robots execute fresh instructions in paired blind trials. Human evaluators choose the better run or a tie, and comparisons determine ratings for each embodiment. Everyday Nov 2026 Not in cards
RoboTeam Agents control four to sixteen simulated robot arms in real time using camera images. The evaluator checks which cooperative tasks the arms complete. University Nov 2026 80.6% Not in cards
RSLiveBench-UAV Models answer questions about vehicles in drone images. The benchmark checks perception and spatial reasoning across different views and adds newly captured scenes. Everyday Nov 2026 59.9 Not in cards
Science's Last Exam Agents conduct scientific research through tools, code, desktop software, laboratory recordings, and withheld references across 113 tasks in nine disciplines. The official composite equally averages six assessment scores on a zero-to-100 scale. Checks combine reference comparisons, executable tests, and literature audits. Research frontier Nov 2026 44.7 Not in cards
Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. Everyday Nov 2026 42.9% Not in cards
ShadowBench Models pose two 3D hands between a lamp and a screen, one tool call at a time, until their shadow matches one of 20 hand-shadow plates from books of 1859 to about 1900. The environment refuses any pose real hands can't make. Score, out of 100, mixes the overlap with the target and how closely the outlines match. Everyday Nov 2026 48.8 Not in cards
Snowflake AI_EXTRACT Document Extraction Evaluation Models extract fields, lists and tables across 2,789 annotated tasks from 11 datasets. Missing items, extra items and incorrect cells reduce the score. University Nov 2026 Not in cards
Social Signal Bench Models identify overlapping social signals in conversations from audio or video. The score weights each label by annotator agreement and gives credit for related predictions. Everyday Nov 2026 45.8 Not in cards
SolidWorks Bench Agents model the custom parts of 73 real machines in SolidWorks and assemble them from raster drawing packages, through screenshots and the SolidWorks API. A grader checks the geometry against the reference and reads mates, motion and feature trees from the native files. Expert Nov 2026 0.57 Not in cards
Spatial Intelligence Index v2 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. University Nov 2026 88.7 Not in cards
Terminal-Bench Math Modeling Agents use a terminal to solve 18 mathematical modeling problems and submit numerical answers. Feasibility checks and numerical scoring compare answers with reference solutions. Competition Nov 2026 64.3 Not in cards
TriPhysBench Models answer quantitative questions about physical events in videos. Some questions require measuring an object or looking up its specifications. School Nov 2026 50.4% Not in cards
VibePrevisBench Coding agents write Blender Python that turns a script, photos, an opening frame or reference footage into a graybox 4D previs scene and render it. Rule checks score the exported scene against hidden facts, and a VLM judge checks that those facts are visible in the render. Expert Nov 2026 77.7 Not in cards
VideoACE Models answer questions about fast actions in short videos. They can inspect selected frames more closely and request hints from a simulated viewer. Everyday Nov 2026 88.1% Not in cards
WorkIndex Agents complete everyday WorkBuddy tasks. Tencent combines eight tests covering office files, audio and video work, and visual questions. University Nov 2026 Not in cards
4DCodeBench Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion. Expert Oct 2026 0.79 Not in cards
AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. Everyday Oct 2026 Not in cards
ABACUS Models add, remove, or change a specified number of objects in an image. The evaluator checks the count and preservation of the surrounding scene. School Oct 2026 0.31 Not in cards
AD4Edu-Bench Models write audio descriptions for university lectures. A rubric checks accessibility rules, and the authors compare the rankings with blind and low-vision listeners' preferences. University Oct 2026 0.86 Not in cards
AdSpark-Bench Models generate advertisement videos from product images and prompts. The benchmark checks whether products remain consistent and whether each shot conveys the requested selling point. University Oct 2026 66.9 Not in cards
ANCHOR Models explain answers to audiovisual questions and locate the supporting evidence. The benchmark includes time intervals, image regions, and relevant sounds. Everyday Oct 2026 3.36 Not in cards
AVPS-360 Models locate and track the person making a sound in panoramic video. The annotations connect each sound source to a person across frames. Everyday Oct 2026 Not in cards
BenchCAD 2.0 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 59% Not in cards
BenchCAD 2.1 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 83.5% In use
BenchCAD 2.1-beta Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 Not in cards
BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. Expert Oct 2026 79.9 Not in cards
BURGER Robot policies draw shapes and objects from language instructions. The authors compare how well the resulting trajectories follow the requested drawing. Everyday Oct 2026 0.56 Not in cards
ChemArgus Models solve chemistry competition and textbook problems with text and figures. A model judge awards partial credit for each subquestion. Competition Oct 2026 64.9% Not in cards
CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. Expert Oct 2026 76.6 Not in cards
CIPHERGRID Models infer an encoded grid-world rule system from examples and produce a valid path for a new grid. Dynamic programming verifies the path. Expert Oct 2026 Not in cards
Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. Expert Oct 2026 $9,514 Not in cards
ClothDojo Robot policies use both hands to flatten and fold garments. The simulator checks success on familiar garments and garments withheld from training. Everyday Oct 2026 44.7% Not in cards
Co-Design Bench Agents revise existing applications from multimodal user requests. A judge checks browser and repository evidence against 2,331 expert criteria across 273 tasks. Expert Oct 2026 69.9% Not in cards
Code2Video-Compass Multimodal models write programs that render complete videos from text, an image or a video. The default Overall board ranks models by their mean rank across quality metrics, so a lower score is better. Everyday Oct 2026 1.07 Not in cards
CodeCap-Bench Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail. Everyday Oct 2026 73.2 Not in cards
CREATE Models find a word that connects three photographs. A judge grades each answer against the intended connection. Everyday Oct 2026 42.2% Not in cards
DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. Everyday Oct 2026 40% Not in cards
Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. University Oct 2026 957 In use
DexArena Robots use both hands to complete household and industrial tasks. The simulator checks each stage and the final outcome across ten multistage tasks. Everyday Oct 2026 52.1% Not in cards
Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. Everyday Oct 2026 Not in cards
Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. Expert Oct 2026 59.8 Not in cards
Earth-Bench V2 Agents analyze prepared satellite observations. The corrected release measures final answer accuracy, tool selection, argument accuracy, and execution efficiency. University Oct 2026 80% Not in cards
EduExplain Models select resources and arrange them into a storyboard to explain a science concept. Human-validated sets and storyboards provide references for 321 learner questions. University Oct 2026 Not in cards
Ego2Act Models generate first-person video showing how to complete a task from an initial image. A judge checks the steps and their physical plausibility. Everyday Oct 2026 83.3 Not in cards
Embodied Agent Arena Vision-language agents solve geometry, spatial reasoning, affordance, planning, and manipulation tasks. The 1,000-case evaluation combines 32 established sources with GeoProbe. Everyday Oct 2026 Not in cards
EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. Everyday Oct 2026 52% Not in cards
EmbodiedLab Agents choose measurements, operate a simulated robot arm, and read instruments through cameras to complete 35 laboratory experiments. University Oct 2026 41.4% Not in cards
EvaOmni Models answer questions about videos and identify the audio and visual evidence behind their answers. The score accounts for both correctness and evidence timing. Everyday Oct 2026 79.4% Not in cards
FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. Expert Oct 2026 0.67 Not in cards
FrontierCAD Parts Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score. Expert Oct 2026 0.63 Not in cards
GeoFlip Models generate videos from paired scenes where a geometry change reverses the expected physical outcome. Model judges check the event and the scene structure. School Oct 2026 3.17 Not in cards
GeoGuessBench (Nutlope) Models place a pin after seeing one street photo. Each game has five rounds, and closer guesses earn more points. Everyday Oct 2026 23412.0 Not in cards
GhostInk: Honest OCR on Bill Photos Models read text from phone photos and mask unreadable characters. The grader penalizes guessed text, even when the guess happens to match. Everyday Oct 2026 0% Not in cards
HealBench-Eval Models generate images intended to provide psychological comfort. The authors compare 14 image generators and prompt strategies on 1,095 cases. Everyday Oct 2026 26.1 Not in cards
HealthAgentBench V2 Agents complete 75 healthcare tasks across nine categories in terminal environments. Task-specific verifiers check outputs against hidden labels, with model grading for radiology report correction. Expert Oct 2026 57.3% Not in cards
HearingBench Models identify sounds from spectrogram images without hearing the audio. The test covers everyday sounds, instruments, vocals, words and sentences. University Oct 2026 46% Not in cards
HMU-Bench Models answer Chinese and English questions about musical notation and recordings. The authors score responses against reference answers. University Oct 2026 2.47 Not in cards
Human Verification Gap Models respond to paired requests about the same synthetic image. The evaluation measures how the stated purpose changes their refusal rate. Everyday Oct 2026 1.00 Not in cards
Humanity's Sixth Sense Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric. Everyday Oct 2026 75.9% In use
HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. Everyday Oct 2026 49.4% Not in cards
Hy-Nomas-Bench Interactive video world models follow text and camera controls across multiple turns and retain spatial memory. Expert Oct 2026 Not in cards
ImageJevBench v0.1.5 Models answer decisions about images and assign probabilities to the answers. The maker ranks capability using accuracy and calibration within cost and latency limits. Everyday Oct 2026 16.1 Not in cards
ImageJevBench v0.3.0 Models make decisions about images and report probabilities. This version draws 300 public and 1,200 sealed items from a new pool. Everyday Oct 2026 17.3 Not in cards
ImgMT Models translate text embedded in diagrams, maps, and infographics. Evaluation measures text recognition, region matching, and translation with CER, F1, chrF, and COMET. Everyday Oct 2026 61.3 Not in cards
InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. Everyday Oct 2026 41.7% Not in cards
IntPhys2-Timing Models identify when a physical rule breaks in a video. The authors add violation intervals to IntPhys2 and score temporal localization. Everyday Oct 2026 77.4% Not in cards
LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. Everyday Oct 2026 63.3% Not in cards
LightGenBench Models predict which parts of a 3D surface emit light and what color they emit. The authors compare generated emission with reference assets. University Oct 2026 Not in cards
Mind the Gap Models describe visible events during natural pauses in a video's soundtrack. The evaluator renders the narration and checks its timing and coverage. University Oct 2026 0.94 Not in cards
Mitosis Bench Agents mark every dividing cell in a tumor image. Each mark must fall within 30 pixels of an expert label. The grader rejects extra or missing marks. Expert Oct 2026 32.4% Not in cards
MMEnterprise-Bench Enterprise agents answer 1,613 questions over workplace documents, chats and figures that change over time. They must resolve conflicting versions and say when evidence is not enough. Expert Oct 2026 68% Not in cards
MoI-Bench Models generate and edit images for mobile users. The authors compare image quality across 64 subtasks and measure the cost of running models on phones. Everyday Oct 2026 0.97 Not in cards
OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. Everyday Oct 2026 Not in cards
Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. Everyday Oct 2026 20.2% Not in cards
Omni-InteractScope Explicit Request Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7. Everyday Oct 2026 20.2% Not in cards
Omni-InteractScope Proactive Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall. Everyday Oct 2026 46.4% Not in cards
Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. Everyday Oct 2026 0% Not in cards
PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. Everyday Oct 2026 68.1% Not in cards
ReactionBench Models rank a viewer’s reactions, match a reaction to one of four video clips, and explain which cue prompted it. Everyday Oct 2026 0.51 Not in cards
Read My Aura Models respond to requests for aura readings. The tasks probe invented perceptions, flattery, and whether a change in a name or image tint changes the reading. Everyday Oct 2026 1.00 Not in cards
RGraph-Eval Models generate robot manipulation videos. The evaluator compares objects, changing relationships, and three-dimensional motion paths with a reference demonstration. Everyday Oct 2026 0.89 Not in cards
RISEBench++ Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types. University Oct 2026 56.6% Not in cards
Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. Expert Oct 2026 32.7% Not in cards
Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. Everyday Oct 2026 Not in cards
RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. Everyday Oct 2026 19.1% Not in cards
RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. Expert Oct 2026 0.06 Not in cards
RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. Everyday Oct 2026 17% Not in cards
RSNA Knee AUC10 Models label twelve knee findings from images, radiology reports, or both. The grader measures how well their predictions distinguish findings on ten reference studies. Expert Oct 2026 0.92 Not in cards
RVG-Bench Models locate the video interval that answers a question about sounds and images. The evaluator measures overlap with the annotated interval. Everyday Oct 2026 49.6% Not in cards
Safety-Scene Vision Benchmark Models identify visible hazards in everyday photos. The evaluation compares answers under a neutral prompt and a prompt claiming an inspector found danger. Everyday Oct 2026 0.86 Not in cards
SAVU-Bench Models answer spatial questions about real scenes using sound and video. The authors report accuracy across seven tasks. Everyday Oct 2026 55.3% Not in cards
ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. Expert Oct 2026 0.53 Not in cards
ScribbleEdit Models edit images using a scribble as the only instruction. The evaluator checks whether the change matches the intended operation and location. Everyday Oct 2026 0.26% Not in cards
Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. Expert Oct 2026 $2,949 Not in cards
SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. Expert Oct 2026 100% Not in cards
SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. Expert Oct 2026 Not in cards
Spatial Agent Index v1 Agents analyze geospatial files and produce numbers, maps, or lists. Axis Spatial averages the scores for each included evaluation. University Oct 2026 Not in cards
Spatial Intelligence Index v1 Models analyze geospatial data, follow guided workflows, and answer knowledge questions. Axis Spatial also grades image interpretation in its vision ranking. University Oct 2026 Not in cards
SplatBench A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100. University Oct 2026 49.6 Not in cards
StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. Everyday Oct 2026 Not in cards
TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. Everyday Oct 2026 Not in cards
TextVBench Models render requested text in video or preserve text from a reference image. The score tracks spelling, placement, and consistency across frames. Everyday Oct 2026 68.3 Not in cards
TireSidewall-Bench: Multimodal Zero-Shot Tire OCR Models read the size markings on tire sidewalls from photographs. The grader compares extracted specifications with reference annotations. Everyday Oct 2026 46% Not in cards
VCIS Models inspect a grid and decide whether a proposed movement plan reaches its goal. Separate tasks check object coordinates and the step where a plan fails. School Oct 2026 100% Not in cards
VIALS 2 Models interpret 161 professional life-science visual artifacts. The score is judged accuracy averaged across three rollouts. Expert Oct 2026 49.9% Not in cards
Video-Index Models answer 840 multiple-choice questions about videos, the 210 hardest in each of four groups: perception, time, space and reasoning. Questions that shortcut attacks could answer without the video were left out. Everyday Oct 2026 68.1% In use
VisR-RAG-Fin Models retrieve pages from financial documents and answer questions using their contents. The evaluator measures page ranking and checks answers against references. University Oct 2026 Not in cards
VISTA (Visually Impaired Scene and Task Annotations) Models describe scenes, answer questions, and give guidance to blind or low-vision users from glasses recordings. A judge scores responses from 0 to 5. Everyday Oct 2026 4 Not in cards
WeGenEditBench Models edit images from instructions that can include text or multiple references. A shared checklist scores each requested change and the details that should stay intact. Everyday Oct 2026 4.14 Not in cards
WeRef-Bench Models locate objects described in English or Chinese and return no boxes when the image contains no match. Everyday Oct 2026 50.7% Not in cards
WorldState-Bench Video models generate two clips from the same image and action. A hidden state changes which ending the rule requires. A vision model checks both clips. Everyday Oct 2026 32% Not in cards
Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. Expert Sep 2026 40% Not in cards
APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. Everyday Sep 2026 60.2% Not in cards
Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. Expert Sep 2026 72.2% System cards only
BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. Expert Sep 2026 76.4 In use
CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. Expert Sep 2026 8.60% Not in cards
Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. University Sep 2026 0.62 Not in cards
Decision Index Vision Models make decisions over images and screens. The score is the published equal-weight mean of public and private vision scores. Everyday Sep 2026 63.4 Not in cards
DrivingBench Vision-language models drive a real Toyota Corolla through a cone course with up to three attempts in one conversation. Scored on the furthest progress along the course. Everyday Sep 2026 100% Not in cards
DrugTargetWorld Agents interpret cardiac MRI and identify proteins associated with causal effects in 20 simulated biobanks. The verifier checks their conclusions against hidden ground truth. Expert Sep 2026 40.0 Not in cards
Earth-Bench-Pro Agents complete Earth observation workflows under three conditions. They follow instructions, plan from supplied observations, or obtain observations at runtime. University Sep 2026 Not in cards
Furniture Assembly Benchmark Model gets an IKEA manual and a photo of a partly built item, then names the faulty step or next step. Accuracy on 60 photos. Everyday Sep 2026 90% Not in cards
GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. Everyday Sep 2026 80.2 Not in cards
GenPuzzle Models generate images that solve visual puzzles such as mazes, tangrams, and Sudoku. The authors select an automated judge using human reference scores. School Sep 2026 40.6 Not in cards
GenScale Models generate or edit images containing objects with known relative sizes. A judge calibrated against human ratings scores the relative scale of object pairs. Everyday Sep 2026 Not in cards
GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Scored by a distance-based GeoGuessr-style reward from 0 to 1. Everyday Sep 2026 0.64 Not in cards
HappyWorld Embodied The HappyWorld arena publishes a separate embodied vote pool and Elo ranking. Everyday Sep 2026 1,074 Not in cards
HappyWorld Spatial Reconstruction The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking. Everyday Sep 2026 1,252 Not in cards
HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. Everyday Sep 2026 Not in cards
HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Research frontier Sep 2026 77.9% In use
JusticeAxis Models assess evidence and produce judgments for 256 criminal cases from 18 countries. The evaluator checks charges and outcomes against lawyer-written references. Expert Sep 2026 84.4% Not in cards
Last Translation Benchmark Models translate inputs that other systems struggled with. A model checks each translation against human-written rules. A translation passes only if it meets every rule. Research frontier Sep 2026 45.3% Not in cards
metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. Everyday Sep 2026 Not in cards
MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. Expert Sep 2026 82.2% In use
MindWorldBench Models animate a scene to show how a person would act given their beliefs or intentions. Paired prompts change mental states while holding the scene fixed. Everyday Sep 2026 Not in cards
Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. Expert Sep 2026 34% System cards only
MUSE Models answer questions about artworks used in language education. The tasks cover visual details, emotions, cultural context, and relationships within the image. School Sep 2026 56.7% Not in cards
OmniEchoBench Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency. Everyday Sep 2026 43.9% Not in cards
OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. Everyday Sep 2026 72.7 Not in cards
OpenRSI Index Molmo2 Video-Pointing Inference Strategy Agents tune video inputs and inference for a frozen Molmo2 model. The score is held-out spatiotemporal pointing soft F1. Research frontier Sep 2026 0.46 Not in cards
OpenScore String Quartets Models transcribe images of string quartet sheet music into symbolic notation. Scored as 1 minus normalized edit distance from the reference encoding. University Sep 2026 0.84 In use
ORAV Bench Models combine reference images, sounds, and videos into a requested audiovisual scene. A judge compares pairs of outputs against each reference's assigned role. Everyday Sep 2026 0.70 Not in cards
Panda-CVL Models accept correct image-grounded responses or locate and replace their first erroneous text span. The test expands 652 conversations into 2,126 instances. Its overall F1 combines correct acceptance with exact correction of faulty responses. University Sep 2026 17.1% Not in cards
Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. Everyday Sep 2026 36.3% Not in cards
Parametric CAD Bench Agents create and edit parametric FreeCAD models from text and engineering drawings. Programmatic checks award continuous task rewards for geometry and specification consistency. Expert Sep 2026 61.0 Not in cards
Periapical Lesions Bench Agents locate and grade lesions around tooth roots in panoramic dental X-rays. A perfect report matches every lesion and grade without adding extra boxes. Expert Sep 2026 18.7% Not in cards
PhysVista Vision-language models perceive, reason about and judge whether videos follow real-world physics. The benchmark also tests if they can judge the physical realism of generated videos. Everyday Sep 2026 85% Not in cards
ReactHuman Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate. Everyday Sep 2026 63.7% Not in cards
Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. Everyday Sep 2026 52.5% Not in cards
RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. Everyday Sep 2026 65.7% Not in cards
Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. Everyday Sep 2026 20% System cards only
Runway Model Router Benchmark Video generation systems produce clips for 250 image-to-video prompts across ten production categories and are compared by usable rate and cost. Expert Sep 2026 Not in cards
ScreenASR-Bench Models transcribe speech using on-screen text, visible objects, or clues from the scene. The authors report word error rate and recall. Everyday Sep 2026 3.47% Not in cards
Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. Expert Sep 2026 Not in cards
SolveEdit Models edit an image to solve a visual problem. The evaluator checks whether the goal holds and whether unrelated scene details survive the edit. School Sep 2026 57% Not in cards
StateBench (video continuation) Models continue a video while tracking what happened in earlier segments. A judge checks whether visible objects, hidden processes, and later state changes remain consistent. Everyday Sep 2026 32.5 Not in cards
StationeryBench Robots use both arms to handle common desk supplies. A human grades progress from zero to 100 on five tasks. Everyday Sep 2026 46.0 Not in cards
SWE-bench Multimodal v2 Agents patch JavaScript repositories using issue reports that include images. Automated tests check whether each patch fixes the issue. Expert Sep 2026 70.2% System cards only
TimelineBench Agents turn raw footage and a client brief into a finished video. The edit must pass delivery checks, meet the brief, and clear a quality threshold. University Sep 2026 32.1% Not in cards
TrafficImag Models edit roadside traffic videos to show a different sequence of events. The evaluator checks routes, interactions, and preservation of the surrounding scene. Everyday Sep 2026 90.0 Not in cards
TriWorldBench World models generate synchronized robot-view videos from head and wrist cameras. Nineteen signals measure cross-view agreement, task alignment, physical coherence, motion, temporal stability, and visual quality. Everyday Sep 2026 Not in cards
UFO-Bench Models generate customized images from text and visual references. The evaluator breaks each request into individual requirements and checks how they interact. Everyday Sep 2026 0.73 Not in cards
VANTAGE-BENCH Models answer questions, describe events, and locate objects or actions in infrastructure-camera footage. Task-specific metrics measure answer accuracy, caption quality, localization, and tracking. Everyday Sep 2026 75.3 Not in cards
VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. Expert Sep 2026 60.5% System cards only
VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. Everyday Sep 2026 79.0 Not in cards
VIF-Bench Models generate images from multiple references and visual instructions. Judges check whether the result follows controls such as pose or lighting without copying unwanted features. Everyday Sep 2026 7.60 Not in cards
VisEditBench (TikZ diagram editing) Models edit TikZ code to follow instructions drawn on a diagram. A judge scores adherence, preservation of other content, and readability. University Sep 2026 83.6% Not in cards
WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. Expert Sep 2026 46.5% Not in cards
WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. Everyday Sep 2026 42.3% Not in cards
ADVANCE audio and image retrieval (MTEB) Models match field recordings and aerial images from 5,075 locations. The correct recording and image come from the same place. Everyday Aug 2026 Not in cards
AnomalyXL Models locate and describe anomalies in long synthetic time series. The benchmark checks multiple-choice answers and scores precise answers using the known anomaly properties. Expert Aug 2026 0.33 Not in cards
Bio-VQA Models identify organisms, describe visible traits, judge spatial relationships, and recognize individual animals across photographs. The benchmark has 24 tasks. University Aug 2026 0.91 Not in cards
Brennan Center election image identification Chatbots inspect 14 synthetic election images in English and Spanish. Researchers count how many images each chatbot identifies as synthetic in each language. Everyday Aug 2026 Not in cards
Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. Expert Aug 2026 90.7% In use
CharXiv Extended Models answer descriptive and reasoning questions about scientific charts. Mercor adds 150 new expert-written tasks to the 5000 public ones and reports both scores. University Aug 2026 87.1% Not in cards
ConceptEdit-Bench Models edit images according to 1,000 instructions across six categories. A vision-language judge scores instruction following, visual quality, and consistency in unedited regions from 0 to 10. The per-case score averages those three dimensions. Everyday Aug 2026 66.2 Not in cards
Cyc3D Models generate 3D assets from images. The evaluator checks mesh quality and measures how much shape and identity drift across repeated rendering and reconstruction. University Aug 2026 Not in cards
Effective FPS Models decide whether video frames are consecutive or duplicated to simulate a higher capture rate. The evaluator checks each prediction against the construction of the clip. University Aug 2026 Not in cards
EmbodiedCLUE-VLA RoboTwin Evaluation Robot policies perform ten two-handed manipulation tasks under randomized RoboTwin 2.0 Hard conditions. The score averages task success rates over twenty episodes per task. Everyday Aug 2026 Not in cards
Escalation Router Models predict whether Qwen2.5-VL 7B correctly identifies visible hands or grippers in a clip. The reference label comes from two observed runs per clip. University Aug 2026 Not in cards
H2R-Bench Models turn first-person demonstrations into videos of a specified robot performing the task. The benchmark checks contacts, object responses, and completion of the goal. Everyday Aug 2026 81.0 Not in cards
Hands Visible Models inspect 12 frames per clip and determine whether a human hand or robot gripper appears. The 45-clip evaluation uses provisional machine-generated labels. Everyday Aug 2026 Not in cards
HarnessEval-W Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning. Expert Aug 2026 75.5 Not in cards
HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. Expert Aug 2026 77.3% Not in cards
HM3D-MFMON Agents navigate across floors and find a sequence of objects. Everyday Aug 2026 Not in cards
LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. Everyday Aug 2026 66.9% Not in cards
LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. Everyday Aug 2026 85.6% In use
MedXpertQA MM Extended Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores. Expert Aug 2026 71.2% Not in cards
MineAmongUs Vision-language agents play an embodied Among Us game with verbal and nonverbal deception. Everyday Aug 2026 81.3% Not in cards
MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. University Aug 2026 61.3% Not in cards
OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. University Aug 2026 100% Not in cards
OmniMech Models write CAD programs to reconstruct parts from dimensioned drawings. The evaluator compares the reconstructed geometry with reference parts and measures relative volume error. University Aug 2026 15.9% Not in cards
OmniPhys Models solve Chinese physics problems from middle school through university and draw diagrams. The evaluator grades answers, reasoning, and generated figures. University Aug 2026 80.4 Not in cards
PatternEval Models answer 2,415 image prompts in thinking and non-thinking modes. A model judge flags exposed draft reasoning, repetition, contradictions, and unsupported reasoning. The Trigger rate measures responses with any flagged failure. Everyday Aug 2026 0.78% Not in cards
PAWBench Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior. Everyday Aug 2026 Not in cards
PlayWorld An agent player pursues objectives inside video world models. Gemini 3.1 Pro applies weighted rubrics for geometry, interaction, and state evolution across the resulting videos. Everyday Aug 2026 Not in cards
QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. University Aug 2026 44.6% In use
SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. Everyday Aug 2026 28.3 Not in cards
VGI-Bench Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame. Everyday Aug 2026 51% Not in cards
VIALS Vision models answer 161 questions about lab images such as gel blots, plasmid maps, microscopy and flow cytometry plots. An LLM judge grades each answer against a key. Expert Aug 2026 65.8% Not in cards
VideoGAIA Agents inspect selected video segments, search the web, and read pages before answering questions. Model judges check answers, and the harness reports accuracy by category and difficulty. Everyday Aug 2026 58.3% Not in cards
VWE-Bench Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics. Expert Aug 2026 57.3% Not in cards
WildArtifactBench Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo. Expert Aug 2026 Not in cards
WorldExam Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states. Everyday Aug 2026 72.8 Not in cards
ActiveVision Models inspect images repeatedly to count regions, trace structures, and compare visual attributes. The released split has five examples for each of 17 tasks. Exact match checks the final answer after normalizing case, spacing, and separators. Everyday Jul 2026 50.6% Not in cards
Agent Memory Leaderboard Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework. Everyday Jul 2026 Not in cards
Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. Expert Jul 2026 78.3% System cards only
ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. Everyday Jul 2026 47.6% Not in cards
ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. Everyday Jul 2026 System cards only
AstroCLIMB Models compare pairs of astronomy figures or captions. They determine whether the pair shares a figure, shares a paper, comes from citation-linked papers, or comes from unrelated papers. University Jul 2026 Not in cards
AutoCAD Bench Computer-use agents turn a dimensioned reference drawing into a correct, editable DWG in AutoCAD through the screen alone, with mouse and keyboard. 50 tasks span 2D drafting and 3D modeling at four difficulty levels. Expert Jul 2026 46% Not in cards
BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. Everyday Jul 2026 Not in cards
Blind-Spots-Bench Models answer 235 text and visual problems chosen to expose failures on tasks that people find simple. A Gemini 3 Flash grader compares each response with a structured reference solution and gives binary credit. The grader can execute code to check exact constraints. Everyday Jul 2026 Not in cards
CHAOS-Bench Models read deliberately corrupted words in 500 academic page images. Scores measure faithful recognition of the visible strings instead of correction to plausible words. Expert Jul 2026 29.8% In use
CLBench-V Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition. Everyday Jul 2026 28.5% Not in cards
DexVerse Robot policies perform tabletop manipulation tasks in Isaac Lab, including tool use, bimanual coordination, and sequences with several goals. Task-specific success conditions check completion under different visual settings and robot embodiments. Everyday Jul 2026 34% Not in cards
Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. Expert Jul 2026 97.3% Not in cards
EduArt Models answer art history questions from Italian secondary schools and US Advanced Placement exams. The authors compare 12 models across seven question formats. School Jul 2026 82.8% Not in cards
EgoProceVQA Models watch first-person clips and answer questions about the steps in a procedure. Some questions ask when a step occurred or which step is missing. Everyday Jul 2026 51.6% Not in cards
EmbodiedCLUE-World 2.0 World models predict outcomes in embodied settings under the second evaluation protocol. Everyday Jul 2026 41.8 Not in cards
Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. Everyday Jul 2026 Not in cards
GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. Everyday Jul 2026 Not in cards
Gemini Robotics On-Device data scaling evaluation Robots learn tasks on hardware first introduced during post-training. Google compares task success as the amount of adaptation data increases. Everyday Jul 2026 Not in cards
Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Everyday Jul 2026 93% In use
HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. Everyday Jul 2026 11.1% Not in cards
KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. Everyday Jul 2026 0.81 Not in cards
KWV Bench Models answer atomic visual questions extracted from tasks distilled from real knowledge-work scenarios. Expert Jul 2026 64.7% System cards only
MM-ToolSandbox Visual tool-calling agents complete 258 multi-image, multi-turn scenarios across 16 application domains and more than 500 tools. Rubric and entity-diff checks score the runs. Expert Jul 2026 Not in cards
Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Everyday Jul 2026 91.3% In use
MPIE-Bench Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other. Everyday Jul 2026 0.65 Not in cards
MultiRef-Compass Models generate audio-video content from several visual references and text instructions across 350 samples. Fourteen metrics assess quality, reference fidelity, audio-visual consistency, and instruction following. The evaluator combines automatic measurements with multimodal model judgments. Everyday Jul 2026 Not in cards
OCT-Bench Models answer 10,076 questions about retinal scans, from anatomy to treatment and prognosis. The evaluator measures answer accuracy and counts invalid responses as incorrect. Expert Jul 2026 62% Not in cards
OmniRouting Models route copper traces between pads on printed circuit boards. The evaluator counts required nets that connect without design rule violations. University Jul 2026 28% Not in cards
Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. Expert Jul 2026 69.2% System cards only
PerceptionBench (Moonshot) Moonshot in-house benchmark of atomic visual perception skills. Scoring details unpublished. Everyday Jul 2026 59.7% In use
Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Everyday Jul 2026 57.4% In use
RadLE 2.0 Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors. Expert Jul 2026 758.0 Not in cards
RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. Everyday Jul 2026 27.6% In use
RoughCut-Bench Models select and order video clips or mark speech to remove from a transcript. The evaluation compares cut plans with professional editors’ annotations. University Jul 2026 0.53 Not in cards
S-EMBER Models answer memory questions from first-person video using only frames available before the query time. The benchmark scores multiple-choice accuracy and temporal overlap between predicted evidence intervals and annotated intervals. Everyday Jul 2026 54.4% Not in cards
SoundingEarth audio and image retrieval (MTEB) Models match a field recording to an aerial image of the same location, or retrieve the recording from the image. Each direction has 1,000 queries and 2,048 candidates. Everyday Jul 2026 Not in cards
Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. Expert Jul 2026 81.6% Not in cards
UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. Everyday Jul 2026 47.5% In use
WorldModelGym World models predict returns for five candidate action sequences across 100+ Atari, Meta-World, DMC and classic-control tracks. Scored by normalized regret. Jul 2026 Not in cards
ZendoWorld Models infer hidden rules in the board game Zendo from labeled examples. Everyday Jul 2026 Not in cards
Act2Answer Vision-language-action models answer commonsense and world-knowledge questions by acting. Each question becomes a short tabletop episode, and the robot answers by placing an object on one of the candidate answers. The score is the success rate per knowledge category, which reduces the effect of weak low-level control. Everyday Jun 2026 68% Not in cards
ATOM-Bench Robot policies combine manipulation skills on single arm and dual arm platforms. Everyday Jun 2026 48.2% Not in cards
AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. Expert Jun 2026 66.5 Not in cards
ClinHallu Models answer medical image questions and produce intermediate reasoning. The evaluation reports answer accuracy and separate hallucination rates for visual recognition, knowledge recall, and reasoning integration against annotated reasoning traces. Expert Jun 2026 80.1% Not in cards
CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. Everyday Jun 2026 55.2 Not in cards
DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. Everyday Jun 2026 10.6% Not in cards
EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. Everyday Jun 2026 Not in cards
Ego-MC-Bench Models watch a cook follow spoken instructions and tell them when to correct a mistake. The evaluator checks intervention timing and recipe step completion. Everyday Jun 2026 0.23 Not in cards
FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. University Jun 2026 77.8% In use
GeoDrive-Bench Models choose driving actions from road images and local traffic conventions. The benchmark covers six countries and omits explicit country labels. Everyday Jun 2026 91.9% Not in cards
Harbor-Index Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. Research frontier Jun 2026 28.1% In use
HealthAgentBench Agents use terminal tools to complete 54 clinical and biomedical tasks across seven categories. Task-specific verifiers measure success, including a model judge for radiology report correction. Expert Jun 2026 55% Not in cards
Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. Everyday Jun 2026 48.0 In use
K9-Bench Models answer 4,744 questions about dog behavior in 907 videos. The evaluator compares each free-form answer with five options that the model never sees. School Jun 2026 40.1% Not in cards
KilometerVision Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures. Everyday Jun 2026 Not in cards
LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. Expert Jun 2026 61.3% In use
LLM Agronomy Benchmark Image Identification (four choices) Models choose among four labels for each photo of a crop disease, insect pest, or weed. One accuracy score combines the 150 questions. University Jun 2026 90% Not in cards
MarineEVT Models answer 2,000 held-out questions about underwater events, including species interactions, movement, and timing. The authors report accuracy across five reasoning categories. University Jun 2026 43.7% Not in cards
MedBench 5.0 (OpenCompass) Models reason through clinical cases as information changes. The benchmark assesses intermediate decisions, use of tools, and hallucinations. Expert Jun 2026 69.2 Not in cards
MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. Everyday Jun 2026 Not in cards
OmniCap-IF Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance. Everyday Jun 2026 80.7% In use
OmniLayout Models place components inside a board outline. The evaluator checks overlap, routing, and electrical functionality against the schematic and the reference layout. University Jun 2026 0.76 Not in cards
OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. Everyday Jun 2026 59.2% Not in cards
P3D-Bench Models generate parametric 3D objects from text, images, or assembly instructions. The evaluator checks whether outputs run and preserve geometry, topology, and part structure. University Jun 2026 69.4 Not in cards
PhysicalRealismBench-U VLM watches 1,300 synthetic Isaac Sim videos, flags physics-law violations, names object and frames. Scored by joint realism F1. Everyday Jun 2026 60.8% Not in cards
Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. Everyday Jun 2026 10.9 Not in cards
RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. Everyday Jun 2026 49.9% Not in cards
Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. Everyday Jun 2026 Not in cards
Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. Expert Jun 2026 Not in cards
SLoMO Audio Description Competition 2026 Models write one sentence for each gap in a film’s dialogue. Judges score how well the descriptions convey visual information and match professional references. University Jun 2026 Not in cards
SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. Everyday Jun 2026 14.4% Not in cards
SpatialWorld Physical Agents complete tasks in physical environments. The score is the published overall success rate across physical task categories. Everyday Jun 2026 14.4% Not in cards
SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. University Jun 2026 63.7% In use
TEM-VQA Models give short written answers about structures, image features and experimental design in transmission electron micrographs. Expert Jun 2026 40.4 Not in cards
UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. Everyday Jun 2026 55.8% Not in cards
Vision2Web S3-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Jun 2026 74.6% Not in cards
WorldBench Models answer hand-written reasoning questions about images drawn from a taxonomy of thousands of visual concepts. Questions were kept only if frontier models failed them. Expert Jun 2026 64% System cards only
WorldOlympiad Models generate long videos for gaming, robotics, and everyday scenes. The evaluator checks physical rules, reconstructed geometry, and instruction following across video segments. Everyday Jun 2026 Not in cards
X-Stream Models answer questions that combine concurrent video streams. A model judge grades answers, while forward-looking questions also require responses within the correct time window. Everyday Jun 2026 49.6 Not in cards
3DCodeBench Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. May 2026 55.4% In use
ACII DaiKon Challenge Models predict interpersonal influence, turn taking and rapport trajectories from multimodal dyadic conversations across five languages. Expert May 2026 Not in cards
AerialSim Agents fly a simulated drone using camera observations or write a program for the flight. The simulator checks the recorded trajectory against the mission requirements. Expert May 2026 73.2% Not in cards
Agentick Agents solve procedurally generated gridworld tasks involving navigation, planning, reasoning, memory, generalization, and coordination. Scores normalize task performance between random and oracle reference policies. Everyday May 2026 0.63 Not in cards
AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. Expert May 2026 38.4% In use
Animation2Code Models recreate a web animation from a video using HTML, CSS, and JavaScript. The evaluator renders their code and compares appearance and motion separately. University May 2026 0.84 Not in cards
BenchCAD Programmatic CAD: 17,900 execution-verified CadQuery programs over 106 industrial part families. Four matched tasks scored on geometry. University May 2026 95.9% In use
Blueprint-Bench 2 Agents turn about 20 interior photos of each of 50 apartments into 2D floor plans. Scores compare room connectivity with ground truth, normalized against random. Everyday May 2026 58.6% In use
CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. University May 2026 45.1% System cards only
CC-OCR-Bench-V2 Models process 7,093 hard real-world document samples across five OCR tracks, including parsing and document QA. Each track uses its own metric. Everyday May 2026 76.6% In use
EmbodiedCLUE-World World models predict outcomes in embodied settings. Everyday May 2026 76.2 Not in cards
ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. Everyday May 2026 70% Not in cards
EvalVerse Video models generate cinematic clips from prompts and reference material derived from professional videos. Film experts compare outputs across visual design, acting, cinematography, aesthetics, emotion, shot sequences, and sound. A calibrated vision-language evaluator also estimates these judgments. Expert May 2026 0.73 Not in cards
HiDrive Driving policies navigate unusual traffic situations in a closed-loop simulator. The evaluator measures route completion and applies penalties for driving errors. Everyday May 2026 Not in cards
LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. Everyday May 2026 0.90 Not in cards
MDMR-Bench Models generate one image from several references. A model judge checks whether the output preserves requested elements, follows instructions, and combines them correctly. Everyday May 2026 88.8% Not in cards
MemEye Multimodal agent memory systems answer questions that need fine visual evidence kept from earlier sessions, which captions or text traces alone cannot supply. Everyday May 2026 49.4% Not in cards
MemLens Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths. Everyday May 2026 53.6% Not in cards
Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. Expert May 2026 37.6% Not in cards
MM-OCEAN Models rate apparent personality traits in short videos, explain those ratings, and answer questions about observable cues. Evaluation reports rating accuracy, absolute error, cue-question accuracy, and model-judged explanation scores. Everyday May 2026 Not in cards
MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. Expert May 2026 16.2% Not in cards
OmniMatBench Models answer questions and perform calculations across 19 materials science fields. The authors compare automated grades with human assessments. University May 2026 0.37 Not in cards
OncoTriad-QA Models answer questions about cancer patients using radiology, pathology, and genomic evidence. The authors score multiple-choice answers and compare written responses with reference answers. Expert May 2026 46% Not in cards
PaintBench Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks. Everyday May 2026 17.1% Not in cards
Polaris-Bench Models solve the same 53 visual reasoning tasks in Cartesian and polar layouts. The benchmark compares accuracy across the paired layouts. School May 2026 39.2% Not in cards
ProactiveVideoQA Models answer spoken questions when a streamed video first reveals the answer. Everyday May 2026 33.5 Fading
QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. University May 2026 1,884 In use
RepCount-A Models count repeated actions in streamed video and are scored by distance from the true count. Everyday May 2026 35.4 Fading
RoboChallenge Table30 v2 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. Everyday May 2026 0.73% Not in cards
RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. Everyday May 2026 21.5% Not in cards
SFI-Bench Multimodal models answer video questions about spatial layouts, object affordances and functional reasoning in indoor scenes. University May 2026 Not in cards
SpaceDG-Bench Models answer spatial questions under clean images and nine forms of visual degradation. Choice and binary answers use accuracy, while numerical answers use relative-error thresholds. The test has 1,102 verified questions from scenes excluded from the training set. Everyday May 2026 56.7% Not in cards
SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. Expert May 2026 40.5% Not in cards
TeachObs Models identify teaching practices in classroom video and assess entire lessons. Human annotations cover 39 observable behaviors in 5,158 segments from eight countries. University May 2026 0.28 Not in cards
Tricky Edits of Collected and Curated Images Models edit a source image to follow a written instruction. People and a model judge instruction following, preservation of unaffected content, and visual quality. Everyday May 2026 19.5% Not in cards
TVRBench Agents move through simulated indoor scenes to reproduce a target image's viewpoint. Success requires stopping at exactly the target pose on the simulator's discrete grid. The evaluation split has 500 tasks across 80 scenes. Everyday May 2026 12% Not in cards
Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. Expert May 2026 74.2% Not in cards
Vals Multimodal Index v1.1 Models complete finance, coding, and education tasks that contribute to a weighted index. Expert May 2026 70.7% Not in cards
VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. Everyday May 2026 3.40 Not in cards
ViMU Models explain what a short video implies and select the cues that support their reading. They also identify rhetorical devices and social signals. Everyday May 2026 46.9% Not in cards
Visual Aesthetic Benchmark Models see a set of images with the same subject and intent and must pick the best one, and in a harder setting both the best and the worst. The 400 sets hold 1,195 commissioned works across fine art, photography and illustration. Labels come from expert judgments, and only sets where experts largely agreed are kept. Expert May 2026 40.3% Not in cards
WikiVQABench Vision-language models answer multiple-choice questions that require image evidence and linked Wikipedia and Wikidata knowledge. University May 2026 Not in cards
WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. May 2026 69.5% Not in cards
YoCausal Video generation models are tested for causal and arrow-of-time understanding by comparing real videos with their time-reversed versions, in two levels. Everyday May 2026 54.2 Not in cards
Aether-Bench Models retouch photographs automatically, match a requested style, or apply specified adjustments. The style test measures fidelity on 100 image pairs from unseen presets, including peak signal-to-noise ratio. University Apr 2026 79.7% Not in cards
AstroVLBench Models interpret galaxy images, light curves, and spectra across five observing tasks. The authors compare their classifications with reference labels and examine their explanations. University Apr 2026 68.8% Not in cards
Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. University Apr 2026 78.3% In use
Claw-Eval-MM Agents complete the 101 multimodal tasks of Claw-Eval, such as video questions, document extraction, and webpage generation. Rubric items graded from run traces set scores. University Apr 2026 54.7% In use
CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. Expert Apr 2026 45.1% Not in cards
DrawingVQA Models answer 92 multiple-choice questions about 33 construction drawings. They read symbols, cross-reference details, and calculate quantities. University Apr 2026 77.2% Not in cards
GraphicDesignBench Models analyze and produce graphic designs in 50 tasks spanning layout, typography, infographics, design semantics, and animation. The suite uses professional layered design templates. Expert Apr 2026 0.94 Not in cards
GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. Expert Apr 2026 11.4% Not in cards
Image-to-WebDev Arena Users upload a screenshot, wireframe or mockup, two anonymous models build the website in Code Arena, and users vote for the better result. Votes become ratings with bootstrapped confidence intervals. Everyday Apr 2026 1,756 Not in cards
ImageMining Multimodal agents retrieve and reason over image information for image mining tasks. Expert Apr 2026 30.7% Fading
Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. Everyday Apr 2026 65.7% Fading
K-MetBench Models answer 1,774 questions from the Korean National Meteorological Engineer Examination under a zero-shot protocol. The benchmark reports answer accuracy across five subjects and separate visual and Korean-specific subsets. Expert rationales support an additional rubric score for reasoning. Expert Apr 2026 93.7% Not in cards
KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. University Apr 2026 42% Fading
LVSum Models summarize long videos with timestamped segments, relevance scores and descriptions using video frames, transcripts or both. Expert Apr 2026 Not in cards
MMEB-V3 Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval. Everyday Apr 2026 32.1% Not in cards
MMSearch Multimodal agents search and answer questions that require visual and textual evidence. Expert Apr 2026 72.9% Fading
neuralCAD-Edit Agents edit existing CAD models from text or recorded instructions. CAD experts rate whether they would accept each edit. Software also compares the resulting geometry. University Apr 2026 0.25 Not in cards
PulseBench-Tab Models convert 1,820 table images in nine languages into HTML tables. T-LAG represents adjacent cells as directed edges and matches their text and structure against human annotations. The reported score is F1 over matched edge weights. Everyday Apr 2026 0.89 Not in cards
RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. Everyday Apr 2026 46.2% In use
SHROOM-Visions 2026 Models inspect an image and a written response. They mark unsupported characters, estimate the probability of an error and classify each error in four languages. Everyday Apr 2026 Not in cards
TriScore Models identify composers and instruments from recordings, score images, and symbolic notation. They also retrieve excerpts from the same composition. University Apr 2026 Not in cards
VagueBench Agents follow paired instructions in indoor environments. The authors compare performance when a clear instruction and a vague instruction describe the same task. Everyday Apr 2026 43.5% Not in cards
VideoMME v2 (with subtitles) Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. Everyday Apr 2026 68.3% In use
VSAS-Bench Streaming vision-language models answer tasks over online video frames while evaluation measures accuracy, response timing and consistency. Expert Apr 2026 Not in cards
ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. Everyday Mar 2026 100% In use
AsgardBench Agents adapt high-level action plans from images, action history and lightweight success signals in a controlled simulator. Expert Mar 2026 76.5% Not in cards
ATM-Bench Agents answer questions about a person’s history using images, videos, and email records. The Harbor package contains 31 hard questions with schema-guided memory. University Mar 2026 86% Not in cards
BigEarthNet.txt Models answer land cover questions, caption satellite images, and locate regions described in text. The authors checked annotations for 1,082 pairs of radar and optical images. University Mar 2026 0.79% Not in cards
BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. University Mar 2026 Not in cards
CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. University Mar 2026 32.3% Not in cards
Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Everyday Mar 2026 81.4% In use
CoVR-R Models find the video that matches a reference clip after applying a change described in text. School Mar 2026 47.6% Not in cards
Crate Bench Models identify an album and its artist from the cover image. The benchmark provides 300 test covers and supports closed-book and web-search modes. Everyday Mar 2026 Not in cards
DesignJanus People rate generated objects from six viewpoints and mark inconsistencies in shape, materials, and identity. The harness compares human ratings with automatic measures. Everyday Mar 2026 Not in cards
HCSU-Bench Models match a Chinese calligraphy sample to one of eight styles. A second task asks models to describe the brushwork in fewer than 50 words. University Mar 2026 34.6% Not in cards
LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. Everyday Mar 2026 71.4% Not in cards
Live Gaming Benchmark Models watch game clips and decide when to speak. They narrate alone, join another commentator, or guide a player. A judge scores their commentary. Everyday Mar 2026 28.4% Not in cards
LVOmniBench Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track. Everyday Mar 2026 65.8% In use
ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. Everyday Mar 2026 Not in cards
MDPBench Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage. Everyday Mar 2026 86.4 Not in cards
MediaPerf Models tag and summarize video advertisements. The benchmark measures tag accuracy, summary quality, cost and throughput. University Mar 2026 0.42 Not in cards
MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. Everyday Mar 2026 46.8% System cards only
OmniGUI Models use screenshots, sound, and video to predict the next action in 29 phone apps. The grader compares each action with a human demonstration. Everyday Mar 2026 65.4% Not in cards
OmniSch Models convert circuit schematics into graphs of components, pins, and connections. A second task lets models search schematic images for individual features. University Mar 2026 0.28 Not in cards
PerceptionComp 1,114 five-choice questions on 279 complex videos. Each question needs evidence from several separate moments, so no single frame answers it. Everyday Mar 2026 39.4% System cards only
PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. Everyday Mar 2026 75.1% Not in cards
Real5-OmniDocBench Models parse the same 1,355 document pages under five physical capture conditions. Text edit distance, formula matching, and table similarity determine the overall score. Expert Mar 2026 93.7% In use
RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. Everyday Mar 2026 10.4% Not in cards
RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. Everyday Mar 2026 23.9% Not in cards
RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. Everyday Mar 2026 0% Not in cards
RVMS-Bench Agents search open-domain videos and localize moments using global impression, key moment, temporal context and auditory memory cues. Expert Mar 2026 Not in cards
TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. Everyday Mar 2026 0.92 Not in cards
TextEdit Real Models edit text in real images in the full TextEdit test set. The score is the published judged average. Everyday Mar 2026 0.90 Not in cards
TextEdit Virtual Models edit text in virtual images in the full TextEdit test set. The score is the published judged average. Everyday Mar 2026 0.90 Not in cards
VAREX-Bench Models extract structured fields from 1,777 government forms using a supplied JSON schema. Normalized exact match checks field values, while string similarity gives partial credit. Array entries use order-independent matching. Everyday Mar 2026 97.3% Not in cards
Video-Oasis A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions. Everyday Mar 2026 34.6% Not in cards
Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Mar 2026 82.3% In use
Vision2Web S1-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Mar 2026 52.7% Not in cards
Vision2Web S2-2026 Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. Expert Mar 2026 43.5% Not in cards
WebVR Models recreate a webpage from a screen recording and supplied assets. A model judge scores the rendered page against visual and interaction criteria. University Mar 2026 79.1 Not in cards
Wild-OmniDocBench Models parse 1,350 photographed document pages with geometric distortion, lighting changes, and other capture artifacts. OmniDocBench metrics score text, formulas, tables, and reading order. Expert Mar 2026 92.4% In use
Accessible Structured Subtitle Generation 2026 Systems produce timed Chinese subtitles for deaf and hard-of-hearing viewers. The structured track also scores speaker labels, key information, and readability. University Feb 2026 Not in cards
BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. Expert Feb 2026 52.3% Fading
CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. Everyday Feb 2026 Not in cards
DETOUR Agents ask a simulated user for clues and identify a partially remembered item. Everyday Feb 2026 Not in cards
DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. Everyday Feb 2026 28.7% Not in cards
HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Expert Feb 2026 48.2% In use
Human SxS image editing Elo (Google) Google compares edited images side by side and converts human preferences into Elo ratings. Everyday Feb 2026 1,065 Fading
HY3D-Bench Models generate textured 3D meshes from text and images, with semantic and geometric quality evaluated across categories. Expert Feb 2026 Not in cards
Korean Dental Licensing Examination 2024 and 2025 Models answer 642 Korean dental licensing questions from two new exams. The authors score accuracy and break it down by subject and image type. Expert Feb 2026 81.5% Not in cards
KorMedMCQA-Mixed Models take combined text and image exams. The 2022 set contains 281 questions and the 2023 set contains 307. Each year has a separate score. Expert Feb 2026 97.3% Not in cards
KorMedMCQA-V Models answer Korean medical licensing questions using clinical images. The full test contains 1,534 questions and 2,043 images from 2012 through 2023. Expert Feb 2026 96.9% Not in cards
LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. Expert Feb 2026 60.9% In use
LABBench2 FigQA2 Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2. Expert Feb 2026 42.6% Not in cards
LABBench2 FigQA2 (image) Models answer research questions about figures supplied as images. Expert Feb 2026 66.3% Not in cards
LABBench2 TableQA2 (image) Models answer research questions about tables supplied as images. Expert Feb 2026 95% Not in cards
Life-Bench Models answer multimodal questions about synthetic personal lifelogs. Tasks cover concept identification, event understanding, and reasoning over multiple pieces of evidence. Expert Feb 2026 56% Not in cards
MarsRetrieval Models match Mars images to captions, retrieve examples of named landforms, and locate features in a global mosaic of the planet. University Feb 2026 26.5 Not in cards
MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. Everyday Feb 2026 37.7% Fading
OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. University Feb 2026 62.5% Fading
OpenEarthAgent Agents use tools such as QGIS to measure features in satellite images and maps. The authors grade tool decisions and final answers. University Feb 2026 44.5% Not in cards
Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. Expert Feb 2026 86.2% System cards only
Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. Expert Feb 2026 79.6% System cards only
RefAdv-S Models locate the object a referring expression describes among hard distractors in real images. Scored by accuracy at 0.5 box overlap on 1,142 cases. Everyday Feb 2026 68% In use
SA-1B Hard human pose evaluation Models locate body and foot joints in photographs with unusual viewpoints, poses, and occlusions. Meta measures the fraction of correctly placed joints. School Feb 2026 Not in cards
SAM 3D Body synthetic pose evaluation Models estimate human pose and shape from synthetic images of individual people. Meta compares predicted joints and mesh vertices with the rendering parameters. School Feb 2026 Not in cards
Snowflake Financial PDF Table Parsing Models reconstruct tables from 50 manually annotated financial documents. Evaluators compare complete tables, table dimensions and individual cells with reference Markdown. University Feb 2026 Not in cards
SPM-Bench Models interpret microscopy images and select all correct answers. The primary score gives partial credit and penalizes incorrect selections. Expert Feb 2026 0.94 Not in cards
VBVR-Bench Video generation models solve visual reasoning tasks by generating a video from a starting image and instruction. Rule-based scorers check the result instead of a judge model. Tasks cover abstraction, knowledge, perception, spatial reasoning and transformation, with in-domain and out-of-domain splits. Everyday Feb 2026 0.56 Not in cards
VBVR-Pro The current official results table evaluates the Pro release. Original VBVR results must remain separate. Everyday Feb 2026 0.56 Not in cards
VDR-Bench Models answer 2,000 visual questions that require finding entities in images and researching related facts. A Qwen3-VL judge checks final answers against references. Entity recall separately measures whether searches retrieve the relevant entities. University Feb 2026 27.2% Not in cards
WorldArena Embodied world models generate RoboTwin 2.0 task videos from first frame plus text or actions. 15 video metrics (EWMScore) plus utility tracks. Feb 2026 63.1% Not in cards
WTP-Bench Models identify Pokémon from official artwork and silhouettes across 1,160 entries. The scorer checks exact forms, species, and evolutionary families. School Feb 2026 79% Not in cards
Artificial Analysis Intelligence Index v4.0 Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests. Jan 2026 60.2 Fading
AVMeme Exam Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge. Everyday Jan 2026 69.5% Fading
BabyVision Model solves 388 perception-heavy visual puzzles (discrimination, tracking, spatial, patterns) that young children solve. Accuracy versus 94% human baseline. Everyday Jan 2026 91.3% In use
CT Dataset 1 Models classify seven conditions on CT scans of the head, chest, and abdomen. Google balances the examples for each condition and reports macro accuracy. Expert Jan 2026 Not in cards
DatBench Models answer visual questions drawn from 33 datasets. The suite reports separate scores across nine visual capabilities. University Jan 2026 64% Not in cards
FutureOmni Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together. Everyday Jan 2026 66.7% Fading
LLM Benchmark: Danske Idiomer Models match Danish idioms to pictures, complete expressions and write definitions. GPT-4.1 grades definitions, and answer keys cover the other tasks. Everyday Jan 2026 97.5% Not in cards
LMArena Image-to-Video Arena Video generation models animate the same input image with the same prompt, anonymously and side by side. Votes give Elo-style ratings. Everyday Jan 2026 1,503 Not in cards
MedGenVidQA 2026 Visual Answer Localization Models mark the start and end of the video segment that answers a medical question. The organizers created 80 test examples from professional medical videos. University Jan 2026 Not in cards
Mem-Gallery Multimodal agents hold long conversations over many sessions that mix images and text, then answer questions that need what they remembered. Questions test memory extraction and test-time adaptation, reasoning over memory, and managing knowledge that changes over time. Everyday Jan 2026 Not in cards
MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. Expert Jan 2026 42.2% Not in cards
MRI Dataset 1 Models classify ten conditions on MRI scans of the head, abdomen, and knee. Google compares macro accuracy on balanced samples. Expert Jan 2026 Not in cards
NarrativeTrack Video-language models answer entity-centric questions about existence, changes and ambiguity as narratives unfold across clips. Expert Jan 2026 Not in cards
OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. Expert Jan 2026 81% Not in cards
PaperScope Agents search multiple scientific papers to compare figures, summarize results, and solve problems. Each question can require evidence from several documents. Expert Jan 2026 41.0 Not in cards
Progress-Bench Vision-language models estimate how far a task has progressed from partial observations, such as frames of a robot or human doing a job. The benchmark varies how the demonstration is shown and the camera viewpoint, and it includes cases with no valid answer. Everyday Jan 2026 20.9 Not in cards
RBench 650 image+text prompts for robot video generation (tasks and embodiments). MLLM judge scores task completion, physical plausibility, stability. Jan 2026 60.7% System cards only
RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. Everyday Jan 2026 76.3% Not in cards
RoboRewardBench Vision-language models act as reward models on real robot videos from Open X-Embodiment and RoboArena, judging task success and partial progress. Counterfactual relabeling adds calibrated failures and near misses. Everyday Jan 2026 0.63 Not in cards
VDEBench Image editing models make local text and table edits to 3,506 Chinese and English document images. Scores check the edit and that the rest of the page is preserved. Everyday Jan 2026 0.77 Not in cards
Video Browser Showdown 2026 marine video search Participants use interactive retrieval systems to find five underwater clips after seeing each target. Each search has a five-minute limit. Everyday Jan 2026 Not in cards
VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. University Jan 2026 76% In use
WorldVQA Model names visual entities in 3,500 image-question pairs across nine categories, isolating memorized visual world knowledge. Exact-answer accuracy. University Jan 2026 61.1% In use
WSI-Path Models generate a diagnosis from a whole slide pathology image. Google compares each answer with the final diagnosis in the original report using ROUGE. Expert Jan 2026 Not in cards
Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. Expert 2026 Not in cards
AECV-Bench (2026 release) Models count four types of floor plan elements and answer questions about engineering drawings. Counting uses exact matches. A model grades the drawing answers. University Dec 2025 71.4% Not in cards
AMUSE Multimodal models reason over multi-speaker audio-visual interactions in zero-shot, guided and agentic modes across six task families. Expert Dec 2025 Not in cards
AV-SpeakerBench Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together. Everyday Dec 2025 77.6% Fading
BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. Competition Dec 2025 53.5% Fading
CADExpert Models write CadQuery programs from dimensioned engineering drawings. The graders check whether each program runs and compare its 3D shape with the reference. University Dec 2025 35.6% Not in cards
CAIS Vision Capabilities Index Averages model scores on five vision tests run by the Center for AI Safety: ERQA, MindCube, SpatialViz-Bench, IntPhys 2 and EnigmaEval. Research frontier Dec 2025 84.0 Not in cards
CrossPoint-Bench Models see two views of a scene and must find the point in the second view that matches a point in the first. Tasks go from perception and reasoning to exact coordinate prediction. Everyday Dec 2025 72.9% Not in cards
CubeBench Agents solve Rubik's Cube tasks across symbolic, visual and partially observed settings to measure spatial reasoning and long-horizon state tracking. Expert Dec 2025 Not in cards
FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. University Dec 2025 78.3% Fading
FACTS Multimodal Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions. University Dec 2025 60.2% Not in cards
JointAVBench Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans. Everyday Dec 2025 58% Fading
LongShOTBench Models answer single-turn and multi-turn questions about long videos using visual, spoken, and ambient audio evidence. Weighted rubrics award partial credit. Everyday Dec 2025 64.2% Not in cards
MMSI-Video-Bench Human-annotated questions on video-based spatial intelligence, organized in four levels from perception to planning and prediction. Everyday Dec 2025 38% System cards only
Multimodal RewardBench 2 Reward models compare two outputs containing text and images. Accuracy measures agreement with expert preferences across four tasks. University Dec 2025 76.3% Not in cards
OSI-Bench Spatial reasoning questions on outdoor pedestrian-view videos with metric 3D ground truth from synchronized stereo cameras, LiDAR and GPS. 8,736 questions range from relative direction to metric and motion estimates. Everyday Dec 2025 30.6% System cards only
PE-AV internal video-to-audio retrieval Models match videos to music tracks. Meta compares retrieval accuracy on internal clips that last from five to thirty seconds. Everyday Dec 2025 Not in cards
RealKIE-FCC Verified Model extracts structured fields from scanned FCC invoices into a JSON schema. Amazon human-verified labels. Field-level similarity (ALNS). Everyday Dec 2025 67% Fading
REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. Everyday Dec 2025 53.1% Not in cards
SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. Everyday Dec 2025 57.8% Not in cards
SAM Audio-Bench Models isolate a requested sound from an audio mixture using text, visual, or temporal prompts. Listeners judge how well the output preserves the target. Everyday Dec 2025 Not in cards
SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. Expert Dec 2025 33.8% In use
SO-Bench Multimodal models produce schema-compliant structured outputs grounded in images from four visual domains. Static checks and rubric-based grading score semantic correctness and schema compliance. University Dec 2025 Not in cards
SpatialTree Organizes spatial abilities into a hierarchy, from low-level perception to mental mapping, simulation and agentic use, and tests vision-language models at each level. Everyday Dec 2025 58.1% System cards only
SUPERChem Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. Expert Dec 2025 66.4% In use
TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. Everyday Dec 2025 62.9% Fading
TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. Everyday Dec 2025 Not in cards
Video Reality Test Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection. Everyday Dec 2025 76.3% Not in cards
VIST Image models generate a new image in a reference brand style. Human evaluators compare pairs of outputs. University Dec 2025 Not in cards
ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. Expert Dec 2025 14.5% System cards only
ArtEdit-Bench-Eval Models rate photo edits against English instructions. Spearman and Pearson correlations measure agreement with human ratings on 200 cases. Everyday Nov 2025 0.62 Not in cards
ArtEdit-Bench-Lr Models retouch photos from English or Chinese instructions. GPT-4o scores instruction following and visual quality. Pixel errors measure fidelity to reference edits. University Nov 2025 8.62 Not in cards
CrochetBench Models identify stitches, select instructions that match a photo, write crochet patterns, and translate patterns into programs for a crochet simulator. University Nov 2025 60.9 Not in cards
CrossVid Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats. Everyday Nov 2025 45.2% Fading
DravidianLangTech 2026 Multilevel Political Meme Classification in Tamil and Malayalam Models identify a political meme's stance and target from its image and text in Tamil or Malayalam. Everyday Nov 2025 Not in cards
EvaHan 2026 Models transcribe printed and handwritten Chinese books and locate page elements such as illustrations and annotations. University Nov 2025 Not in cards
GEO-Bench-2 Models interpret Earth imagery across 19 tasks, including field boundaries, forest biomass and tree species. The authors normalize task scores to compare model capabilities. University Nov 2025 Not in cards
MarineEval Models answer 2,000 questions about marine images and documents. The authors combine answer checks with model judgments. They report total accuracy and mean accuracy across seven tasks. University Nov 2025 50.4% Not in cards
MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. School Nov 2025 63.8% System cards only
MME-CC Models answer 1,173 expert-annotated vision-grounded questions in 11 task types grouped as spatial, geometric and visual knowledge reasoning. Everyday Nov 2025 40.3% Fading
Multi-Crit Multimodal judges compare responses under several criteria. Human judgments provide the reference for each criterion and reveal trade-offs between criteria. University Nov 2025 43% Not in cards
NeuroDiscoveryBench Models analyze neuroscience, genetics and immunology datasets and answer questions grounded in figures and data workflows. Expert Nov 2025 Not in cards
OCR Arena Users upload a document, compare two anonymous OCR or vision-language models side by side, and vote. Votes become Elo ratings. Everyday Nov 2025 1,781 Not in cards
Qualcomm Interactive Cooking Models watch a cook follow a recipe. They give timed instructions, recognize completed steps, and warn about mistakes. The benchmark scores completion accuracy and mistake detection. Everyday Nov 2025 23.1% Not in cards
SAM 3D Artist Objects Models reconstruct an object's shape and placement from a single photograph. The benchmark compares predictions with meshes created by 3D artists. School Nov 2025 Not in cards
SAM 3D Objects human preference evaluation Models reconstruct objects from images of real scenes. People compare the shape and texture of competing reconstructions, including objects seen from an egocentric viewpoint. Everyday Nov 2025 Not in cards
Segment Anything with Concepts Models find every instance of a concept named in a prompt. The benchmark compares image masks and video tracks with human annotations. Everyday Nov 2025 Not in cards
Table Judge Benchmark Vision language models judge whether rendered table images match candidate HTML. Controlled corruptions test false positives, true positives and injected-error detection. Expert Nov 2025 94.9% Not in cards
TIR-Bench Models solve 13 kinds of image tasks that need tool use on images, such as rotation, jigsaw, and maze. Most items are multiple choice. Everyday Nov 2025 61.6% Fading
UniREditBench Image editing models carry out 2,700 edits that need reasoning, in real-world scenes and in game worlds with human-defined rules, across 8 primary and 18 secondary dimensions. A multimodal judge scores each edit against both a text reference and a ground-truth image. Everyday Nov 2025 57% Not in cards
VSI-Bench-Debiased Models answer the 2,362 VSI-Bench questions left after removing 2,768 that text-only models could solve from answer priors or world knowledge. Everyday Nov 2025 41.5% Not in cards
VSI-SUPER Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours. Everyday Nov 2025 Not in cards
ADQA Models describe film scenes. An evaluator answers questions from the descriptions and dialogue, then checks whether the context supports each correct answer. Everyday Oct 2025 Not in cards
AGI Score (A Definition of AGI) Experts test models on ten cognitive domains from Cattell-Horn-Carroll theory, such as reasoning, memory and perception, with adapted human psychometric tests. Each domain counts 10% toward a score out of 100%. Expert Oct 2025 57% Not in cards
BlueprintSymVL Models use an example symbol to find matching components in piping and instrumentation diagrams. They must give both the correct count and the component labels. School Oct 2025 Not in cards
Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. Everyday Oct 2025 37% Not in cards
CRAG-MM Multimodal retrieval agents answer single-turn and multi-turn questions grounded in images and changing web content. The benchmark includes wearable-style egocentric images. Expert Oct 2025 Not in cards
Dense 200 (bbox) Model detects and boxes every object in 200 densely annotated images (109 categories, about 91 boxes each). Box F1 at IoU thresholds. Everyday Oct 2025 76.9% In use
DSI-Bench Video questions about dynamic scenes where both the observer and objects move, testing whether models separate camera motion from object motion. Everyday Oct 2025 46.9% System cards only
FenBench Models identify chess pieces, locate them, count them or describe a board position in Forsyth-Edwards Notation. The test contains 200 examples. Everyday Oct 2025 Not in cards
HLE-Rolling A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log. Research frontier Oct 2025 Not in cards
LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. Everyday Oct 2025 69.6% Not in cards
LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. Everyday Oct 2025 53% Not in cards
Math Kangaroo 2025 (MathArena) Vision language models answer 168 multiple-choice problems from the 2025 Math Kangaroo, given as screenshots across six grade groups. Many problems need reading a diagram. School Oct 2025 94.9% Fading
Math-VR Models answer 5,000 English and Chinese math questions whose solutions need visual reasoning, such as drawing auxiliary lines or plotting functions. Answers are checked against ground truth. School Oct 2025 66.8% System cards only
MathCanvas-Bench Models solve 3,000 math problems that call for interleaved visual and textual solutions, such as drawing auxiliary lines in geometry. School Oct 2025 44.1% Fading
MatSciBench Models solve college materials science problems, including questions with figures. The authors report accuracy separately for text and image questions. University Oct 2025 64% Not in cards
MeasureBench Vision-language models read values from images of measuring instruments, such as gauges and rulers, in real and synthesized photos. Scored by matching the reading to the key. Everyday Oct 2025 62.9% In use
MeasureBench Real-world Models read values and units from real instrument images. The score is the published overall measurement accuracy. Everyday Oct 2025 41.7% Not in cards
MeasureBench Synthetic Models read values and units from generated instrument images. The score is the published overall measurement accuracy. Everyday Oct 2025 39.2% Not in cards
MM-HELIX Multimodal models solve 1,260 items from 42 synthetic tasks that need long reflective reasoning with backtracking. University Oct 2025 84.5% Not in cards
NautData test set Models answer questions, describe underwater scenes, and locate organisms in images. The paper reports 7,916 test examples on 3,920 images. University Oct 2025 91.9% Not in cards
Omni-Cloze Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail. Everyday Oct 2025 64.8% Fading
OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. University Oct 2025 61.8% Fading
PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. Everyday Oct 2025 61.9% Not in cards
PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. School Oct 2025 63.3% Not in cards
Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. Everyday Oct 2025 1.37 System cards only
RefSpatial-Expand-Bench Models point to an object or free space described by multi-step spatial relations, in more indoor scenes plus new outdoor streets and parking lots. Scored by mask hit. Everyday Oct 2025 Not in cards
RefSpatial-Expand-Bench Location Models solve spatial location questions in the RefSpatial expansion set. The score is the published overall accuracy. Everyday Oct 2025 Not in cards
RefSpatial-Expand-Bench Placement Models solve spatial placement questions in the RefSpatial expansion set. The score is the published overall accuracy. Everyday Oct 2025 Not in cards
RoboChallenge Fine-tuned policies run 10 real rollouts on each of 30 tabletop tasks across four robot types. Scored by success rate and progress. Everyday Oct 2025 42.7% Not in cards
SPBench Spatial perception and reasoning questions built from ScanNet scenes. A single-image set covers distance, size and direction, and a multi-view set adds counting across views. Answers are numbers or multiple choice. Everyday Oct 2025 63.8% System cards only
SuperCLUE-Edit Models edit an input image according to a Chinese instruction. The evaluation measures whether the requested change preserves relevant details. Everyday Oct 2025 82.5 Not in cards
UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. Everyday Oct 2025 76.5% Dropped
USIM underwater robot tasks Agents follow instructions to navigate, grasp, transport, and track objects underwater. The simulator measures task completion across 700 trials. Everyday Oct 2025 37.6% Not in cards
UWBench Models answer questions about underwater scenes, write captions, and locate objects from descriptions. Questions cover visibility, materials, and spatial relationships. Everyday Oct 2025 Not in cards
Vals Multimodal Index v1 Models complete finance, coding, and education tasks that contribute to a weighted index. Expert Oct 2025 71.1% Not in cards
VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. University Oct 2025 44.8% Not in cards
ViVerBench Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX. School Oct 2025 74.4% System cards only
World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. Everyday Oct 2025 64.8% Not in cards
Arti4D Models watch first-person recordings of people opening furniture. They predict whether parts rotate or slide and estimate the axis of motion. School Sep 2025 Not in cards
ASIMOV Models spot physical hazards in text and generated videos from real injury reports, time safety interventions in video, and follow robot payload and handling constraints. Sep 2025 97.9% Fading
BEHAVIOR Challenge 2025 50 long-horizon household activities on a Galaxea R1 Pro in OmniGibson, trained from 10,000 demos. Scored by Q-score and success. Everyday Sep 2025 Fading
BioML-bench Agents build and evaluate machine learning systems for biomedical experiments spanning proteins, drugs, single cells, imaging, and biomarkers. Task-specific metrics compare submissions with experimental reference results. Expert Sep 2025 Not in cards
Blueprint-Bench Models turn about 20 interior photos of each of 50 apartments into a 2D floor plan. Scored by similarity of room connectivity and size ranking to the true plan. Everyday Sep 2025 0.43 Not in cards
CircuitSense Models read circuit schematics and block diagrams, derive equations, and select designs. SymPy checks symbolic answers. Other tasks use exact matches, a model judge, or simulation. University Sep 2025 68.9% Not in cards
ClockBench Models read analog clock faces and answer questions about the time shown. Most clocks stay private to avoid training leakage. Everyday Sep 2025 87.8% Not in cards
CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. Expert Sep 2025 55.3% System cards only
Earth-Bench Agents analyze satellite observations with Earth science tools. The evaluator checks the final answer and compares tool calls with a reference sequence. University Sep 2025 66% Not in cards
Ego3D-Bench Models answer spatial reasoning questions about ego-centric multi-view scenes, such as those from self-driving cars. Some items ask for distances, scored by error. Everyday Sep 2025 7.40 In use
GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Expert Sep 2025 87.9% In use
GSM8K-V Vision-language models solve GSM8K word problems that are redrawn as multi-image visual scenes, so the facts must be read from pictures. School Sep 2025 59.1% Not in cards
HiPhO Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. Competition Sep 2025 78.3% In use
LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. Competition Sep 2025 Not in cards
MapAnything dense multi-view reconstruction Models reconstruct a scene from multiple images. The benchmark compares point geometry, metric scale, depth, and camera poses with reference measurements. School Sep 2025 Not in cards
MapAnything single-view calibration Models infer camera intrinsics from a single image. The benchmark measures angular error between predicted camera rays and reference rays. School Sep 2025 Not in cards
MatCha Models answer multiple-choice questions about material structure, shape and properties using images from materials characterization instruments. Expert Sep 2025 68.1% Not in cards
MulSeT Models answer spatial questions about direction, distance and masks across views. The paper studies single images, multiple views and video. University Sep 2025 36.7% Not in cards
OceanGym Agents interpret underwater images and sonar, then navigate simulated search and inspection tasks. The simulator scores how closely they reach target locations. University Sep 2025 14.6 Not in cards
ODV-Bench Streaming video models answer real-time questions about autonomous driving footage. Everyday Sep 2025 55.6% Fading
Radiology's Last Exam (RadLE) Multimodal models make a single spot diagnosis from images alone on 50 hard radiology cases. Board-certified radiologists scored about 83%. Expert Sep 2025 30% Not in cards
ROME Models solve reasoning problems selected to limit contamination. University Sep 2025 69.4 Not in cards
Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. Everyday Sep 2025 90% Fading
Synthetic Sheet Music Reasoning Benchmark Models answer questions about rhythm, chords, intervals, and scales. Each test question appears in text notation and as an image. School Sep 2025 90.5% Not in cards
VideoScore-Bench v2 Models score 500 generated videos on visual quality, agreement with the prompt, and physical consistency. The evaluation checks their ratings against human scores. Everyday Sep 2025 28.9% Not in cards
Visual Reasoning Benchmark (Fab AI) Models choose which picture completes a pattern, matches another shape, or shows a folded figure. Questions come from school exams in Zambia and India. School Sep 2025 89.9% Not in cards
VisualProbe Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key. Everyday Sep 2025 65% Fading
VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. Everyday Sep 2025 72.1% Not in cards
VoxelBench Language models turn text or image prompts into voxel structures. Voters compare anonymous builds from the same prompt, and Glicko-2 ratings rank the models. Everyday Sep 2025 2,607 Not in cards
VoxelBench Image Models turn image prompts into voxel structures. Human comparisons determine Glicko-2 ratings in a separate image-prompt vote pool. Everyday Sep 2025 2,215 Not in cards
Arctic-Extract Business Document Benchmark Models answer questions about business documents and extract information from varied layouts. The internal visual tasks include contracts, signatures and checkboxes. University Aug 2025 58.5 Not in cards
Artificial Analysis Intelligence Index v2.2 Version 2.2 of Artificial Analysis's index, which swapped the older math tests for AIME 2025 and added instruction following and long-context reasoning. Aug 2025 Not in cards
BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. Expert Aug 2025 59% Fading
CountQA Models count objects in over 1,500 real-world photos with dense clutter and occlusion. Scoring is accuracy on the count. Everyday Aug 2025 42.9% In use
DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. Everyday Aug 2025 39% Not in cards
EgoCross Models answer questions about events in first-person videos. The challenge scores accuracy on 957 four-choice questions across four domains. Everyday Aug 2025 Not in cards
EncQA Vision-language models answer chart questions that isolate visual encoding channels and analytic operations. University Aug 2025 Not in cards
HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. Expert Aug 2025 16% Fading
KRETA Models read and reason over Korean text in images across diverse visual contexts. It is a text-rich visual question answering test for Korean. Everyday Aug 2025 85.4% Fading
M3-Bench Multimodal agents answer questions about long videos while using long-term memory and multimodal retrieval. Expert Aug 2025 Not in cards
MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. Expert Aug 2025 24.3% Fading
MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. Expert Aug 2025 41.4% Fading
MOSEv2 Models track and segment objects in videos with complex scenes. Objects may disappear, reappear, or blend into a crowd. Everyday Aug 2025 Not in cards
RQA-70K weather radar forecast evaluation Models compare forecast radar images or sequences with observations. They rate errors such as missed storms and false alarms and write assessments of forecast quality. University Aug 2025 52.7% Not in cards
SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. Everyday Aug 2025 0.92 Fading
SatLidar canopy height evaluation Models predict canopy height from satellite images. Meta compares height estimates with airborne lidar measurements and reports errors on held-out regions. University Aug 2025 Not in cards
WildScore Models read score excerpts from Reddit music theory discussions and choose among answers drawn from the comments. University Aug 2025 75% Not in cards
AgroBench Models answer illustrated questions from agronomists about crops, diseases, pests, weeds, machinery and farming practices. The overall score averages accuracy across seven tasks. University Jul 2025 73.5% Not in cards
EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. Expert Jul 2025 Not in cards
Fundus-MMBench Models identify retinal structures, classify diseases, and grade their severity in fundus photographs. Each of the 31 tasks contains 20 questions. Expert Jul 2025 Not in cards
MMCircuitEval Models answer questions about circuit diagrams, specifications, and design decisions. The authors combine text similarity with a language model’s judgment. University Jul 2025 69.4% Not in cards
MWAHAHA Models write jokes from word pairs or news headlines in English, Spanish, and Chinese. They also caption animated GIFs in English. Everyday Jul 2025 Not in cards
OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. Everyday Jul 2025 53.4% Not in cards
PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. Expert Jul 2025 Not in cards
RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. Everyday Jul 2025 44.3% Fading
SciVid Video models answer questions about scientific videos across several disciplines. The benchmark tests whether models recognize procedures, objects, and events in specialized footage. University Jul 2025 Not in cards
SF20K Competition 2025 Models answer story questions about short films. GPT-4.1-nano compares their answers with human references. Films released during the competition determine the final rankings. Everyday Jul 2025 Not in cards
SpatialViz-Bench Models answer 1,180 generated multiple choice puzzles on mental rotation, mental folding, visual penetration, and mental animation. Scoring is accuracy. Everyday Jul 2025 41.4% System cards only
TimeScope Models answer questions about short clips inserted into videos lasting up to eight hours. Accuracy measures retrieval, synthesis and temporal perception. Everyday Jul 2025 97.3% Not in cards
TreeBench Models answer 405 visual questions about small targets in dense SA-1B images and give bounding boxes as evidence. Scored by answer accuracy, with box overlap as a second measure. University Jul 2025 71.1% Fading
US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. Expert Jul 2025 Not in cards
34th Japanese National Examination for Dental Hygienists Models answer 213 Japanese dental hygiene questions, including 139 with visual material such as instrument photographs and radiographs. University Jun 2025 Not in cards
Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. Everyday Jun 2025 64.7% Fading
Argus Inspection Models inspect images for small details that change the sensible response to a question. A judge scores whether the response identifies the relevant detail and chooses an appropriate action. The score combines those judgments and assigns zero when a separate description fails to recognize the detail. Everyday Jun 2025 0.44 Not in cards
BM-Bench Image editing models perform instruction-guided non-rigid image manipulations on the ByteMorph benchmark dataset. Expert Jun 2025 Not in cards
CausalVQA Models answer questions about real egocentric videos that probe physical cause and effect, in counterfactual, hypothetical, anticipation, planning and descriptive types. Everyday Jun 2025 70% System cards only
FieldWorkArena V2.0 Agents inspect workplace images, videos, manuals, and logs to plan actions and report operational events. The evaluator compares their answers and actions with reference results. It combines correctness judgments with partial credit for numerical agreement. University Jun 2025 52% Not in cards
Fire360 Models interpret panoramic video from firefighter training drills. They answer questions, caption actions, assess procedures, locate equipment, and match damaged objects to clean examples. Expert Jun 2025 53.8% Not in cards
IntPhys 2 Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy. Everyday Jun 2025 68.3% In use
K12-PEBench Models identify incorrect reasoning steps in solutions to school problems. The evaluator compares their decisions with human annotations for 3,033 examples. School Jun 2025 63.4% Not in cards
K12Vista Models solve Chinese primary and secondary school problems with text and images. The benchmark covers five subjects and includes multiple-choice, fill-in, and written answers. School Jun 2025 55.4% Not in cards
Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. Everyday Jun 2025 Not in cards
LMArena Image Edit Arena Image editing models apply the same user edit request to the same image, anonymously and side by side. Community votes give Elo-style ratings. Everyday Jun 2025 1,519 Not in cards
MindCube Models answer 21,154 multiple choice questions about scenes seen from a few views. Questions test positions, orientations, and what-if movements. Everyday Jun 2025 47.6% Fading
MIRAGE-MMMT Models decide whether to ask for more information or answer a grower during an agricultural consultation. They then write the question or answer. University Jun 2025 65.5% Not in cards
MIRAGE-MMST Models answer growers’ questions using photographs and written descriptions. They identify the cause of a problem and recommend how to manage it. University Jun 2025 68.0 Not in cards
MMArt-Bench Models retouch 200 photos from editing instructions. Gemini scores visual quality and instruction following, and image comparisons measure how closely edits match reference photos. University Jun 2025 8.34 Not in cards
MORSE-500 Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning. Everyday Jun 2025 23.6% Fading
MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. University Jun 2025 33.5 Not in cards
MusiXQA Models read printed music and answer questions about notes, chords, text, and page layout. The authors measure answer accuracy and string similarity. School Jun 2025 12.8% Not in cards
MVPBench Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail. Everyday Jun 2025 72.5% Dropped
OmniSpatial Vision language models answer over 8,400 spatial reasoning questions in four groups: dynamic reasoning, complex spatial logic, spatial interaction and perspective taking. Everyday Jun 2025 58.6% Fading
OpenLMM Spatial Leaderboard Models answer spatial reasoning tasks involving images. OpenCompass compares performance across the component evaluations. University Jun 2025 62.5 Not in cards
PST50 Models transfer a reference look to a photograph. The paired test compares 50 outputs with reference grades that professionals refined in DaVinci Resolve. University Jun 2025 Not in cards
PuzzleWorld Models solve 667 puzzle-hunt problems that require varied reasoning strategies. Competition Jun 2025 18% Not in cards
RefSpatial-Bench Real images where the model points to an object or free space defined by multi-step spatial relations. Scored by mask hit. Everyday Jun 2025 73.6% Fading
RoboArena Double-blind pairwise A/B trials of generalist DROID policies on real Franka arms, free-form tasks by volunteer evaluators. Elo-style rating. Everyday Jun 2025 1,608 Not in cards
RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. Everyday Jun 2025 58.3% Not in cards
Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. Expert Jun 2025 60.7% Fading
STARE Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation. Everyday Jun 2025 61.8% System cards only
V-JEPA 2 zero-shot robot manipulation Robots use a goal image to reach, grasp, or place an object. Meta measures task success on physical robot arms in two laboratories. Everyday Jun 2025 Not in cards
Visual Strategic Bench Models identify objects, predict another player’s next action, and choose their own moves in ten games. The authors score each ability separately. Everyday Jun 2025 84.9% Not in cards
VRBench Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps. Everyday Jun 2025 68.2% Not in cards
WildKhmerST Models find and read Khmer text in photographs of public places. Annotations mark text lines with polygons and transcriptions. Everyday Jun 2025 Not in cards
AgroMind (original) Models answer questions about farmland images across 13 tasks. They identify crops, assess crop health and interpret spatial relationships. University May 2025 35.8% Not in cards
ChartMuseum Models answer 1,162 expert-annotated questions about real-world charts from 184 sources. Many questions need visual reasoning that text cannot replace. Expert May 2025 80.7% Fading
CoralVQA Models answer questions about coral identity, growth, and health in reef photographs. GPT-4o checks answers against references across 16 question types. University May 2025 41.8% Not in cards
CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. School May 2025 49.6% System cards only
Daily-Omni Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy. Everyday May 2025 85.1% Fading
Do You See Me Models answer 2,612 visual perception questions on 1,758 images across seven 2D and 3D subtasks drawn from human psychology. Questions are multiple choice or numeric. Everyday May 2025 51.1% System cards only
DreamGen Bench Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline. May 2025 Not in cards
EmotionTalk Models infer emotions and sentiment from Chinese conversations. Everyday May 2025 50.9 Not in cards
General-Bench Models perform multimodal understanding and generation tasks. General-Level averages task scores and measures performance above specialist baselines at progressively broader levels. University May 2025 8.41 Not in cards
GIE-Bench Image-editing models follow text instructions while preserving unedited regions. GPT-4o question answers and masked image metrics score functional correctness and preservation. University May 2025 Not in cards
ImgEdit-Bench Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5. Everyday May 2025 4.12 System cards only
IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. Everyday May 2025 55.3 Dropped
KORGym Models play 51 text and visual games designed to test interactive reasoning. Everyday May 2025 0.82 Not in cards
KRIS-Bench Image editing models follow 1,267 instructions that need factual, conceptual or procedural knowledge across 22 tasks. A GPT-4o judge scores the edits. School May 2025 66.2 System cards only
LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. Everyday May 2025 72.4% Fading
LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. Everyday May 2025 72.5% Dropped
MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. University May 2025 70.7% Fading
MME-Reasoning Multimodal models answer logic questions grouped into inductive, deductive and abductive reasoning. University May 2025 59.9% Not in cards
MME-VideoOCR Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning. Everyday May 2025 73.7% Fading
MMEB-V2 Multimodal embedding models are scored on 78 tasks over images, videos and visual documents. V2 adds visual document retrieval, video retrieval, temporal grounding, video classification and video QA. Everyday May 2025 77.9% Fading
MMLongBench Vision-language models solve 13,331 long-context examples in five task types, such as visual RAG, needle search and many-shot in-context learning. Each example comes at five lengths from 8K to 128K tokens. Everyday May 2025 78.3% Fading
MMSI-Bench Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data. Everyday May 2025 43.1% Fading
MSEarth Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split. Expert May 2025 68.8% Fading
OCR-Reasoning Models answer 1,069 human-annotated questions that need reasoning over text-rich images. The questions cover 6 reasoning abilities and 18 practical tasks. Everyday May 2025 44.4% Fading
OmniEarth-Bench Models interpret satellite images and observational data across Earth systems, answering questions about environmental processes and their interactions. Multiple-choice accuracy and model judgments score answers. Separate measures check localization, captions, and annotated reasoning steps. Expert May 2025 31.5% Not in cards
PhyX Models solve 3,000 image-based physics problems across six domains such as mechanics and optics. Versions include multiple choice and open-ended answers. University May 2025 45.8% Fading
Point-Bench Models point to the image location a language request describes, in about 1,000 tasks across five reasoning categories. A point inside the target mask counts as correct. Everyday May 2025 67.1% Fading
RBench-M Models answer 665 graduate-level multimodal reasoning questions in English and Chinese. Top-1 accuracy compares selected answers with the answer key. Expert May 2025 69.9% Not in cards
Roboflow 100-VL Models locate objects named in natural language across image collections from different domains. The benchmark scores predicted bounding boxes using average precision. Everyday May 2025 7.60% Not in cards
Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. University May 2025 59.3% System cards only
SITE Models answer multiple-choice spatial questions over single images, multiple images and video, drawn from about 30 vision datasets. Tasks span figural to environmental scale. Everyday May 2025 32.8% Fading
ThinkGeo Agents use remote sensing tools to interpret aerial and satellite images. The authors compare individual tool calls and use GPT-4o mini to grade final answers. University May 2025 100% Not in cards
Video-Holmes Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film. Everyday May 2025 68.2% Fading
Video-SafetyBench Video-language models receive 2,264 synthesized videos paired with a harmful or a benign-looking query across 13 unsafe categories and 48 subcategories. An LLM judge scores harm with a confidence-weighted metric called RJScore. Everyday May 2025 1.30 Not in cards
VideoEval-Pro Video models give open-ended short answers to questions that need the whole long video, which removes the guessing that multiple choice allows. Everyday May 2025 40.8% Fading
VideoReasonBench Video models watch a sequence of fine-grained operations on a partly hidden state and answer questions that recall, infer or predict that state. Everyday May 2025 56% Fading
ViewSpatial-Bench Models answer 5,712 questions that locate objects from the camera view and from the view of people in the scene. Five task types test egocentric and allocentric reasoning. Everyday May 2025 43.2% Fading
ViStoryBench Models turn stories and character references into coherent image sequences. Model judgments and image features measure prompt alignment, character consistency, style, and image quality. University May 2025 2.00 Not in cards
VLMsAreBiased Models count or identify features in altered images of famous subjects, such as a logo with an added stripe, across 7 domains. Scoring is accuracy. Everyday May 2025 84.1% Fading
VSTI-Bench Video questions about how spatial relationships change over time in 3D scenes, over 138,000 pairs across five tasks. Everyday May 2025 67.3% System cards only
Word-Level BISINDO Models identify 32 signs from the Banten variety of Indonesian Sign Language in video clips. Everyday May 2025 Not in cards
3D Animation Arena Models reconstruct a moving 3D human from a video. Voters compare motion fidelity, ground contact, smoothness, and overall appearance. Everyday Apr 2025 Not in cards
All-Angles Bench Models answer over 2,100 multi-view questions about 90 real scenes. Six tasks cover counting, attributes, relative distance and direction, object manipulation and camera pose. Everyday Apr 2025 72.1% Fading
AstroLLaVA image and text retrieval Models match astronomical images with descriptions. Google evaluates whether an image retrieves the matching description among its top five results. University Apr 2025 Not in cards
ChartQA Pro Model answers 1,948 diverse questions over 1,341 charts, infographics and dashboards, including conversational and unanswerable ones. Accuracy. Everyday Apr 2025 72.3% Fading
ColorBench Models answer image questions about color perception, color reasoning, and robustness to color changes. The evaluator checks the selected answers against a single correct option. Everyday Apr 2025 59.6% Not in cards
FortisAVQA Models answer questions about sounds and visible events in music videos. Accuracy is reported separately for frequent and rare question distributions after the original questions are rephrased. Everyday Apr 2025 48.7% Not in cards
GEdit-Bench Image editing models follow real-world user edit instructions. A model judge scores semantic consistency, perceptual quality and an overall score. Everyday Apr 2025 6.52 Dropped
Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. Everyday Apr 2025 System cards only
Japanese National Dental Examination 2025 Models answer 350 questions from Japan's 2025 dental licensing exam. The study reports accuracy separately for 202 text questions and 148 visual questions. University Apr 2025 Not in cards
Japanese National Examination for Occupational Therapists 2025 Models answer 199 Japanese occupational therapy licensing questions. The authors score answers against the official key and compare performance on text and visual questions. University Apr 2025 88.4% Not in cards
JEEM Vision-language models caption images and answer visual questions in four Arabic dialects. The evaluation compares outputs with reference annotations. Everyday Apr 2025 Not in cards
Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. University Apr 2025 52.9% System cards only
LiveSports-3K Video models give real-time commentary and answer questions on sports clips. An LLM judge scores the free-form commentary. Everyday Apr 2025 78.1% Fading
MIEB Image and image-text embedding models are scored on 130 tasks in 38 languages, grouped into 8 task types such as retrieval, clustering and classification. Everyday Apr 2025 Not in cards
MM-IFEval Multimodal models follow image-grounded instructions with exact output constraints. Checks mix rule-based verification and a judge model. Everyday Apr 2025 64.6% In use
MMK12 Models use images to answer school mathematics and science questions. The test split contains 2,000 problems with reference answers. School Apr 2025 73.9% Not in cards
Open RAG Benchmark RAG systems answer 3,045 queries over 1,000 arXiv PDFs with text, tables, images, and hard negatives. Expert Apr 2025 Not in cards
OpenLMM Subjective Leaderboard Models generate open-ended answers to multimodal tasks. The board reports subjective evaluation results. University Apr 2025 76.6 Not in cards
Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. Everyday Apr 2025 System cards only
PLM-VideoBench Models answer questions about activities, describe events involving a particular person or object, and locate those events in time. Meta also tests first-person video understanding. Everyday Apr 2025 42.5 Not in cards
PVD Bench Models match captions with short videos in both retrieval directions. Meta measures retrieval on held-out video and caption pairs. Everyday Apr 2025 Not in cards
RISEBench Image editing models follow 360 instructions that need temporal, causal, spatial or logical reasoning. An LMM judge scores instruction reasoning, appearance consistency and visual plausibility. University Apr 2025 51.4% System cards only
RSCC Models describe disaster damage from satellite images taken before and after an event. The authors reserve 988 image pairs from 19 events for testing. University Apr 2025 79.2 Not in cards
Snowflake Multimodal PDF Retrieval Evaluation Retrievers find relevant pages in technical manuals, SEC filings and presentation slides. The tasks include 143 chart questions and 69 text questions about manuals. University Apr 2025 Not in cards
SPACE Models solve spatial cognition tasks involving maps, object layouts, spatial attention and memory in text and image presentations. Accuracy measures task performance. University Apr 2025 Not in cards
Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. Everyday Apr 2025 System cards only
VCR-Bench Video models answer 1,034 questions on 859 videos, and graders score both the final answer and each tagged chain-of-thought step against human rationales. Everyday Apr 2025 56.7% Fading
VisualPuzzles Models solve visual puzzles that need algorithmic, analogical, deductive, inductive and spatial reasoning but little domain knowledge. Scored by accuracy. Everyday Apr 2025 57% Fading
VisuLogic Models answer 1,000 visual reasoning puzzles in six categories that cannot be solved by describing the image in words. Scored by accuracy. Everyday Apr 2025 54.3% Fading
WorldScore 3,000 next-scene generation tasks with camera-trajectory layouts. Scores 3D/4D and video world generators on controllability, quality, dynamics. Apr 2025 Not in cards
3DGen-Bench People compare generated 3D objects in pairs. The authors use their judgments to evaluate automated scoring methods for text-to-3D and image-to-3D generation. Everyday Mar 2025 Not in cards
ASIMOV v1 Models judge whether robot actions in scenes generated from real images and hospital injury reports are undesirable or unsafe. Scored by alignment with human labels. Everyday Mar 2025 Not in cards
AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. Expert Mar 2025 22% Not in cards
AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. Everyday Mar 2025 40% Not in cards
AVUT Models answer questions about videos where the audio carries the key information. The benchmark removes questions that text alone can answer. Everyday Mar 2025 75.7% Fading
AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. Everyday Mar 2025 64% Dropped
BLUR Models identify a remembered item from incomplete clues in several languages and media. Everyday Mar 2025 54% Not in cards
Creation-MMBench Models write creative pieces grounded in up to nine images plus a role, background and instructions, across 765 cases in 51 tasks. A judge scores quality and visual fact alignment against case-specific criteria. Everyday Mar 2025 4.48 Not in cards
CrossWordBench Models solve generated crossword grids from text or images. The tasks vary the grid size and the share of letters already filled in. School Mar 2025 70.7% Not in cards
EgoBlind Models answer questions from blind users using only video recorded before each question. GPT-4o mini checks each answer against human references. Everyday Mar 2025 59.3% Not in cards
EgoLife (with memory) Models answer life-assistant questions about a week of egocentric video from six people, such as recalling past events and habits. Everyday Mar 2025 80.3% In use
EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. Everyday Mar 2025 71.8% Fading
EgoToM Video models infer a camera wearer’s goals, beliefs, and future actions from egocentric clips. Multiple-choice questions test theory of mind. University Mar 2025 Not in cards
ERQA 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. Everyday Mar 2025 78.5% In use
FiVE-Bench Models edit objects in videos from written instructions. A vision language model checks whether each edit succeeded, and other metrics measure changes to the background. University Mar 2025 Not in cards
Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. Everyday Mar 2025 Not in cards
GeoBench Models geolocate GeoGuessr photos from community maps by giving coordinates and a country. Scored by country accuracy and a distance-based score. Everyday Mar 2025 95% Dropped
LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. Everyday Mar 2025 72% Not in cards
m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. Everyday Mar 2025 72% Dropped
MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. Everyday Mar 2025 64.1% Fading
MicroVQA Models answer expert-written multiple choice questions about biology microscopy images that test image understanding, hypothesis generation and experiment proposal. Expert Mar 2025 59.4% Fading
OmniMMI Omni models answer 2,290 questions on 1,121 streaming videos, including proactive tasks where they must decide when to respond. Everyday Mar 2025 53% Fading
PHYSICS Models solve 1,297 PhD qualifying exam physics problems across six subfields, 298 of them with images. Answers are checked by symbolic equivalence. Expert Mar 2025 59.9% Dropped
QIVD Models answer spoken questions about short phone videos in real time, as in a face to face talk. Each item has a short answer and a timestamp. Everyday Mar 2025 69.6% Fading
RefAV Models find driving scenarios described in natural language. They identify when each scenario occurs and track the objects involved. University Mar 2025 Not in cards
RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. Everyday Mar 2025 50% Fading
ROOMELSA Models use a marked region in a room panorama and a written description to retrieve the matching 3D object. They return their ten closest matches. Everyday Mar 2025 Not in cards
SPAR-Bench Spatial perception and reasoning questions over single and multiple views of indoor 3D scenes, from depth and distance estimates to object relations and spatial imagination. Everyday Mar 2025 64.2% System cards only
Video SimpleQA Video models answer multi-hop fact-seeking questions that need knowledge beyond the video and evidence from its time segments. Short definitive answers are graded for factual correctness. University Mar 2025 66.3% Fading
ViSpeak-Bench Streaming video models must notice instructions given visually, such as a hand wave, and respond in seven subtasks of visual instruction feedback. Everyday Mar 2025 59.8% Fading
VLADBench Models answer closed-form questions about driving scenes, from static traffic knowledge to reasoning about dynamic road situations. University Mar 2025 58% In use
XLRS-Bench Models answer perception and reasoning questions about extremely large, ultra-high-resolution remote sensing images. Expert Mar 2025 41.1% Fading
AGI-Eval Community Image and Video Understanding Models interpret images and videos. The board reports visual question accuracy and separate comparison win rates. Everyday Feb 2025 86.5 Not in cards
Artificial Analysis Intelligence Index v2 Artificial Analysis's index of seven public reasoning, knowledge, math and coding benchmarks, each run by Artificial Analysis. Feb 2025 67.7 Dropped
Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. Expert Feb 2025 System cards only
EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. Everyday Feb 2025 50.5% Not in cards
EmbodiedBench EB-ALFRED Agents follow household instructions through high-level actions in AI2-THOR. The score is the published average task success percentage across EB-ALFRED capability groups. Everyday Feb 2025 67.7% Not in cards
EmbodiedBench EB-Habitat Agents follow instructions to find and rearrange objects through high-level skills in Habitat 2.0. The score is the published average task success percentage. Everyday Feb 2025 58.7% Not in cards
EmbodiedBench EB-Manipulation Agents use visual observations to control a robot arm for object manipulation in simulation. The score is the published average task success percentage. Everyday Feb 2025 28.5% Not in cards
EmbodiedBench EB-Navigation Agents use visual observations and low-level movement actions to reach target objects in AI2-THOR. The score is the published average task success percentage. Everyday Feb 2025 53.7% Not in cards
EmoBench-M Multimodal models are tested on emotional intelligence across 13 scenarios in three dimensions: emotion recognition, conversational emotion understanding and socially complex emotion analysis. Everyday Feb 2025 70.5% Not in cards
EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. Competition Feb 2025 39.3% Not in cards
Flame-React-Eval Models write React components from UI design images. Rendered outputs are compared with gold components by image similarity and scored as pass@k. University Feb 2025 93.8% Fading
HD-EPIC VQA Models watch first-person kitchen videos and choose among five answers. Questions ask what people cooked, where they moved objects, and where they looked. Everyday Feb 2025 28.7% Not in cards
Massive Visualized IR Benchmark Models retrieve screenshots that match a search query. The benchmark compares performance across four retrieval task groups. Everyday Feb 2025 Not in cards
MCS-Bench Models read seal inscriptions, identify calligraphy, and answer questions about Chinese paintings and artifacts. The overall score averages task scores. Expert Feb 2025 49.3 Not in cards
MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. Everyday Feb 2025 82.5% Not in cards
MME-CoT Models reason step by step on image questions in math, science, OCR, logic, space-time and general scenes. A judge scores reasoning quality, robustness and efficiency. University Feb 2025 64.2% Not in cards
Multimodal RewardBench Reward models choose between two answers to a visual prompt. Accuracy measures agreement with annotated preferences across six areas. University Feb 2025 62.4% Not in cards
Open FlagEval VLM Leaderboard Models answer questions about images. The leaderboard compares their scores across visual understanding tasks. University Feb 2025 70.7 Not in cards
RivaBench Models answer questions about stand-up routines and academic talks using speech and visual cues. They also classify clips as real or synthetic. University Feb 2025 Not in cards
SimpleVQA Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. University Feb 2025 81.7% In use
Virology Capabilities Test (VCT) Models answer 322 multimodal virology troubleshooting questions written by expert virologists. Multiple-select variant requires all correct options. Expert Feb 2025 67.4% Fading
VisFactor Models take 20 vision-centric subtests adapted from a human cognitive psychology test battery, covering perception primitives and visual memory. Scored by accuracy. Everyday Feb 2025 55.9% In use
WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. Everyday Feb 2025 45.8% Not in cards
WorldSense (omnimodal) Models answer multiple choice questions about real-world videos that need both the visual and the audio track. Scored by accuracy. Everyday Feb 2025 67.3% Fading
ZeroBench 100 hand-made visual questions that all frontier models originally failed. Scored by exact answer, pass@1 and pass@5. University Feb 2025 54.6% In use
DriveBench Vision-language models answer 20,498 questions about 19,200 driving frames, covering perception, prediction, planning and behavior. Inputs come in 17 settings: clean images, corrupted images and text only. The test checks whether answers rest on the image or on general knowledge. Everyday Jan 2025 48.0 Not in cards
DriveBench Behavior Models answer driving behavior questions. The score is the published GPT evaluation score for this task family. Everyday Jan 2025 54.6 Not in cards
DriveBench Perception Models answer driving perception questions. The score is the published GPT evaluation score for this task family. Everyday Jan 2025 35.2 Not in cards
DriveBench Planning Models answer driving planning questions. The score is the published GPT evaluation score for this task family. Everyday Jan 2025 57.0 Not in cards
DriveBench Prediction Models answer driving prediction questions. The score is the published GPT evaluation score for this task family. Everyday Jan 2025 48.9 Not in cards
EMMA Models solve math, physics, chemistry and coding problems that need reasoning over text and images together, not each alone. Answers are checked against a key. University Jan 2025 42.5% Fading
GeoMap-Bench Models answer 3,864 questions about 124 geologic maps. They locate map features, interpret legends, and use geological evidence to explain conditions at a site. Expert Jan 2025 0.37 Not in cards
HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. Expert Jan 2025 23.4 Fading
Humanity's Last Exam 2,500 expert-written closed-ended questions across 100+ subjects. Exact-match or multiple-choice answers graded by an LLM judge. Expert Jan 2025 54.8% In use
Japanese Nephrology Board Renewal Questions 2014 to 2023 Models answer 209 Japanese nephrology board renewal questions from ten exam years, including questions with clinical images. Researchers compare answers with the society’s key. Expert Jan 2025 81.3% Not in cards
MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. Expert Jan 2025 52.6% Fading
MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. Expert Jan 2025 80.4% In use
MER-UniBench Models describe emotions from audio, video, and text across nine datasets. Label normalization supports hit rate, weighted F1, and emotion-wheel F1. The reported mean averages the primary metric from each dataset. Everyday Jan 2025 Not in cards
MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. University Jan 2025 82.4% In use
MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. Everyday Jan 2025 75.2% Fading
OVO-Bench Video models answer questions asked at given timestamps of a video stream in 12 tasks over 644 videos. Tasks cover backward tracing, real-time understanding and forward active responding. Everyday Jan 2025 80.7% Fading
Physics-IQ Video models continue 396 filmed physical events from a frame or 3 s clip. Motion compared with the real continuation. Jan 2025 57.7% System cards only
StreamBench (StreamChat) Video models answer questions on streaming videos across media types, including multi-turn interaction and reasoning tasks. Everyday Jan 2025 59.1% Dropped
Turkish Dental Specialty Examination 2020 and 2021 Models answer Turkish dental specialty entrance questions in basic and clinical sciences. The two official answer keys cover 238 scored questions, including questions with images. Expert Jan 2025 Not in cards
Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. University Jan 2025 88.7% In use
VideoGen-RewardBench Models rank two generated videos for the same prompt. The scorer compares their rankings with human preferences for visual quality, motion, prompt alignment, and overall quality. Everyday Jan 2025 Not in cards
VPCT Models view an image of a ball, ramps, and three buckets and predict which bucket the ball falls into. Scored by accuracy on 100 problems. Everyday Jan 2025 91% Fading
3DSRBench Models answer questions about 3D spatial relations in images, such as height, location, orientation and relations between objects. Everyday Dec 2024 62.4% Fading
BdSLW401 Models identify one of 401 Bangla Sign Language words in a video. The collection records signs from the front and side. Everyday Dec 2024 Not in cards
CC-OCR Models read text in 7,058 images across four tracks: scene text, multilingual text, document parsing, and key information extraction. Each track has its own metric. Everyday Dec 2024 83.4% Fading
CG-Bench Video models answer 12,129 questions about 1,219 long videos and must also find the clue segments that support each answer. Everyday Dec 2024 33% Fading
ICDAR 2025 MapText Competition Models detect and read words on historical maps and link them into phrases. This edition adds Taiwanese maps with Chinese labels and revises how links score. School Dec 2024 Not in cards
Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. Everyday Dec 2024 99% System cards only
Labelbox Multimodal Reasoning Models interpret images, identify differences, write captions, and reason about spatial relationships. Human raters compare the responses. Everyday Dec 2024 1,220 Not in cards
LMAct Multimodal language models imitate long demonstrations of embodied actions. The benchmark compares zero-shot, few-shot, and many-shot performance. University Dec 2024 Not in cards
Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. Expert Dec 2024 63.1% System cards only
Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. Everyday Dec 2024 81% System cards only
OCRBench v2 Visual-text localization and reasoning tasks across 31 scenarios (documents, scenes, charts). Averaged task accuracy. Everyday Dec 2024 68.3% In use
Open LMM Reasoning Leaderboard Models solve multimodal reasoning tasks. OpenCompass ranks the models using the component results. University Dec 2024 61.6 Not in cards
OVBench Video models answer questions about past, current and future moments of an online video stream in 6 task types and 16 subtasks. Everyday Dec 2024 70% Fading
SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. Everyday Dec 2024 76.7% System cards only
ScanNet++ v2 Models reconstruct indoor scenes from images of real scans. The second release adds scene types and updates the annotations and camera poses. School Dec 2024 Not in cards
U-MATH Models solve 1,100 unpublished open-ended university math problems from teaching materials, balanced across six subjects. About 20% include images, and an LLM judge checks each solution. University Dec 2024 86.8% Not in cards
Video-MonetBench Models rank two videos generated for the same prompt. The test compares each ranking with a human preference, including ties. Everyday Dec 2024 Not in cards
VisionArena-Bench Models answer visual questions from Arena users. A vision language model judges their answers to approximate human preference rankings. Everyday Dec 2024 88 Not in cards
VISTA Models answer 758 single-turn prompts about one image each, covering perception and reasoning. Three LLM judges grade answers with yes/no rubric questions. The main score is average task accuracy. Expert Dec 2024 54.6% Not in cards
VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. Everyday Dec 2024 9.84% Not in cards
VSI-Bench Models watch egocentric videos of 288 indoor scenes and answer over 5,000 questions on object counts, distances, sizes, room layout and route planning. Everyday Dec 2024 69.5% Fading
BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. Everyday Nov 2024 68.3% Not in cards
GEOBench-VLM Models interpret satellite and aerial images, identify land use, and track changes across observations. Some tasks ask models to locate objects or outline regions. University Nov 2024 0.41 Not in cards
HourVideo Multimodal models answer questions about first-person videos that run 20 to 120 minutes. The 500 videos come from Ego4D, and the 12,976 five-way multiple-choice questions cover summarization, perception, visual reasoning and navigation. Score is accuracy. Everyday Nov 2024 Not in cards
MaCBench Vision language models answer chemistry and materials questions from images. Tasks cover reading data, understanding experiments and interpreting results. Expert Nov 2024 72.6% Not in cards
RD-TableBench Models extract 1,000 hand-labeled complex tables from document images, including scans, handwriting and merged cells. Scored by Levenshtein-based row and cell alignment. Everyday Nov 2024 57.4% System cards only
RoboSpatial-Home Models answer 350 spatial questions on home scenes for robots, in three types: object context, compatibility of placements and spatial configuration. Everyday Nov 2024 73.9% Dropped
StreamingBench Models answer questions while a video streams in, as a human viewer would. It tests real-time visual understanding, omni-source understanding and context understanding. Everyday Nov 2024 56.4% Fading
SURDS Models answer spatial questions about objects in nuScenes driving camera images across six single-object and multi-object tasks. Scoring is task accuracy. Everyday Nov 2024 35.7% In use
VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. Everyday Nov 2024 86.9% Not in cards
VL-RewardBench Vision-language reward models pick the better of two responses on general queries, hallucination detection and reasoning tasks. Expert Nov 2024 53.9% Not in cards
DynaMath Models answer visual math questions generated from 501 seed programs that vary values and graphs. Scored by average and worst-case accuracy across variants. School Oct 2024 88% Fading
Harmony4D Models recover the body shapes and poses of people interacting at close range. The benchmark includes sports where people occlude each other. School Oct 2024 Not in cards
HumanEval-V Models read a diagram and a function signature, then write Python code that must pass test cases. Tasks span six types of visual reasoning. University Oct 2024 40.6% Fading
Image2Struct Vision-language models turn images of webpages, LaTeX and musical scores into code that renders them. The render is compared with the input image. University Oct 2024 94.7 Not in cards
Khmer Handwritten Dataset (4.2k) Models transcribe line images from a handwriting collection that includes Khmer and English text. Everyday Oct 2024 Not in cards
LEVIR-CDC Models describe changes between aerial images and mark changed buildings. The authors paired LEVIR-CC captions with binary building masks. Everyday Oct 2024 Not in cards
LOKI Models distinguish synthetic content from real content across images, videos, audio, text, and 3D data. The benchmark checks real-or-synthetic labels and annotated abnormal details, with separate tasks for explaining anomalies. Everyday Oct 2024 60.8% Not in cards
ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. Everyday Oct 2024 Not in cards
MEGA-Bench Models solve 505 real-world multimodal tasks with over 8,000 samples. Outputs include numbers, code, LaTeX, coordinates and JSON, scored by more than 40 task metrics. University Oct 2024 54.3% System cards only
MM-MT-Bench Models hold multi-turn conversations about images such as charts, documents and photos. GPT-4o grades each answer against a reference on a 1 to 10 scale. Everyday Oct 2024 8.50 Dropped
MMEB Multimodal embedding models are scored on classification, visual question answering, multimodal retrieval and visual grounding, with 16 evaluation datasets in and out of distribution. Everyday Oct 2024 80.1% Not in cards
MMIE Models interpret and generate interleaved text and images. MMIE-Score, a fine-tuned InternVL-2-4B model, grades the resulting multimodal responses. University Oct 2024 Not in cards
NaturalBench Models answer 10,000 human-checked questions on natural images. Each question pairs with two images that need different answers, so a model cannot pass without looking. Everyday Oct 2024 55.1% Not in cards
OBI-Bench Models locate inscriptions, rejoin bone fragments, classify characters, retrieve matches, and explain glyphs. The authors publish separate scores for each task. Expert Oct 2024 0.61 Not in cards
QAG-360K Models compare two satellite images, answer questions about land cover, and mark the relevant pixels. The authors score answer accuracy and mask overlap separately. School Oct 2024 Not in cards
R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. Everyday Oct 2024 75.4% System cards only
SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests. Expert Oct 2024 74.1% In use
TaxaBench-8k Models identify species or match observations across photographs, sounds, and other field data. The test contains 8,813 observations paired across six types of data. Expert Oct 2024 Not in cards
TemporalBench Video models answer about 10K questions on fine-grained temporal dynamics, built from about 2K detailed human annotations of video clips. Everyday Oct 2024 28.1% Dropped
TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. Everyday Oct 2024 79.5% Fading
TVBench Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. Everyday Oct 2024 81.9% In use
VDC Benchmark Models describe the contents of a video in detail. The evaluation checks descriptions against human annotations using questions about the video. Everyday Oct 2024 1.96 Not in cards
VHELM Vision-language models are scored across nine aspects, including perception, reasoning, knowledge, bias, fairness, multilinguality, robustness, toxicity and safety, from many existing datasets. University Oct 2024 87.1 Not in cards
Vinoground Models match short videos with captions that distinguish events by their temporal order. School Oct 2024 Not in cards
WHU-CDC Models compare aerial photos of Christchurch, describe what changed, and mark changed pixels. Reference captions describe reconstruction after the earthquake. Everyday Oct 2024 Not in cards
WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. Oct 2024 Not in cards
xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. Everyday Oct 2024 70.1 System cards only
xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. University Oct 2024 41.4% System cards only
CHART-Info 2024 Models classify chart images, identify text and axes, and recover plotted values. The authors compare baselines on five of the seven tasks. School Sep 2024 Not in cards
DeepMind Control Vision Benchmark Agents control simulated bodies while visual distractors vary across locomotion and navigation tasks. The benchmark measures robustness of representation learning for control. University Sep 2024 Not in cards
DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. Expert Sep 2024 89.9% Dropped
EventHallusion Models answer questions about video events that can prompt false descriptions. The evaluation tests whether answers stay grounded in the clip. Everyday Sep 2024 76.5% Not in cards
MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. University Sep 2024 84.6% In use
OmniBench Models answer multiple choice questions that need an image, an audio clip and text together. Scored by accuracy. Everyday Sep 2024 56.1% Dropped
OpenVLM Video Leaderboard Models complete video understanding tasks. OpenCompass compares models across video evaluation datasets. University Sep 2024 35.4 Not in cards
PixMo-Count Models count objects in natural images and give the exact number. Scored by accuracy on the test split. Everyday Sep 2024 85.2% Fading
Q-Bench-Video Models judge the quality of natural, AI-generated and computer-graphics videos through yes-or-no, what-how, open-ended and pairwise comparison questions. Everyday Sep 2024 52.2% Not in cards
RoboTwin 1.0 Dual-arm policies train on generated digital-twin demos and are tested on simulated single-arm and dual-arm tasks, with matching real-world data. Success rate. Everyday Sep 2024 Not in cards
Vistra Systems translate text in natural images across multiple language directions. University Sep 2024 70.7 Fading
GeoSound Models match environmental recordings to satellite images of their locations and retrieve recordings from images. The test holds out geographic areas and contains 9,931 examples. Everyday Aug 2024 Not in cards
GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. Expert Aug 2024 43.4% Not in cards
HR-Bench Models answer multiple-choice questions about fine details in 4K and 8K resolution images. Results are reported separately for the 4K and 8K sets. Everyday Aug 2024 83% Fading
Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. Everyday Aug 2024 8.16 System cards only
MME-RealWorld Models answer human-annotated multiple choice questions about high-resolution real-world images in scenarios such as remote sensing, autonomous driving, monitoring and OCR. Everyday Aug 2024 50% Dropped
MMIU Vision-language models answer 11,000 multiple-choice questions over sets of images. 52 tasks cover 7 kinds of multi-image relationship. Everyday Aug 2024 27.8% System cards only
Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. School Aug 2024 System cards only
Baba Is AI Models see a puzzle and write a plan to reach the winning object. Some puzzles require them to push words together to change the rules. Everyday Jul 2024 43.9% Not in cards
ChipGPTV Models write Verilog from circuit diagrams and descriptions with different levels of detail. Testbenches check whether the generated circuits behave correctly. University Jul 2024 71.8% Not in cards
CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. Expert Jul 2024 Not in cards
DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. Everyday Jul 2024 34.4% Dropped
GTA Agents answer 229 human-written queries that need 1 to 4 real deployed tools for perception, operation, logic and creativity, often with image inputs. Queries leave the tools and steps implicit. Everyday Jul 2024 46.6% Not in cards
LAB-Bench Multiple-choice tasks on practical biology research skills: literature, protocols, sequence and figure reasoning, database lookup. Expert Jul 2024 80.7% System cards only
LAB-Bench FigQA Models interpret figures from biology papers and answer multiple-choice questions. FigQA subset of LAB-Bench. Expert Jul 2024 80.4% Fading
LogicVista Models answer 448 multiple-choice questions on spatial, deductive, inductive, numeric, and mechanical reasoning set in images. Scored by accuracy. School Jul 2024 91.9% In use
LongVideoBench Models answer referring reasoning questions over 3,763 web videos up to an hour long, with interleaved subtitles. Scored by multiple-choice accuracy. Everyday Jul 2024 80.6% Fading
MIA-Bench Models answer 400 image prompts that stack several strict instructions on format, length and content. An LLM judge scores how well each answer follows every instruction. Everyday Jul 2024 91% Dropped
MJ-Bench Models compare image pairs for text alignment, safety, visual quality, and bias. Scores measure preference accuracy, with separate dispersion and equality measures for bias. Everyday Jul 2024 44.5% Not in cards
OpenCanopy Models estimate tree canopy height from satellite images. The evaluation compares predicted height maps with measurements from airborne laser scans. University Jul 2024 Not in cards
QEVD-FIT-COACH Models watch a workout and decide when to give feedback. The benchmark scores feedback timing and the content of the coaching advice. Everyday Jul 2024 Not in cards
ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. Everyday Jul 2024 Not in cards
SA-37 interactive segmentation Models refine an object mask after a user clicks on an image. The benchmark averages overlap across datasets at different numbers of clicks. Everyday Jul 2024 Not in cards
Segment Anything Video Models predict masks for prompted objects throughout a video. The evaluation compares predicted boundaries and regions with annotated object tracks. Everyday Jul 2024 Not in cards
TV-AD Models describe what viewers can see during clips from television episodes. The test pairs each clip with a timed reference description. Everyday Jul 2024 Not in cards
VLMs Are Blind Models answer seven simple visual tasks such as counting line crossings or overlapping circles, which young children solve easily. Scored by accuracy. Everyday Jul 2024 98.6% Fading
We-Math Models solve visual math problems decomposed into sub-problems by knowledge concept. Scoring compares sub-problem and full-problem results to diagnose knowledge and generalization gaps. School Jul 2024 89% Fading
1X World Model Challenge Predict future egocentric frames of 1X's EVE humanoid from video and state. Compression scored by token cross-entropy, Sampling by PSNR on private tests. Jun 2024 Not in cards
A-Bench Models answer multiple-choice questions about AI-generated images, covering both meaning and low-level visual quality, to test them as judges of image generators. Everyday Jun 2024 81% Not in cards
BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. School Jun 2024 72.4% Dropped
ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original. University Jun 2024 60.3 System cards only
CharXiv Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. University Jun 2024 93.5% In use
CharXiv (DQ) Models answer descriptive questions about charts from arXiv papers, such as titles, axis labels and legend entries. It is the descriptive half of CharXiv. University Jun 2024 95% System cards only
ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. University Jun 2024 69.7% Dropped
CVBench Models answer 2,638 manually checked 2D and 3D vision questions built from ADE20K, COCO, and Omni3D. Scoring is accuracy. Everyday Jun 2024 88% Fading
CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. Everyday Jun 2024 74.1% System cards only
EmbSpatialBench Models answer multiple-choice questions on six spatial relations between objects in egocentric embodied scenes. Scored by accuracy. Everyday Jun 2024 85.4% Fading
GenAI-Arena Users compare outputs from two anonymous models for image generation, image editing and video generation, and vote. Votes give Elo ratings. Everyday Jun 2024 1,133 Not in cards
GenAI-Arena Image Editing Models edit images from user instructions. Anonymous human comparisons determine Elo in this independent vote pool. Everyday Jun 2024 Not in cards
Humor in AI: Cartoon Captioning Models write captions for New Yorker cartoons. Human judges and language models compare the captions with contest entries from different ranking groups. Everyday Jun 2024 73.9% Not in cards
INQUIRE-Fullrank Models retrieve wildlife photographs from five million iNaturalist images for ecological queries. The benchmark measures average precision among the first 50 results. University Jun 2024 Not in cards
INQUIRE-Rerank Models reorder 100 photographs supplied for each ecological query. The benchmark measures average precision among the first 50 results. University Jun 2024 62.4% Not in cards
K-Viscuit Models answer multiple-choice visual questions about Korean culture. Questions were written through human and VLM collaboration. Everyday Jun 2024 89.5% Fading
LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Everyday Jun 2024 91.7% In use
M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. School Jun 2024 62.4% System cards only
MIRB Models compare, analyze and reason across several images in four categories: perception, visual world knowledge, reasoning and multi-hop reasoning. Everyday Jun 2024 37.7% System cards only
MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Everyday Jun 2024 90.8% In use
MM-NIAH Vision-language models answer questions about long documents of interleaved images and text. Tasks are retrieval, counting and reasoning over needles hidden in the document. Everyday Jun 2024 11.6% Not in cards
MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. Everyday Jun 2024 2.15 Dropped
MMDU Vision-language models hold long conversations about many images, up to 20 images and 27 turns per dialog. A GPT-4o judge scores the answers. Everyday Jun 2024 62.6 Not in cards
MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. School Jun 2024 83% In use
MuirBench Models answer multiple choice questions over sets of several images, including unanswerable variants. Scored by accuracy. Everyday Jun 2024 81.8% In use
Multi-Scenario Anomaly Detection Models score video frames for unusual events across 14 settings. The two protocols hold out different videos and score frame-level detection. Everyday Jun 2024 Not in cards
NOTA Models extract facts from sheet music, convert score images to ABC notation, and analyze musical structure. The authors report extraction rates and separate transcription distances. University Jun 2024 17.8% Not in cards
OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. Competition Jun 2024 40.5% Dropped
Ref-L4 Models locate the object described by a long referring expression in an image. The set has 45,341 annotations over 365 object categories, scored by box accuracy. Everyday Jun 2024 89.5% Fading
ReMI Models reason over several images per question in math, physics, logic, code, charts, and spatial or temporal tasks. Scored by accuracy. School Jun 2024 67.3% Fading
RoboCasa Simulated mobile manipulators and humanoids do 100 kitchen tasks, 25 atomic and 75 composite, across 120 scenes. Scored by success rate. Everyday Jun 2024 70.8% Fading
SF20K-Test Models answer questions about the story in a short film. The authors check multiple-choice answers against a key and grade free responses with GPT-3.5. Everyday Jun 2024 62.8% Not in cards
SpatialEval Maze-Nav Models answer questions about routes and turns in mazes given as text or images. 1,500 multiple-choice questions scored by accuracy. Everyday Jun 2024 41% System cards only
SpatialEval Spatial-Map Models answer questions about relations and counts among objects placed on a map given as text or images. 1,500 questions scored by accuracy. Everyday Jun 2024 64.3% System cards only
VCG+ Diverse Models answer questions about videos drawn from varied settings. The evaluation grades their ability to describe and explain video content. Everyday Jun 2024 43.5 Not in cards
VCR (Visual Caption Restoration) Models restore caption text that is partly covered in an image, using only the visible pixel fragments. Scored by exact match and Jaccard similarity of the words. Everyday Jun 2024 75.6% System cards only
VELOCITI Models distinguish a video's correct description from a caption that assigns an action to the wrong person. School Jun 2024 43.1% Not in cards
VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. Everyday Jun 2024 59.6% Fading
VideoScore-Bench Models rate generated videos or choose which video better matches a human preference. The four test sets compare model judgments with human judgments. Everyday Jun 2024 39.7 Not in cards
Vision Arena People send an image with a prompt, compare answers from two anonymous vision-language models, and vote for the better one. Votes become Bradley-Terry ratings. Everyday Jun 2024 1,309 Not in cards
VRSBench Models caption aerial images, locate objects described in words, and answer visual questions. The held-out files contain 9,350 captions, 16,159 object descriptions, and 37,409 questions. Everyday Jun 2024 65.6% Not in cards
WeatherQA (severe weather) Models read weather maps to identify a threatened area and classify its severe weather potential. The second score includes only cases with the correct area. Expert Jun 2024 62.7% Not in cards
Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. Everyday Jun 2024 64% System cards only
WildVision Vision-language models answer 500 real user image prompts from the WildVision Arena. A GPT-4 judge compares each answer with a fixed baseline model. Everyday Jun 2024 74.1 Dropped
AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. Expert May 2024 37.8% Not in cards
CompassArena Users chat with two anonymous models side by side, mostly in Chinese, and vote for the better answer. A Bradley-Terry model turns the votes into Elo-style ratings. Everyday May 2024 1,062 Not in cards
Mantis-Eval Models answer questions that need reasoning across several interleaved images. Scored by accuracy. Everyday May 2024 51.1% Dropped
MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. Everyday May 2024 56.6% Fading
PitVQA Models answer questions about a frame from pituitary surgery. Questions cover the surgical step, the instruments in view, and what the surgeon is doing. Expert May 2024 Not in cards
SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. Everyday May 2024 80% In use
Vibe-Eval Models answer 269 image-based prompts, 100 of them hard, that have expert gold responses. A Reka Core judge rates each answer. Everyday May 2024 81.4% Dropped
Video-MME Models answer 2,700 multiple-choice questions about 900 videos from 11 seconds to 1 hour long, with and without subtitles. Scored by accuracy. Everyday May 2024 90.4% In use
Assembly101-O Models watch people assemble toy vehicles and flag the first procedural mistake. Training uses correct procedures, and evaluation stops at the first mistake in an incorrect sequence. Everyday Apr 2024 Not in cards
BLINK Models answer 3,807 multiple choice questions built from 14 classic computer vision tasks that use one or more images. Scoring is accuracy. Everyday Apr 2024 81.4% In use
CMD-AD-Eval Models describe what happens on screen during gaps in movie dialogue. The authors compare the descriptions with timed human narration using text overlap and model judgments. Everyday Apr 2024 Not in cards
DesignQA Models read Formula SAE rules and inspect CAD views and drawings of a student race car. They retrieve rules, identify parts, and check compliance. University Apr 2024 0.72 Not in cards
Epic-tent-O Models watch someone pitch a tent and flag procedural mistakes using only the video so far. The test includes 15 videos. Everyday Apr 2024 Not in cards
Gecko People compare text to image outputs for prompt alignment. The benchmark studies model ordering, pairwise instance scoring, and pointwise instance scoring. Everyday Apr 2024 Not in cards
Japanese National Dental Examination 2024 Models select answers to 353 Japanese dental licensing questions, including 149 that use visual material. The exam covers 17 dental specialties. University Apr 2024 81.9% Not in cards
Long-term Video Object Segmentation v2 Models track object masks through long videos. The evaluation tests whether they can recover an object's identity after an extended absence. Everyday Apr 2024 Not in cards
MMBench-EN v1.1 Models answer English multiple-choice image questions across 20 ability dimensions, using CircularEval with shuffled options. Scored by accuracy on the dev split. Everyday Apr 2024 88.5% In use
MMT-Bench Vision-language models answer 31,325 multiple-choice questions across many multimodal tasks such as recognition, localization, reasoning and planning. Expert Apr 2024 62.4% In use
OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. Everyday Apr 2024 57.9 Dropped
RealWorldQA Models answer a question about each of 700+ real-world photos, many taken from vehicles, that test basic spatial understanding. Answers are easy to verify. Everyday Apr 2024 88.5% In use
SEED-Bench-2-Plus Multimodal models answer multiple-choice questions about text-rich images such as charts, maps and web pages. Everyday Apr 2024 73.8% System cards only
ArAIEval 2024 Models mark persuasive passages in Arabic text and name the technique. They also classify Arabic memes using the image, its extracted text, or both. University Mar 2024 Not in cards
BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. Everyday Mar 2024 Not in cards
CLIP representation bias evaluation Models associate images of everyday objects with gender terms. The evaluation measures how far the associations depart from an equal preference across groups. Everyday Mar 2024 Not in cards
Design2Code Models turn screenshots of 484 real webpages into HTML and CSS. Automatic metrics compare the rendered page with the original on layout, text, position, color and CLIP similarity. Everyday Mar 2024 87.2 Fading
EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. School Mar 2024 44% Not in cards
HM3DSem-Walks Models use indoor color and depth scans to find objects described by room and floor. The original retrieval test covers eight scenes. Everyday Mar 2024 Not in cards
HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. Everyday Mar 2024 Not in cards
JA-VLM-Bench-In-the-Wild Models answer Japanese questions about images of places, objects and culture in Japan. Everyday Mar 2024 41.1 Not in cards
LHRS-Bench Models answer 690 single-choice questions about 108 remote sensing images. The questions cover land use, object locations, image resolution, and other properties. University Mar 2024 39.1% Not in cards
MathVerse Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram. School Mar 2024 89.7% Fading
MMStar Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. School Mar 2024 85.9% In use
Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. Everyday Mar 2024 99.2% System cards only
TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. Everyday Mar 2024 89.6% Fading
TGVE+ Models follow video editing instructions on the original 76 TGVE videos. The extension adds object removal, object addition, and texture changes. University Mar 2024 Not in cards
1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. Everyday Feb 2024 89.1% System cards only
ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. Everyday Feb 2024 20.9 Not in cards
ContPhy Models infer physical properties such as mass and density of fluids and soft bodies from videos and predict their dynamics. Everyday Feb 2024 67.4% Fading
CRPE Models answer single-choice questions about objects and their relations in images, covering subject, predicate and object. Each question is asked again with options rotated (CircularEval). Everyday Feb 2024 79.2% Dropped
LoCoMo Models answer questions, summarize events and continue dialogue over very long multi-session chats of about 300 turns and 9K tokens, up to 35 sessions. Answers are scored against references. Everyday Feb 2024 Not in cards
MAD-Bench Multimodal models answer image questions containing deceptive prompt information about objects, counts and spatial relationships. University Feb 2024 Not in cards
MATH-Vision (MathVision) 3,040 competition math problems with visual context. Answer accuracy. Competition Feb 2024 97.7% In use
MedVidQA 2024 Query-Focused Instructional Step Captioning Models divide a medical demonstration into steps that answer a question. They give each step a caption and start and end times. Everyday Feb 2024 Not in cards
MedVidQA 2024 Video Corpus Visual Answer Localization Models search 48,605 medical videos for demonstrations that answer 52 new questions. They return ranked videos and the start and end times of each answer. Everyday Feb 2024 Not in cards
OlympiadBench Models solve bilingual olympiad-level math and physics problems in Chinese and English, many with images. Answers are checked against a key. Competition Feb 2024 18% Dropped
OmniMedVQA Models answer questions about 118,010 medical images across 12 imaging modalities. The authors measure accuracy against the correct answer option. Expert Feb 2024 50.7% Not in cards
PAIRS (Parallel Images) Models answer questions about people in paired images. The evaluation tests whether changing demographic characteristics changes judgments about the subjects. Everyday Feb 2024 97.2% Not in cards
PCA-Bench Models interpret visual scenes, explain their reasoning, and select actions for embodied tasks. GPT-4-0125 compares perception concepts, reasoning, and actions with annotated references. Everyday Feb 2024 0.70 Not in cards
RRG24 Models write findings and impressions from one or more chest X-rays per study. Organizers score each section separately using five metrics. Expert Feb 2024 Not in cards
VLGuard Vision language models answer safe and unsafe image and text pairs from the VLGuard test set. Responses are judged for harmfulness and refusal. Everyday Feb 2024 9.10 System cards only
WildVision Arena Users upload an image with a question, compare answers from two anonymous vision-language models, and vote. Votes give Elo ratings. Everyday Feb 2024 1,100 Not in cards
CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. University Jan 2024 36.8% Not in cards
CMMU Models answer Chinese multimodal questions across seven school subjects. School Jan 2024 30.9 Not in cards
ICDAR 2024 MapText Competition Models locate and transcribe words on historical maps and group them into place names. The tasks use David Rumsey maps and French land registers. School Jan 2024 Not in cards
Mementos Models describe what happens across a sequence of images. The evaluation checks whether their descriptions capture the objects and actions. Everyday Jan 2024 40.6% Not in cards
MMVP Models answer 300 questions about 150 image pairs that CLIP embeds as near duplicates. A pair counts only when both answers are right. Everyday Jan 2024 87% Fading
MultipanelVQA Models answer questions about images arranged into panels. The benchmark tests whether a model can locate the relevant panels and compare their contents. Everyday Jan 2024 72.3% Not in cards
OpenVLM Leaderboard OpenCompass runs vision-language models through VLMEvalKit on a fixed set of image benchmarks, such as MMBench, MMStar and MathVista. The headline is the average score across that set. University Jan 2024 80.1 Not in cards
RTVLM Vision language models answer red-teaming image prompts on faithfulness, privacy, safety and fairness. GPT-4V scores responses from 0 to 10. Everyday Jan 2024 7.92 System cards only
SciMMIR Models match figures and tables from scientific papers with their descriptions. Expert Jan 2024 3.35 Not in cards
BdSLW60 Models identify an isolated Bangla Sign Language word in a video clip. Accuracy measures how often the predicted word matches its label. Everyday Dec 2023 Not in cards
Ego-Exo4D Models estimate human pose during skilled activities filmed from first-person and external viewpoints. Meta reports joint and mesh errors for physical and procedural activities. School Dec 2023 Not in cards
Ego4D Goal-Step Models find when a goal or step occurs in first-person video. They also identify the current step or locate a step described in words. Everyday Dec 2023 Not in cards
EgoPER Error Detection Models watch first-person cooking videos and flag incorrect steps. The evaluator checks accuracy by step and measures error detection by frame. Everyday Dec 2023 Not in cards
LingoQA Models answer free-form questions about short driving videos. A trained classifier, Lingo-Judge, scores whether each answer is truthful. Everyday Dec 2023 84.8% In use
SMILE Video Laugh Reasoning Models explain why people laugh during sitcoms and TED talks. The authors compare generated explanations with human references. Everyday Dec 2023 Not in cards
SVG-Bench (StarVector) Models generate SVG code from images or text. The RLRF port uses the SVG-Stack image-to-SVG task and rewards the quality of the rendering. University Dec 2023 Not in cards
V* Models answer 191 multiple choice questions about small details in high-resolution images, such as an object's color or relative position. Everyday Dec 2023 96.9% Fading
AMBER Multimodal models are checked for object, attribute and relation hallucinations in generative and discriminative tasks, scored without an LLM judge. Everyday Nov 2023 84.0 Not in cards
Bingo (Visual Hallucination) Models answer questions about images. The benchmark tests whether prior expectations or interference from other visual information cause false answers. Everyday Nov 2023 Not in cards
GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. Everyday Nov 2023 66.1% Fading
MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. Everyday Nov 2023 Not in cards
MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. University Nov 2023 86% In use
MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. Everyday Nov 2023 79% Fading
SEED-Bench-2 Multimodal models answer multiple-choice questions that cover comprehension of interleaved image and text inputs and image generation, in hierarchical capability levels. Everyday Nov 2023 71.9% Not in cards
Unicorn (Vision Safety) Models answer questions about objects in unusual images and sketches. The benchmark tests how well visual judgments hold up when the images change. Everyday Nov 2023 91.2% Not in cards
VideoCon Models distinguish descriptions supported by a video from captions that contradict it. School Nov 2023 Not in cards
CCBench (MMBench Chinese Culture) Vision language models answer 510 Chinese-language multiple-choice questions about images of Chinese culture, such as calligraphy, painting, food and historical figures. Everyday Oct 2023 86.9% Dropped
HallusionBench Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. Everyday Oct 2023 73.5% In use
MathVista Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. School Oct 2023 90.7% In use
Ref-ZOM Models mark the pixels described by a phrase in a COCO image. A phrase can refer to zero, one or several objects. Everyday Oct 2023 Not in cards
Fin-Fact Models judge whether evidence supports a financial claim, contradicts it, or leaves too little information. The original evaluation also tests models with images. University Sep 2023 61% Not in cards
FORB Models retrieve other images of the same flat object despite changes in viewpoint or lighting. School Sep 2023 Not in cards
HoloAssist Mistake Detection Models watch a person perform a task and classify each annotated action as correct or mistaken. The challenge reports an F-score. Everyday Sep 2023 Not in cards
Object Detection Leaderboard Models locate and classify objects in images. The board compares average precision. Everyday Sep 2023 Not in cards
Q-Bench Multimodal models are tested on low-level vision: perceiving image distortions, describing them and predicting quality scores. Everyday Sep 2023 73.9% Not in cards
Q-Bench Image-Pair Perception Models answer multiple-choice questions comparing low-level attributes across image pairs. The score is overall answer accuracy in the A1-Pair table. Everyday Sep 2023 Not in cards
Q-Bench Single-Image Perception Models answer multiple-choice questions about low-level attributes of individual images. The score is overall answer accuracy in the A1-Single table. Everyday Sep 2023 Not in cards
XTD-200 Models retrieve images and captions across languages using translations of the XTD test set. Results measure how often the matching item ranks first. School Sep 2023 Not in cards
EgoSchema Models answer five-way multiple-choice questions about three-minute egocentric video clips from Ego4D. Scored by accuracy on 5,031 questions. Everyday Aug 2023 76.2% Dropped
EMDB Models reconstruct human pose and body shape in everyday scenes. The benchmark compares joint positions with electromagnetic measurements. School Aug 2023 Not in cards
FinEval Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings. University Aug 2023 69.4% Not in cards
MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. Everyday Aug 2023 Not in cards
MM-Vet Models answer open questions about images that combine recognition, OCR, knowledge, spatial reasoning, language generation and math. GPT-4 grades each answer against a reference. Everyday Aug 2023 82.8% Dropped
ReasonSeg Models infer which object an instruction describes and return its mask. Instructions can require reasoning about the image. Everyday Aug 2023 Not in cards
BLUEX Models answer questions from two Brazilian university entrance examinations. The questions cover high school subjects in Portuguese. School Jul 2023 95% Not in cards
COCO-O Models detect objects under natural changes in image appearance. The benchmark compares detection on drawings, cartoons, and other visual domains. Everyday Jul 2023 Not in cards
ExFunTube Models explain why a moment in a short video is funny. The authors compare explanations with human references and ask viewers to rate them. Everyday Jul 2023 Not in cards
ImageNet-UA Models classify images under distortions that differ from those used during training. The updated suite measures accuracy across 18 new distortions. Everyday Jul 2023 Not in cards
Malaysian Sign Language Medical Dataset Models identify Malaysian Sign Language terms used in health care. The training and test lists share all seven signer names. Everyday Jul 2023 Not in cards
MMBench Vision language models answer multiple-choice questions covering 20 perception and reasoning skills. CircularEval rotates the options. Scored by accuracy. Everyday Jul 2023 88.8% Fading
OPT-RSVG Models use a written description to locate an object in an aerial image. The test pairs 24,477 descriptions with reference boxes. Everyday Jul 2023 Not in cards
PUMaVOS Models track masks for object parts across video frames. A prompt can identify a small region, such as one cheek of a person's face. Everyday Jul 2023 Not in cards
RSIEval Models describe scenes in aerial photographs and answer questions about objects, positions, and road directions. Human graders score captions and answers separately. Everyday Jul 2023 Not in cards
SEED-Bench Models answer 19,000 human-annotated multiple-choice questions over 12 dimensions of image and video understanding. Scored by accuracy against the answer key. Everyday Jul 2023 70.4% Fading
Aria Digital Twin Models infer the layout of objects from first-person images. The SAM 3D Objects evaluation compares predicted positions and orientations with reference scene geometry. School Jun 2023 Not in cards
Babel-ImageNet Models classify ImageNet images using class descriptions in different languages. The benchmark measures how recognition accuracy changes with the prompt language. Everyday Jun 2023 Not in cards
Encyclopedic VQA Models answer questions about detailed properties of pictured objects and places. The Gemini Embedding 2 report measures whether they retrieve the correct answer. Everyday Jun 2023 Not in cards
LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. Everyday Jun 2023 97.2% Fading
M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. School Jun 2023 Dropped
MME Models answer yes or no questions about images across 14 perception and cognition subtasks with hand-written instructions. Labs usually report the perception sum, out of 2000. Everyday Jun 2023 2525.9 Fading
NAVI Models match points on the same object across photographs taken from different viewpoints. Reference geometry supplies the correct correspondences. School Jun 2023 Not in cards
ReMuQ Models retrieve a knowledge passage that answers a question about an image. School Jun 2023 Not in cards
SugarCrepe Models choose between a caption that describes an image and a plausible caption with a changed detail. School Jun 2023 95.2% Not in cards
Video-ChatGPT video conversation benchmark Models answer questions about video content. The evaluation grades correctness, detail, context, timing, and consistency. Everyday Jun 2023 Not in cards
EDIS Models retrieve news images and their associated text using queries that refer to specific people, places, or events. School May 2023 Not in cards
LOVEU-TGVE-2023 Models edit 76 videos with four instructions per video. People compare each submission with Tune-A-Video on text alignment, structure, and visual quality. University May 2023 Not in cards
OCRBench Models answer 1,000 checked questions on text recognition, scene text, documents, key information extraction, and handwritten math. The score counts correct answers. Everyday May 2023 94% In use
Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. Everyday May 2023 78.4% Dropped
PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. University May 2023 66.2% Fading
POPE Vision language models answer yes or no questions about whether objects appear in COCO images. Scored by accuracy or F1 on object hallucination. Everyday May 2023 90.7% Fading
DataComp image encoder evaluation Models trained on different image and text collections perform the same set of downstream tasks. The evaluation compares transfer performance across 37 tasks. Everyday Apr 2023 Not in cards
Large-Vocabulary Video Instance Segmentation Models detect, segment, and track object instances across video frames. The benchmark includes categories outside the training vocabulary. Everyday Apr 2023 Not in cards
LLaVA-Bench (In-the-Wild) Multimodal assistants answer open questions about a small set of diverse real images. A GPT-4 judge scores each answer relative to a reference answer. Everyday Apr 2023 Not in cards
OmniLabel Models locate objects described by category names and free-form phrases. The benchmark compares predicted boxes with annotated objects. Everyday Apr 2023 Not in cards
SoccerNet-Caption Models write timestamped commentary for soccer broadcasts. The set has almost 37k commentaries over 715.9 hours of video. Streaming reports also score when the model chooses to speak. Everyday Apr 2023 Not in cards
VALOR-32K Models match captions to video and audio clips. Meta reports separate retrieval results for text paired with audio and text paired with video. Everyday Apr 2023 Not in cards
Hi4D Models reconstruct the bodies of people interacting at close range. The SAM 3D Body evaluation compares joint and mesh errors with motion capture references. School Mar 2023 Not in cards
MAD-Eval Models write descriptions for scenes from ten movies. One set of references names characters. Another replaces their names with the word “SOMEONE.” Everyday Mar 2023 Not in cards
MammalNet Models identify mammals and their behavior in videos. The benchmark covers 173 mammal categories and 12 behaviors, including nursing and hunting. School Mar 2023 Not in cards
CountBench Models count the objects in natural images from the CountBench image-text set. Lab cards often use a question-answer version, CountBenchQA, scored by accuracy. Everyday Feb 2023 97.8% In use
Event-based Segmentation Dataset Models segment objects in cluttered indoor scenes filmed from a moving robot arm. The benchmark provides reference masks for comparison. Everyday Feb 2023 Not in cards
MedVidQA 2023 Medical Instructional Question Generation Models write a question that a supplied medical video segment answers. They can use both the images and the subtitles. Everyday Feb 2023 Not in cards
MedVidQA 2023 Video Corpus Visual Answer Localization Models search 12,657 medical videos and return the times when a video shows the answer. Half of the 40 questions require visual evidence. Everyday Feb 2023 Not in cards
MOSE Models follow and segment objects in videos with crowds, occlusions, and objects leaving the screen. The benchmark compares masks against human annotations. Everyday Feb 2023 Not in cards
OVEN Models identify the Wikipedia entity that answers a question about an image. University Feb 2023 Not in cards
GeoDE Models recognize objects in images collected across different world regions. The evaluation compares accuracy across regions and object classes. Everyday Jan 2023 Not in cards
MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. Expert Jan 2023 Not in cards
RadSum23 Models write the impression section of radiology reports from findings. One track includes chest images. Organizers rank submissions by RadGraph F1. Expert Jan 2023 38.0 Not in cards
Video Object Segmentation under Transformations Models follow objects whose appearance changes as people manipulate them. The benchmark compares predicted masks with annotations through these transformations. Everyday Dec 2022 Not in cards
ArtCap Models associate paintings with descriptions of their contents. Google compares image encoders by how often the matching caption appears in their top five. Everyday Nov 2022 Not in cards
CLIPSep Sound Separation Models isolate a requested sound from a mixture using a text or image query. The two tests mix music or everyday sounds with interfering audio. University Nov 2022 Not in cards
KiloGram Models choose which of ten tangram pictures matches a description. The descriptions can name the whole shape or explain what its pieces represent. Everyday Nov 2022 Not in cards
NEVIS’22 Models learn 106 image classification tasks in chronological order. The evaluation compares error rates against the computation each model uses. Everyday Nov 2022 Not in cards
TAP-Vid Models track individual points across frames and predict when each point disappears. The benchmark compares their trajectories and visibility predictions with reference tracks. Everyday Nov 2022 Not in cards
DIOR-RSVG Models draw a box around the object named in a description of an aerial image. The released test split contains 7,500 descriptions. Everyday Oct 2022 Not in cards
Language-Table A simulated robot moves colored blocks in response to natural language instructions. The benchmark measures success and success weighted by path length across five task families and 696 task variations. Everyday Oct 2022 Not in cards
Video to Valence Models predict which of two videos viewers find more pleasant. Human preference annotations supply the reference judgments. Everyday Oct 2022 Not in cards
VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. Everyday Oct 2022 Not in cards
BURST Models recognize, segment, and track objects across video frames. The benchmark tests how well they associate the same object across time. Everyday Sep 2022 Not in cards
MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. Everyday Sep 2022 62.3% System cards only
New Yorker Caption Contest Humor Understanding Models match a cartoon to its caption, choose the funnier of two captions, and explain a joke. They receive the cartoon or its description. Everyday Sep 2022 70.8% Not in cards
Robust Multi-view Depth Models estimate depth from multiple photographs of a scene. The benchmark compares relative depth errors across datasets from different environments. School Sep 2022 Not in cards
ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. School Sep 2022 97.5% Dropped
EPIC-KITCHENS VISOR Hand Object Segmentation Models outline each hand and the object it touches in a kitchen image. They also identify which hand appears and whether it touches anything. Everyday Aug 2022 Not in cards
EPIC-KITCHENS VISOR Semi-Supervised Video Object Segmentation Models receive masks for hands and objects in the first frame of a kitchen video. They track those objects through later frames, including after occlusion. Everyday Aug 2022 Not in cards
Google Universal Image Embedding Challenge Models retrieve images of the same item across several visual domains using a single image representation. The challenge compares results against labeled matches. Everyday Aug 2022 Not in cards
EgoProceL Models find recurring steps in first-person videos and mark when each step occurs. The videos show 16 tasks, including cooking and assembling computers. Everyday Jul 2022 Not in cards
LineEX Models extract labels and numerical series from scientific line charts. The hub port uses a 20,000-image test split and grades the structured output. University Jul 2022 Not in cards
Omni3D A large benchmark for 3D object detection from a single image, built by unifying indoor and outdoor datasets such as SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI and nuScenes. Vision-language models answer its 3D grounding questions. Everyday Jul 2022 53.4% System cards only
A-OKVQA Models answer about 25,000 visual questions that need world knowledge beyond the image, in multiple-choice and direct-answer forms. Everyday Jun 2022 Not in cards
EndoVis18-VQA Models answer questions about organs, tools, tool positions, and interactions in images from robotic kidney surgery. The benchmark accepts class labels or sentence answers in separate tasks. Expert Jun 2022 Not in cards
Meta-Album Models classify unfamiliar images after seeing a few labeled examples. Tasks span ten domains, including microscopy, remote sensing and wildlife. School Jun 2022 Not in cards
MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. Everyday Jun 2022 49% Fading
RICH Models reconstruct people interacting with their surroundings. The evaluation compares body joints and mesh vertices with the reference geometry. School Jun 2022 Not in cards
SignBD-Word Models identify isolated Bangla signs from video or body pose. The authors score top one and top five accuracy. Everyday Jun 2022 Not in cards
Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. School May 2022 0.65 Not in cards
Animal Kingdom Models identify actions in wildlife videos that can show several behaviors at once. The benchmark also tests pose estimation and locating events described in text. School Apr 2022 Not in cards
Google Scanned Objects Models synthesize an object from a new viewpoint. Researchers compare the rendered result with reference views of scanned household items. School Apr 2022 Not in cards
LEVIR-CC Models compare aligned aerial images from Texas and describe what changed. Five annotators wrote reference captions for each of the 1,929 test pairs. Everyday Apr 2022 Not in cards
NWPU-Captions Models describe aerial scenes across 45 land use categories. The authors reserve 3,150 images for testing and provide five captions per image. School Apr 2022 Not in cards
VSR Models judge whether captions with 65 kinds of spatial relation, such as under or facing, are true of an image. Over 10,000 text and image pairs. Everyday Apr 2022 Not in cards
VSR Random Split Models judge whether image captions correctly describe spatial relations on the randomly partitioned test set. The score is accuracy over 2,195 test examples. Everyday Apr 2022 Not in cards
VSR Zero-Shot Split Models judge spatial captions on a test set whose object concepts do not appear in training. The score is accuracy over 616 test examples. Everyday Apr 2022 Not in cards
Winoground Models match two images to two captions that use the same words in a different order. School Apr 2022 Not in cards
AmsterTime Models match historic photographs of Amsterdam with contemporary street views. The benchmark measures recognition of the same place across time. Everyday Mar 2022 Not in cards
ChartQA Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. School Mar 2022 90.9% In use
HierText Models locate and recognize text arranged in lines and paragraphs in real scenes. Google also uses the annotations to compare text and image retrieval. School Mar 2022 Not in cards
ImageCoDe Models select the image that matches a description from a set of visually similar alternatives. Everyday Mar 2022 Not in cards
MET-Meme Models classify Chinese and English memes by metaphor, sentiment, intent, and degree of offensiveness. Everyday Feb 2022 Not in cards
MuMiN Models use social media context to classify claims as factual or misleading. A second task asks which verdict applies to the claim a post discusses. University Feb 2022 Not in cards
CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. Everyday Dec 2021 Not in cards
CDVQA Models compare two aerial images and answer questions about which land cover changed and by how much. The two tests use the same 968 image pairs with different answer distributions. They contain 39,686 and 31,036 questions, respectively. School Dec 2021 Not in cards
ODinW-13 Models detect objects in 13 varied real-world datasets, such as aquarium and aerial images, from text category names. Scored by mean average precision. Everyday Dec 2021 51.8% Fading
SLIP zero-shot transfer suite Models assign labels to images using text descriptions of the classes. The evaluation averages zero-shot accuracy across 26 recognition tasks. Everyday Dec 2021 Not in cards
ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. Everyday Nov 2021 53.7% Dropped
Clinical Laboratory Test Reports Models extract laboratory results from report images into structured JSON. Google compares predictions with reference fields using macro and micro F1. School Nov 2021 Not in cards
Natural Adversarial Objects Models identify objects in unmodified photographs that caused errors in earlier detectors. Human annotations identify the objects. Everyday Nov 2021 Not in cards
Ego4D State Change Object Detection Models locate the object that changes in a first-person video. The official test scores its bounding box in the frame where the change begins. Everyday Oct 2021 Not in cards
LoveDA Models label land cover in aerial images of urban and rural areas. The benchmark supports comparisons across these two settings. University Oct 2021 Not in cards
SciCap Models describe figures from scientific publications. Researchers compare generated captions with the text that accompanies each figure. School Oct 2021 Not in cards
TEACh Agents complete household tasks in simulated homes, such as making coffee or preparing breakfast, guided by a dialogue between a Commander who knows the task and a Follower who acts. The dataset holds over 3,000 human dialogues and defines three benchmarks: acting from dialogue history, predicting the trajectory from a dialogue, and completing the task as two agents. Scores are task success rates. Everyday Oct 2021 Not in cards
GD-VCR Models answer questions about what people are doing and why. The test pairs 328 images, mainly from film and television, with 886 questions across four regions. Everyday Sep 2021 Not in cards
CIRR Models find an image that matches a reference photograph after applying a change described in text. Everyday Aug 2021 22% Not in cards
TREK-150 Models follow a marked object through first-person kitchen videos. In the one-pass test, they receive its bounding box in the first frame and track it to the end. Everyday Aug 2021 Not in cards
Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. Expert Jul 2021 Not in cards
QVHighlights Natural-language queries over YouTube videos. Model localizes relevant moments, scored by Recall@1 at IoU 0.5. Everyday Jul 2021 77.2% Fading
Inter-GPS (Geometry3K) Models solve high school geometry problems that combine a diagram and text from the Geometry3K test set. Scored by accuracy. School May 2021 48.6% Dropped
LAD2000 Models identify unusual events in videos and score when they happen. The test contains 280 videos with unusual events and 280 without them. Everyday May 2021 Not in cards
NExT-QA Models answer causal, temporal and descriptive questions about short videos of daily life. Labs usually report the multiple choice split. Everyday May 2021 86.2% Dropped
InfographicVQA Models answer about 30,000 questions on about 5,000 infographics that mix text, layout, charts, and graphics. Scoring is ANLS. School Apr 2021 92.6% In use
Unidentified Video Objects Models segment and track objects in video without a fixed list of object categories. The evaluation compares predicted masks and tracks with annotated instances. Everyday Apr 2021 Not in cards
iNaturalist 2021 Models classify nature photographs by species. The 2021 competition uses a new collection of images and species labels. University Mar 2021 Not in cards
Natural World Tasks (NeWT 2021) Models distinguish pairs of labels in wildlife photographs. The 164 tasks include recognizing life stages, animal behavior and scene context. University Mar 2021 Not in cards
PPR10K Models retouch groups of portraits against three expert retouchers’ Adobe Camera Raw edits. The test measures accuracy on people and color consistency across each group. University Mar 2021 Not in cards
Country211 Models predict the country where a photograph was taken. University Feb 2021 Not in cards
Occluded Video Instance Segmentation Models track and segment objects that overlap or disappear behind other objects. The evaluation compares predicted masks and identities with annotated video instances. Everyday Feb 2021 Not in cards
SLAKE Models answer English and Chinese questions about radiology images of the head, neck, chest, abdomen, and pelvis. Scoring is accuracy. University Feb 2021 90.8% Fading
Long Range Arena Transformers solve six long-sequence tasks covering list operations, text, retrieval, image, path and path crossing. Expert Jan 2021 Not in cards
WebSRC Models answer questions about a web page from its screenshot, HTML or both. The set has 400K question and answer pairs from 6.4K web pages. Scored by exact match and F1. Everyday Jan 2021 Not in cards
Cross-lingual Test Dataset Models retrieve images from captions written in different languages. The benchmark compares retrieval against paired photographs and descriptions. School Dec 2020 Not in cards
Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. Everyday Dec 2020 Dropped
Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. Everyday Nov 2020 13.5 Fading
INCLUDE Indian Sign Language Recognition Models identify isolated Indian Sign Language words in video. The authors provide separate test files for the full vocabulary and its smaller selection. Everyday Oct 2020 Not in cards
Kinetics-700-2020 Models recognize human actions in short videos from the 2020 edition of Kinetics-700. Everyday Oct 2020 62% Not in cards
MECCANO Action Recognition Models identify an assembly action from a first-person video clip of someone building a toy motorcycle. The answer key has 61 action classes. Everyday Oct 2020 Not in cards
MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. University Oct 2020 Not in cards
Room-Across-Room (RxR) Agents follow navigation instructions through photo-realistic indoor scenes to reach a goal. The instructions are in English, Hindi and Telugu, and each word is time-aligned to the annotator's position. Scores include success rate and how closely the path follows the reference route. Everyday Oct 2020 Not in cards
Widget Captioning Models describe an individual interface element in a mobile screenshot. The benchmark compares their captions with human descriptions of the same element. School Oct 2020 Not in cards
Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. Everyday Aug 2020 Not in cards
DocVQA Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers. Everyday Jul 2020 96.9% In use
EPIC-KITCHENS-100 action anticipation Models predict a person's next action from first-person kitchen video. The benchmark scores predicted verbs, nouns, and their combinations. Everyday Jun 2020 Not in cards
ImageNet ReaL Models classify ImageNet validation images against revised human annotations. An image can have more than one acceptable label. Everyday Jun 2020 Not in cards
ImageNet-R Models classify renditions of objects in drawings, sculptures, and other media. The benchmark measures recognition when appearance differs from ordinary photographs. Everyday Jun 2020 Not in cards
RP2K Models identify retail products from photographs taken in stores. The test distinguishes products with similar packaging. Everyday Jun 2020 Not in cards
Hateful Memes Challenge Models decide whether a meme is hateful by interpreting its image and caption together. Paired examples change one modality while preserving the other. Everyday May 2020 Not in cards
How2QA Models answer multiple-choice questions about instructional videos. The task tests understanding of the actions shown in each clip. Everyday May 2020 Not in cards
PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. School May 2020 Not in cards
Taiwan Mandarin Speech with Video: Speech Enhancement Models use lip video to remove noise from Mandarin sentences. Researchers compare the enhanced signal with clean speech for quality and intelligibility. University May 2020 Not in cards
Google Landmarks Dataset v2 Models recognize landmarks or retrieve photographs of the same landmark. The benchmark uses images from around the world. Everyday Apr 2020 Not in cards
VGGSound Models identify sound events in video clips where the sound source is visible. Everyday Apr 2020 Not in cards
Virtual KITTI 2 Models interpret simulated driving videos with reference geometry and pixel labels. Meta uses the sequences to compare video object segmentation models. Everyday Jan 2020 Not in cards
ALFRED Agents follow natural-language instructions to finish household tasks in the AI2-THOR simulator from first-person views, such as rinsing a mug and putting it in the coffee maker. The benchmark has expert demonstrations for 25,000 directives with both high-level goals and step-by-step instructions. Scores count tasks completed and goal conditions met in seen and unseen rooms. Everyday Dec 2019 Not in cards
ARC-AGI Few-shot grid puzzles: infer a hidden transformation from about three examples and produce the output grid. Semi-private sets verified by ARC Prize. Everyday Nov 2019 98.5% In use
ScanNet-1500 Models match features between pairs of indoor views and estimate relative camera poses. The evaluation measures pose accuracy on 1,500 image pairs. School Nov 2019 Not in cards
Jester gesture recognition Models classify hand gestures in short videos. The task distinguishes movements with similar appearance but different directions or timing. Everyday Oct 2019 Not in cards
FreiHAND Models recover a hand's joint positions and surface from a color image. The evaluation compares predictions with reference hand geometry. School Sep 2019 Not in cards
OCR-VQA Models answer questions about book covers by reading visible text. The benchmark compares answers with facts such as a book’s title or author. School Sep 2019 82% Not in cards
RLBench Robot arm policies attempt 100 hand-designed manipulation tasks in simulation, from reaching a target and opening a door to multi-stage tasks such as putting a tray in an oven. Each task has camera, depth and joint observations and an unlimited supply of motion-planned demonstrations. Scores are task success rates. Everyday Sep 2019 Not in cards
DIODE Models estimate depth in indoor and outdoor scenes. The dataset provides depth measurements from a laser scanner for comparison. School Aug 2019 Not in cards
DIOR Models locate and classify objects in remotely sensed images. The benchmark measures detection accuracy across different object sizes and backgrounds. University Aug 2019 Not in cards
FairFace Models classify face images using the dataset’s annotated age, gender, and race categories. The benchmark compares recognition performance across demographic groups. Everyday Aug 2019 Not in cards
LVIS Models locate objects and outline each instance across a large vocabulary of categories. Meta reports box and mask average precision. Everyday Aug 2019 Not in cards
SPair-71k Models match corresponding parts of objects across image pairs. The benchmark includes changes in viewpoint, scale, and occlusion. School Aug 2019 Not in cards
SpatialSense VQA Models judge whether a stated spatial relation between two objects holds in an image. The relations were crowdsourced to defeat simple cues. Everyday Aug 2019 60% Dropped
ImageNet-A Models classify real photographs that commonly fool image classifiers. The test uses unmodified images from ImageNet categories. Everyday Jul 2019 Not in cards
Kinetics-700 Models classify short video clips into 700 human action categories. Meta compares encoders on this expanded Kinetics edition. Everyday Jul 2019 Not in cards
ActivityNet-QA Models answer open-ended questions about web videos of human activities, from 58,000 QA pairs on 5,800 videos. Scored by accuracy. Everyday Jun 2019 66.7% Dropped
Fashion IQ Models retrieve a garment using a reference photograph and text describing how the target garment differs. Everyday May 2019 Not in cards
ImageNet-Sketch Models classify black-and-white sketches using ImageNet category labels. The benchmark tests recognition when images lack photographic texture and color. Everyday May 2019 Not in cards
iSAID Models detect and segment individual objects in aerial images. The benchmark measures how well predicted masks match annotated object instances. University May 2019 Not in cards
OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. Everyday May 2019 60.2% Dropped
ST-VQA Models answer questions by reading text in a scene. The benchmark compares each answer with the text required by the question. School May 2019 Not in cards
YouTube-VIS Models detect, segment, and track object instances in YouTube videos. The SAM 3 report evaluates the 2021 edition. Everyday May 2019 Not in cards
MUSIC-21 Instrument Separation Models separate performances from a mixture using video of the musicians. When two musicians play the same instrument, their movements help identify which sound belongs to each. University Apr 2019 Not in cards
TextVQA Models answer 45,336 questions about 28,408 images that require reading text in the image. Scoring is VQA accuracy against human answers. Everyday Apr 2019 85% In use
VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. Everyday Apr 2019 71.3 Fading
VATEX ZH Models write Chinese captions for short videos of human activities from the VATEX dataset. Captions are scored with CIDEr against human references. Everyday Apr 2019 59.7 Dropped
COIN instructional video analysis Models recognize steps in videos of people completing everyday tasks. The benchmark supplies annotated actions and their time intervals. Everyday Mar 2019 Not in cards
ImageNet-C Models classify images under common corruptions at different severities. The benchmark measures errors under blur, noise, weather, and digital distortions. Everyday Mar 2019 Not in cards
Meta-Dataset Models learn from episodes sampled across diverse image datasets and classify novel classes from a few examples. University Mar 2019 Not in cards
Nuscene Models localize named objects as 3D boxes in nuScenes driving camera images. Scored by mAP. Everyday Mar 2019 16.0 Fading
nuScenes Detection Models detect 3D objects in nuScenes sensor data. The official test leaderboard ranks submissions by nuScenes Detection Score. Everyday Mar 2019 Not in cards
GQA Models answer compositional questions about real-world images, generated from scene graphs. Scored by exact-match accuracy, often on the balanced test-dev split. Everyday Feb 2019 62% Fading
ImageNetV2 Models classify a newly collected set of photographs using the original ImageNet labels. The test measures how accuracy changes on fresh images. Everyday Feb 2019 Not in cards
CheXpert Models read chest radiographs and label up to 14 observations. The test labels come from board-certified radiologists. Expert Jan 2019 Not in cards
MIMIC-CXR (classification) Models read chest X-rays from Beth Israel Deaconess and say whether five common conditions are present. Scored by average F1 over the conditions. Expert Jan 2019 Not in cards
MIMIC-CXR report generation Models write the findings of a radiology report for a MIMIC-CXR chest X-ray. Scored against the radiologist's report, most often by RadGraph F1. Expert Jan 2019 Not in cards
nocaps Models caption photographs that contain objects outside the caption training vocabulary. The benchmark compares generated captions with human references. Everyday Dec 2018 Not in cards
Dillen herbarium specimen label benchmark Models transcribe specimen labels from images of pressed plants. The collection pairs 1,800 images from nine European herbaria with transcribed records. School Nov 2018 Not in cards
NLVR2 Models judge whether a sentence is true of a pair of photographs. The benchmark measures agreement with annotated truth values. Everyday Nov 2018 Not in cards
VQA-RAD Vision-language models answer clinicians' questions about 315 radiology images, including head CTs, MRIs and chest X-rays. Questions are closed yes or no, or open free text. Expert Nov 2018 Not in cards
BabyAI Agents follow compositional English instructions to navigate rooms, open doors, and move objects in a grid world. The environment checks whether each instruction was completed, and evaluation reports task success rates. Everyday Oct 2018 37.5% Not in cards
MELD Models identify emotions in utterances from multiparty conversations. The dataset includes speech, video, and text. Everyday Oct 2018 Not in cards
TallyQA Models answer open-ended counting questions about images, including complex ones that need object relationships or attributes. Scored by exact count accuracy. Everyday Oct 2018 61.5% Dropped
3DPW Models recover a person's three-dimensional pose in outdoor video. Reference poses come from wearable sensors and a moving camera. School Sep 2018 Not in cards
Diving48 Models classify competitive dives from videos by distinguishing their movement sequences. Expert Sep 2018 Not in cards
TVQA Models answer 152,545 multiple-choice questions about clips from 6 TV shows, using video frames and subtitles. Scoring is accuracy. Everyday Sep 2018 87.3% Dropped
YouCook2 retrieval Models retrieve cooking videos from text queries in the YouCook2 collection. University Sep 2018 52.5 Fading
YouTube-VOS Models segment and track objects across frames in YouTube videos. The evaluation distinguishes categories seen during training from previously unseen categories. Everyday Sep 2018 Not in cards
Kinetics-600 Models recognize one of 600 human actions in short video clips. Everyday Aug 2018 Not in cards
PatchCamelyon Models identify whether a microscope image patch contains metastatic tumor tissue. Expert Jun 2018 69.3% Not in cards
CrisisMMD Models classify disaster photographs and their accompanying posts for humanitarian use. School May 2018 Not in cards
RAVDESS Models identify an actor's expressed emotion from recorded speech or song. Everyday May 2018 Not in cards
MSD-I Multimodal Genre Classification Models predict a track’s genre from its audio and album cover. The published test contains 4,649 tracks across 15 genres. Everyday Apr 2018 Not in cards
MUSIC Instrument Separation Models use images of musicians to isolate an instrument from a mixture of performances. The authors measure separation on mixtures of solo recordings, where each instrument has a reference track. University Apr 2018 Not in cards
Places Audio Captions (Hindi) Models retrieve photographs from spoken Hindi descriptions or retrieve descriptions from photographs. The score counts how often the correct match appears among the top results. Everyday Apr 2018 Not in cards
Audio-Visual Event Localization Models identify an event that can be both seen and heard in a video and locate when it occurs. Everyday Mar 2018 Not in cards
Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. Everyday Mar 2018 Not in cards
Revisited Oxford and Paris Models retrieve photographs of the same landmark from Oxford and Paris image collections. The revised benchmark defines medium and hard evaluation protocols. Everyday Mar 2018 Not in cards
iNaturalist 2018 Models identify species in photographs submitted by naturalists. The 2018 competition uses an expanded set of plant and animal categories. University Feb 2018 Not in cards
VizWiz Models answer over 31,000 spoken questions that blind people asked about photos they took with phones. Some questions are unanswerable from the image. Everyday Feb 2018 Not in cards
MSRVTT-QA Models answer short open-ended questions about web video clips from the MSR-VTT captioning dataset. The question and answer pairs were generated automatically from the clip descriptions. Scored by answer accuracy. Everyday Oct 2017 Not in cards
RSICD Models describe aerial scenes in a sentence. The authors provide five reference captions per image and reserve 1,093 images for testing. Everyday Sep 2017 Not in cards
DiDeMo Models locate the time interval in a video that matches a natural language description. Everyday Aug 2017 Not in cards
EuroSAT Models classify land use and land cover in Sentinel-2 satellite images. University Aug 2017 83.6% Not in cards
Animals with Attributes 2 Models use shared attributes to recognize animal classes they did not see during training. The benchmark provides 37,322 photographs across 50 classes with 85 attributes. School Jun 2017 Not in cards
ActivityNet Captions Models match descriptions of events to segments of untrimmed videos. Meta evaluates retrieval in both directions. Everyday May 2017 Not in cards
Charades-STA Models read a sentence describing an action and return its start and end time in a Charades indoor video. Scored by temporal IoU with the labeled moment. Everyday May 2017 64% Fading
ChestX-ray14 Models read frontal chest X-rays from the NIH Clinical Center and say whether thoracic conditions are present. Expert May 2017 Not in cards
Kinetics-400 Models recognize one of 400 human actions in short video clips. Everyday May 2017 0.25% Not in cards
DAVIS 2017 Models track and segment individual objects throughout a video. The evaluation measures region overlap and boundary accuracy. Everyday Apr 2017 Not in cards
iNaturalist 2017 Models identify the species in photographs submitted by naturalists. Meta evaluates transfer to fine-grained species classification. University Apr 2017 Not in cards
TGIF-QA Models answer questions about actions, repetitions, and changes over time in animated clips. Everyday Apr 2017 Not in cards
NWPU-RESISC45 Models classify remote sensing images into 45 scene categories. University Mar 2017 88.3% Not in cards
YouCook2 Models describe the procedure steps in YouTube cooking videos. Scored by CIDEr against human step captions. Everyday Mar 2017 188.3 Dropped
YouCook2 Dense Video Description Models locate cooking steps and describe each step. The score is CIDEr for the complete dense video description task. Everyday Mar 2017 Not in cards
ScanNet Models infer indoor scene geometry from images. Meta evaluates depth predictions against measurements from scanned indoor environments. School Feb 2017 Not in cards
VQAv2 Open-ended questions about COCO images, balanced to reduce language priors. Scored by consensus accuracy against 10 human answers. Everyday Dec 2016 88.4% Fading
LSMDC Models fill in a missing word in movie audio descriptions using the matching video clip. Scoring is accuracy. Everyday Nov 2016 45.7% Dropped
ADE20K Models label the pixels in a scene by object or region. Meta measures overlap between predicted and reference segments. Everyday Aug 2016 Not in cards
RefCOCO / RefCOCO+ / RefCOCOg Model grounds a referring expression to a bounding box in a COCO image. Scored by accuracy at IoU 0.5. Everyday Jul 2016 95% In use
Sydney-Captions Models describe land use in aerial images of Sydney. The dataset pairs 613 images with five human captions each. Everyday Jul 2016 Not in cards
UCM-Captions Models describe aerial land use scenes from the UC Merced collection. The dataset pairs 2,100 images with five human captions each. Everyday Jul 2016 Not in cards
MSR-VTT Models write captions for short web videos. Human-written sentences provide multiple descriptions of each clip. Everyday Jun 2016 Not in cards
MSR-VTT retrieval Models retrieve videos from text queries in the MSR-VTT collection. University Jun 2016 68.0 Fading
Cityscapes Models assign a class to each pixel in street scenes filmed from a vehicle. The benchmark compares predicted segments with annotated road scenes. Everyday Apr 2016 Not in cards
AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. School Mar 2016 96.9% Fading
Stanford Online Products Models retrieve photographs of the same product. The test compares learned image features on products outside the training set. Everyday Nov 2015 Not in cards
SUNRGBD Models detect objects in about 10,000 indoor RGB-D images and predict their 3D bounding boxes. Scored by average precision. Everyday Jun 2015 48.3 Fading
COCO Captions Models write a caption for each image, and outputs are compared with five human captions per image using BLEU, METEOR, ROUGE, and CIDEr. Everyday Apr 2015 116.0 Dropped
Food-101 Models identify the dish in a food photograph from 101 categories. Everyday Sep 2014 Not in cards
TACoS Models find the time span in a cooking video that matches a sentence. Labs score temporal grounding by mean IoU with the annotated span. Everyday Sep 2014 49.6% Dropped
Birdsnap Models identify the North American bird species shown in a photograph. University Jun 2014 Not in cards
PASCAL-Context Models assign a semantic class to each pixel in a photograph. The benchmark includes objects and background regions. Everyday Jun 2014 Not in cards
COCO Models detect and locate objects in everyday photos from the Microsoft COCO dataset. Scored by mean average precision over box overlap thresholds. Everyday May 2014 78.7% In use
Stanford Cars Models identify a car's make, model, and year from a photograph. The benchmark measures fine-grained vehicle recognition. University Dec 2013 Not in cards
Describable Textures Dataset Models assign a texture description to a photograph based on how its surface looks. Everyday Nov 2013 Not in cards
EVVE Models retrieve other videos showing the same real-world event as a query video. School Jun 2013 Not in cards
FGVC-Aircraft Models identify aircraft variants from photographs. University Jun 2013 39.7% Not in cards
UCF101 Models recognize one of 101 human actions in video clips. Everyday Dec 2012 Not in cards
PASCAL VOC 2012 Models locate objects and label their pixels in photographs. Meta reports detection and segmentation results on the 2012 challenge. Everyday May 2012 Not in cards
HMDB51 Models identify human actions in short video clips across 51 categories. Everyday Nov 2011 Not in cards
Leeds Sports Pose Extended Models locate body joints in photographs of athletes. Meta compares the fraction of predicted joints within a distance threshold. School Jun 2011 Not in cards
MIT-Adobe FiveK Models adjust photographic tone to match human edits. Five photographers each edited 5,000 raw photos in Lightroom. The original evaluation compares luminance with each reference edit. University Jun 2011 Not in cards
STL-10 Models classify images into ten object categories using limited labeled training data. Everyday Jun 2011 Not in cards
German Traffic Sign Recognition Benchmark Models identify the class of a traffic sign in a cropped road image. Everyday Dec 2010 Not in cards
Caltech-UCSD Birds 200 Models identify bird species from photographs. The benchmark distinguishes species with similar appearance. University Sep 2010 Not in cards
SUN397 Models recognize the scene category in indoor and outdoor photographs. The classification task covers 397 categories. Everyday Jun 2010 Not in cards
CIFAR-10 Models classify small color images into ten object categories. Everyday Apr 2009 Not in cards
CIFAR-100 Models classify small color images into 100 object categories. School Apr 2009 Not in cards
Oxford Flowers 102 Models identify flower categories from photographs. The benchmark tests recognition across changes in shape, scale, and lighting. University Dec 2008 Not in cards
PASCAL VOC 2007 Models locate and classify objects in photographs. Meta evaluates object detection using the 2007 Visual Object Classes challenge images. Everyday Apr 2007 Not in cards
3D Arena Users compare two 3D models generated from the same input and vote for the better one. A live leaderboard ranks the generators. Everyday Not in cards
3D Worlds Arena Models generate explorable 3D scenes from an image and caption. Voters inspect two anonymous scenes from the same input and choose a favorite. Everyday Not in cards
AA-Image-Editing Validated private evaluators compare two image edits made from the same instruction and vote blind. Votes become Elo ratings. Everyday Not in cards
AA-Video-Editing Voters compare two edits of the same video made from one instruction and pick the better one. Votes become Elo ratings. Everyday Not in cards
Animal-Bench Models answer 41,839 questions about animal videos. The overall score averages accuracy across 13 tasks, including action prediction and recognizing predator and prey behavior. School Not in cards
Animation Bench Agents recreate web animations in HTML from reference frames and a network capture. The scorer compares appearance, motion and layout. University 0.61 Not in cards
Artificial Analysis Video Arena Voters compare videos from two text-to-video models for the same prompt and pick the better one. Votes become Elo ratings. Everyday Not in cards
AudioCaps Models match natural language descriptions to audio clips. Meta also compares how accurately encoders match audio, video, and text from the same clip. Everyday Not in cards
Caltech-101 Models identify object categories in photographs. Meta compares the classification accuracy of image encoders on this collection. Everyday Not in cards
Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. Everyday 100% System cards only
COCO-Stuff 164K Models label each pixel with an object or background category. The benchmark adds annotations for regions such as sky, grass, and pavement. Everyday Not in cards
Common Objects in 3D v2 Models estimate camera poses from images of objects viewed from different angles. Meta compares predicted poses with the reference poses. School Not in cards
CompassArena Multimodal Users compare anonymous multimodal model answers and vote for the better response. A Bradley-Terry model estimates ratings from these votes. Everyday 1,173 Not in cards
CURIE Biodiversity Georeferencing Models read a map and its caption from an ecology paper and return the geographic bounds. The score measures overlap with an expert's bounding box. University 19.4 Not in cards
DeepSDO Description Models describe solar features visible in full-disk images from the Solar Dynamics Observatory. The scorer compares generated captions with human descriptions. University Not in cards
DesignArena Agentic Image to HTML Coding agents turn a user-uploaded image into a self-contained HTML website. Users compare how well each page reproduces the design and vote. Everyday 1,247 Not in cards
DesignArena Image Editing Image models edit a user image following a text instruction. Users compare the two edits and vote for the better one. Everyday 1,368 Not in cards
DesignArena Image to Frontend Coding agents turn a reference image into a multi-file frontend web app. Users compare the running apps and vote for the better one. Everyday 1,299 Not in cards
DesignArena Image to HTML Models recreate a reference image or mockup as a working single-file HTML website. Users compare the rendered pages and vote for the closer, better one. Everyday 1,300 Not in cards
DesignArena Image to Video Video models animate a user image into a short video. Users compare two anonymous videos and vote for the better one. Everyday 1,319 Not in cards
DesignArena Multi-Image to Video Video models compose one video from several reference images. Users compare two anonymous videos and vote for the better one. Everyday 1,329 Not in cards
DesignArena Multimodal to Video Video models make a video from a mix of text, images and other inputs. Users compare two anonymous videos and vote for the better one. Everyday 1,305 Not in cards
DesignArena Video Editing Video models edit a user video following a text instruction. Users compare the two results and vote for the better one. Everyday 1,373 Not in cards
DesignArena Video to Website Models watch a reference video and recreate its content as a working single-file HTML website. Users compare the rendered pages and vote. Everyday 1,274 Not in cards
Docci Models retrieve matching documents and images using the DOCCI corpus. University 93.4 Fading
Dollar Street Models recognize household objects in photos from families at different income levels. The benchmark supports comparisons across regions and income groups. Everyday Not in cards
DTU Multi-View Stereo Models reconstruct objects from multiple camera views. The benchmark measures the accuracy and completeness of reconstructed surfaces against scanned reference geometry. School Not in cards
Dubai-CC Models describe land use changes between two satellite views of Dubai. The Change3D team used 150 of the 500 image pairs for testing. School Not in cards
EmbodiedCLUE-VLA (planning) Models reason about actions in physical environments. Everyday 86.8 Not in cards
EmbodiedCLUE-VLA (tabletop tasks) Robot policies carry out tasks on a tabletop. Everyday 49.3 Not in cards
EmbodiedVerse-Bench Models answer questions about spatial relationships and actions from visual observations. The leaderboard compares models across embodied reasoning tasks. Everyday 50.7 Not in cards
EndoVis 2018 Robotic Scene Segmentation Models segment structures and surgical instruments in video from robot-assisted operations. The evaluation compares predicted masks with reference labels. Everyday Not in cards
ETH3D Models reconstruct scene geometry from images. Meta evaluates depth predictions against reference measurements on the ETH3D test scenes. School Not in cards
Flickr30k Models describe photographs of people, objects, and activities. Meta compares model captions with the human reference captions. Everyday Not in cards
Floorplan Arena Models furnish hand-drawn room sketches. People compare the layouts and vote for the better floorplan. Everyday 1,807 Not in cards
Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. Not in cards
GeoBiaset Models predict photo coordinates before and after researchers insert a person into the scene. The coordinates stay fixed across all edits of each background. University 69.2% Not in cards
GeoGuessBench (panoramic) Models see four Google Street View images taken at right angles. They predict coordinates and report their confidence. Closer guesses earn more points. Everyday 4771.5 Not in cards
Goliath human mesh recovery Models reconstruct a person's body from images of studio performances. Meta measures joint and surface errors against captured three-dimensional motion. School Not in cards
Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. Expert System cards only
Image Matching Challenge Models match photographs of landmarks taken from different viewpoints. The evaluation measures the accuracy of camera poses recovered from the matches. School Not in cards
ImageNet-1K Models assign one of a thousand object labels to each photograph. Meta reports classification accuracy on the validation images. Everyday Not in cards
KITTI depth prediction Models estimate scene depth from road images. The evaluation compares model predictions with depth measurements collected from a moving vehicle. School Not in cards
LiveOCRVQA Models read stylized text in images and answer questions about it. Everyday 80.5 Not in cards
LMArena Video Edit Arena Video editing models apply the same edit request to the same clip, anonymously and side by side. Community votes give Elo-style ratings. Everyday 1,424 Not in cards
MIEB Lite Embedding models complete the MIEB Lite task set. The official API reports the mean across task types. 65.7 System cards only
MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. Everyday 88.1% Fading
MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. Everyday 72.1% Not in cards
MSVD Models retrieve short videos from human descriptions and retrieve descriptions from videos. Meta reports retrieval accuracy in both directions. Everyday Not in cards
MuseBench (Music Score and Performance Understanding) Models answer music theory questions, read piano scores, and interpret recorded performances. The authors report accuracy separately for text, images, and audio. University 52.8% Not in cards
NeuroscienceBench Agents analyze neuroscience research data and report a scientific conclusion. The score counts correct conclusions among graded runs. Expert 41% Not in cards
NYU Depth V2 Models estimate depth from indoor photographs. The benchmark compares predictions with depth measurements captured by a sensor. School Not in cards
ObjectNet Models identify familiar objects photographed with controlled changes in viewpoint, orientation, and background. Meta reports results on classes shared with ImageNet. Everyday Not in cards
OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. System cards only
Oxford-IIIT Pet Dataset Models identify the breed of a cat or dog from a photograph. Meta measures breed classification across 37 categories. School Not in cards
PCB-Bench Models answer placement and routing questions and describe circuit boards from editor screenshots. The authors report choice accuracy and separate measures for free-form answers. University Not in cards
Personal Agent Bench Personal assistants complete everyday tasks using connected accounts. Experts check whether each assistant completes the work and respects the user’s instructions. Everyday Not in cards
Places205 Models classify photographs by scene type. Meta compares scene recognition on the 205-category edition of Places. Everyday Not in cards
Razavi-Bench Models explain what happens in analog circuits and why. A language model awards up to four points per answer against a reference and a rubric. University 98.3% Not in cards
RealEstate10K Models estimate relative camera poses across frames from real estate videos. The benchmark supplies camera trajectories for comparison. School Not in cards
Recipe1M image and recipe retrieval Models match food photographs with recipes. Google evaluates image retrieval of the matching ingredients and instructions separately. Everyday Not in cards
Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. Everyday Not in cards
SHREC 2026 Human Perceived Visual Complexity of 3D Shapes Models rank CAD parts, primate teeth, and fractals by visual complexity. People compared pairs of CAD parts and fractals. A zoological measure ranks the teeth. Everyday Not in cards
Something-Something V2 Models recognize actions involving people manipulating everyday objects. The benchmark distinguishes actions that differ in direction, ordering, or outcome. Everyday 74% Not in cards
STAR situated video reasoning Models answer multiple-choice questions about how people interact with objects in videos. The questions cover interactions, sequences, predictions, and feasibility. Everyday Not in cards
SuperCLUE Arena Image Editing Models edit images in response to instructions. The arena ranks systems from pairwise comparisons. University 1,308 Not in cards
SuperCLUE Arena Image to Video Models animate images. The arena ranks systems from pairwise comparisons. University Not in cards
SuperCLUE Arena Reference to Video Models generate videos from references. The arena ranks systems from pairwise comparisons. University Not in cards
SuperCLUE-AdsVU Models interpret video advertisements. Everyday 89.1 Not in cards
SuperCLUE-I2V Models generate videos from still images. University 71.5 Not in cards
SuperCLUE-R2V Models generate videos conditioned on reference material. University 64.9 Not in cards
SuperCLUE-V Models answer Chinese questions about visual inputs. Everyday 75.1 Not in cards
SuperCLUE-VideoEdit Models edit videos in response to Chinese instructions. University 74.2 Not in cards
SuperCLUE-VLM Vision language models interpret images and answer questions in Chinese. Everyday 89.0 Not in cards
SuperCLUE-VLR Models solve problems that require visual reasoning. University 72.1 Not in cards
SuperCLUE-Voice (audiovisual interaction) Models respond to audio and video inputs in real time. Everyday Not in cards
SuperCLUE-World World models generate visual sequences. University 74.6 Not in cards
TartanAir v2 Models reconstruct camera motion and scene geometry from simulated trajectories. The evaluation compares predictions with known poses and depths. School Not in cards
TextCaps Models retrieve matching text and images using the TextCaps corpus. University 89.6 Fading
The Met Dataset Models match photographs of artworks to catalog images from the Metropolitan Museum of Art. The evaluation measures instance recognition across photographic conditions. Everyday Not in cards
TissueMicroscopyBench Agents examine tissue images and answer questions about cells, staining, and changes over time. Expert 34% Not in cards
Toloka Visual Question Answering Benchmark Models answer questions about images by locating the object that answers each question. The benchmark compares predicted and reference bounding boxes. Everyday Not in cards
TRUE (Text Recognition and Understanding) Models recognize text in images and answer questions about its content. Everyday 81.6 Not in cards
VIPSeg Models label video pixels and keep instance identities consistent across frames. The benchmark includes both countable objects and background regions. Everyday Not in cards
WMArena Video Arena Models generate a video from a starting image and an action. Viewers compare anonymous clips and vote for the more convincing result. University 1,487 Not in cards
Wojak-Bench 3D Models turn the same drawing into a three-dimensional figure. Voters compare two anonymous prints or renders and choose which better matches the drawing. University 1,273 Not in cards