Benchmark Almanac

Image, video and 3D generation

Models make images, video or 3D models. We count 401 benchmarks here, 66 of them arenas. A benchmark belongs here when output includes images, video or 3D.

How fast it grows

116 were released since July, more than the 63 in the first half of 2026.

  1. Earlier 69
  2. 2024 H1 15
  3. 2024 H2 26
  4. 2025 H1 34
  5. 2025 H2 28
  6. 2026 H1 63
  7. 2026 H2 116

How much is still open

No model has passed 85% on 50 of the 59 benchmarks here that score in percent, 85% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
3DArena-Modeling Agents create 3D models for 120 tasks covering precision and appearance. Tencent converts Elo ratings from blind comparisons into percentages. University Nov 2026 Not in cards
3DSceneBench Agents reconstruct interactive MuJoCo scenes from images and task descriptions. Verifiers measure appearance, geometry, and physical functionality, then average the three scores. Expert Nov 2026 59.5 Not in cards
AudioVideoBench Agents produce and edit audio and video in containers. The tasks include compositing, mixing, repairing media, reframing shots, and transcribing recordings. University Nov 2026 Not in cards
Construction AI Leaderboard Models answer construction questions, inspect drawings, edit building models, and generate validation rules. ONESTRUCTION averages the representative scores from five evaluations. University Nov 2026 73.7 Not in cards
EVF-Bench Models generate product videos with visual effects. The evaluator checks motion, product appearance, text stability, and whether the effects fit the request. University Nov 2026 51.6 Not in cards
Frontier Design Benchmark Professional designers vote blind between two models’ designs for the same real client brief from Contra’s marketplace, such as a marketing site. Models are ranked by Elo from the votes. Expert Nov 2026 1,713 Not in cards
Frontier3DBench Agents create editable Blender objects and scenes or executable Unreal Engine worlds across 145 tasks and nine task families. Model judgments and native project checks assess appearance, geometry, requirements, and interactions. The overall score weights object, scene, and world scores by 30%, 30%, and 40%. Expert Nov 2026 66.0 Not in cards
HandWorldBench Models generate videos of human or robotic hands moving objects. The benchmark combines 17 measures of hand motion, object state, and video quality. Everyday Nov 2026 52.2 Not in cards
InfiHall-Bench Models generate images from 411 prompts that combine visual requirements. The evaluator checks details such as object parts, actions, and spatial relations separately. Everyday Nov 2026 67.1 Not in cards
LongBanana Models synthesize images from many reference images and textual constraints. Atomic rubric checks score content fidelity, relationships, and global constraints across 90 tasks. Everyday Nov 2026 74.6 Not in cards
MedHEB Models retrieve related medical images, reports, and image regions. The authors evaluate how models rank relevant evidence across 107 retrieval tasks. University Nov 2026 24.6 Not in cards
MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. Expert Nov 2026 56.5% In use
PhyRubric-Bench Models generate videos of physical events. A judge checks caption-specific requirements against evidence at particular times in each clip. School Nov 2026 36.2 Not in cards
Pixel2Perspective Models locate and segment objects from aerial and ground views. Tests vary the camera angle, weather, and use of visible or infrared images. Everyday Nov 2026 Not in cards
Rapidata Multi-Step Image Editing (Identity Drift) Image editors apply five sequential changes to 199 photographs, and people compare the final faces for identity drift. Expert Nov 2026 1,866 Not in cards
Real2Sim Coding agents get a robot recording from DROID (three camera videos, stereo calibration, joint and gripper commands and the task instruction) and build an interactive simulation of the scene. Each scene is scored on how closely it matches the recording, on replaying the recorded actions, and on whether a fixed robot policy completes the task in it. Expert Nov 2026 0.60 Not in cards
SolidWorks Bench Agents model the custom parts of 73 real machines in SolidWorks and assemble them from raster drawing packages, through screenshots and the SolidWorks API. A grader checks the geometry against the reference and reads mates, motion and feature trees from the native files. Expert Nov 2026 0.57 Not in cards
T2I-NSI Models generate images from paired positive and negative prompts. The index measures whether removing a requested concept also removes it from the image. Everyday Nov 2026 0.1 Not in cards
VAHBench Agents turn short drama briefs into finished videos with specified characters, dialogue, and durations. The evaluator checks completeness, quality, reliability, and accuracy of production reports. University Nov 2026 51.8 Not in cards
VibePrevisBench Coding agents write Blender Python that turns a script, photos, an opening frame or reference footage into a graybox 4D previs scene and render it. Rule checks score the exported scene against hidden facts, and a VLM judge checks that those facts are visible in the render. Expert Nov 2026 77.7 Not in cards
WorkflowArena Agents write executable workflows that coordinate other agents on professional tasks. Hidden verifiers check the completed deliverables. Expert Nov 2026 37% Not in cards
AA-3D Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings. Expert Oct 2026 1,318 In use
ABACUS Models add, remove, or change a specified number of objects in an image. The evaluator checks the count and preservation of the surrounding scene. School Oct 2026 0.31 Not in cards
AdSpark-Bench Models generate advertisement videos from product images and prompts. The benchmark checks whether products remain consistent and whether each shot conveys the requested selling point. University Oct 2026 66.9 Not in cards
BenchCAD 2.0 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 59% Not in cards
BenchCAD 2.1 Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 83.5% In use
BenchCAD 2.1-beta Models reconstruct parts, assemblies, and circuit boards from drawings and images. The Core set contains 100 problems with partial credit. Expert Oct 2026 Not in cards
BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. Expert Oct 2026 79.9 Not in cards
CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. Expert Oct 2026 76.6 Not in cards
Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. Expert Oct 2026 $9,514 Not in cards
Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. University Oct 2026 957 In use
Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. Everyday Oct 2026 Not in cards
Ego2Act Models generate first-person video showing how to complete a task from an initial image. A judge checks the steps and their physical plausibility. Everyday Oct 2026 83.3 Not in cards
FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. Expert Oct 2026 0.67 Not in cards
FrontierCAD Assemblies Agents place supplied CAD parts into 16 assemblies. The score is the published mean assembly geometry score. Expert Oct 2026 0.71 Not in cards
FrontierCAD Parts Agents build 39 CAD parts from engineering drawings. The score is the published mean geometry score. Expert Oct 2026 0.63 Not in cards
GeoFlip Models generate videos from paired scenes where a geometry change reverses the expected physical outcome. Model judges check the event and the scene structure. School Oct 2026 3.17 Not in cards
HealBench-Eval Models generate images intended to provide psychological comfort. The authors compare 14 image generators and prompt strategies on 1,095 cases. Everyday Oct 2026 26.1 Not in cards
Infographic factuality (Google) Image generation systems produce infographics and an automated rater scores factuality. Everyday Oct 2026 0.52 In use
Kaggle Sort Viz Arena Models use Manim to animate sorting algorithms and explain their time complexity. A submission must produce a valid video under the file size limit. Votes determine the arena rating. University Oct 2026 1,650 Not in cards
LightGenBench Models predict which parts of a 3D surface emit light and what color they emit. The authors compare generated emission with reference assets. University Oct 2026 Not in cards
LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. Expert Oct 2026 66.5% Not in cards
LogicVal Models generate images from prompts with formal constraints on objects, attributes, counts, and relationships. The evaluator checks which constraints the image satisfies. School Oct 2026 50.9% Not in cards
MarathonDocBench Models parse 100 documents spanning 103 to 300 pages. Evaluation scores content, cross-page continuations, figures, reading order, and heading hierarchy. Expert Oct 2026 77.1% In use
MoI-Bench Models generate and edit images for mobile users. The authors compare image quality across 64 subtasks and measure the cost of running models on phones. Everyday Oct 2026 0.97 Not in cards
Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. Expert Oct 2026 0.49 Not in cards
PROWBench Programmable world models must render video that follows the rules and interactions their program specifies. The test checks visual fidelity to fine-grained rules. Everyday Oct 2026 68.0 Not in cards
RGraph-Eval Models generate robot manipulation videos. The evaluator compares objects, changing relationships, and three-dimensional motion paths with a reference demonstration. Everyday Oct 2026 0.89 Not in cards
RISEBench++ Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types. University Oct 2026 56.6% Not in cards
RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. Expert Oct 2026 62% Not in cards
ScribbleEdit Models edit images using a scribble as the only instruction. The evaluator checks whether the change matches the intended operation and location. Everyday Oct 2026 0.26% Not in cards
Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. Expert Oct 2026 $2,949 Not in cards
SimReadyBench NV100 Agents reconstruct an articulated 3D asset from one image and nominal dimensions. The verifier grades formal, visual, and kinematic fidelity separately across 100 assets. Expert Oct 2026 100% Not in cards
SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. Expert Oct 2026 Not in cards
SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. Expert Oct 2026 27.8% Not in cards
SplatBench A model rebuilds an object in 3D from eight photos by placing up to 300 Gaussians itself through tool calls, with no optimizer. Half of the 30 tasks start from an empty scene, and half ask it to reassemble a scrambled fit. SplatScore compares renders from eight angles the model never sees, with an empty scene at 0 and gradient descent at 100. University Oct 2026 49.6 Not in cards
StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. Everyday Oct 2026 Not in cards
TextVBench Models render requested text in video or preserve text from a reference image. The score tracks spelling, placement, and consistency across frames. Everyday Oct 2026 68.3 Not in cards
UltraText Bench Image generators render dense bilingual text in many regions from 432 prompts. Scores check string accuracy, placement and legibility. Everyday Oct 2026 99.3 Not in cards
VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. Expert Oct 2026 4.11 Not in cards
VTR-Bench Models generate videos containing visible text. The evaluation checks the rendered words and how well the text persists across frames. Everyday Oct 2026 0.89 Not in cards
WeGenEditBench Models edit images from instructions that can include text or multiple references. A shared checklist scores each requested change and the details that should stay intact. Everyday Oct 2026 4.14 Not in cards
WorldState-Bench Video models generate two clips from the same image and action. A hidden state changes which ending the rule requires. A vision model checks both clips. Everyday Oct 2026 32% Not in cards
AV-SafetyBench Models generate videos with sound from written prompts. Judges review each clip with sound, in silence, and as audio alone to identify unsafe outputs. Everyday Sep 2026 49.4% Not in cards
BlenderBench (Refresh) Agents edit Blender scenes across 20 tasks. Each task grades the submitted scene using its own scoring configuration. University Sep 2026 0.69 Not in cards
BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. Expert Sep 2026 76.4 In use
CAD Arena Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match. Expert Sep 2026 0.75 Not in cards
Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. Expert Sep 2026 1,585 Not in cards
Code4Scene Agents build and edit Unreal Engine scenes. The evaluator inspects the scene itself for correct edits, intact assets, and physical validity. University Sep 2026 0.62 Not in cards
CutCraft Benchmark Models generate multi-shot videos under editing instructions. The evaluator checks transitions, shot structure, and the timing of audio relative to cuts. University Sep 2026 0.64 Not in cards
EngiWorld Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software. Expert Sep 2026 44.7% Not in cards
GenPuzzle Models generate images that solve visual puzzles such as mazes, tangrams, and Sudoku. The authors select an automated judge using human reference scores. School Sep 2026 40.6 Not in cards
GenScale Models generate or edit images containing objects with known relative sizes. A judge calibrated against human ratings scores the relative scale of object pairs. Everyday Sep 2026 Not in cards
HappyWorld Embodied The HappyWorld arena publishes a separate embodied vote pool and Elo ranking. Everyday Sep 2026 1,074 Not in cards
HappyWorld Spatial Reconstruction The HappyWorld arena publishes a separate spatial reconstruction vote pool and Elo ranking. Everyday Sep 2026 1,252 Not in cards
HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. Everyday Sep 2026 Not in cards
MindWorldBench Models animate a scene to show how a person would act given their beliefs or intentions. Paired prompts change mental states while holding the scene fixed. Everyday Sep 2026 Not in cards
OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. Everyday Sep 2026 72.7 Not in cards
ORAV Bench Models combine reference images, sounds, and videos into a requested audiovisual scene. A judge compares pairs of outputs against each reference's assigned role. Everyday Sep 2026 0.70 Not in cards
OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. Expert Sep 2026 73.7% Not in cards
Parametric CAD Bench Agents create and edit parametric FreeCAD models from text and engineering drawings. Programmatic checks award continuous task rewards for geometry and specification consistency. Expert Sep 2026 61.0 Not in cards
Runway Model Router Benchmark Video generation systems produce clips for 250 image-to-video prompts across ten production categories and are compared by usable rate and cost. Expert Sep 2026 Not in cards
SolveEdit Models edit an image to solve a visual problem. The evaluator checks whether the goal holds and whether unrelated scene details survive the edit. School Sep 2026 57% Not in cards
StateBench (video continuation) Models continue a video while tracking what happened in earlier segments. A judge checks whether visible objects, hidden processes, and later state changes remain consistent. Everyday Sep 2026 32.5 Not in cards
T2LSC-Bench Models render text at a specified location in an image. Judges check the lettering and whether its meaning alters other objects or the background. Everyday Sep 2026 99% Not in cards
TimelineBench Agents turn raw footage and a client brief into a finished video. The edit must pass delivery checks, meet the brief, and clear a quality threshold. University Sep 2026 32.1% Not in cards
TrafficImag Models edit roadside traffic videos to show a different sequence of events. The evaluator checks routes, interactions, and preservation of the surrounding scene. Everyday Sep 2026 90.0 Not in cards
TriWorldBench World models generate synchronized robot-view videos from head and wrist cameras. Nineteen signals measure cross-view agreement, task alignment, physical coherence, motion, temporal stability, and visual quality. Everyday Sep 2026 Not in cards
UFO-Bench Models generate customized images from text and visual references. The evaluator breaks each request into individual requirements and checks how they interact. Everyday Sep 2026 0.73 Not in cards
VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. Everyday Sep 2026 79.0 Not in cards
VIF-Bench Models generate images from multiple references and visual instructions. Judges check whether the result follows controls such as pose or lighting without copying unwanted features. Everyday Sep 2026 7.60 Not in cards
ADVANCE audio and image retrieval (MTEB) Models match field recordings and aerial images from 5,075 locations. The correct recording and image come from the same place. Everyday Aug 2026 Not in cards
AEC-Geometric-Bench Models locate doors, windows, and fixtures on construction drawings and trace walls and enclosed spaces. The scorer matches objects and compares wall and area masks. University Aug 2026 0.06 Not in cards
Brennan Center election image generation audit Image generators receive requests for misleading election scenes. Researchers count how often each system produces an image, including images that omit requested details. Everyday Aug 2026 100% Not in cards
ConceptEdit-Bench Models edit images according to 1,000 instructions across six categories. A vision-language judge scores instruction following, visual quality, and consistency in unedited regions from 0 to 10. The per-case score averages those three dimensions. Everyday Aug 2026 66.2 Not in cards
Cyc3D Models generate 3D assets from images. The evaluator checks mesh quality and measures how much shape and identity drift across repeated rendering and reconstruction. University Aug 2026 Not in cards
H2R-Bench Models turn first-person demonstrations into videos of a specified robot performing the task. The benchmark checks contacts, object responses, and completion of the goal. Everyday Aug 2026 81.0 Not in cards
HarnessEval-W Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning. Expert Aug 2026 75.5 Not in cards
OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. University Aug 2026 100% Not in cards
OmniPhys Models solve Chinese physics problems from middle school through university and draw diagrams. The evaluator grades answers, reasoning, and generated figures. University Aug 2026 80.4 Not in cards
PAWBench Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior. Everyday Aug 2026 Not in cards
PersonaShot Models generate multi-shot videos about a person. The evaluator checks character continuity, emotional changes, and camera choices within and between shots. Everyday Aug 2026 0.67 Not in cards
PlayWorld An agent player pursues objectives inside video world models. Gemini 3.1 Pro applies weighted rubrics for geometry, interaction, and state evolution across the resulting videos. Everyday Aug 2026 Not in cards
QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. University Aug 2026 69.9% In use
SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. Everyday Aug 2026 28.3 Not in cards
VGI-Bench Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame. Everyday Aug 2026 51% Not in cards
VWE-Bench Multimodal agents build interactive 3D worlds from a user request. They infer intent, plan the scene layout, call 3D tools for asset retrieval, editing and rendering, and revise from rendered feedback. The benchmark has 2,616 3D assets, 323 human-annotated seed worlds and 6,828 queries. Verified queries have ground truth, and the rest are scored with rubrics. Expert Aug 2026 57.3% Not in cards
WildArtifactBench Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo. Expert Aug 2026 Not in cards
WorldExam Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states. Everyday Aug 2026 72.8 Not in cards
AutoCAD Bench Computer-use agents turn a dimensioned reference drawing into a correct, editable DWG in AutoCAD through the screen alone, with mouse and keyboard. 50 tasks span 2D drafting and 3D modeling at four difficulty levels. Expert Jul 2026 46% Not in cards
Blind-Spots-Bench Models answer 235 text and visual problems chosen to expose failures on tasks that people find simple. A Gemini 3 Flash grader compares each response with a structured reference solution and gives binary credit. The grader can execute code to check exact constraints. Everyday Jul 2026 Not in cards
EmbodiedCLUE-World 2.0 World models predict outcomes in embodied settings under the second evaluation protocol. Everyday Jul 2026 41.8 Not in cards
KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. Everyday Jul 2026 0.81 Not in cards
MPIE-Bench Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other. Everyday Jul 2026 0.65 Not in cards
MultiRef-Compass Models generate audio-video content from several visual references and text instructions across 350 samples. Fourteen metrics assess quality, reference fidelity, audio-visual consistency, and instruction following. The evaluator combines automatic measurements with multimodal model judgments. Everyday Jul 2026 Not in cards
SoundingEarth audio and image retrieval (MTEB) Models match a field recording to an aerial image of the same location, or retrieve the recording from the image. Each direction has 1,000 queries and 2,048 candidates. Everyday Jul 2026 Not in cards
Arena-T2I Hard Models generate images from detailed user prompts. A judge checks whether the images satisfy individual requirements and their dependencies. University Jun 2026 0.85 Not in cards
BIM-Edit Agents create, change, or delete elements in IFC building models. The evaluator checks the resulting geometry, attributes, and relationships. University Jun 2026 87.7% Not in cards
BuildArena 2.0 Construction Challenge Agents build and test machines in Besiege for seasonal engineering challenges. The organizers score each entry using simulation results and the rules for its track. Competition Jun 2026 Not in cards
CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. Everyday Jun 2026 55.2 Not in cards
Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. Everyday Jun 2026 48.0 In use
Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. Everyday Jun 2026 10.9 Not in cards
TailOR Models generate images and videos of regular, unconventional, and impossible tool interactions. Annotators score instruction adherence, interaction accuracy, physical realism, and perceptual quality. Everyday Jun 2026 Not in cards
WorldOlympiad Models generate long videos for gaming, robotics, and everyday scenes. The evaluator checks physical rules, reconstructed geometry, and instruction following across video segments. Everyday Jun 2026 Not in cards
3DCodeArena Models generate 3D assets through code. Human preference votes determine the published Bradley-Terry Elo rating. Expert May 2026 1,230 Not in cards
3DCodeBench Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. May 2026 55.4% In use
AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. Expert May 2026 38.4% In use
Blueprint-Bench 2 Agents turn about 20 interior photos of each of 50 apartments into 2D floor plans. Scores compare room connectivity with ground truth, normalized against random. Everyday May 2026 58.6% In use
CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. University May 2026 45.1% System cards only
EduVideoBench Models generate classroom videos intended to teach facts, skills, or attitudes. Experts assess instructional accuracy, presentation, and educational safety. University May 2026 0.59 Not in cards
EmbodiedCLUE-World World models predict outcomes in embodied settings. Everyday May 2026 76.2 Not in cards
EvalVerse Video models generate cinematic clips from prompts and reference material derived from professional videos. Film experts compare outputs across visual design, acting, cinematography, aesthetics, emotion, shot sequences, and sound. A calibrated vision-language evaluator also estimates these judgments. Expert May 2026 0.73 Not in cards
LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. Everyday May 2026 0.90 Not in cards
MDMR-Bench Models generate one image from several references. A model judge checks whether the output preserves requested elements, follows instructions, and combines them correctly. Everyday May 2026 88.8% Not in cards
MPDocBench-Parse Models parse complete multipage documents while preserving content and cross-page structure. Evaluation scores text, formulas, tables, figure extraction, reading order, and heading hierarchy. Expert May 2026 84.6% In use
MultiMedia-TerminalBench Agents inspect and edit media in persistent terminal workspaces. The 105 tasks require verifiable output files across five areas of professional multimedia work. Expert May 2026 16.2% Not in cards
PaintBench Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks. Everyday May 2026 17.1% Not in cards
Parametric CAD Bench V1 Agents write FreeCAD scripts that build editable parametric parts from plain-English descriptions and dimension tables. Each of 100 tasks scores the harmonic mean of geometry similarity and spec consistency. Expert May 2026 In use
PRISM-Bench (audio and video generation) Models generate speech, music, or environmental sounds with video. A judge compares outputs with reference clips across 35 criteria, including synchronization and prompt following. Everyday May 2026 91.9 Not in cards
Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. Everyday May 2026 64.7 Not in cards
Tricky Edits of Collected and Curated Images Models edit a source image to follow a written instruction. People and a model judge instruction following, preservation of unaffected content, and visual quality. Everyday May 2026 19.5% Not in cards
WBench Interactive video world models run 289 multi-turn cases with navigation, subject actions, event edits and perspective switches. 22 metrics cover quality, adherence, consistency and physics. Everyday May 2026 79.0 Not in cards
WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. May 2026 69.5% Not in cards
Aether-Bench Models retouch photographs automatically, match a requested style, or apply specified adjustments. The style test measures fidelity on 100 image pairs from unseen presets, including peak signal-to-noise ratio. University Apr 2026 79.7% Not in cards
AVGen-Bench Models generate videos with audio from prompts across eleven real-world categories. Specialist models and multimodal judges measure perceptual quality, synchronization, and adherence to detailed visual, speech, music, and physical requirements. University Apr 2026 67.9 Not in cards
DataSciBench Agents complete open-ended data-science tasks that require data cleaning, analysis, visualization and modeling. Component checks and overall task success score the work. Expert Apr 2026 Not in cards
GraphicDesignBench Models analyze and produce graphic designs in 50 tasks spanning layout, typography, infographics, design semantics, and animation. The suite uses professional layered design templates. Expert Apr 2026 0.94 Not in cards
neuralCAD-Edit Agents edit existing CAD models from text or recorded instructions. CAD experts rate whether they would accept each edit. Software also compares the resulting geometry. University Apr 2026 0.25 Not in cards
WISE Verified Models generate images from the revised WISE prompts. The verified evaluation changes about 200 prompts and the judge, so original scores remain separate. School Apr 2026 0.82 Not in cards
BIBIMBAP Agents read and edit IFC building files. Executable tests compare their changes and answers with reference results. University Mar 2026 50.2% Not in cards
BioVITA retrieval benchmark Models match animal images, audio recordings, and descriptions. The benchmark checks whether the retrieved items match the reference species, genus, or family. University Mar 2026 Not in cards
DesignJanus People rate generated objects from six viewpoints and mark inconsistencies in shape, materials, and identity. The harness compares human ratings with automatic measures. Everyday Mar 2026 Not in cards
Teaching Monster Challenge Agents generate instructional videos for a specified topic and learner. A model screens submissions, viewers compare videos in pairs, and an expert panel sets the final ranking. University Mar 2026 Not in cards
TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. Everyday Mar 2026 0.92 Not in cards
TextEdit Real Models edit text in real images in the full TextEdit test set. The score is the published judged average. Everyday Mar 2026 0.90 Not in cards
TextEdit Virtual Models edit text in virtual images in the full TextEdit test set. The score is the published judged average. Everyday Mar 2026 0.90 Not in cards
AIDABench Agents analyze documents and spreadsheets to answer questions, draw charts, and generate files across 603 tasks. They can execute Python in a sandbox. Separate model evaluators check answer correctness, chart correctness and readability, and spreadsheet structure and content. Expert Feb 2026 59.4% Not in cards
GEBench Image generation models create interactive GUI states from prompts. A judge scores goal, logic, consistency, interface and quality across five task types. Everyday Feb 2026 63.2 Not in cards
Human SxS image editing Elo (Google) Google compares edited images side by side and converts human preferences into Elo ratings. Everyday Feb 2026 1,065 Fading
Human SxS image generation Elo (Google) Google compares generated images side by side and converts human preferences into Elo ratings. Everyday Feb 2026 1,114 In use
HY3D-Bench Models generate textured 3D meshes from text and images, with semantic and geometric quality evaluated across categories. Expert Feb 2026 Not in cards
MarsRetrieval Models match Mars images to captions, retrieve examples of named landforms, and locate features in a global mosaic of the planet. University Feb 2026 26.5 Not in cards
PhyWorldBench Video generators create scenes that follow physical laws across object motion, energy conservation, rigid interactions and human or animal motion. Expert Feb 2026 Not in cards
SA-1B Hard human pose evaluation Models locate body and foot joints in photographs with unusual viewpoints, poses, and occlusions. Meta measures the fraction of correctly placed joints. School Feb 2026 Not in cards
SAM 3D Body synthetic pose evaluation Models estimate human pose and shape from synthetic images of individual people. Meta compares predicted joints and mesh vertices with the rendering parameters. School Feb 2026 Not in cards
VBVR-Bench Video generation models solve visual reasoning tasks by generating a video from a starting image and instruction. Rule-based scorers check the result instead of a judge model. Tasks cover abstraction, knowledge, perception, spatial reasoning and transformation, with in-domain and out-of-domain splits. Everyday Feb 2026 0.56 Not in cards
VBVR-Pro The current official results table evaluates the Pro release. Original VBVR results must remain separate. Everyday Feb 2026 0.56 Not in cards
WorldArena Embodied world models generate RoboTwin 2.0 task videos from first frame plus text or actions. 15 video metrics (EWMScore) plus utility tracks. Feb 2026 63.1% Not in cards
CVTG-2K Image models render text in images across regions and languages. CVTG-2K reports normalized edit distance, CLIPScore and word accuracy. Everyday Jan 2026 0.91 Fading
LMArena Image-to-Video Arena Video generation models animate the same input image with the same prompt, anonymously and side by side. Votes give Elo-style ratings. Everyday Jan 2026 1,503 Not in cards
LMArena Text-to-Video Arena Video generation models make clips from the same text prompt, anonymously and side by side. Votes from logged-in users give Elo-style ratings. Everyday Jan 2026 1,540 Not in cards
LongText-Bench Image models render long text in images. The benchmark reports separate English and Chinese scores. Everyday Jan 2026 0.97 Fading
MineBench Models turn a prompt into a Minecraft-style voxel build as raw 3D coordinates or tool calls. Users compare builds blind, and votes become rankings. Everyday Jan 2026 2,169 Not in cards
OneIG-Bench Image models generate images from prompts with English and Chinese evaluation splits. Everyday Jan 2026 0.53 Fading
RBench 650 image+text prompts for robot video generation (tasks and embodiments). MLLM judge scores task completion, physical plausibility, stability. Jan 2026 60.7% System cards only
SpatialGenEval Text-to-image models generate scenes from 1,230 prompts, and vision-language judges answer 12,300 spatial questions about the resulting images. University Jan 2026 Not in cards
TIIF-Bench Image models generate short and long images scored on overall image quality. Everyday Jan 2026 81.0 Fading
VDEBench Image editing models make local text and table edits to 3,506 Chinese and English document images. Scores check the edit and that the rest of the page is preserved. Everyday Jan 2026 0.77 Not in cards
Video Browser Showdown 2026 marine video search Participants use interactive retrieval systems to find five underwater clips after seeing each target. Each search has a five-minute limit. Everyday Jan 2026 Not in cards
GenEval 2 Models generate images from compositional prompts. A visual question answering model checks whether each requested object, attribute, count, and relation appears. Everyday Dec 2025 Not in cards
PAI-Bench Physical-AI video generation from 1,044 prompts in six domains, VLM-QA plus quality metrics. Companion control and understanding tracks. Dec 2025 83.7% System cards only
TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. Everyday Dec 2025 Not in cards
VIST Image models generate a new image in a reference brand style. Human evaluators compare pairs of outputs. University Dec 2025 Not in cards
ArtEdit-Bench-Lr Models retouch photos from English or Chinese instructions. GPT-4o scores instruction following and visual quality. Pixel errors measure fidelity to reference edits. University Nov 2025 8.62 Not in cards
AssurAI Models respond to Korean prompts across 35 risk categories. Model judges rate the safety of text, image, audio, and video outputs from 1 to 5. Everyday Nov 2025 Not in cards
GEO-Bench-2 Models interpret Earth imagery across 19 tasks, including field boundaries, forest biomass and tree species. The authors normalize task scores to compare model capabilities. University Nov 2025 Not in cards
SAM 3D Artist Objects Models reconstruct an object's shape and placement from a single photograph. The benchmark compares predictions with meshes created by 3D artists. School Nov 2025 Not in cards
SAM 3D Objects human preference evaluation Models reconstruct objects from images of real scenes. People compare the shape and texture of competing reconstructions, including objects seen from an egocentric viewpoint. Everyday Nov 2025 Not in cards
Segment Anything with Concepts Models find every instance of a concept named in a prompt. The benchmark compares image masks and video tracks with human annotations. Everyday Nov 2025 Not in cards
UniREditBench Image editing models carry out 2,700 edits that need reasoning, in real-world scenes and in game worlds with human-defined rules, across 8 primary and 18 secondary dimensions. A multimodal judge scores each edit against both a text reference and a ground-truth image. Everyday Nov 2025 57% Not in cards
LeWitt-Bench Image models interpret instructions for Sol LeWitt wall drawings. People compare the resulting pictures. Everyday Oct 2025 Not in cards
MathCanvas-Bench Models solve 3,000 math problems that call for interleaved visual and textual solutions, such as drawing auxiliary lines in geometry. School Oct 2025 44.1% Fading
MBench Models generate human motion from 450 prompts. The evaluation measures motion quality and uses a vision-language model to judge prompt matching and generalization. Everyday Oct 2025 Not in cards
PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. Everyday Oct 2025 61.9% Not in cards
SuperCLUE-Edit Models edit an input image according to a Chinese instruction. The evaluation measures whether the requested change preserves relevant details. Everyday Oct 2025 82.5 Not in cards
UWBench Models answer questions about underwater scenes, write captions, and locate objects from descriptions. Questions cover visibility, materials, and spatial relationships. Everyday Oct 2025 Not in cards
Arti4D Models watch first-person recordings of people opening furniture. They predict whether parts rotate or slide and estimate the axis of motion. School Sep 2025 Not in cards
Blueprint-Bench Models turn about 20 interior photos of each of 50 apartments into a 2D floor plan. Scored by similarity of room connectivity and size ranking to the true plan. Everyday Sep 2025 0.43 Not in cards
MapAnything dense multi-view reconstruction Models reconstruct a scene from multiple images. The benchmark compares point geometry, metric scale, depth, and camera poses with reference measurements. School Sep 2025 Not in cards
MapAnything single-view calibration Models infer camera intrinsics from a single image. The benchmark measures angular error between predicted camera rays and reference rays. School Sep 2025 Not in cards
PRISM-Bench (image generation) Models generate images from English or Chinese prompts. Vision-language judges score visual appeal and prompt alignment across seven tracks, including composition, lettering, and style. Everyday Sep 2025 86.3 Not in cards
VoxelBench Language models turn text or image prompts into voxel structures. Voters compare anonymous builds from the same prompt, and Glicko-2 ratings rank the models. Everyday Sep 2025 2,607 Not in cards
VoxelBench Image Models turn image prompts into voxel structures. Human comparisons determine Glicko-2 ratings in a separate image-prompt vote pool. Everyday Sep 2025 2,215 Not in cards
7Bench Layout-guided image models generate 224 text-and-layout samples across seven scenarios, scored for semantic and spatial alignment. University Aug 2025 Not in cards
MOSEv2 Models track and segment objects in videos with complex scenes. Objects may disappear, reappear, or blend into a crowd. Everyday Aug 2025 Not in cards
SatLidar canopy height evaluation Models predict canopy height from satellite images. Meta compares height estimates with airborne lidar measurements and reports errors on held-out regions. University Aug 2025 Not in cards
UniGenBench Text-to-image models draw images for 600 prompts across 5 themes and 20 subthemes, in English and Chinese and in short and long forms. A multimodal model judge checks each image on 10 primary and 27 secondary semantic dimensions, such as attributes, actions, relationships, layout, text and logical reasoning. Everyday Aug 2025 95.8% System cards only
BM-Bench Image editing models perform instruction-guided non-rigid image manipulations on the ByteMorph benchmark dataset. Expert Jun 2025 Not in cards
Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. Everyday Jun 2025 Not in cards
LMArena Image Edit Arena Image editing models apply the same user edit request to the same image, anonymously and side by side. Community votes give Elo-style ratings. Everyday Jun 2025 1,519 Not in cards
MMArt-Bench Models retouch 200 photos from editing instructions. Gemini scores visual quality and instruction following, and image comparisons measure how closely edits match reference photos. University Jun 2025 8.34 Not in cards
PST50 Models transfer a reference look to a photograph. The paired test compares 50 outputs with reference grades that professionals refined in DaVinci Resolve. University Jun 2025 Not in cards
DreamGen Bench Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline. May 2025 Not in cards
EWMBench Embodied world models generate robot manipulation videos from language. Scored on scene consistency, motion correctness, semantic alignment. May 2025 Not in cards
General-Bench Models perform multimodal understanding and generation tasks. General-Level averages task scores and measures performance above specialist baselines at progressively broader levels. University May 2025 8.41 Not in cards
GIE-Bench Image-editing models follow text instructions while preserving unedited regions. GPT-4o question answers and masked image metrics score functional correctness and preservation. University May 2025 Not in cards
ImgEdit-Bench Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5. Everyday May 2025 4.12 System cards only
IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. Everyday May 2025 55.3 Dropped
KRIS-Bench Image editing models follow 1,267 instructions that need factual, conceptual or procedural knowledge across 22 tasks. A GPT-4o judge scores the edits. School May 2025 66.2 System cards only
NarrLV Models generate videos from prompts that describe changes in a scene, an object, or an action. A multimodal judge checks what appears and in what order. Everyday May 2025 67.1 Not in cards
ViStoryBench Models turn stories and character references into coherent image sequences. Model judgments and image features measure prompt alignment, character consistency, style, and image quality. University May 2025 2.00 Not in cards
3D Animation Arena Models reconstruct a moving 3D human from a video. Voters compare motion fidelity, ground contact, smoothness, and overall appearance. Everyday Apr 2025 Not in cards
GEdit-Bench Image editing models follow real-world user edit instructions. A model judge scores semantic consistency, perceptual quality and an overall score. Everyday Apr 2025 6.52 Dropped
RISEBench Image editing models follow 360 instructions that need temporal, causal, spatial or logical reasoning. An LMM judge scores instruction reasoning, appearance consistency and visual plausibility. University Apr 2025 51.4% System cards only
WorldScore 3,000 next-scene generation tasks with camera-trajectory layouts. Scores 3D/4D and video world generators on controllability, quality, dynamics. Apr 2025 Not in cards
3DGen-Bench People compare generated 3D objects in pairs. The authors use their judgments to evaluate automated scoring methods for text-to-3D and image-to-3D generation. Everyday Mar 2025 Not in cards
AutoAdvExBench Agents inspect adversarial-example defenses and produce attacks that perturb images within a fixed bound. The benchmark measures attack success against 51 real defense implementations. It reports defense accuracies and separate results on pedagogical and real-world subsets. Expert Mar 2025 22% Not in cards
ETVA Text-to-video systems are scored on fine-grained prompt alignment through generated atomic questions and video-language answers. Expert Mar 2025 Not in cards
FiVE-Bench Models edit objects in videos from written instructions. A vision language model checks whether each edit succeeded, and other metrics measure changes to the background. University Mar 2025 Not in cards
FunGraph3D Models identify objects and their controls in indoor scans. They predict which part operates which object, such as a switch that controls a light. Everyday Mar 2025 Not in cards
LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. Everyday Mar 2025 72% Not in cards
OpenLex3D Models label points and retrieve objects in indoor 3D scans. Annotators distinguish an object’s synonyms from pictures on its surface and visually similar objects. Everyday Mar 2025 Not in cards
VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. Everyday Mar 2025 66.7% Not in cards
VideoPhy-2 200 action-centric prompts. Generated videos human-rated for semantic adherence, physical commonsense and physical-rule grounding. Mar 2025 24.9% System cards only
WISE Text-to-image models draw 1,000 prompts that need world knowledge in culture, space, time and natural science. A vision model judge gives the WiScore. School Mar 2025 0.82 System cards only
AGI-Eval Community Text to Image Models generate images from text. The board reports objective, subjective and combined scores. Everyday Feb 2025 0.64 Not in cards
AGI-Eval Community Text to Video Models generate videos from text. The board reports objective, subjective and combined scores. Everyday Feb 2025 62.5 Not in cards
Massive Visualized IR Benchmark Models retrieve screenshots that match a search query. The benchmark compares performance across four retrieval task groups. Everyday Feb 2025 Not in cards
WorldModelBench Video generators judged as world models on instruction following and physics adherence in application domains. Human labels and a fine-tuned judge. Feb 2025 Not in cards
Physics-IQ Video models continue 396 filmed physical events from a frame or 3 s clip. Motion compared with the real continuation. Jan 2025 57.7% System cards only
ViGiL3D Models locate objects in indoor 3D scans from English descriptions. A description can match several objects or none. Models must return the matching set. Everyday Jan 2025 Not in cards
Articulate3D Models locate movable parts and handles in scanned rooms. They predict how each part moves, including its axis and the location of its hinge. School Dec 2024 Not in cards
Gray Swan Jailbreak Arena Human attackers try to jailbreak models for violent, self-harm and malicious code outputs. Scored as attack success rate per category. Expert Dec 2024 3.60% System cards only
Labelbox Image Generation v2 Models generate images from text prompts. Human raters compare how well the outputs meet the request. Everyday Dec 2024 1,069 Not in cards
Labelbox Video Generation v2 Models generate videos from text prompts. Human raters compare the outputs for visual quality and adherence to the prompt. Everyday Dec 2024 1,268 Not in cards
LMArena Text-to-Image Arena Image generation models answer the same user prompt side by side and anonymously. Community votes between the two images give Elo-style ratings. Everyday Dec 2024 1,428 Not in cards
ScanNet++ v2 Models reconstruct indoor scenes from images of real scans. The second release adds scene types and updates the annotations and camera poses. School Dec 2024 Not in cards
GEOBench-VLM Models interpret satellite and aerial images, identify land use, and track changes across observations. Some tasks ask models to locate objects or outline regions. University Nov 2024 0.41 Not in cards
VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. Everyday Nov 2024 86.9% Not in cards
Harmony4D Models recover the body shapes and poses of people interacting at close range. The benchmark includes sports where people occlude each other. School Oct 2024 Not in cards
LEVIR-CDC Models describe changes between aerial images and mark changed buildings. The authors paired LEVIR-CC captions with binary building masks. Everyday Oct 2024 Not in cards
MMIE Models interpret and generate interleaved text and images. MMIE-Score, a fine-tuned InternVL-2-4B model, grades the resulting multimodal responses. University Oct 2024 Not in cards
Movie Gen Video Bench Video generation models make clips from a fixed prompt set, and people compare them. Meta released non-cherry-picked Movie Gen videos for comparison. Everyday Oct 2024 Not in cards
OBI-Bench Models locate inscriptions, rejoin bone fragments, classify characters, retrieve matches, and explain glyphs. The authors publish separate scores for each task. Expert Oct 2024 0.61 Not in cards
QAG-360K Models compare two satellite images, answer questions about land cover, and mark the relevant pixels. The authors score answer accuracy and mask overlap separately. School Oct 2024 Not in cards
TaxaBench-8k Models identify species or match observations across photographs, sounds, and other field data. The test contains 8,813 observations paired across six types of data. Expert Oct 2024 Not in cards
WHU-CDC Models compare aerial photos of Christchurch, describe what changed, and mark changed pixels. Reference captions describe reconstruction after the earthquake. Everyday Oct 2024 Not in cards
WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. Oct 2024 Not in cards
Labelbox Image Generation (original) Models generate images from text prompts. Human raters compare how well the outputs meet the request. Everyday Sep 2024 Not in cards
Labelbox Video Generation (original) Models generate videos from text prompts. Human raters compare the outputs for visual quality and adherence to the prompt. Everyday Sep 2024 Not in cards
Text2CAD Models generate parametric CAD designs from text prompts at different levels of detail. The authors compare generated geometry with the reference and count invalid designs. University Sep 2024 Not in cards
GeoSound Models match environmental recordings to satellite images of their locations and retrieve recordings from images. The test holds out geographic areas and contains 9,931 examples. Everyday Aug 2024 Not in cards
K-Sort Arena Users rank outputs from K image or video generation models at once instead of a pair. Bayesian updates turn the rankings into ratings. Everyday Aug 2024 31 Not in cards
OpenCanopy Models estimate tree canopy height from satellite images. The evaluation compares predicted height maps with measurements from airborne laser scans. University Jul 2024 Not in cards
ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. Everyday Jul 2024 Not in cards
SA-37 interactive segmentation Models refine an object mask after a user clicks on an image. The benchmark averages overlap across datasets at different numbers of clicks. Everyday Jul 2024 Not in cards
Segment Anything Video Models predict masks for prompted objects throughout a video. The evaluation compares predicted boundaries and regions with annotated object tracks. Everyday Jul 2024 Not in cards
1X World Model Challenge Predict future egocentric frames of 1X's EVE humanoid from video and state. Compression scored by token cross-entropy, Sampling by PSNR on private tests. Jun 2024 Not in cards
ChronoMagic-Bench Video models generate time-lapse scenes from 1,649 prompts covering biological, human-made, weather, and physical changes. Automatic metrics measure the amount of change and temporal coherence, with a separate GPT-4o assessment of change. Everyday Jun 2024 Not in cards
GeckoNum Models generate images from prompts about quantities. Human raters check exact counts, approximate quantities, and reasoning about numerical concepts. Everyday Jun 2024 47.6% Not in cards
GenAI-Arena Users compare outputs from two anonymous models for image generation, image editing and video generation, and vote. Votes give Elo ratings. Everyday Jun 2024 1,133 Not in cards
GenAI-Arena Image Editing Models edit images from user instructions. Anonymous human comparisons determine Elo in this independent vote pool. Everyday Jun 2024 Not in cards
GenAI-Arena Image Generation Models generate images from text prompts. Anonymous human comparisons determine Elo in this independent vote pool. Everyday Jun 2024 1,133 Not in cards
GenAI-Arena Video Generation Models generate videos from text prompts. Anonymous human comparisons determine Elo in this independent vote pool. Everyday Jun 2024 Not in cards
SuperCLUE-Image Models generate images from Chinese prompts. The benchmark scores text rendering, visual quality, prompt adherence and scene composition. Everyday Jun 2024 94.3 Not in cards
VideoPhy Text-to-video models generate clips from captions of physical interactions between solids and fluids. Human raters judge caption match and physical commonsense. Everyday Jun 2024 Not in cards
Long-term Video Object Segmentation v2 Models track object masks through long videos. The evaluation tests whether they can recover an object's identity after an extended absence. Everyday Apr 2024 Not in cards
DPG-Bench Text-to-image models draw 1,000 dense prompts with many objects, attributes and relations. A VQA model checks each element of the prompt graph in the image. Everyday Mar 2024 84.2% Fading
SceneFun3D Models identify movable parts in indoor scans, locate the part needed for a written task, and estimate how it moves. Everyday Mar 2024 Not in cards
TGVE+ Models follow video editing instructions on the original 76 TGVE videos. The extension adds object removal, object addition, and texture changes. University Mar 2024 Not in cards
ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. Everyday Feb 2024 20.9 Not in cards
SuperCLUE-Video Models generate videos from Chinese prompts. A judging model examines video frames and scores them against criteria for the prompt. School Feb 2024 54.3 Not in cards
Ego-Exo4D Models estimate human pose during skilled activities filmed from first-person and external viewpoints. Meta reports joint and mesh errors for physical and procedural activities. School Dec 2023 Not in cards
HEIM Text-to-image models are scored on 12 aspects, such as alignment, quality, aesthetics, originality, reasoning, knowledge, bias and toxicity, with human raters and automatic metrics. Everyday Nov 2023 Not in cards
VBench Video generation models make clips from prompts, and automatic checks score 16 dimensions of quality and prompt adherence. Everyday Nov 2023 84.2% Fading
EvalCrafter Text-to-video models generate clips for 700 prompts drawn from real user requests. The clips are scored with 17 automatic metrics for visual quality, content quality, motion and text alignment. Weights fitted to human ratings combine the metrics into one score. Everyday Oct 2023 Not in cards
GenEval Text-to-image models draw prompts about object count, color, position and co-occurrence, and an object detector checks each image. Scored by the share of correct images. Everyday Oct 2023 90.1% Fading
Ref-ZOM Models mark the pixels described by a phrase in a COCO image. A phrase can refer to zero, one or several objects. Everyday Oct 2023 Not in cards
Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. School Sep 2023 81.7% System cards only
EMDB Models reconstruct human pose and body shape in everyday scenes. The benchmark compares joint positions with electromagnetic measurements. School Aug 2023 Not in cards
MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. Everyday Aug 2023 Not in cards
ReasonSeg Models infer which object an instruction describes and return its mask. Instructions can require reasoning about the image. Everyday Aug 2023 Not in cards
OpenSUN3D Open-Vocabulary 3D Object Instance Search Models read a description and mark every matching object in a scanned room. The organizers rank predictions by mean average precision across 25 test scenes. Everyday Jul 2023 Not in cards
PUMaVOS Models track masks for object parts across video frames. A prompt can identify a small region, such as one cheek of a person's face. Everyday Jul 2023 Not in cards
Aria Digital Twin Models infer the layout of objects from first-person images. The SAM 3D Objects evaluation compares predicted positions and orientations with reference scene geometry. School Jun 2023 Not in cards
LOVEU-TGVE-2023 Models edit 76 videos with four instructions per video. People compare each submission with Tune-A-Video on text alignment, structure, and visual quality. University May 2023 Not in cards
Large-Vocabulary Video Instance Segmentation Models detect, segment, and track object instances across video frames. The benchmark includes categories outside the training vocabulary. Everyday Apr 2023 Not in cards
Hi4D Models reconstruct the bodies of people interacting at close range. The SAM 3D Body evaluation compares joint and mesh errors with motion capture references. School Mar 2023 Not in cards
Event-based Segmentation Dataset Models segment objects in cluttered indoor scenes filmed from a moving robot arm. The benchmark provides reference masks for comparison. Everyday Feb 2023 Not in cards
MOSE Models follow and segment objects in videos with crowds, occlusions, and objects leaving the screen. The benchmark compares masks against human annotations. Everyday Feb 2023 Not in cards
Video Object Segmentation under Transformations Models follow objects whose appearance changes as people manipulate them. The benchmark compares predicted masks with annotations through these transformations. Everyday Dec 2022 Not in cards
Inappropriate Image Prompts Models generate images from prompts drawn from real users. The evaluation measures how often the outputs contain inappropriate material. Everyday Nov 2022 Not in cards
Common Syntactic Processes in Text-to-Image Generation Models generate images from sentences with different grammatical structures. Evaluators check whether each image preserves the relationships expressed in the sentence. School Oct 2022 Not in cards
BURST Models recognize, segment, and track objects across video frames. The benchmark tests how well they associate the same object across time. Everyday Sep 2022 Not in cards
Robust Multi-view Depth Models estimate depth from multiple photographs of a scene. The benchmark compares relative depth errors across datasets from different environments. School Sep 2022 Not in cards
EPIC-KITCHENS VISOR Hand Object Segmentation Models outline each hand and the object it touches in a kitchen image. They also identify which hand appears and whether it touches anything. Everyday Aug 2022 Not in cards
EPIC-KITCHENS VISOR Semi-Supervised Video Object Segmentation Models receive masks for hands and objects in the first frame of a kitchen video. They track those objects through later frames, including after occlusion. Everyday Aug 2022 Not in cards
Omni3D A large benchmark for 3D object detection from a single image, built by unifying indoor and outdoor datasets such as SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI and nuScenes. Vision-language models answer its 3D grounding questions. Everyday Jul 2022 53.4% System cards only
Testing Relational Understanding in Text-Guided Image Generation Models depict relationships between named entities. Human participants judge whether the generated scenes match each caption. Everyday Jul 2022 Not in cards
PartiPrompts Models generate images from English prompts spanning subjects and difficulty types. Human raters compare image realism and fit to the prompt. Everyday Jun 2022 Not in cards
RICH Models reconstruct people interacting with their surroundings. The evaluation compares body joints and mesh vertices with the reference geometry. School Jun 2022 Not in cards
Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. School May 2022 0.65 Not in cards
DrawBench Models generate images from text prompts that test counting, composition, and other skills. Human raters compare image quality and prompt alignment. Everyday May 2022 Not in cards
Google Scanned Objects Models synthesize an object from a new viewpoint. Researchers compare the rendered result with reference views of scanned household items. School Apr 2022 Not in cards
HierText Models locate and recognize text arranged in lines and paragraphs in real scenes. Google also uses the annotations to compare text and image retrieval. School Mar 2022 Not in cards
PaintSkills Models generate images of specified objects, counts, and spatial relationships. Detectors check whether an image satisfies the requested composition. Everyday Feb 2022 Not in cards
ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. Everyday Nov 2021 53.7% Dropped
Ego4D State Change Object Detection Models locate the object that changes in a first-person video. The official test scores its bounding box in the frame where the change begins. Everyday Oct 2021 Not in cards
LoveDA Models label land cover in aerial images of urban and rural areas. The benchmark supports comparisons across these two settings. University Oct 2021 Not in cards
TREK-150 Models follow a marked object through first-person kitchen videos. In the one-pass test, they receive its bounding box in the first frame and track it to the end. Everyday Aug 2021 Not in cards
Unidentified Video Objects Models segment and track objects in video without a fixed list of object categories. The evaluation compares predicted masks and tracks with annotated instances. Everyday Apr 2021 Not in cards
PPR10K Models retouch groups of portraits against three expert retouchers’ Adobe Camera Raw edits. The test measures accuracy on people and color consistency across each group. University Mar 2021 Not in cards
Occluded Video Instance Segmentation Models track and segment objects that overlap or disappear behind other objects. The evaluation compares predicted masks and identities with annotated video instances. Everyday Feb 2021 Not in cards
Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. Everyday Dec 2020 Dropped
Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. Everyday Nov 2020 13.5 Fading
MedICaT Models locate panels within a biomedical figure and match each panel to the relevant words in its caption. The original test scores word overlap. University Oct 2020 Not in cards
Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. Everyday Aug 2020 Not in cards
Virtual KITTI 2 Models interpret simulated driving videos with reference geometry and pixel labels. Meta uses the sequences to compare video object segmentation models. Everyday Jan 2020 Not in cards
ARC-AGI Few-shot grid puzzles: infer a hidden transformation from about three examples and produce the output grid. Semi-private sets verified by ARC Prize. Everyday Nov 2019 98.5% In use
ScanNet-1500 Models match features between pairs of indoor views and estimate relative camera poses. The evaluation measures pose accuracy on 1,500 image pairs. School Nov 2019 Not in cards
FreiHAND Models recover a hand's joint positions and surface from a color image. The evaluation compares predictions with reference hand geometry. School Sep 2019 Not in cards
DIODE Models estimate depth in indoor and outdoor scenes. The dataset provides depth measurements from a laser scanner for comparison. School Aug 2019 Not in cards
LVIS Models locate objects and outline each instance across a large vocabulary of categories. Meta reports box and mask average precision. Everyday Aug 2019 Not in cards
iSAID Models detect and segment individual objects in aerial images. The benchmark measures how well predicted masks match annotated object instances. University May 2019 Not in cards
YouTube-VIS Models detect, segment, and track object instances in YouTube videos. The SAM 3 report evaluates the 2021 edition. Everyday May 2019 Not in cards
Nuscene Models localize named objects as 3D boxes in nuScenes driving camera images. Scored by mAP. Everyday Mar 2019 16.0 Fading
nuScenes Detection Models detect 3D objects in nuScenes sensor data. The official test leaderboard ranks submissions by nuScenes Detection Score. Everyday Mar 2019 Not in cards
3DPW Models recover a person's three-dimensional pose in outdoor video. Reference poses come from wearable sensors and a moving camera. School Sep 2018 Not in cards
YouCook2 retrieval Models retrieve cooking videos from text queries in the YouCook2 collection. University Sep 2018 52.5 Fading
YouTube-VOS Models segment and track objects across frames in YouTube videos. The evaluation distinguishes categories seen during training from previously unseen categories. Everyday Sep 2018 Not in cards
Places Audio Captions (Hindi) Models retrieve photographs from spoken Hindi descriptions or retrieve descriptions from photographs. The score counts how often the correct match appears among the top results. Everyday Apr 2018 Not in cards
Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. Everyday Mar 2018 Not in cards
DAVIS 2017 Models track and segment individual objects throughout a video. The evaluation measures region overlap and boundary accuracy. Everyday Apr 2017 Not in cards
ScanNet Models infer indoor scene geometry from images. Meta evaluates depth predictions against measurements from scanned indoor environments. School Feb 2017 Not in cards
ADE20K Models label the pixels in a scene by object or region. Meta measures overlap between predicted and reference segments. Everyday Aug 2016 Not in cards
Cityscapes Models assign a class to each pixel in street scenes filmed from a vehicle. The benchmark compares predicted segments with annotated road scenes. Everyday Apr 2016 Not in cards
SUNRGBD Models detect objects in about 10,000 indoor RGB-D images and predict their 3D bounding boxes. Scored by average precision. Everyday Jun 2015 48.3 Fading
PASCAL-Context Models assign a semantic class to each pixel in a photograph. The benchmark includes objects and background regions. Everyday Jun 2014 Not in cards
PASCAL VOC 2012 Models locate objects and label their pixels in photographs. Meta reports detection and segmentation results on the 2012 challenge. Everyday May 2012 Not in cards
Leeds Sports Pose Extended Models locate body joints in photographs of athletes. Meta compares the fraction of predicted joints within a distance threshold. School Jun 2011 Not in cards
MIT-Adobe FiveK Models adjust photographic tone to match human edits. Five photographers each edited 5,000 raw photos in Lightroom. The original evaluation compares luminance with each reference edit. University Jun 2011 Not in cards
3D Arena Users compare two 3D models generated from the same input and vote for the better one. A live leaderboard ranks the generators. Everyday Not in cards
3D Worlds Arena Models generate explorable 3D scenes from an image and caption. Voters inspect two anonymous scenes from the same input and choose a favorite. Everyday Not in cards
AA-Image-Editing Validated private evaluators compare two image edits made from the same instruction and vote blind. Votes become Elo ratings. Everyday Not in cards
AA-Video-Editing Voters compare two edits of the same video made from one instruction and pick the better one. Votes become Elo ratings. Everyday Not in cards
Artificial Analysis Image Arena Voters compare images from two text-to-image models for the same prompt and pick the better one. Votes become Elo ratings. Everyday Not in cards
Artificial Analysis Video Arena Voters compare videos from two text-to-video models for the same prompt and pick the better one. Votes become Elo ratings. Everyday Not in cards
Chart Arena Models create charts. Pairwise votes determine the published Arena rating. Everyday 1,615 Not in cards
COCO-Stuff 164K Models label each pixel with an object or background category. The benchmark adds annotations for regions such as sky, grass, and pavement. Everyday Not in cards
Common Objects in 3D v2 Models estimate camera poses from images of objects viewed from different angles. Meta compares predicted poses with the reference poses. School Not in cards
DesignArena 3D Design Models build an interactive 3D scene that runs in the browser as a single HTML file. Users explore the scenes and vote for the better one. Everyday 1,535 In use
DesignArena Blender Agents build a 3D scene in Blender from a user prompt. Users compare the rendered scenes and vote for the better one. Everyday 1,381 Not in cards
DesignArena Code to Video Agents write code that renders a video from a user prompt. Users compare the two videos and vote for the better one. Everyday 1,454 Not in cards
DesignArena Graphic Design Image models make a graphic design piece from a user prompt. Users compare two anonymous designs and vote for the better one. Everyday 1,436 Not in cards
DesignArena Image Image models generate a picture from a user prompt. Users compare two anonymous images and vote for the better one. Everyday 1,381 Not in cards
DesignArena Image Editing Image models edit a user image following a text instruction. Users compare the two edits and vote for the better one. Everyday 1,368 Not in cards
DesignArena Image to Video Video models animate a user image into a short video. Users compare two anonymous videos and vote for the better one. Everyday 1,319 Not in cards
DesignArena Logo Image models design a logo from a user prompt. Users compare two anonymous logos and vote for the better one. Everyday 1,365 Not in cards
DesignArena Multi-Image to Video Video models compose one video from several reference images. Users compare two anonymous videos and vote for the better one. Everyday 1,329 Not in cards
DesignArena Multimodal to Video Video models make a video from a mix of text, images and other inputs. Users compare two anonymous videos and vote for the better one. Everyday 1,305 Not in cards
DesignArena Video Arena Users compare videos from anonymous video generation models for the same prompt and vote for the better one. Votes become Elo ratings. Everyday 1,369 Not in cards
DesignArena Video Editing Video models edit a user video following a text instruction. Users compare the two results and vote for the better one. Everyday 1,373 Not in cards
DTU Multi-View Stereo Models reconstruct objects from multiple camera views. The benchmark measures the accuracy and completeness of reconstructed surfaces against scanned reference geometry. School Not in cards
EndoVis 2018 Robotic Scene Segmentation Models segment structures and surgical instruments in video from robot-assisted operations. The evaluation compares predicted masks with reference labels. Everyday Not in cards
ETH3D Models reconstruct scene geometry from images. Meta evaluates depth predictions against reference measurements on the ETH3D test scenes. School Not in cards
Floorplan Arena Models furnish hand-drawn room sketches. People compare the layouts and vote for the better floorplan. Everyday 1,807 Not in cards
GameAdsBench Models generate video ads for games. Viewers watch two anonymous ads and choose the one they prefer. University 1,283 Not in cards
GenAI-Bench Image generation systems are compared by human side-by-side preference on text-to-image prompts. Everyday 1,079 Fading
GenExam Image generation systems are evaluated on subject-focused educational tasks with expert ratings. University Not in cards
Goliath human mesh recovery Models reconstruct a person's body from images of studio performances. Meta measures joint and surface errors against captured three-dimensional motion. School Not in cards
Image Matching Challenge Models match photographs of landmarks taken from different viewpoints. The evaluation measures the accuracy of camera poses recovered from the matches. School Not in cards
KITTI depth prediction Models estimate scene depth from road images. The evaluation compares model predictions with depth measurements collected from a moving vehicle. School Not in cards
LMArena Video Edit Arena Video editing models apply the same edit request to the same clip, anonymously and side by side. Community votes give Elo-style ratings. Everyday 1,424 Not in cards
NYU Depth V2 Models estimate depth from indoor photographs. The benchmark compares predictions with depth measurements captured by a sensor. School Not in cards
OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact. Everyday 91.0 Not in cards
RealEstate10K Models estimate relative camera poses across frames from real estate videos. The benchmark supplies camera trajectories for comparison. School Not in cards
StudyArena Image Models produce image responses to user prompts. Anonymous human comparisons determine the published rating in this separate vote pool. Everyday 1,284 Not in cards
SuperCLUE Arena Image Editing Models edit images in response to instructions. The arena ranks systems from pairwise comparisons. University 1,308 Not in cards
SuperCLUE Arena Image to Video Models animate images. The arena ranks systems from pairwise comparisons. University Not in cards
SuperCLUE Arena Reference to Video Models generate videos from references. The arena ranks systems from pairwise comparisons. University Not in cards
SuperCLUE Arena Text to Image Models generate images from text. The arena ranks systems from pairwise comparisons. University 1,458 Not in cards
SuperCLUE Arena Text to Video Models generate videos from text. The arena ranks systems from pairwise comparisons. University Not in cards
SuperCLUE-ComicShorts Models generate short animated comics. University 38.5 Not in cards
SuperCLUE-I2V Models generate videos from still images. University 71.5 Not in cards
SuperCLUE-R2V Models generate videos conditioned on reference material. University 64.9 Not in cards
SuperCLUE-VideoEdit Models edit videos in response to Chinese instructions. University 74.2 Not in cards
SuperCLUE-World World models generate visual sequences. University 74.6 Not in cards
TartanAir v2 Models reconstruct camera motion and scene geometry from simulated trajectories. The evaluation compares predictions with known poses and depths. School Not in cards
VIPSeg Models label video pixels and keep instance identities consistent across frames. The benchmark includes both countable objects and background regions. Everyday Not in cards
WMArena Video Arena Models generate a video from a starting image and an action. Viewers compare anonymous clips and vote for the more convincing result. University 1,487 Not in cards
Wojak-Bench 3D Models turn the same drawing into a three-dimensional figure. Voters compare two anonymous prints or renders and choose which better matches the drawing. University 1,273 Not in cards