Benchmark Almanac

Audio and video

We track 240 benchmarks in audio and video. They run from everyday level to research frontier level.

30 in use 19 past 85% 121 with public code

Qwen and ByteDance Seed held the top audio and video score about equally, 31% and 30% of the contested time

That's about 14 of the 46 months since January 2023, averaged over these benchmarks. We only count time while other labs are still testing, and stop once a model passes 85%. Faded stretches no longer count.

Other labs Qwen Google DeepMind ByteDance Seed Moonshot AI Shanghai AI Lab Meituan OpenAI Anthropic Thinking Machines Microsoft Xiaomi Corp Baidu NVIDIA Amazon
Everyday Most adults, without training
VideoMME v2 (with subtitles) 10 leaders
Qwen 3.8 Max 68.3%
LVBench Passed 85% in Aug 2026, by Gemini 3.7 Flash
Gemini 4 Argon 91.7%
MLVU Passed 85% in Feb 2026, by Qwen3.5 397B-A17B
Qwen 3.8 Max 90.8%
MVBench Not tested since Aug 2026
Qwen3.5-Omni-Plus 79%
TempCompass Passed 85% in Dec 2025, by Seed1.8
Seed2.0 Pro 89.6%
LongVideoBench Not tested since Jun 2026
Seed2.1 Pro 80.6%
VoiceBench Passed 85% in Sep 2025, by Qwen3-Omni-30B-A3B
Qwen3.5-Omni-Plus 93.1%
vCX Benchmark Passed 85% in Aug 2025, by gpt-realtime
GPT-6.1 Sol 93.3%
MotionBench Not tested since Jun 2026
Seed2.0 Pro 75.2%
Audio MC 3 leaders
Gemini 3.8 Flash 60.4%
University Undergraduates and entry-level professionals
MMVU Not tested since Aug 2026
Qwen 3.8 Max 82.4%
Video-MMMU Passed 85% in Nov 2025, by Gemini 3 Pro
Qwen 3.8 Max 88.7%
MMAU v05.15.25 Not tested since Dec 2025
Nova 2 Omni Preview 77.8%

Sources: Official leaderboards and labs’ model cards

Each benchmark counts from the one source its own page leads with: its official leaderboard when that is current, else Epoch AI’s own runs, else labs’ own numbers. We date independent runs by the model’s release and lab reports by when they came out.

Other labs Reka Shanghai AI Lab Kuaishou OpenAI Google DeepMind ByteDance Seed Alibaba Group Qwen Z.ai Fixie.ai (Ultravox) Moonshot AI NVIDIA Anthropic Thinking Machines
Everyday Most adults, without training
TempCompass Not tested since Aug 2025
Keye-VL-1.5-8B 75.5%
VideoMME v2 (with subtitles) Not tested since Apr 2026
Gemini 3 Pro 49.4%
LVBench Not tested since May 2025
Seed1.5-VL 64.6%
MLVU Not tested since Oct 2024
VideoLLAMA3-7B 47.7%
LongVideoBench Not tested since Nov 2024
GPT-4o 66.7%
TOMATO Not tested since Dec 2024
Qwen2-VL-72B-Instruct 37.9%
VoiceBench Passed 85% in Sep 2025, by Qwen3-Omni-30B-A3B
Nemotron 3 Nano Omni 30B-A3B 89.4%
MotionBench Not tested since Nov 2025
Gemini 3 Pro 70.4%
vCX Benchmark Passed 85% in Aug 2025, by gpt-realtime
GPT-6.1 Sol 93.3%
Daily-Omni Passed 85% in Sep 2026, by Qwen3.8-Omni-Flash
Qwen3.8-Omni-Flash 85.1%
Audio MC 3 leaders
Gemini 3.8 Flash 60.4%
University Undergraduates and entry-level professionals
MMVU Not tested since Jan 2025
o1 75.5%
Video-MMMU Not tested since Aug 2025
GPT-5 84.6%

Source: Official leaderboards

Every independent run: each benchmark’s own leaderboard, Epoch AI’s runs and boards Epoch AI copied, dated by the model’s release. We leave out benchmarks that only labs report.

Other labs Qwen ByteDance Seed Shanghai AI Lab Reka OpenAI Google DeepMind Moonshot AI Alibaba Group Z.ai Fixie.ai (Ultravox) Anthropic Thinking Machines Microsoft Xiaomi Corp Baidu NVIDIA Amazon
Everyday Most adults, without training
MVBench Not tested since Aug 2026
Qwen3.5-Omni-Plus 79%
TempCompass Passed 85% in Dec 2025, by Seed1.8
Seed2.0 Pro 89.6%
VideoMME v2 (with subtitles) 12 leaders
Qwen 3.8 Max 68.3%
LVBench Passed 85% in Aug 2026, by Gemini 3.7 Flash
Gemini 4 Argon 91.7%
MLVU Passed 85% in Feb 2026, by Qwen3.5 397B-A17B
Qwen 3.8 Max 90.8%
LongVideoBench Not tested since Jun 2026
Seed2.1 Pro 80.6%
TOMATO Not tested since Jun 2026
Seed2.1 Pro 79.5%
VoiceBench Passed 85% in Sep 2025, by Qwen3-Omni-30B-A3B
Qwen3.5-Omni-Plus 93.1%
vCX Benchmark Passed 85% in Aug 2025, by gpt-realtime
GPT-6.1 Sol 93.3%
Audio MC 4 leaders
Qwen3.8-Omni-Flash 71.5%
University Undergraduates and entry-level professionals
MMAU v05.15.25 Not tested since Dec 2025
Nova 2 Omni Preview 77.8%
MMVU Not tested since Aug 2026
Qwen 3.8 Max 82.4%
Video-MMMU Passed 85% in Nov 2025, by Gemini 3 Pro
Qwen 3.8 Max 88.7%

Sources: Official leaderboards and labs’ model cards

Every score, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

From everyday to research frontier level

Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.

Everyday

Most adults, without training

School

Students, from grade school to high school

University

Undergraduates and entry-level professionals

Expert

Specialists with years of training, often over hours of work

Research frontier

Researchers, on open problems or tasks that take days

By kind

Video understanding

BenchmarkLevelScoredBest scoreStatus
Video-MME Models answer 2,700 multiple-choice questions about 900 videos from 11 seconds to 1 hour long, with and without subtitles. Scored by accuracy. Everyday Checked automatically 90.4% In use
Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. University Checked automatically 88.7% In use
MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. University Checked automatically 82.4% In use
MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. Everyday Checked automatically 79% Fading
BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. Expert Checked automatically 76.4 Not in cards
VideoMME v2 (with subtitles) Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. Everyday Checked automatically 68.3% In use
TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. Everyday Checked automatically 79.5% Fading
TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. Everyday Checked automatically 89.6% Fading
MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. Everyday Checked automatically 75.2% Fading
Daily-Omni Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy. Everyday Checked automatically 85.1% Fading
WorldSense (omnimodal) Models answer multiple choice questions about real-world videos that need both the visual and the audio track. Scored by accuracy. Everyday Checked automatically 67.3% Fading
TVBench Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. Everyday Checked automatically 81.9% In use
VideoReasonBench Video models watch a sequence of fine-grained operations on a partly hidden state and answer questions that recall, infer or predict that state. Everyday Checked automatically 56% Fading
CrossVid Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats. Everyday Checked automatically 45.2% Fading
OVBench Video models answer questions about past, current and future moments of an online video stream in 6 task types and 16 subtasks. Everyday Checked automatically 70% Fading
OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. Everyday Checked automatically 59.2% Not in cards
Video-Holmes Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film. Everyday Checked automatically 68.2% Fading
MORSE-500 Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning. Everyday Checked automatically 23.6% Fading
CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. Expert Graded by a model 76.6 Not in cards
OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. University Checked automatically 61.8% Fading
QIVD Models answer spoken questions about short phone videos in real time, as in a face to face talk. Each item has a short answer and a timestamp. Everyday Graded by a model 69.6% Fading
TemporalBench Video models answer about 10K questions on fine-grained temporal dynamics, built from about 2K detailed human annotations of video clips. Everyday Checked automatically 28.1% Dropped
ActivityNet-QA Models answer open-ended questions about web videos of human activities, from 58,000 QA pairs on 5,800 videos. Scored by accuracy. Everyday Graded by a model 66.7% Dropped
OVO-Bench Video models answer questions asked at given timestamps of a video stream in 12 tasks over 644 videos. Tasks cover backward tracing, real-time understanding and forward active responding. Everyday Checked automatically 80.7% Fading
StreamingBench Models answer questions while a video streams in, as a human viewer would. It tests real-time visual understanding, omni-source understanding and context understanding. Everyday Checked automatically 56.4% Fading
VCR-Bench Video models answer 1,034 questions on 859 videos, and graders score both the final answer and each tagged chain-of-thought step against human rationales. Everyday Graded by a model 56.7% Fading
AV-SpeakerBench Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together. Everyday Checked automatically 77.6% Fading
JointAVBench Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans. Everyday Checked automatically 58% Fading
EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. Everyday Checked automatically 71.8% Fading
ODV-Bench Streaming video models answer real-time questions about autonomous driving footage. Everyday Checked automatically 55.6% Fading
Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. Everyday Checked automatically 78.4% Dropped
QVHighlights Natural-language queries over YouTube videos. Model localizes relevant moments, scored by Recall@1 at IoU 0.5. Everyday Checked automatically 77.2% Fading
VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. University Checked automatically 76% In use
Charades-STA Models read a sentence describing an action and return its start and end time in a Charades indoor video. Scored by temporal IoU with the labeled moment. Everyday Checked automatically 64% Fading
AVUT Models answer questions about videos where the audio carries the key information. The benchmark removes questions that text alone can answer. Everyday Checked automatically 86.6% Fading
FutureOmni Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together. Everyday Checked automatically 66.7% Fading
Video Reality Test Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection. Everyday Checked automatically 76.3% Not in cards
ViSpeak-Bench Streaming video models must notice instructions given visually, such as a hand wave, and respond in seven subtasks of visual instruction feedback. Everyday Checked automatically 87% Fading
Omni-InteractScope Explicit Request Models answer explicit requests during audio and video streams. The score is the published average across tasks T1 through T7. Everyday Graded by a model 20.2% Not in cards
Omni-InteractScope Proactive Models provide timely proactive assistance during audio and video streams. The score is the published T8 assistance recall. Everyday Graded by a model 46.4% Not in cards
Video-Oasis A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions. Everyday Checked automatically 34.5% Not in cards
EgoHome Models annotate 137 egocentric clips of household chores, about two hours in all, with subtasks, hand actions, hand captions and hand boxes. Segments are as short as 0.1 seconds. A model judge matches segment labels, and boxes are scored by centroid hits. Everyday Graded by a model 0.80 Not in cards
Q-Bench-Video Models judge the quality of natural, AI-generated and computer-graphics videos through yes-or-no, what-how, open-ended and pairwise comparison questions. Everyday Graded by a model 52.2% Not in cards
LSMDC Models fill in a missing word in movie audio descriptions using the matching video clip. Scoring is accuracy. Everyday Checked automatically 45.7% Dropped
MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. Everyday Checked automatically 66.6% Fading
MME-VideoOCR Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning. Everyday Checked automatically 77% Fading
MSR-VTT retrieval Models retrieve videos from text queries in the MSR-VTT collection. University Checked automatically 68.0 Fading
NExT-QA Models answer causal, temporal and descriptive questions about short videos of daily life. Labs usually report the multiple choice split. Everyday Checked automatically 86.2% Dropped
OmniMMI Omni models answer 2,290 questions on 1,121 streaming videos, including proactive tasks where they must decide when to respond. Everyday Checked automatically 53% Fading
ProactiveVideoQA Models answer spoken questions when a streamed video first reveals the answer. Everyday Graded by a model 33.5 Fading
RepCount-A Models count repeated actions in streamed video and are scored by distance from the true count. Everyday Graded by a model 35.4 Fading
StreamBench (StreamChat) Video models answer questions on streaming videos across media types, including multi-turn interaction and reasoning tasks. Everyday Checked automatically 72.8% Dropped
TACoS Models find the time span in a cooking video that matches a sentence. Labs score temporal grounding by mean IoU with the annotated span. Everyday Checked automatically 49.6% Dropped
TVQA Models answer 152,545 multiple-choice questions about clips from 6 TV shows, using video frames and subtitles. Scoring is accuracy. Everyday Checked automatically 87.3% Dropped
VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. Everyday Checked automatically 59.6% Fading
YouCook2 retrieval Models retrieve cooking videos from text queries in the YouCook2 collection. University Checked automatically 52.5 Fading
PerceptionComp 1,114 five-choice questions on 279 complex videos. Each question needs evidence from several separate moments, so no single frame answers it. Everyday Checked automatically 39.4% System cards only
TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. Everyday Checked automatically Fading
MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. Everyday Checked automatically Not in cards
Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. Expert Checked automatically Not in cards
MSRVTT-QA Models answer short open-ended questions about web video clips from the MSR-VTT captioning dataset. The question and answer pairs were generated automatically from the clip descriptions. Scored by answer accuracy. Everyday Checked automatically Not in cards
Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. Everyday Graded by a model Not in cards
ReasonVOS Models segment and track objects in videos from explicit referring queries and implicit reasoning queries. The score is overall J&F on ReVOS. Everyday Checked automatically Not in cards
Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. Everyday Checked automatically Not in cards
Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. Everyday Checked automatically Not in cards

Long videos

BenchmarkLevelScoredBest scoreStatus
LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Everyday Checked automatically 91.7% In use
MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Everyday Checked automatically 90.8% In use
LongVideoBench Models answer referring reasoning questions over 3,763 web videos up to an hour long, with interleaved subtitles. Scored by multiple-choice accuracy. Everyday Checked automatically 80.6% Fading
MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. Everyday Graded by a model 2.15 Dropped
EgoSchema Models answer five-way multiple-choice questions about three-minute egocentric video clips from Ego4D. Scored by accuracy on 5,031 questions. Everyday Checked automatically 76.2% Dropped
1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. Everyday Checked automatically 89.1% System cards only
VRBench Models answer multi-step reasoning questions about long narrative videos, which average 1.6 hours. The 1,010 videos come with 9,468 human-labeled question and answer pairs and 30,292 timestamped reasoning steps across seven reasoning types. Multiple-choice answers score the outcome, and a model judge scores the reasoning steps. Everyday Checked automatically 68.2% Not in cards
VideoEval-Pro Video models give open-ended short answers to questions that need the whole long video, which removes the guessing that multiple choice allows. Everyday Graded by a model 40.8% Fading
CG-Bench Video models answer 12,129 questions about 1,219 long videos and must also find the clue segments that support each answer. Everyday Checked automatically 33% Fading
MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. University Graded by a model 70.7% Fading
LVOmniBench Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track. Everyday Checked automatically 65.8% In use
MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. Everyday Checked automatically 46.8% System cards only
EgoLife (with memory) Models answer life-assistant questions about a week of egocentric video from six people, such as recalling past events and habits. Everyday Checked automatically 80.3% In use
LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. Everyday Checked automatically 85.6% In use
Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. Everyday Checked automatically 87.3% System cards only
ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. Expert Checked automatically 14.5% System cards only
APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. Everyday Graded by a model 60.2% Not in cards
HourVideo Multimodal models answer questions about first-person videos that run 20 to 120 minutes. The 500 videos come from Ego4D, and the 12,976 five-way multiple-choice questions cover summarization, perception, visual reasoning and navigation. Score is accuracy. Everyday Checked automatically Not in cards
MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. Expert Graded by a model Not in cards
VSI-SUPER Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours. Everyday Checked automatically Not in cards

Video captioning

BenchmarkLevelScoredBest scoreStatus
DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. Everyday Graded by a model 34.4% Dropped
OmniCap-IF Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance. Everyday Graded by a model 80.7% In use
VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. Everyday Checked automatically 71.3 Fading
CodeCap-Bench Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail. Everyday Graded by a model 73.2 Not in cards
Omni-Cloze Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail. Everyday Checked automatically 64.8% Fading
LiveSports-3K Video models give real-time commentary and answer questions on sports clips. An LLM judge scores the free-form commentary. Everyday Graded by a model 78.1% Fading
VATEX ZH Models write Chinese captions for short videos of human activities from the VATEX dataset. Captions are scored with CIDEr against human references. Everyday Checked automatically 59.7 Dropped
YouCook2 Models describe the procedure steps in YouTube cooking videos. Scored by CIDEr against human step captions. Everyday Checked automatically 188.3 Dropped
SoccerNet-Caption Models write timestamped commentary for soccer broadcasts. The set has almost 37k commentaries over 715.9 hours of video. Streaming reports also score when the model chooses to speak. Everyday Checked automatically Not in cards
YouCook2 Dense Video Description Models locate cooking steps and describe each step. The score is CIDEr for the complete dense video description task. Everyday Checked automatically Not in cards

Speech recognition and translation

BenchmarkLevelScoredBest scoreStatus
vCX Benchmark Models run inside a Retell voice agent on pass or fail checks for business rules, grounding, tool calls and speech-ready replies. Each suite reports its own pass rate. Everyday Checked automatically 93.3% Not in cards
CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. University Checked automatically 40.7 Fading
FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. Everyday Checked automatically 4.00% In use
MMSU Models answer questions about spoken language that need its content, its paralinguistic cues such as emotion and pitch, and its phonology. University Checked automatically 86.5% Fading
Common Voice Crowd-sourced multilingual read-speech corpus. Used here as an ASR test set scored by word error rate. Everyday Checked automatically 5.50% Fading
τ-Voice Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction. Everyday Checked automatically 81.7% In use
LibriSpeech Models transcribe English audiobook speech. Scored by word error rate on the test-clean and test-other splits. Everyday Checked automatically 1.53% Fading
EVA-Bench Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes. Everyday Graded by a model 0.74 In use
VoiceCodeBench Speech-to-text models transcribe 300 English workplace recordings. They are scored on whether every target entity, such as an address or email, survives exactly. Everyday Checked automatically 67.7% Not in cards
WildSpeech-Bench Models hold spoken conversations built from real chat data, with prosody, homophones and stuttering. A judge model scores the replies. Everyday Graded by a model 75.4 Fading
WenetSpeech Models transcribe Mandarin speech from many domains. Scored by character error rate on the test-net and test-meeting sets. Everyday Checked automatically Fading
AISHELL-1 Models transcribe Mandarin read speech. Scored by character error rate. Everyday Checked automatically 0.31% Dropped
Audio Arena Speech-to-speech models hold one long spoken conversation per scenario and call tools when needed, such as booking a dental appointment or ordering groceries. Six scenarios add up to 221 spoken turns with corrections, false memory traps, prompt injection and state tracking. An LLM judge grades each turn against a golden answer. Everyday Graded by a model 85.7 Not in cards
AISHELL-2 Models transcribe 1,000 hours of Mandarin speech recorded on several devices. Scored by character error rate. Everyday Checked automatically 2.56% Dropped
Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. Everyday Checked automatically 6.14% Fading
Seed-TTS-eval Models clone a voice from a short prompt and speak new text in English and Mandarin. Scored by word error rate from an ASR model and by speaker similarity. Everyday Checked automatically 0.77% Fading
Coval Speech-to-Text WER Speech models transcribe call-like audio. The score is mean word error percentage over the published pooled 24-hour window. Everyday Checked automatically 1.79% Not in cards
Full-Duplex-Bench Full-duplex spoken dialogue models are scored on pause handling, backchanneling, smooth turn-taking and user interruption. Automatic metrics include takeover rate and latency. Everyday Checked automatically 0.40 In use
Speech-to-Speech Index Artificial Analysis's combined score for speech-to-speech models. Research frontier Graded by a model 86.9 In use
SpeechRole Speech role-playing agents talk in character across 98 roles. A model judge scores interaction ability, speech expressiveness and role fidelity from the generated speech. Everyday Graded by a model 123.5 Fading
Full-Duplex-Bench v1.5 Full-duplex speech models are tested on overlap scenarios: listener backchannel, side conversation and ambient speech. Metrics cover how the model yields, continues or responds. Everyday Checked automatically 77.8 Fading
Hume AI Overall Quality Index Listeners rate generated speech on reliability and expressiveness. The benchmark reports an overall quality index. Everyday Ranked by votes 0.9 In use
Kimi-Audio speech conversation evaluation People rate spoken replies on speed, accent and emotion control, empathy and style control. Scored on a 1 to 5 scale and averaged. Everyday Graded by people 3.90 System cards only
MU-Bench Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts. Everyday Checked automatically 14.9% Not in cards
Artificial Analysis average word error rate Speech transcription systems are scored by average word error rate across the Artificial Analysis evaluation set. Everyday Checked automatically 4.00% In use
Context Aware Automatic Speech Recognition Speech transcription systems are scored for semantic accuracy with and without free-form context. Everyday Graded by a model 44.6% In use
CueSpeak Models speak at the correct moment and give the expected semantic response during simultaneous speech. Everyday Graded by a model 81.7% Fading
Full-Duplex-Bench v3 Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking. Everyday Checked automatically 68% Fading
Hume AI Voice Design Benchmark Listeners rate generated voices for English voice design, multilingual control, accents and voice qualities. Everyday Ranked by votes 71.4 In use
InstructTTSEval Text-to-speech models follow natural-language style instructions in English and Chinese across acoustic parameter, descriptive style and role-play tasks. Gemini judges each clip. Everyday Graded by a model 83% Fading
Kimi-ASR Internal Testset Subset 1 Models transcribe speech from Moonshot internal subset 1. The score is the published word error rate. Everyday Checked automatically 1.42% Not in cards
Kimi-ASR Internal Testset Subset 2 Models transcribe speech from Moonshot internal subset 2. The score is the published word error rate. Everyday Checked automatically 2.44% Not in cards
Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. Everyday Checked automatically 4.70 Fading
Real-World Audio Recording Speech transcription systems are scored by word error rate on real-world recordings across multiple languages. Everyday Checked automatically 9.60% In use
Speech Agent Arena People hold live conversations with two hidden speech-to-speech agents on the same scenario, some with tool calls, then pick the one they preferred. Votes are fit to an Elo rating, and the arena also tracks task success. Everyday Ranked by votes 1,070 In use
TimeSpeak Models initiate speech at specified times while producing the requested content. Everyday Graded by a model 64.7% Fading
URO-Bench Spoken dialogue models answer speech with speech across Understanding, Reasoning and Oral conversation. It has a basic track of 16 datasets and a pro track of 20. Everyday Graded by a model 84.1% Fading
VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. Everyday Graded by people 3.44 Not in cards
Coval Text-to-Speech TTFA Speech models synthesize customer-service prompts. The score is median time to first audible sample in milliseconds over the published 24-hour window. Everyday Checked automatically 273.8 Not in cards
MultiChallenge Audio (Xiaomi) Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house. Everyday Graded by a model 15.2% System cards only
Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. Everyday Graded by people 1.37 System cards only
TurnBench Turn-taking systems listen to two-person conversations and must detect when a speaker's turn ends and when the other speaker interrupts. The test set has 116 English conversations in six interaction styles, each labeled by three annotators. Systems rank by recall, and a system whose false-positive rate exceeds 0.15 ranks below every qualifying system. Latency after each event is also reported. Everyday Checked automatically 23.3% Not in cards
Coval Voice AI Benchmarks Speech-to-text, text-to-speech and speech-to-speech models from almost 40 providers are scored every 30 minutes on call-like audio with accents, noise and echo. Everyday Checked automatically Not in cards
CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. Everyday Checked automatically System cards only
Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. Everyday Checked automatically Not in cards
Full-Duplex-Bench v2 An automated examiner holds real-time multi-turn audio conversations with the model over WebRTC or WebSocket. A model judge scores task completion under conversational constraints. Everyday Graded by a model Not in cards
Kimi-ASR Internal Testset Models transcribe speech from two internal Moonshot test subsets. Scored by word error rate, lower is better. Everyday Checked automatically System cards only
MiniMax TTS Multilingual Test Set Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity. Everyday Checked automatically System cards only
Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. School Checked automatically System cards only
SUPERB Self-supervised speech models are tested as frozen encoders on a range of speech tasks, including speech recognition, keyword spotting, speaker identification, intent classification and emotion recognition. Each task trains only a light prediction head on top of the shared model. Everyday Checked automatically Not in cards
WenetSpeech Test_Meeting Models transcribe spontaneous Mandarin speech from real meetings in the Test_Meeting set. The baseline score is mixture error rate, counting Mandarin characters and English words. Everyday Checked automatically Not in cards
WenetSpeech Test_Net Models transcribe Mandarin internet speech from the Test_Net set. The baseline score is mixture error rate, counting Mandarin characters and English words. Everyday Checked automatically Not in cards

Audio understanding and voice

BenchmarkLevelScoredBest scoreStatus
VoiceBench Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. Everyday Graded by a model 93.1% In use
MMAU 10K audio clips with questions on speech, sound and music needing expert knowledge and reasoning. Multiple-choice accuracy. University Checked automatically 82.2% In use
Audio MC Spoken multi-turn conversations test inference memory, instruction retention, self-coherence and voice editing. O4-mini judge applies per-conversation rubrics. Everyday Graded by a model 60.4% In use
OmniBench Models answer multiple choice questions that need an image, an audio clip and text together. Scored by accuracy. Everyday Checked automatically 56.1% Dropped
Big Bench Audio Models answer 1,000 spoken questions from four Big Bench Hard tasks: formal fallacies, navigate, object counting and web of lies. Built to test reasoning over audio input. School Checked automatically 98.5% In use
MMAU-Pro Models answer 5,305 expert-written questions about speech, sounds, music and their mixes, covering 49 listening skills. Some clips run up to 10 minutes. University Checked automatically 72.2% Fading
MSEB Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction. Everyday Checked automatically 39.2 Fading
AHELM Audio-language models are scored across 10 aspects with standard prompts. The suite combines existing datasets with two new ones, PARADE for stereotypes and CoRe-Bench for conversational reasoning. Everyday Checked automatically 80.3 Not in cards
Voice Arena Hindi Models generate speech in Hindi. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,084 System cards only
VoxMem Audio language models answer questions that need memory of earlier spoken sessions. Questions cover what was said, who said it, how it was spoken and what was audible. Everyday Checked automatically 38.5% Not in cards
DECIDE Models select decisions from short voice-call clips and conversation history across 4,867 questions, 1,081 clips, and 32 tasks. Human reference labels determine accuracy. The default board ranks by DECIDE Balance Score, which combines accuracy, cost, and latency on a zero-to-100 scale. Everyday Checked automatically 66.6 Not in cards
MuChoMusic Models answer multiple-choice questions about music recordings. It tests music knowledge and reasoning in audio-language models. University Checked automatically 72.6% Fading
UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. Everyday Graded by a model 76.5% Dropped
MMAR Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge. University Checked automatically 80% Fading
Voice Arena Arabic MSA Models generate speech in Arabic MSA. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,192 System cards only
Voice Arena Brazilian Portuguese Models generate speech in Brazilian Portuguese. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,127 System cards only
Voice Arena Mexican Spanish Models generate speech in Mexican Spanish. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,138 System cards only
Voice Arena US English Models generate speech in US English. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,081 System cards only
Voice Arena Vietnamese Models generate speech in Vietnamese. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,141 System cards only
AVMeme Exam Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge. Everyday Checked automatically 69.5% Fading
MMAU v05.15.25 Models answer multiple-choice questions about speech, sound and music clips that need expert knowledge and reasoning. Scored by accuracy on test-mini or the hidden-answer test set. University Checked automatically 77.8% Not in cards
MusTBench Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items. Expert Checked automatically 50.6% In use
Voice Arena Japanese Models generate speech in Japanese. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,188 System cards only
Cekura Agent Performance Scenario reliability across three repeated calls (Pass³). Everyday Graded by a model 81.7% Not in cards
Cekura Turn Bench Mean end-of-turn score across six line conditions, as published by Cekura. Everyday Graded by a model 84.7% Not in cards
Clotho-AQA Models answer crowdsourced yes or no and single word questions about short environmental audio clips. Scored by accuracy. Everyday Checked automatically 73.7% Dropped
SpotSoundBench Models find the time span of short sound events hidden in long, unstructured YouTube recordings. It has 400 audio, query and timestamp triplets. Everyday Checked automatically 67.2% In use
DesignArena Audio Realism Bench Voice models produce conversational speech, and users vote on which sounds more real. DesignArena built it because 1 to 5 opinion scores for TTS had saturated. Everyday Ranked by votes 1,179 Not in cards
DesignArena Text-to-Speech Speech models read the same text aloud. Users listen to two anonymous clips and vote for the more natural one. Everyday Ranked by votes 1,226 Not in cards
OpenAudioBench Models answer spoken questions in five sets: Reasoning QA, Llama Questions, Web Questions, TriviaQA and AlpacaEval. Knowledge sets use accuracy and the open sets use a model score. Everyday Graded by a model 80.1% Dropped
OpenAudioBench AlpacaEval Models respond to spoken AlpacaEval instructions. The score is the published judge quality score. Everyday Graded by a model 7.79 Not in cards
OpenAudioBench Llama Questions Models answer spoken Llama Questions. The score is the published accuracy. Everyday Graded by a model 78.5% Not in cards
OpenAudioBench Reasoning QA Models answer spoken reasoning questions. The score is the published accuracy. Everyday Graded by a model 50% Not in cards
OpenAudioBench TriviaQA Models answer spoken TriviaQA questions. The score is the published judge quality score. Everyday Graded by a model 6.19 Not in cards
OpenAudioBench Web Questions Models answer spoken Web Questions. The score is the published judge quality score. Everyday Graded by a model 6.86 Not in cards
Music Arena Users write a music prompt, listen to tracks from two text-to-music systems, and vote. Preferences, listening data and feedback feed a leaderboard. Everyday Ranked by votes 1,394 Not in cards
Voice Arena Bahasa Models generate speech in Bahasa. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,138 Not in cards
Voice Arena Tagalog Models generate speech in Tagalog. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,130 Not in cards
AIR-Bench (Audio Instruction Benchmark) Audio language models answer questions about speech, natural sounds and music. A foundation set has 19 tasks with about 19k single-choice questions, and a chat set has 2k open-ended questions scored by GPT-4. Everyday Graded by a model 6.98 System cards only
Artificial Analysis Speech Arena Voters listen to two speech clips made from the same text and pick the more natural one. Votes become Elo ratings. Everyday Ranked by votes 1,211 Fading
Audio Understanding Benchmark (Mistral) Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription. Expert Graded by a model 86.6% Dropped
Cekura Speech to Text Overall word error rate across the published audio clips. Everyday Checked automatically 3.73% Not in cards
HumMusQA Models answer 320 hand-written questions about music, curated and checked by people with musical training. Expert Checked automatically 75.8% In use
LongAudioSpan Models answer 3,240 questions about 360 English and Chinese recordings from 10 minutes to over 2 hours. Each question is scored as multiple choice and as open-ended. Everyday Graded by a model 82.7% In use
MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. Everyday Graded by a model 30.5% Dropped
Music Arena Vocal Models generate music with vocals. Listeners compare anonymous tracks, and separate vocal preference votes determine the Arena score. Everyday Ranked by votes 1,107 Not in cards
StudyArena Audio Models produce audio responses to user prompts. Anonymous human comparisons determine the published rating in this separate vote pool. Everyday Ranked by votes 1,216 Not in cards
TTS Arena Users listen to two anonymous text-to-speech models read the same text and vote for the more natural one. Votes become Elo ratings. Everyday Ranked by votes 1,545 Not in cards
Voice Arena French Models generate speech in French. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,084 Not in cards
Voice Arena German Models generate speech in German. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,034 Not in cards
Voice Arena Indian English Models generate speech in Indian English. Listeners compare anonymous clips, and preference votes determine independent language Elo ratings. Everyday Ranked by votes 1,066 Not in cards
Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. Everyday Graded by a model 6.28 System cards only
MAEB Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages. Everyday Checked automatically 49.9 System cards only
Speech Arena Models generate speech. Listeners compare outputs, and pairwise votes determine Elo ratings. Everyday Ranked by votes 1,566 Not in cards
AIR-Bench Foundation Models answer foundational audio understanding questions. The score is the official average accuracy across nineteen tasks. Everyday Checked automatically Not in cards
AudioBench Audio language models follow instructions about speech, voice and environmental sounds. The paper covers 8 tasks and 26 datasets, including speech recognition, spoken question answering, voice traits and audio scenes. The repository has since grown to more than 50 datasets. Everyday Not in cards
Cekura Bench Models complete separately scored speech recognition, speech generation, voice-agent, real-call, turn-taking and voice-quality benchmarks. Cekura publishes no combined score. Everyday Graded by a model Not in cards
Cekura Production Voice Agents Published composite score for production voice agents. Everyday Graded by a model Not in cards
Cekura Real Human Calls Real caller reliability across three repeated calls (Pass³). Everyday Graded by a model Not in cards
Cekura Text to Speech Round-trip word error rate averaged across the two transcribers, as displayed by Cekura. Everyday Checked automatically Not in cards
DesignArena Speech-to-Speech Voice models answer spoken input with speech. Users listen to two anonymous replies and vote for the better one. Everyday Ranked by votes Not in cards
Dynamic-SUPERB Phase-2 Spoken language models follow text instructions about audio across 180 tasks in speech, music and other sounds. Tasks include classification, regression and sequence generation, and researchers keep contributing new ones. Models run zero-shot, without task-specific training. Everyday Not in cards
MMAE Audio models edit audio from instructions across a broad range of real-world editing tasks and sound types. Everyday Graded by a model Not in cards

Video generation

BenchmarkLevelScoredBest scoreStatus
LMArena Text-to-Video Arena Video generation models make clips from the same text prompt, anonymously and side by side. Votes from logged-in users give Elo-style ratings. Everyday Ranked by votes 1,540 Not in cards
LMArena Image-to-Video Arena Video generation models animate the same input image with the same prompt, anonymously and side by side. Votes give Elo-style ratings. Everyday Ranked by votes 1,503 Not in cards
DesignArena Code to Video Agents write code that renders a video from a user prompt. Users compare the two videos and vote for the better one. Everyday Ranked by votes 1,341 Not in cards
Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. Expert Graded by a model 1,585 Not in cards
Code2Video-Compass Multimodal models write programs that render complete videos from text, an image or a video. The default Overall board ranks models by their mean rank across quality metrics, so a lower score is better. Everyday Graded by a model 1.07 Not in cards
DesignArena Video Arena Users compare videos from anonymous video generation models for the same prompt and vote for the better one. Votes become Elo ratings. Everyday Ranked by votes 1,369 Not in cards
Code2Video Arena Models create videos through code. Pairwise votes determine the published Arena rating. Everyday Ranked by votes 1,599 Not in cards
Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. Everyday Graded by a model 64.7 Not in cards
AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. Expert Graded by a model 38.4% In use
DesignArena Image to Video Video models animate a user image into a short video. Users compare two anonymous videos and vote for the better one. Everyday Ranked by votes 1,319 Not in cards
LMArena Video Edit Arena Video editing models apply the same edit request to the same clip, anonymously and side by side. Community votes give Elo-style ratings. Everyday Ranked by votes 1,424 Not in cards
VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. Expert Graded by a model 4.11 Not in cards
DesignArena Multi-Image to Video Video models compose one video from several reference images. Users compare two anonymous videos and vote for the better one. Everyday Ranked by votes 1,329 Not in cards
DesignArena Multimodal to Video Video models make a video from a mix of text, images and other inputs. Users compare two anonymous videos and vote for the better one. Everyday Ranked by votes 1,305 Not in cards
VBench Video generation models make clips from prompts, and automatic checks score 16 dimensions of quality and prompt adherence. Everyday Checked automatically 84.2% Fading
DesignArena Video Editing Video models edit a user video following a text instruction. Users compare the two results and vote for the better one. Everyday Ranked by votes 1,373 Not in cards
IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. Everyday Graded by a model 55.3 Dropped
SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. Everyday Graded by a model 28.3 Not in cards
CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. Everyday Graded by a model 55.2 Not in cards
VBench image-to-video Models animate input images. Automatic checks score image conditioning and video quality across the complete image-to-video suite. Everyday Checked automatically 86.9% Not in cards
VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. Everyday Graded by a model 66.7% Not in cards
AA-Video-Editing Voters compare two edits of the same video made from one instruction and pick the better one. Votes become Elo ratings. Everyday Ranked by votes Not in cards
Artificial Analysis Video Arena Voters compare videos from two text-to-video models for the same prompt and pick the better one. Votes become Elo ratings. Everyday Ranked by votes Not in cards
EvalCrafter Text-to-video models generate clips for 700 prompts drawn from real user requests. The clips are scored with 17 automatic metrics for visual quality, content quality, motion and text alignment. Weights fitted to human ratings combine the metrics into one score. Everyday Graded by a model Not in cards
KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. Everyday Graded by a model Not in cards
Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. Everyday Graded by people Not in cards
LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. Everyday Graded by a model Not in cards
Movie Gen Video Bench Video generation models make clips from a fixed prompt set, and people compare them. Meta released non-cherry-picked Movie Gen videos for comparison. Everyday Graded by people Not in cards
OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. Everyday Graded by a model Not in cards
StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. Everyday Graded by a model Not in cards