Benchmark Almanac

MMAR

Qwen3.5-Omni-Plus 80% leads since March 2026
Qwen3-Omni-30B-A3B-Thinking 66.4% leads since September 2025
Qwen3.5-Omni-Plus 80% leads since March 2026
In use

The official leaderboard hasn’t scored a model released after Qwen3-Omni-30B-A3B-Thinking (September 2025). Labs report their own numbers for 5 newer models, such as Qwen3.5-Omni-Plus at 80%.

Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge.

Repo has over 200 GitHub stars.

Released
May 2025
Kind
Audio understanding and voice
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Audio → Text
Interaction
Single turn
Items from
Real work and use
Refresh
One fixed set
Run by
Anyone
In views
Audio and speech
Strongest model tested
o3 #84 on the capability index
Last in a launch post
October 2026
Last model added
October 2026
Scores refreshed
November 20th, 2026

In the authors’ words

We introduce MMAR, a new benchmark designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR comprises 1,000 meticulously curated audio-question-answer triplets, collected from real-world internet videos and refined through iterative error corrections and quality checks to ensure high quality. Unlike existing benchmarks that are limited to specific domains of sound, music, or speech, MMAR extends them to a broad spectrum of real-world audio scenarios, including mixed-modality combinations of sound, music, and speech. Each question in MMAR is hierarchically categorized across four reasoning layers: Signal, Perception, Semantic, and Cultural, with additional sub-categories within each layer to reflect task diversity and complexity. To further foster research in this area, we annotate every question with a Chain-of-Thought (CoT) rationale to promote future advancements in audio reasoning. Each item in the benchmark demands multi-step deep reasoning beyond surface-level understanding. Moreover, a part of the questions requires graduate-level perceptual and domain-specific knowledge, elevating the benchmark's difficulty and depth. We evaluate MMAR using a broad set of models, including Large Audio-Language Models (LALMs), Large Audio Reasoning Models (LARMs), Omni Language Models (OLMs), Large Language Models (LLMs), and Large Reasoning Models (LRMs), with audio caption inputs. The performance of these models on MMAR highlights the benchmark's challenging nature, and our analysis further reveals critical limitations of understanding and reasoning capabilities among current models. We hope MMAR will serve as a catalyst for future advances in this important but little-explored area.

Abstract of MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (arXiv:2505.13032, CC0)
5
Models scored in lab cards
4
Launch posts list it, from 2 labs
3
System cards report it, from 2 labs

Qwen3.5-Omni-Plus leads with 80%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

Qwen3-Omni-30B-A3B-Thinking leads with 66.4% across all independent runs

Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.

Qwen3.5-Omni-Plus leads with 80%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.

  1. 1 Qwen3.5-Omni-Plus 80%
  2. 2 Qwen3.8-Omni-Flash 79.8%
  3. 3 Qwen3.5-Omni-Flash 74%
  4. 4 NVIDIA Nemotron 3.5 Transcribe 30B-A4B 66.7%
  5. 5 MiMo-Audio-7B-Instruct 63.6%

Source: Labs’ model cards

Source: Official MMAR leaderboard

#ModelScoreSettingDateSource
1 Qwen3.5-Omni-Plus 80% Mar 2026 Lab model card
2 Qwen3.8-Omni-Flash 79.8% Sep 2026 Lab model card
3 Qwen3.5-Omni-Flash 74% Mar 2026 Lab model card
4 NVIDIA Nemotron 3.5 Transcribe 30B-A4B 66.7% Oct 2026 Lab model card
5 MiMo-Audio-7B-Instruct 63.6% Sep 2025 Lab model card
#ModelScoreSettingDateSource
1 Qwen3-Omni-30B-A3B-Thinking 66.4% Sep 2025 Official leaderboard
2 o3 54.7% Caption Apr 2025 Official leaderboard
3 o1 53% Caption Dec 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Qwen3.5-Omni-Plus 80% Mar 2026 Lab model card
2 Qwen3.8-Omni-Flash 79.8% Sep 2026 Lab model card
3 Qwen3.5-Omni-Flash 74% Mar 2026 Lab model card
4 NVIDIA Nemotron 3.5 Transcribe 30B-A4B 66.7% Oct 2026 Lab model card
5 Qwen3-Omni-30B-A3B-Thinking 66.4% Sep 2025 Official leaderboard
6 MiMo-Audio-7B-Instruct 63.6% Sep 2025 Lab model card
7 o3 54.7% Caption Apr 2025 Official leaderboard
8 o1 53% Caption Dec 2024 Official leaderboard

The benchmark's own leaderboard has 35 entries, 3 of them models we track. Switch the scores above to rank them together.

Rankings by part

We read 7 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Sound, Music and Speech. Qwen3-Omni-30B-A3B-Thinking leads every one of them. Arrows show how far a model moved from its overall rank among the models we track.

Sound

3 models we track have a score here.

  1. 1 Qwen3-Omni-30B-A3B-Thinking 57.6%
  2. 2 o3 49.7%
  3. 3 o1 48.5%

2 models have held number one in labs’ own numbers since its release in May 2025 2 models have held number one across all independent runs since its release in May 2025 3 models have held number one with claims included since its release in May 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Qwen Xiaomi Corp
Own leaderboard Not tested since Sep 2025
Lab claims 2 leaders

Sources: Official MMAR leaderboard and labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Qwen3.5-Omni-Plus Qwen3-Omni-30B-A3B-Thinking o3 0% 20% 40% 60% 80% 100% 2025 2026 2027 Near the ceiling (85%) Qwen3.5-Omni-Plus Qwen3-Omni-30B-A3B-Thinking

Sources: Official MMAR leaderboard and labs’ model cards

2 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out. Qwen accounts for 3 of the 5 releases that named it since 2024.

Qwen
NVIDIA
DateLaunch postScoreAs the post writes it
Oct 28, 2026 NVIDIA Nemotron 3.5 Transcribe 30B-A4B 66.7 MMAR — Audio reasoning
Sep 18, 2026 Qwen3.8-Omni-Flash 79.8 MMAR Audio Understanding
Mar 30, 2026 Qwen3.5-Omni-Flash 74 MMAR, Qwen3.5-Omni-Flash
Mar 30, 2026 Qwen3.5-Omni-Plus 80 MMAR

3 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Sep 22, 2026 Qwen3.8-Omni-Flash technical report 79.8 MMAR (Table 4) Capabilities
Apr 17, 2026 Qwen3.5-Omni-Plus technical report 80 MMAR (Table 5) Capabilities
Sep 18, 2025 MiMo-Audio-7B-Instruct technical report 63.6 MMAR (Table 8) Capabilities

More in Audio and video

All 695 in Audio and video
  • Video-MMMU In use 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. Qwen 3.8 Max 88.7%
  • LVBench In use Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. Gemini 4 Argon 91.7%
  • MMVU In use 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. Qwen 3.8 Max 82.4%
  • MLVU In use Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. Qwen 3.8 Max 90.8%
  • FLEURS In use Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. Phi-4-multimodal 4.00%
  • VoiceBench In use Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. Qwen3.5-Omni-Plus 93.1%

Data audit

All 10 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (35 entries), launch posts (4 scores) and system cards (3 scores).
  • Pass Models scored 8 models we track, plus 32 other entries on its leaderboard.
  • Pass Newest score The newest is from October 2026, for NVIDIA Nemotron 3.5 Transcribe 30B-A4B.
  • Note Frontier models None of the top 10 has a score. The strongest tested is o3 (#84).
  • Pass Headroom The best is 80%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on github.com: 3 rows for models we track and 32 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check