MMAR
The official leaderboard hasn’t scored a model released after Qwen3-Omni-30B-A3B-Thinking (September 2025). Labs report their own numbers for 5 newer models, such as Qwen3.5-Omni-Plus at 80%.
Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge.
Repo has over 200 GitHub stars.
- Made by
- Ma et al. (arXiv 2505.13032)
- Released
- May 2025
- Kind
- Audio understanding and voice
- Level
- University
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Real work and use
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Audio and speech
- Strongest model tested
- o3 #84 on the capability index
- Last in a launch post
- October 2026
- Last model added
- October 2026
- Scores refreshed
- November 20th, 2026
In the authors’ words
We introduce MMAR, a new benchmark designed to evaluate the deep reasoning capabilities of Audio-Language Models (ALMs) across massive multi-disciplinary tasks. MMAR comprises 1,000 meticulously curated audio-question-answer triplets, collected from real-world internet videos and refined through iterative error corrections and quality checks to ensure high quality. Unlike existing benchmarks that are limited to specific domains of sound, music, or speech, MMAR extends them to a broad spectrum of real-world audio scenarios, including mixed-modality combinations of sound, music, and speech. Each question in MMAR is hierarchically categorized across four reasoning layers: Signal, Perception, Semantic, and Cultural, with additional sub-categories within each layer to reflect task diversity and complexity. To further foster research in this area, we annotate every question with a Chain-of-Thought (CoT) rationale to promote future advancements in audio reasoning. Each item in the benchmark demands multi-step deep reasoning beyond surface-level understanding. Moreover, a part of the questions requires graduate-level perceptual and domain-specific knowledge, elevating the benchmark's difficulty and depth. We evaluate MMAR using a broad set of models, including Large Audio-Language Models (LALMs), Large Audio Reasoning Models (LARMs), Omni Language Models (OLMs), Large Language Models (LLMs), and Large Reasoning Models (LRMs), with audio caption inputs. The performance of these models on MMAR highlights the benchmark's challenging nature, and our analysis further reveals critical limitations of understanding and reasoning capabilities among current models. We hope MMAR will serve as a catalyst for future advances in this important but little-explored area.
Qwen3.5-Omni-Plus leads with 80%
Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.
Qwen3-Omni-30B-A3B-Thinking leads with 66.4% across all independent runs
Best score per model across the official leaderboard, Epoch AI’s own runs and any board Epoch AI copied. Dotted bars are reruns by Epoch AI or another board, and each row names its source.
Qwen3.5-Omni-Plus leads with 80%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model. Striped bars are claims, and dotted bars are reruns by Epoch AI or another board.
- 1
Qwen3.5-Omni-Plus 80%
- 2
Qwen3.8-Omni-Flash 79.8%
- 3
Qwen3.5-Omni-Flash 74%
- 4
NVIDIA Nemotron 3.5 Transcribe 30B-A4B 66.7%
- 5 MiMo-Audio-7B-Instruct 63.6%
Source: Labs’ model cards
- 1
Qwen3-Omni-30B-A3B-Thinking 66.4%
- 2 o3 54.7%
- 3 o1 53%
Source: Official MMAR leaderboard
- 1
Qwen3.5-Omni-Plus 80%
- 2
Qwen3.8-Omni-Flash 79.8%
- 3
Qwen3.5-Omni-Flash 74%
- 4
NVIDIA Nemotron 3.5 Transcribe 30B-A4B 66.7%
- 5
Qwen3-Omni-30B-A3B-Thinking 66.4%
- 6 MiMo-Audio-7B-Instruct 63.6%
- 7 o3 54.7%
- 8 o1 53%
Sources: Labs’ model cards and official MMAR leaderboard
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 80% | Mar 2026 | Lab model card | |
| 2 | | 79.8% | Sep 2026 | Lab model card | |
| 3 | | 74% | Mar 2026 | Lab model card | |
| 4 | | 66.7% | Oct 2026 | Lab model card | |
| 5 | MiMo-Audio-7B-Instruct | 63.6% | Sep 2025 | Lab model card |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 66.4% | Sep 2025 | Official leaderboard | ||
| 2 | o3 | 54.7% | Caption | Apr 2025 | Official leaderboard |
| 3 | o1 | 53% | Caption | Dec 2024 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | 80% | Mar 2026 | Lab model card | ||
| 2 | 79.8% | Sep 2026 | Lab model card | ||
| 3 | 74% | Mar 2026 | Lab model card | ||
| 4 | 66.7% | Oct 2026 | Lab model card | ||
| 5 | 66.4% | Sep 2025 | Official leaderboard | ||
| 6 | MiMo-Audio-7B-Instruct | 63.6% | Sep 2025 | Lab model card | |
| 7 | o3 | 54.7% | Caption | Apr 2025 | Official leaderboard |
| 8 | o1 | 53% | Caption | Dec 2024 | Official leaderboard |
The benchmark's own leaderboard has 35 entries, 3 of them models we track. Switch the scores above to rank them together.
Rankings by part
We read 7 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Sound, Music and Speech. Qwen3-Omni-30B-A3B-Thinking leads every one of them. Arrows show how far a model moved from its overall rank among the models we track.
Sound
3 models we track have a score here.
- 1
Qwen3-Omni-30B-A3B-Thinking 57.6%
- 2 o3 49.7%
- 3 o1 48.5%
Music
3 models we track have a score here.
- 1
Qwen3-Omni-30B-A3B-Thinking 44.7%
- 2 o1 ↑1 43.2%
- 3 o3 ↓1 41.8%
Sound and music
3 models we track have a score here.
- 1
Qwen3-Omni-30B-A3B-Thinking 63.4%
- 2 o3 36.4%
- 3 o1 18.2%
Sound and speech
3 models we track have a score here.
- 1
Qwen3-Omni-30B-A3B-Thinking 75.2%
- 2 o3 60.1%
- 3 o1 56.9%
Music and speech
3 models we track have a score here.
- 1
Qwen3-Omni-30B-A3B-Thinking 69.5%
- 2 o3 52.4%
- 3 o1 45.1%
Sound and music and speech
3 models we track have a score here.
- 1
Qwen3-Omni-30B-A3B-Thinking 79.1%
- 2 o3 54.2%
- 3 o1 45.8%
2 models have held number one in labs’ own numbers since its release in May 2025 2 models have held number one across all independent runs since its release in May 2025 3 models have held number one with claims included since its release in May 2025
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Sources: Official MMAR leaderboard and labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official MMAR leaderboard and labs’ model cards
2 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out. Qwen accounts for 3 of the 5 releases that named it since 2024.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 28, 2026 | 66.7 | MMAR — Audio reasoning | |
| Sep 18, 2026 | 79.8 | MMAR Audio Understanding | |
| Mar 30, 2026 | 74 | MMAR, Qwen3.5-Omni-Flash | |
| Mar 30, 2026 | 80 | MMAR |
3 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Sep 22, 2026 | | 79.8 | MMAR (Table 4) | Capabilities |
| Apr 17, 2026 | | 80 | MMAR (Table 5) | Capabilities |
| Sep 18, 2025 | MiMo-Audio-7B-Instruct technical report | 63.6 | MMAR (Table 8) | Capabilities |
More in Audio and video
- Video-MMMU In use 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy.
Qwen 3.8 Max 88.7%
- LVBench In use Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization.
Gemini 4 Argon 91.7%
- MMVU In use 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy.
Qwen 3.8 Max 82.4%
- MLVU In use Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy.
Qwen 3.8 Max 90.8%
- FLEURS In use Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better.
Phi-4-multimodal 4.00%
- VoiceBench In use Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules.
Qwen3.5-Omni-Plus 93.1%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Pass Independent runs Scores from the official leaderboard (35 entries), launch posts (4 scores) and system cards (3 scores).
- Pass Models scored 8 models we track, plus 32 other entries on its leaderboard.
- Pass Newest score The newest is from October 2026, for NVIDIA Nemotron 3.5 Transcribe 30B-A4B.
- Note Frontier models None of the top 10 has a score. The strongest tested is o3 (#84).
- Pass Headroom The best is 80%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on github.com: 3 rows for models we track and 32 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.