Benchmark Almanac

OmniEchoBench

Qwen3.8-Omni-Flash 43.9% leads since September 2026
Not in model cards

Models answer 2,972 spatial questions from recorded audio and video or navigate toward sound sources in 900 episodes. Rules score extracted spatial answers and multiple-choice letters. Navigation metrics measure success within one meter and path efficiency.

The scorer uses a model to extract answers before deterministic checks. Public annotations and code exist, but the README says media download links remain pending. Navigation stopping rules and annotation paths need explicit versioning.

Released
September 2026
Kind
Embodied reasoning
Level
Everyday
Scored
Checked automatically
Answers
Agent task
Measures
Capability
Language
English
In and out
Text, Audio, Video → Text, Actions
Interaction
Agentic
Environment
A simulator
Items from
Real work and use
Refresh
One fixed set
Run by
A host
Strongest model tested
Gemini 3.8 Flash #24 on the capability index
Last model added
September 2026
Scores refreshed
November 18th, 2026

In the authors’ words

Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose OmniEcho, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges. Our code and data will be available in https://github.com/PKU-VaLuE-Lab/OmniEcho/tree/main

Abstract of OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents (arXiv:2609.23407, CC0)
10
Entries on its own leaderboard, 7 models we track

Qwen3.8-Omni-Flash leads with 43.9% among models released by September 2026 Qwen3.8-Omni-Flash leads with 43.9% among models released by September 2026

Best score per entry from the benchmark's own leaderboard, read on November 18th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in September 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Qwen3.8-Omni-Flash 43.9%
  2. 2 Gemini 3.8 Flash 32%
  3. 3 Gemini 3.7 Flash 31.4%
  4. 4 Muse Spark 1.2 31.3%
  5. 5 OmniEcho (Ours) 28.5%
  6. 6 doubao-seed-2-0-lite-260428 28.3%
  7. 7 Qwen3.5-Omni-Plus 23.3%
  8. 8 Qwen3-Omni-30B-A3B 18.5%
  9. 9 Qwen2.5-Omni-7B 12.6%
  10. 10 SO-7B 11.4%

Source: Official OmniEchoBench leaderboard

#EntryScoreSettingDateSource
1 Qwen3.8-Omni-Flash 43.9% Sep 2026 Official leaderboard
2 Gemini 3.8 Flash 32% Sep 2026 Official leaderboard
3 Gemini 3.7 Flash 31.4% Aug 2026 Official leaderboard
4 Muse Spark 1.2 31.3% Aug 2026 Official leaderboard
5 OmniEcho (Ours) 28.5% Official leaderboard
6 doubao-seed-2-0-lite-260428 28.3% Official leaderboard
7 Qwen3.5-Omni-Plus 23.3% Mar 2026 Official leaderboard
8 Qwen3-Omni-30B-A3B 18.5% Sep 2025 Official leaderboard
9 Qwen2.5-Omni-7B 12.6% Mar 2025 Official leaderboard
10 SO-7B 11.4% Official leaderboard
#ModelScoreSettingDateSource
1 Qwen3.8-Omni-Flash 43.9% Sep 2026 Official leaderboard
2 Gemini 3.8 Flash 32% Sep 2026 Official leaderboard
3 Gemini 3.7 Flash 31.4% Aug 2026 Official leaderboard
4 Muse Spark 1.2 31.3% Aug 2026 Official leaderboard
5 Qwen3.5-Omni-Plus 23.3% Mar 2026 Official leaderboard
6 Qwen3-Omni-30B-A3B 18.5% Sep 2025 Official leaderboard
7 Qwen2.5-Omni-7B 12.6% Mar 2025 Official leaderboard

Rankings by part

We read 5 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Camera rotation, Source direction and 3D localization. Qwen3.8-Omni-Flash leads every one of them. Arrows show how far a model moved from its overall rank among the models we track.

Camera rotation

7 models we track have a score here.

  1. 1 Qwen3.8-Omni-Flash 74.1%
  2. 2 Muse Spark 1.2 ↑2 55%
  3. 3 Gemini 3.8 Flash ↓1 53.7%
  4. 4 Qwen3.5-Omni-Plus ↑1 52.4%
  5. 5 Gemini 3.7 Flash ↓2 45.3%
  6. 6 Qwen3-Omni-30B-A3B 32.7%
  7. 7 Qwen2.5-Omni-7B 24.9%

3 models have held number one on the official leaderboard since its release in September 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind Qwen
Own leaderboard Not tested since Sep 2026

Source: Official OmniEchoBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen3.8-Omni-Flash Gemini 3.8 Flash Qwen3.5-Omni-Plus Qwen3-Omni-30B-A3B Qwen2.5-Omni-7B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen3.8-Omni-Flash Gemini 3.8 Flash

Source: Official OmniEchoBench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Robotics

All 172 in Robotics
  • ERQA In use 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. Gemini Robotics ER 2 78.5%
  • SimplerEnv In use Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. Praxis-1 80%
  • RoboLab In use Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. Praxis-1 46.2%
  • RoboDojo In use Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. GPT-6 Astra 27.6%
  • Progress classification (Google DeepMind) In use Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Gemini Robotics ER 2 57.4%
  • Human proximity (Google DeepMind robotics) In use Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Gemini Robotics ER 2 93%

Data audit

9 of 10 checks pass, 1 warning. Look first at the Task access check. We don’t know whether the tasks are public. Every benchmark

Catalog record 1 warning 1 pass
  • Warning Task access We don’t know whether the tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (10 entries).
  • Pass Models scored 7 models we track, plus 3 other entries on its leaderboard.
  • Pass Newest score The newest is from September 2026, for Qwen3.8-Omni-Flash.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Gemini 3.8 Flash (#24).
  • Pass Headroom The best is 43.9%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on pku-value-lab.github.io: 7 rows for models we track and 3 other entries at the last read.
  • Pass Last read November 18th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check