Benchmark Almanac

RefAdv-S

Qwen3.5-397B-A17B-FP8 68% tops its leaderboard Qwen3.5-27B 67.3% leads the models we track
Qwen 3.8 Max 80.2% leads since August 2026
In use

Models locate the object a referring expression describes among hard distractors in real images. Scored by accuracy at 0.5 box overlap on 1,142 cases.

It is the public subset of the larger Ref-Adv benchmark.

Released
February 2026
Kind
Detection and captioning
Level
Everyday
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Strongest model tested
Qwen 3.8 Max #29 on the capability index
Last in a launch post
August 2026
Last model added
August 2026
Scores refreshed
November 12th, 2026

In the authors’ words

Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.

Abstract of Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (arXiv:2602.23898, CC0)
20
Entries on its own leaderboard, 19 models we track
1
Launch post lists it, from 1 lab

Qwen3.5-397B-A17B-FP8 tops the leaderboard with 68%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen3.5-27B, at 67.3%.

Qwen 3.8 Max leads with 80.2%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in February 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

20 entries. Scroll sideways for the rest.

  1. 68.0 Qwen3.5-397B-A17B-FP8
  2. 67.3 Qwen3.5-27B
  3. 67.2 Qwen3.5-122B-A10B
  4. 67.1 Qwen3-VL-235B-A22B
  5. 66.7 Qwen3.5-35B-A3B
  6. 65.6 Qwen3-VL-32B-Thinking
  7. 64.1 Qwen3-VL-30B-A3B-Thinking
  8. 59.5 Qwen3-VL-8B-Thinking
  9. 59.3 Qwen3-VL-235B-A22B-Instruct
  10. 59.0 Qwen3-VL-32B-Instruct
  11. 57.6 Qwen3-VL-4B-Thinking
  12. 54.0 Qwen2.5-VL-72B
  13. 52.3 Qwen3-VL-8B-Instruct
  14. 52.1 Qwen3-VL-30B-A3B-Instruct
  15. 50.6 Qwen2.5-VL-32B
  16. 44.4 Qwen3-VL-2B-Thinking
  17. 42.5 Qwen3-VL-4B-Instruct
  18. 39.3 Qwen2.5-VL-7B
  19. 25.3 Qwen2.5-VL-3B
  20. 25.2 Qwen3-VL-2B-Instruct
  1. 1 Qwen3.5-397B-A17B-FP8 CoT 68%
  2. 2 Qwen3.5-27B CoT 67.3%
  3. 3 Qwen3.5-122B-A10B CoT 67.2%
  4. 4 Qwen3-VL-235B-A22B CoT 67.1%
  5. 5 Qwen3.5-35B-A3B CoT 66.7%
  6. 6 Qwen3-VL-32B-Thinking CoT 65.6%
  7. 7 Qwen3-VL-30B-A3B-Thinking CoT 64.1%
  8. 8 Qwen3-VL-8B-Thinking CoT 59.5%
  9. 9 Qwen3-VL-235B-A22B-Instruct CoT 59.3%
  10. 10 Qwen3-VL-32B-Instruct CoT 59%
  11. 11 Qwen3-VL-4B-Thinking CoT 57.6%
  12. 12 Qwen2.5-VL-72B without CoT 54%
  13. 13 Qwen3-VL-8B-Instruct CoT 52.3%
  14. 14 Qwen3-VL-30B-A3B-Instruct CoT 52.1%
  15. 15 Qwen2.5-VL-32B CoT 50.6%
  16. 16 Qwen3-VL-2B-Thinking CoT 44.4%
  17. 17 Qwen3-VL-4B-Instruct CoT 42.5%
  18. 18 Qwen2.5-VL-7B without CoT 39.3%
  19. 19 Qwen2.5-VL-3B CoT 25.3%
  20. 20 Qwen3-VL-2B-Instruct CoT 25.2%

Source: Official RefAdv-S leaderboard

20 models. Scroll sideways for the rest.

  1. 80.2 Qwen 3.8 Max
  2. 67.3 Qwen3.5-27B
  3. 67.2 Qwen3.5-122B-A10B
  4. 67.1 Qwen3-VL-235B-A22B
  5. 66.7 Qwen3.5-35B-A3B
  6. 65.6 Qwen3-VL-32B-Thinking
  7. 64.1 Qwen3-VL-30B-A3B-Thinking
  8. 59.5 Qwen3-VL-8B-Thinking
  9. 59.3 Qwen3-VL-235B-A22B-Instruct
  10. 59.0 Qwen3-VL-32B-Instruct
  11. 57.6 Qwen3-VL-4B-Thinking
  12. 54.0 Qwen2.5-VL-72B
  13. 52.3 Qwen3-VL-8B-Instruct
  14. 52.1 Qwen3-VL-30B-A3B-Instruct
  15. 50.6 Qwen2.5-VL-32B
  16. 44.4 Qwen3-VL-2B-Thinking
  17. 42.5 Qwen3-VL-4B-Instruct
  18. 39.3 Qwen2.5-VL-7B
  19. 25.3 Qwen2.5-VL-3B
  20. 25.2 Qwen3-VL-2B-Instruct
  1. 1 Qwen 3.8 Max 80.2%
  2. 2 Qwen3.5-27B CoT 67.3%
  3. 3 Qwen3.5-122B-A10B CoT 67.2%
  4. 4 Qwen3-VL-235B-A22B CoT 67.1%
  5. 5 Qwen3.5-35B-A3B CoT 66.7%
  6. 6 Qwen3-VL-32B-Thinking CoT 65.6%
  7. 7 Qwen3-VL-30B-A3B-Thinking CoT 64.1%
  8. 8 Qwen3-VL-8B-Thinking CoT 59.5%
  9. 9 Qwen3-VL-235B-A22B-Instruct CoT 59.3%
  10. 10 Qwen3-VL-32B-Instruct CoT 59%
  11. 11 Qwen3-VL-4B-Thinking CoT 57.6%
  12. 12 Qwen2.5-VL-72B without CoT 54%
  13. 13 Qwen3-VL-8B-Instruct CoT 52.3%
  14. 14 Qwen3-VL-30B-A3B-Instruct CoT 52.1%
  15. 15 Qwen2.5-VL-32B CoT 50.6%
  16. 16 Qwen3-VL-2B-Thinking CoT 44.4%
  17. 17 Qwen3-VL-4B-Instruct CoT 42.5%
  18. 18 Qwen2.5-VL-7B without CoT 39.3%
  19. 19 Qwen2.5-VL-3B CoT 25.3%
  20. 20 Qwen3-VL-2B-Instruct CoT 25.2%

Sources: Labs’ model cards and official RefAdv-S leaderboard

#EntryScoreSettingDateSource
1 Qwen3.5-397B-A17B-FP8 68% CoT Official leaderboard
2 Qwen3.5-27B 67.3% CoT Feb 2026 Official leaderboard
3 Qwen3.5-122B-A10B 67.2% CoT Feb 2026 Official leaderboard
4 Qwen3-VL-235B-A22B 67.1% CoT Sep 2025 Official leaderboard
5 Qwen3.5-35B-A3B 66.7% CoT Feb 2026 Official leaderboard
6 Qwen3-VL-32B-Thinking 65.6% CoT Oct 2025 Official leaderboard
7 Qwen3-VL-30B-A3B-Thinking 64.1% CoT Oct 2025 Official leaderboard
8 Qwen3-VL-8B-Thinking 59.5% CoT Oct 2025 Official leaderboard
9 Qwen3-VL-235B-A22B-Instruct 59.3% CoT Sep 2025 Official leaderboard
10 Qwen3-VL-32B-Instruct 59% CoT Oct 2025 Official leaderboard
11 Qwen3-VL-4B-Thinking 57.6% CoT Oct 2025 Official leaderboard
12 Qwen2.5-VL-72B 54% without CoT Jan 2025 Official leaderboard
13 Qwen3-VL-8B-Instruct 52.3% CoT Oct 2025 Official leaderboard
14 Qwen3-VL-30B-A3B-Instruct 52.1% CoT Oct 2025 Official leaderboard
15 Qwen2.5-VL-32B 50.6% CoT Mar 2025 Official leaderboard
16 Qwen3-VL-2B-Thinking 44.4% CoT Oct 2025 Official leaderboard
17 Qwen3-VL-4B-Instruct 42.5% CoT Oct 2025 Official leaderboard
18 Qwen2.5-VL-7B 39.3% without CoT Jan 2025 Official leaderboard
19 Qwen2.5-VL-3B 25.3% CoT Jan 2025 Official leaderboard
20 Qwen3-VL-2B-Instruct 25.2% CoT Oct 2025 Official leaderboard
#ModelScoreSettingDateSource
1 Qwen 3.8 Max 80.2% Aug 2026 Lab model card
2 Qwen3.5-27B 67.3% CoT Feb 2026 Official leaderboard
3 Qwen3.5-122B-A10B 67.2% CoT Feb 2026 Official leaderboard
4 Qwen3-VL-235B-A22B 67.1% CoT Sep 2025 Official leaderboard
5 Qwen3.5-35B-A3B 66.7% CoT Feb 2026 Official leaderboard
6 Qwen3-VL-32B-Thinking 65.6% CoT Oct 2025 Official leaderboard
7 Qwen3-VL-30B-A3B-Thinking 64.1% CoT Oct 2025 Official leaderboard
8 Qwen3-VL-8B-Thinking 59.5% CoT Oct 2025 Official leaderboard
9 Qwen3-VL-235B-A22B-Instruct 59.3% CoT Sep 2025 Official leaderboard
10 Qwen3-VL-32B-Instruct 59% CoT Oct 2025 Official leaderboard
11 Qwen3-VL-4B-Thinking 57.6% CoT Oct 2025 Official leaderboard
12 Qwen2.5-VL-72B 54% without CoT Jan 2025 Official leaderboard
13 Qwen3-VL-8B-Instruct 52.3% CoT Oct 2025 Official leaderboard
14 Qwen3-VL-30B-A3B-Instruct 52.1% CoT Oct 2025 Official leaderboard
15 Qwen2.5-VL-32B 50.6% CoT Mar 2025 Official leaderboard
16 Qwen3-VL-2B-Thinking 44.4% CoT Oct 2025 Official leaderboard
17 Qwen3-VL-4B-Instruct 42.5% CoT Oct 2025 Official leaderboard
18 Qwen2.5-VL-7B 39.3% without CoT Jan 2025 Official leaderboard
19 Qwen2.5-VL-3B 25.3% CoT Jan 2025 Official leaderboard
20 Qwen3-VL-2B-Instruct 25.2% CoT Oct 2025 Official leaderboard

Labs report scores for 1 model in their cards. Switch the scores above to rank them together.

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Qwen3-VL-235B-A22B-Instruct
without CoT
59.3%
Qwen3-VL-32B-Instruct
without CoT CoT
59%
Qwen2.5-VL-72B
CoT
54%
Qwen3-VL-8B-Instruct
without CoT CoT
52.3%
Qwen3-VL-30B-A3B-Instruct
without CoT CoT
52.1%
Qwen2.5-VL-32B
without CoT
50.6%
Qwen3-VL-4B-Instruct
without CoT
42.5%
Qwen2.5-VL-7B
CoT
39.3%
Qwen2.5-VL-3B
without CoT
25.3%
Qwen3-VL-2B-Instruct
without CoT
25.2%

Source: Official RefAdv-S leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Qwen3.5-27B CoT 67.3%
Qwen3.5-122B-A10B CoT 67.2%
Qwen3-VL-235B-A22B CoT 67.1%
Qwen3.5-35B-A3B CoT 66.7%
Qwen3-VL-32B-Thinking CoT 65.6%
Qwen3-VL-30B-A3B-Thinking CoT 64.1%
Qwen3-VL-8B-Thinking CoT 59.5%
Qwen3-VL-235B-A22B-Instruct CoT 59.3%
Qwen3-VL-32B-Instruct CoT 59%
Qwen3-VL-4B-Thinking CoT 57.6%
Qwen3-VL-235B-A22B-Instruct without CoT 57.3%
Qwen2.5-VL-72B without CoT 54%
Qwen3-VL-32B-Instruct without CoT 53.4%
Qwen2.5-VL-72B CoT 52.4%
Qwen3-VL-8B-Instruct CoT 52.3%
Qwen3-VL-30B-A3B-Instruct CoT 52.1%
Qwen2.5-VL-32B CoT 50.6%
Qwen2.5-VL-32B without CoT 48%
Qwen3-VL-8B-Instruct without CoT 47.2%
Qwen3-VL-2B-Thinking CoT 44.4%
Qwen3-VL-30B-A3B-Instruct without CoT 44%
Qwen3-VL-4B-Instruct CoT 42.5%
Qwen3-VL-4B-Instruct without CoT 41.9%
Qwen2.5-VL-7B without CoT 39.3%
Qwen2.5-VL-7B CoT 39%
Qwen2.5-VL-3B CoT 25.3%
Qwen3-VL-2B-Instruct CoT 25.2%
Qwen2.5-VL-3B without CoT 23.8%
Qwen3-VL-2B-Instruct without CoT 23.5%

2 models have held number one on the official leaderboard since its release in February 2026 3 models have held number one with claims included since its release in February 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen
Own leaderboard Not tested since Feb 2026
Qwen3.5-27B 67.3%

Source: Official RefAdv-S leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen 3.8 Max Qwen3.5-27B Qwen3-VL-235B-A22B Qwen2.5-VL-72B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen 3.8 Max Qwen3.5-27B

Sources: Official RefAdv-S leaderboard and labs’ model cards

One lab put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DateLaunch postScoreAs the post writes it
Aug 3, 2026 Qwen 3.8 Max 80.2 RefAdv-S

More in Vision

All 281 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%
  • ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Command A Vision 90.9%