Kaleidoscope

Claude 3.5 Sonnet (snapshot unspecified) 62.9% tops its leaderboard

Qwen2.5-VL-72B 52.9% leads the models we track

Qwen2.5-VL-72B 52.9% leads since April 2025

Only in system cards

Official leaderboard Include claimed scores

Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy.

Questions stay in their original language rather than being translated from English.

Made by: Cohere Labs and community contributors
Released: April 2025
Kind: Knowledge in many languages
Level: University
Scored: Checked automatically
Answers: Multiple choice
Tasks: All public

Last in a system card: May 2025
Last model added: May 2025
Scores refreshed: November 12th, 2026

In the authors’ words

The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.

Abstract of Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation (arXiv:2504.07072, CC0)

Entries on its own leaderboard, 7 models we track

System card reports it, from 1 lab

Claude 3.5 Sonnet (snapshot unspecified) tops the leaderboard with 62.9%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen2.5-VL-72B, at 52.9%.

Qwen2.5-VL-72B leads with 52.9%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

Bars Table

The newest model with a score here came out in March 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

1 Claude 3.5 Sonnet (snapshot unspecified) 62.9%
2 Gemini 1.5 Pro (snapshot unspecified) 62.1%
3 GPT-4o (snapshot unspecified) 58.3%
4 Qwen2.5-VL-72B 52.9%
5 Qwen2.5-VL-32B 48.2%
6 Qwen2.5-VL-7B 39.6%
7 Aya Vision 32B 39.3%
8 Qwen2.5-VL-3B 35.6%
9 Aya Vision 8B 35.1%
10 Molmo-7B-D 32.9%
11 Pangea-7B 31.3%

Source: Official Kaleidoscope leaderboard

1 Qwen2.5-VL-72B 52.9%
2 Qwen2.5-VL-32B 48.2%
3 Aya Vision 32B 41.7%
4 Qwen2.5-VL-7B 39.6%
5 Qwen2.5-VL-3B 35.6%
6 Aya Vision 8B 35.1%
7 Molmo-7B-D 32.9%

Sources: Official Kaleidoscope leaderboard and labs’ model cards

#	Entry	Score	Date	Source
1	Claude 3.5 Sonnet (snapshot unspecified)	62.9%		Official leaderboard
2	Gemini 1.5 Pro (snapshot unspecified)	62.1%		Official leaderboard
3	GPT-4o (snapshot unspecified)	58.3%		Official leaderboard
4	Qwen2.5-VL-72B	52.9%	Jan 2025	Official leaderboard
5	Qwen2.5-VL-32B	48.2%	Mar 2025	Official leaderboard
6	Qwen2.5-VL-7B	39.6%	Jan 2025	Official leaderboard
7	Aya Vision 32B	39.3%	Mar 2025	Official leaderboard
8	Qwen2.5-VL-3B	35.6%	Jan 2025	Official leaderboard
9	Aya Vision 8B	35.1%	Mar 2025	Official leaderboard
10	Molmo-7B-D	32.9%	Sep 2024	Official leaderboard
11	Pangea-7B	31.3%		Official leaderboard

#	Model	Score	Date	Source
1	Qwen2.5-VL-72B	52.9%	Jan 2025	Official leaderboard
2	Qwen2.5-VL-32B	48.2%	Mar 2025	Official leaderboard
3	Aya Vision 32B	41.7%	May 2025	Lab model card
4	Qwen2.5-VL-7B	39.6%	Jan 2025	Official leaderboard
5	Qwen2.5-VL-3B	35.6%	Jan 2025	Official leaderboard
6	Aya Vision 8B	35.1%	Mar 2025	Official leaderboard
7	Molmo-7B-D	32.9%	Sep 2024	Official leaderboard

Labs report scores for 1 model in their cards. Switch the scores above to rank them together.

Number one over time Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen

Own leaderboard Not tested since Apr 2025

Qwen2.5-VL-72B 52.9%

Source: Official Kaleidoscope leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far

Sources: Official Kaleidoscope leaderboard and labs’ model cards

Labs report it only in system cards

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
May 13, 2025	Aya Vision 32B technical report	41.7	Kaleidoscope, Aya Vision languages (Table 4)	Capabilities

One lab claim next to the leaderboard

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model	Lab says	Board says	Gap
Aya Vision 32B	41.7% Kaleidoscope, Aya Vision languages (Table 4)	39.3% Leaderboard	+2.5 pts