Benchmark Almanac

LLM Colosseum

gpt-4o (snapshot unspecified) 1,913 tops its leaderboard GPT-4o mini 1,835 leads the models we track
Not in model cards

Two models each control a fighter in Street Fighter III and choose moves from text descriptions of the game. Wins give Elo ratings.

Made at a Mistral hackathon. The repo has about 1,484 stars and was last updated on March 21st, 2025.

Released
March 2024
Kind
Video games
Level
Everyday
Scored
Ranked by matches
Answers
Open-ended
Tasks
All public
Strongest model tested
GPT-4o mini #177 on the capability index
Last model added
July 2024
Scores refreshed
November 12th, 2026
7
Entries on its own leaderboard, 3 models we track

gpt-4o (snapshot unspecified) tops the leaderboard with 1,913

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is GPT-4o mini, at 1,835.

The newest model with a score here came out in July 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

Bars start at 1,200.

  1. 1,913 gpt-4o (snapshot unspecified)
  2. 1,835 GPT-4o mini
  3. 1,655 pixtral-large-latest (snapshot unspecified)
  4. 1,591 pixtral-12b-2409
  5. 1,441 meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
  6. 1,365 Claude 3 Haiku
  7. 1,315 Claude 3 Sonnet

Bars start at 1,200.

  1. 1 gpt-4o (snapshot unspecified) text 1,913
  2. 2 GPT-4o mini vision 1,835
  3. 3 pixtral-large-latest (snapshot unspecified) vision 1,655
  4. 4 pixtral-12b-2409 vision 1,591
  5. 5 meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo text 1,441
  6. 6 Claude 3 Haiku vision 1,365
  7. 7 Claude 3 Sonnet vision 1,315

Source: Official LLM Colosseum leaderboard

#EntryScoreSettingDateSource
1 gpt-4o (snapshot unspecified) 1,913 text, LLM Colosseum Official leaderboard
2 GPT-4o mini 1,835 vision, LLM Colosseum Jul 2024 Official leaderboard
3 pixtral-large-latest (snapshot unspecified) 1,655 vision, LLM Colosseum Official leaderboard
4 pixtral-12b-2409 1,591 vision, LLM Colosseum Official leaderboard
5 meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo 1,441 text, LLM Colosseum Official leaderboard
6 Claude 3 Haiku 1,365 vision, LLM Colosseum Mar 2024 Official leaderboard
7 Claude 3 Sonnet 1,315 vision, LLM Colosseum Feb 2024 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
GPT-4o mini LLM Colosseum
text vision
1,835
Claude 3 Haiku LLM Colosseum
text
1,365
Claude 3 Sonnet LLM Colosseum
text vision
1,315

Source: Official LLM Colosseum leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-4o mini vision LLM Colosseum 1,835
GPT-4o mini text LLM Colosseum 1,671
Claude 3 Haiku vision LLM Colosseum 1,365
Claude 3 Haiku text LLM Colosseum 1,334
Claude 3 Sonnet vision LLM Colosseum 1,315
Claude 3 Sonnet text LLM Colosseum 1,029

3 models have held number one on the official leaderboard since its release in March 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI
Own leaderboard Not tested since Jul 2024
GPT-4o mini 1,835

Source: Official LLM Colosseum leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
1,000 1,200 1,400 1,600 1,800 2,000 2024 1,000 1,200 1,400 1,600 1,800 2,000 2024

Source: Official LLM Colosseum leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Games and long tasks

All 82 in Games and long tasks
  • ARC-AGI-3 In use Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. GPT-6 Astra 99.95%
  • Orak Fading LLM agents play 12 popular video games across most genres through an MCP interface. Each game scores in-game progress or outcomes, and the board ranks models by their average rank across the games. Gemini 2.5 Pro 3.90
  • MineDojo Fading Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. Seed2.0 Pro 49%
  • TextQuests Only in system cards Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. Llama 4 Scout 7.00
  • GameWorld Only in system cards Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. GPT-6.1 Terra 76.8%
  • Swarm Bench Only in system cards Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. Kimi K3 76.3%