Benchmark Almanac

Games

Models play named games, alone or against each other. We count 205 benchmarks here, 87 of them arenas. A benchmark belongs here when environment includes a named game.

How fast it grows

74 were released since July, more than the 50 in the first half of 2026.

  1. Earlier 7
  2. 2024 H1 3
  3. 2024 H2 5
  4. 2025 H1 16
  5. 2025 H2 26
  6. 2026 H1 50
  7. 2026 H2 74

How much is still open

No model has passed 85% on 27 of the 39 benchmarks here that score in percent, 69% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
Backrooms: 3D Clue-Trail Navigation Agents follow visual clues through a first-person maze. Only an escape without a wrong doorway counts as a solve. Everyday Nov 2026 Not in cards
Beyond Survival Models play social deduction games with human-aligned strategies and interaction patterns. Everyday Nov 2026 58.7% Not in cards
Clocktower Bench Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) against each other, with a scripted Storyteller. One model takes every good seat and another every evil seat, so each game tests one model's detection of the Demon against another's deception. A Bradley–Terry fit turns 386 games into Elo ratings. Everyday Nov 2026 1,859 Not in cards
Kaeru Flip (paprika3d) Agents play an 86-note rhythm game from browser pixels. They must finish the song with at most one miss and leave autoplay off. Everyday Nov 2026 Not in cards
KradleBench Models play live multi-agent Minecraft games and act by writing code. The overall score weights collaboration, economy and conflict at 30% each, and logic and creativity at 5% each. Everyday Nov 2026 78.3 Not in cards
MindGames Agents compete in Colonel Blotto, Iterated Prisoner’s Dilemma, Codenames, and Secret Mafia. Everyday Nov 2026 Not in cards
MTGBench Models play multi-turn games that test reasoning, planning, and interaction across varied game types. Everyday Nov 2026 Not in cards
Portal2Bench Agents solve Portal 2 chambers using screenshots, position readings, and game controls. Each chamber counts if the agent reaches the next map within $30 and 300 plans. Everyday Nov 2026 26.0 Not in cards
RedstoneBench Agents design redstone circuits in Minecraft. A game server tests whether each circuit works and measures the space it occupies. Expert Nov 2026 88.7 Not in cards
RollerCoasterTycoonBench Models manage a theme park by choosing weekly actions in RollerCoaster Tycoon 2. Everyday Nov 2026 Not in cards
SpaceChem Bench Agents program reactors to assemble molecules in SpaceChem. The main ranking measures how close their solutions come to the community’s records. Competition Nov 2026 74.3 Not in cards
TIS-100 Bench Agents write and optimize assembly programs for 55 puzzles. The ranking counts puzzles where a program improves on the community’s recorded tradeoffs in cycles, nodes, and instructions. Competition Nov 2026 11.0 Not in cards
Cat-Goric Game Benchmark Models navigate a 2D platform game and submit results to a public decision-making leaderboard. Everyday Oct 2026 20.0 Not in cards
CivBench Models play full games of Civilization V with the Vox Populi mod against each other on matched maps and seeds. Pairwise skill ratings rank long-term strategy. Expert Oct 2026 1,634 Not in cards
Hanabi Agent Benchmark Models play cooperative Hanabi in four-player teams. The benchmark scores shared board progress while hints and lives constrain coordination. Everyday Oct 2026 20.5 Not in cards
Minecraft Bed Wars (YC Labs) Teams of agents play Minecraft Bed Wars against one another. The researchers compare the number of matches each team wins. Everyday Oct 2026 Not in cards
OpenTTD-Bench Agents build transport networks over ten simulated years. The grader rewards cumulative operating profit and the number of rivals each company beats. Expert Oct 2026 Not in cards
PokéBench: Raw Vision + Buttons Models play Pokémon Red from screenshots using eight Game Boy buttons. Game memory verifies ten milestones, with fewer turns breaking ties. Everyday Oct 2026 10.0 Not in cards
Railhead Agents run a transport company in OpenTTD. The evaluator replays the action log and lets the company run without intervention before measuring its performance. Everyday Oct 2026 Not in cards
Robots Evolved GUI lessons Agents use screenshots, clicks, and keys to move sliding robots onto a goal. Ten puzzles enforce a move budget and offer no undo. Everyday Oct 2026 Not in cards
Rotom Champ Bench Models draft six level 50 Pokémon, select three for each match, and choose their moves. The organizer rates them on wins under Champions singles rules. Competition Oct 2026 2,304 Not in cards
Rotom Gen9 Bench Models draft six level 100 Pokémon and play the whole team in Generation 9 OverUsed singles. Match results determine their ratings. Competition Oct 2026 2,318 Not in cards
SettlerBench Season One Four models trade resources, build settlements, and negotiate in public or private messages. Final victory points determine pairwise results for the Bradley-Terry rating. Everyday Oct 2026 1,270 Not in cards
SpireBench Agents play Slay the Spire and are scored on long-horizon deck-building decisions. Everyday Oct 2026 Not in cards
SpireClimb Agents play Slay the Spire 1 and 2 at max difficulty through a computer-use harness on fixed seeds. Score is the mean win rate. Expert Oct 2026 41.3% Not in cards
SpireClimb StS1 A20 Heart Agents play Slay the Spire 1 through its graphical interface at Ascension 20 and must defeat the Heart. Scored by win rate over fixed seeds. Expert Oct 2026 33.3% Not in cards
SpireClimb StS2 A10 Agents play Slay the Spire 2 at Ascension 10 through screenshots and mouse and keyboard, 15 runs per character. Scoring is the win rate. Everyday Oct 2026 49.3% Not in cards
StarSkirmish Bench Each model gets one hour in its lab's own CLI to write a StarCraft: Brood War bot in C++. The score is its win rate against human-written bots. Expert Oct 2026 57.3% Not in cards
TavernBench Agents build a Hearthstone Battlegrounds player through research, simulator updates, and live games. Expert Oct 2026 95.6 Not in cards
The Midnight Index Models lead a nation through fixed scenarios in a strategy game. The simulation records outcomes, escalation, and whether public commitments match later decisions. Everyday Oct 2026 Not in cards
Turn based Benchmarks Models play turn based games including tic tac toe and rock paper scissors. The tasks evaluate game decisions and adaptation to opponents. Everyday Oct 2026 0.69 Not in cards
Bench on the Clocktower Ten models play the social deduction game Blood on the Clocktower together, one seat each, over 300 ten-player games of the Trouble Brewing script. Each model plays 210 games on the good team and 90 on the evil team. The score is the share of games its team won. Everyday Sep 2026 58% Not in cards
Brood War Bench Coding agents play StarCraft: Brood War against each other in real time. Systems are ranked by win rate over matches, with cost per game. Expert Sep 2026 100% Not in cards
CivBench (Civilization VI) Agents play Civilization VI and are evaluated on strategic progress in a long-running world. Expert Sep 2026 73.0 Not in cards
CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. Everyday Sep 2026 19.2% Not in cards
Deep20Bench 1.1 Models identify hidden people, characters, or objects by asking questions. The score averages the number of counted questions, with a fixed penalty for failed rounds, over three rounds for each of ten subjects. Everyday Sep 2026 13.7 Not in cards
Factorio Play Agents build factories one action at a time in a Factorio simulator. The evaluator measures factory output on held-out layouts. University Sep 2026 Not in cards
fAIntasy Football Models draft NFL players, set lineups, bid for free agents, and trade under the same rules. The standings show wins, fantasy points, and lineup efficiency. Everyday Sep 2026 4.00 Not in cards
Game Arena Go Language models play Go against each other through the Game Arena text harness. Ratings come from head-to-head games. Expert Sep 2026 1,877 Not in cards
GlyphBench Agents choose actions from text-rendered game boards. The standard suite contains 303 tasks across six game collections and scores agents by cumulative return. University Sep 2026 0.42 Not in cards
Large Language Model Football League Models draft, trade, and set weekly NFL lineups. Each team earns one result against its opponent and another against the league median. Everyday Sep 2026 Not in cards
LLM Fantasy Football Arena Guillotine Models draft NFL players and bid for free agents. Each week the lowest-scoring team leaves, and the remaining teams bid on its players. Everyday Sep 2026 10.0 Not in cards
Ludobench Backgammon Models play backgammon against opponents using dice rolls and legal moves. Ludobench publishes Glicko-2 ratings and match replays. Everyday Sep 2026 1,723 Not in cards
Ludobench Hokm Models play Hokm in partnerships. One player names trump, and partners try to take seven tricks before the other team. Everyday Sep 2026 100% Not in cards
MOLTRON LLM Showdown v1.0 Models write JavaScript bots for a light-cycle game. Each working bot plays 600 matches against five reference opponents across six maps and both starting positions. University Sep 2026 73% Not in cards
MOLTRON LLM Showdown v3.0 Models write JavaScript bots for a light-cycle game. Each working bot plays 600 matches against five reference opponents across six maps and both starting positions. University Sep 2026 69% Not in cards
Multi-Agent Arena Codenames Two teams of anonymized models play the word game Codenames. The board rates the clue giver and the guessers on separate rating scales. Everyday Sep 2026 1,535 Not in cards
Multi-Agent Arena Diplomacy Seven players, people or anonymized models, negotiate and fight over World War I Europe. Models are ranked by their mean share of the sum-of-squares score, where 14.3 is an even split. Everyday Sep 2026 37.5% Not in cards
PokéBench: Tool Assisted Benchmark Models play Pokémon Red with a screenshot, map, game text, and action shortcuts. They have 1,000 turns to reach ten milestones ending with Brock. Everyday Sep 2026 10.0 Not in cards
RIICHI ORBIT mahjong bot comparison (open practice) Models get 30 minutes to improve the same mahjong bot, with unlimited practice. The maker freezes each bot before evaluating it on separate tile sequences. University Sep 2026 2.41 Not in cards
RIICHI ORBIT mahjong bot comparison (two-stage practice) Models improve a beginner mahjong bot during two timed coding periods. The maker runs the finished bots in four-player matches and reports mean placement by table. University Sep 2026 2.28 Not in cards
WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. Expert Sep 2026 46.5% Not in cards
Agent-Only Fantasy Football League Models draft NFL players, set lineups, and negotiate trades using the same prompt and tools. The organizer compares points, lineup efficiency, and cost per point. Everyday Aug 2026 1371.4 Not in cards
Balatro Bench (Michael Andrzejewski) Agents play one fixed Balatro seed with the Red Deck on White Stake. The maker ranks runs by ante reached, breaking ties by best hand. Everyday Aug 2026 18.0 Not in cards
BoardBench 0.1 Models play Yatzi, Scotland Yard, and Splendor. Game engines identify the best action at each decision and the scorer counts exact matches. Everyday Aug 2026 91.4% Not in cards
Game Arena Nine Men's Morris Language models play Nine Men's Morris. Forming a row of three, called a mill, lets a player remove one of the opponent's pieces. Everyday Aug 2026 1,190 Not in cards
Game Arena Reversi Language models play Reversi, also called Othello, flipping the opponent's disks that they flank in a line. Everyday Aug 2026 1,306 Not in cards
Game Arena Word Art Language models take turns drawing a word as ASCII art and guessing what the other model drew. Everyday Aug 2026 2,528 Not in cards
LLM Fantasy Football Arena Classic Models manage two NFL fantasy teams each. One league allows trades and messages. The other disables both. Each league scores its own draft and season. Everyday Aug 2026 0.90 Not in cards
MineAmongUs Vision-language agents play an embodied Among Us game with verbal and nonverbal deception. Everyday Aug 2026 81.3% Not in cards
WRENCH (Factorio) Agents build Factorio factories and recover from unannounced disruptions. The evaluator measures how much production the factory retains after a disruption. University Aug 2026 1.49 Not in cards
AI Battle Arena Models play poker, Gomoku, and Connect Four through the same interface. The overall leaderboard averages each model’s finishing position across five head-to-head contests. Everyday Jul 2026 88.0 Not in cards
Artificial Turf War Models bid for draft position, pick NFL players, and submit weekly lineups. The league ranks teams by wins and breaks ties with fantasy points. Everyday Jul 2026 6.00 Not in cards
EBR-bench Agent plays the board game Earthborne Rangers 10-30 times in a row. Scored on objectives completed (of 21) to test learning from experience. Everyday Jul 2026 92.4% Not in cards
Game Arena Coin Game Two language models move on a grid and pick up coins. Each should collect the coins that score for it and avoid the rest. Everyday Jul 2026 514 Not in cards
Game Arena Lines of Action Language models play Lines of Action, where a piece moves as far as the number of pieces on its line. The goal is to join all of one's pieces into one group. Everyday Jul 2026 1,274 Not in cards
Game Arena Ultimate Tic-Tac-Toe Language models play nine linked tic-tac-toe boards. Each move decides which small board the opponent must play next. Everyday Jul 2026 720 Not in cards
Gauntlet Agents develop game controllers through practice. The authors freeze each controller and test it on unseen games without further model calls. University Jul 2026 Not in cards
Multi-Agent Arena Risk Four commanders, people or anonymized models, trade sealed messages and then attack for territory on a 42-territory map. Models are ranked by their mean share of territory, where 25 is an even split. Everyday Jul 2026 38.8% Not in cards
Multi-Agent Arena Social Poker No-limit hold’em tournaments with table talk, where people and anonymized models share a table. Models are ranked by a rating fit on who finished ahead at each table. Everyday Jul 2026 1,553 Not in cards
Olam Labs Multi-Agent Arena People and anonymized models share a table in social strategy games: Diplomacy, Social Poker, Codenames and Risk. One rating combines every game, fit on who finished ahead of whom at each table. Everyday Jul 2026 1,544 Not in cards
ParliamentBench Models play Secret Hitler while deceiving, persuading, and reasoning under hidden roles. Everyday Jul 2026 81% Not in cards
Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. Expert Jul 2026 91.3% Not in cards
ZendoWorld Models infer hidden rules in the board game Zendo from labeled examples. Everyday Jul 2026 Not in cards
AgenticSTS Agents play the Silent in Slay the Spire 2 at its lowest difficulty. The authors compare wins with different memory settings and two other agents. Everyday Jun 2026 60% Not in cards
BabaIsBench Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost. Expert Jun 2026 100% Not in cards
BuildArena 2.0 Construction Challenge Agents build and test machines in Besiege for seasonal engineering challenges. The organizers score each entry using simulation results and the rules for its track. Competition Jun 2026 Not in cards
Eleusis Agents play cards to discover a hidden rule. The scorer tests whether a proposed rule makes the same decisions as the hidden rule. Competition Jun 2026 82.2% Not in cards
Evalatro v2 Models play Balatro from the game state and try to clear Ante 12. Illegal moves reduce the score for progress through the game. Everyday Jun 2026 46.1 Not in cards
Game Arena Checkers Language models play American checkers against each other, with forced jumps and crowned kings. Ratings come from head-to-head games. Everyday Jun 2026 1,382 Not in cards
Game Arena Clobber Language models play Clobber, where every move captures an adjacent enemy stone. The player who makes the last legal move wins. Everyday Jun 2026 819 Not in cards
Game Arena Dark Hex Language models play Hex without seeing the opponent's stones, so they must probe cells to learn the hidden board while building a path. Everyday Jun 2026 581 Not in cards
Game Arena Dots and Boxes Language models take turns drawing lines between dots on a grid. Closing the fourth side of a box claims it. Everyday Jun 2026 1,393 Not in cards
Opus Magnum Bench Agents build machines on a hexagonal grid to make a target molecule. Each puzzle score compares the machine’s combined cost, cycles, and area with the human record. Competition Jun 2026 25.5 Not in cards
Poker Arena Models play no-limit Texas Hold’em across repeated sessions. The benchmark scores tournament chips and nine strategic dimensions, including memory and bluffing. Everyday Jun 2026 15730.0 Not in cards
RNG-Bench Agents play Matching Pairs and navigate 3D mazes using remembered observations. Scores measure matched pairs and combine maze success, efficiency, and exploration. Everyday Jun 2026 49.9% Not in cards
RTSGameBench Agents play the open real-time strategy game Beyond All Reason. Expert Jun 2026 Not in cards
Slay the Spire Bench (alavico) Models choose an action from the text state of Slay the Spire. The campaign report measures floors reached and wins at each difficulty. Everyday Jun 2026 51.0 Not in cards
Agent Island Models play multi-agent games that require cooperation, conflict, and persuasion. Everyday May 2026 0.88 Not in cards
AI-Spire (AGI-Eval) Models replay one Slay the Spire 2 seed, build decks, and choose combat moves. The organizers record the floor reached on each attempt. Everyday May 2026 21.6 Not in cards
Cattle Trade Agents play a long economic game with auctions, hidden offers, bargaining, bluffing, and resource limits. Everyday May 2026 Not in cards
euchre-bench Coding agents build a euchre bot and test it against four opponents. Each opponent contributes up to 100 wins to the total. University May 2026 140.0 Not in cards
Game Arena Word Association Teams of language models give one-word clues and guess linked words, in a game like Codenames, against an opposing team. Everyday May 2026 734 Not in cards
Maze Bench Agents explore connected rooms in a 3D puzzle game. Each of the 100 gems contributes one percentage point to the score. Expert May 2026 24% Not in cards
Megaminx solver Models solve a simulated twelve-faced twisty puzzle. They read the puzzle state as text, make moves with tools and receive rewards from the simulator. University May 2026 Not in cards
PTCG-Bench Agents play the Pokémon Trading Card Game by choosing legal actions from a partially visible board. Match results determine their Glicko-2 ratings. School May 2026 1,854 Not in cards
Slitherlink (savi) Models place edges to form a single loop on seven-by-seven Slitherlink boards. The evaluation uses ten boards disjoint from its 15 training examples. School May 2026 Not in cards
AlphaZero Connect Four study Coding agents implement an AlphaZero pipeline for Connect Four. The resulting programs play a round-robin tournament against the Pascal Pons solver. Expert Apr 2026 1,938 Not in cards
Cooperate to Compete (C2C) Multi-player conquest game with private non-binding negotiation and secret asymmetric objectives. LM agents and humans are ranked by Plackett-Luce player strength from their game results. Everyday Apr 2026 0.74 Not in cards
Game Arena Five in a Row Language models play a gomoku-style game and try to line up five of their tokens before the opponent does. Everyday Apr 2026 1,131 Not in cards
GomokuBench Models place stones on a 19 by 19 board against a search engine. The score rewards wins, quicker wins, and longer losses. Everyday Apr 2026 2.90 Not in cards
SpireBench (HermesBridge trial-v0) Models play Slay the Spire 2 from structured game states. Each plays all five characters at the base difficulty, with one run per character. Everyday Apr 2026 28.0 Not in cards
Clocktower Radio Models play the social deduction game Blood on the Clocktower (the Trouble Brewing script) in mirrored pairs of games. In each match one model plays good and the other evil, and then they swap. A coded Storyteller runs the game, and a Bradley–Terry fit over all matches gives ratings where 1500 is average. Everyday Mar 2026 1,708 Not in cards
Cooperative Codenames (Max Bittker) Two models cooperate as cluegiver and guesser to identify target words on a hidden board. The default evaluation uses 200 separately generated boards. Everyday Mar 2026 Not in cards
GTO Wizard Benchmark Models play heads-up no-limit Texas Hold’em against GTO Wizard AI. AIVAT reduces variance while the benchmark reports poker winnings. Expert Mar 2026 -9.23 Not in cards
InteractiveBench Texas Hold’em Models play no-limit Texas Hold’em from partial observations of cards, chip stacks, and recent actions. Repeated matches measure chip gains, with invalid actions receiving one retry before an automatic fold. Competition Mar 2026 31.8 Not in cards
InteractiveBench Trust Game Models choose whether to cooperate or defect in repeated prisoner’s dilemma matches. A round-robin tournament scores average realized payoff per round, with a random stopping rule. Everyday Mar 2026 1.87 Not in cards
NYT Connections (Prime Community) Models identify four groups of related words in New York Times Connections puzzles. The default 100-puzzle evaluation scores completion, efficiency, partial progress, and format. Everyday Mar 2026 Not in cards
PokeAgent Challenge Agents compete in Pokemon battles under partial observability in a Battling Track and play the Pokemon RPG in a Speedrunning Track that needs long-horizon planning. Expert Mar 2026 Not in cards
Game Arena Chess Openings Language models play chess against each other from varied two-move openings sampled from 20 popular human openings. The board is given as text. Expert Feb 2026 1,263 Not in cards
Game Arena Four in a Row Language models play a Connect Four style game and try to line up four of their tokens before the opponent does. Everyday Feb 2026 1,003 Not in cards
Game Arena Poker Language models play heads-up no-limit Texas Hold'em against each other. Models are ranked by mean big blinds won per 100 hands. Expert Feb 2026 30.3 Not in cards
Game Arena Werewolf Teams of language models play the social deduction game Werewolf in natural language. Villagers must find the hidden werewolves, who try to deceive them. Expert Feb 2026 0 Not in cards
LLM-WikiRace Models play the Wikipedia game: starting from one article, they follow one hyperlink at a time to reach a target article. At each step the model sees only the links on the current page, and a game is lost after 30 steps. Three difficulty levels are set by the shortest path between the two pages. Scores are the share of games won at each level. Everyday Feb 2026 23% Not in cards
RuneBench Coding agents play RuneScape on an emulated game server by writing TypeScript against the rs-sdk library, with game-wiki notes for reference. Tasks train one skill or gather gold within 15 or 30 minutes. Skill tasks are scored by the peak XP rate in any 15-second window, and a verifier reads the final game state. Everyday Feb 2026 7.85 Not in cards
WordleArenas Models guess a five-letter word from colored clues. Each arena changes how many clues give false feedback after a guess, from zero to five. Everyday Feb 2026 100% Not in cards
Agent Poker Bench Models play 10-seat no-limit Texas Hold'em tournaments against each other. Chip counts and elimination order feed TrueSkill ratings. Expert Jan 2026 1,132 Not in cards
Buyout Game Eight language models play a money-driven elimination game with private transfers and a buyout endgame. Everyday Jan 2026 1975.4 Not in cards
KradleVerse Agent harnesses compete in a live Minecraft arena. Everyday Jan 2026 Not in cards
mage-bench Models play Magic: The Gathering in a benchmark that compares strategic play across games. Expert Jan 2026 1617.0 Not in cards
MegaGem Three models bid in sealed auctions with private cards. The benchmark rotates starting hands and fits an Elo-scaled rating to the order in which players finish. Everyday Jan 2026 Not in cards
SokoBench Models solve Sokoban puzzles that require long-horizon planning and persistent state tracking. Everyday Jan 2026 Not in cards
Spyfall study Models play Spyfall while inferring hidden locations and managing deceptive dialogue. Everyday Jan 2026 1003.0 Not in cards
TetrisBench Models play Tetris head-to-head in the original benchmark. The Harbor strategy track grades an agent-written JavaScript placement function on 20 fixed piece sequences. Everyday Jan 2026 66.7% Not in cards
CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. Research frontier Dec 2025 66.5 Not in cards
WOLF Models play Werewolf to measure deception production and deception detection as separate abilities. Everyday Dec 2025 Not in cards
CodeClash Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings. Expert Nov 2025 1,385 Not in cards
Mini Amusement Parks Agents build rides, hire staff, and manage an amusement park. The simulator scores the park’s final value. Everyday Nov 2025 371375.0 Not in cards
CATArena Code agents write programs that play board and card games in repeated tournaments. Between rounds they refine their code through self-reflection and by studying rivals' code. Scores are average ranks that separate first-round skill from improvement. Expert Oct 2025 1,799 Not in cards
CATArena Gomoku Code agents compete in Gomoku. The public competition board reports Elo Rating for each agent configuration. Expert Oct 2025 1,799 Not in cards
CATArena Pommerman Code agents compete in Pommerman. The public competition board reports Elo Rating for each agent configuration. Expert Oct 2025 Not in cards
CATArena Tetris Code agents compete in Tetris. The public competition board reports Score for each agent configuration. Expert Oct 2025 Not in cards
CATArena Texas Holdem Code agents compete in Texas Holdem. The public competition board reports Elo Rating for each agent configuration. Expert Oct 2025 1,463 Not in cards
Fruit Box Models select rectangles whose entries sum to ten on a ten-by-seventeen grid. The evaluator compares the final score with an expert trajectory from the training split. School Oct 2025 Not in cards
Mastermind (Prime Intellect) Models guess a hidden sequence and receive feedback after each turn. The default evaluation uses 50 games and rewards success, speed, and deductions. Everyday Oct 2025 1.62 Not in cards
Blackjack Basic Strategy Models play blackjack hands or select a single move from a dealt position. The grader measures the expected value of each decision against basic strategy. Everyday Sep 2025 Not in cards
Lights Out 3×3 Models toggle lights on a three-by-three grid until every light is off. The dataset reserves 30 boards for testing and rewards efficient solutions. School Sep 2025 Not in cards
Mastermind Information Gain Models choose a Mastermind move from earlier guesses and feedback. The evaluator measures information gain for individual moves and success over complete games. School Sep 2025 Not in cards
Multi-Turn Sudoku (m8ngotree) Models fill a nine-by-nine Sudoku board one move at a time. The generator checks for a unique solution, and the scorer rewards valid moves and completion. School Sep 2025 Not in cards
UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. Everyday Sep 2025 59.8 Dropped
Wer wird Millionär? LLM Benchmark Models play 45 games of the German version of Who Wants to Be a Millionaire, each with 15 questions of rising difficulty. Early questions lean on German wordplay and idioms. Everyday Sep 2025 644444.0 Not in cards
AI FPL Models manage Fantasy Premier League squads using the same weekly prompt. Their transfers, captain choices, and starting lineups earn points from real matches. Everyday Aug 2025 Not in cards
BalatroBench Models play the deck-building roguelike Balatro through an API bot. The leaderboard compares how far each model and strategy gets in a run. Everyday Aug 2025 19.3 Not in cards
Game Arena Chess Language models play full chess games against each other with the board given as FEN and PGN text. Ratings come from many games between each pair. Expert Aug 2025 1,646 Not in cards
Kaggle Game Arena Language models play each other in board, card and word games through a text harness. One leaderboard ranks models across all Game Arena games. Expert Aug 2025 638 Not in cards
LLM Chess Arena (Louis Guichard) Models play each other from a text diagram, piece locations, and recent moves. A model forfeits after three failed attempts to make a legal move. School Aug 2025 1,495 Not in cards
ScrabbleBench Models choose a word from a Scrabble rack and board. A solver finds its best legal placement. The score averages the four seats in self-play. Everyday Aug 2025 152.8 Not in cards
FantasyFootballBench Models draft players, set weekly lineups, and negotiate trades in a replay of the 2024 NFL season. Actual player statistics determine points and wins. Everyday Jul 2025 15.0 Not in cards
Husky Hold'em Bench Models write Python poker bots and revise them after practice games. Six bots compete at each table, and the tournament tracks average winnings. University Jul 2025 Not in cards
SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. Everyday Jul 2025 94% Not in cards
Hangman Bench Agents guess letters to identify hidden words in Hangman. The harness tracks wins across difficulty levels. Everyday Jun 2025 Not in cards
HexMachina Models write Catan players in code and compete in the board game. University Jun 2025 Not in cards
lmgame-Bench Models play platformer, puzzle and narrative games such as Sokoban, Tetris, 2048 and Candy Crush through a Gym-style API with perception and memory scaffolds. Everyday May 2025 87.7 Not in cards
Among Us: A Sandbox for Measuring and Detecting Agentic Deception Language agents play a multiplayer social-deception game with hidden roles. Deception Elo and detection Elo measure success over repeated matches. Expert Apr 2025 Not in cards
KUMO Agents make queries and use the responses to identify hidden facts in generated reasoning games. Harbor offers 5,050 easy games and 250 hard games. University Apr 2025 86.3% Not in cards
MafiaBench Models play a social deduction game with hidden roles and public discussion. Everyday Apr 2025 1,561 Not in cards
TextArena Collection of competitive text games for LLM eval and RL, with an online Elo leaderboard. Apr 2025 Not in cards
Dubesor AI Chess Leaderboard Models play chess by continuing game notation or by reasoning from the board and legal moves. The organizer maintains separate Elo ratings for each mode. School Mar 2025 1,907 Not in cards
Factorio Learning Environment Agents write Python programs that control a Factorio factory, in lab tasks with fixed resources and in open play. Scored by production throughput, milestones and lab task success. Expert Mar 2025 293206.0 Not in cards
MC-Bench Models write code that builds a Minecraft structure from a prompt. Users vote between two builds without knowing the models. Everyday Mar 2025 Not in cards
AI Diplomacy Models play the board game Diplomacy against each other, one power each. They negotiate, form alliances and betray each other in written messages. Everyday Feb 2025 60 Not in cards
Collab-Overcooked Two agents exchange messages and use different kitchen equipment to fill orders. The benchmark measures task completion and progress through required cooking steps. Everyday Feb 2025 86.7% Not in cards
Elimination Game Models play a multi-player game of public and private talk, alliances and votes to eliminate each other. A jury of eliminated players picks the winner. Everyday Feb 2025 8 Not in cards
LLM Public Goods Game Agents choose how many tokens to contribute to a common pool. The evaluation reports the fraction each model contributes across repeated games. Everyday Feb 2025 45.2% Not in cards
SnakeBench Two language models each steer a snake on a shared board in a turn-based game, and the one that survives longest wins. Ratings come from many head-to-head games. Everyday Feb 2025 39 Not in cards
Step Race Several models talk in public, then secretly pick moves in a race where colliding moves cancel. The game tests cooperation and deception. Everyday Jan 2025 5 Not in cards
BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. Everyday Nov 2024 68.3% Not in cards
LLM Chess Models play chess in multi-turn dialogs against a random player or the Komodo Dragon engine. The board reports wins, Elo and game duration, which tracks instruction following. Everyday Sep 2024 1,614 Not in cards
VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. Everyday Aug 2024 36.2% Not in cards
Baba Is AI Models see a puzzle and write a plan to reach the winning object. Some puzzles require them to push words together to change the rules. Everyday Jul 2024 43.9% Not in cards
Werewolf Arena Models compete in Werewolf with strategic speaking, deception, deduction, and persuasion. Everyday Jul 2024 26.7 Not in cards
MiniStS Models play card battles with five preset decks. The authors measure final health in four scenarios and how often agents play the Harm card in the fifth. Everyday May 2024 Not in cards
LLM Colosseum Two models each control a fighter in Street Fighter III and choose moves from text descriptions of the game. Wins give Elo ratings. Everyday Mar 2024 1,835 Not in cards
LLM Game Benchmark Models compete in tic-tac-toe, Connect Four, Gomoku, and other grid games through a shared interface. Everyday Jan 2024 185.0 Not in cards
AvalonBench Models play Resistance Avalon while discussing, deceiving, deducing, and negotiating with other agents. Everyday Nov 2023 Not in cards
LLM-Coordination Models play four cooperative games and answer 198 multiple-choice questions about the same games. The authors report game outcomes and question accuracy separately. Everyday Oct 2023 46.6 Not in cards
clembench Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages. Everyday May 2023 88.8 Not in cards
MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. The score is normalized game reward. Annotations also score power-seeking, disutility and ethical violations. Everyday Apr 2023 Not in cards
PushWorld Agents move shapes through puzzles that require tools or pushing obstacles aside. The benchmark includes more than 200 puzzles with goal positions. School Jan 2023 Not in cards
MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. Everyday Jun 2022 49% Fading
TextWorld Cooking (GATA) Agents navigate text-based rooms, gather ingredients, and prepare recipes. The port sums points earned during play and includes separate training and test games at increasing difficulties. Everyday Feb 2020 2.17 Not in cards
ChessBench (MadeByLupu) Models play each other with access to the board, move history and legal moves. The organizer rates each model and thinking setting with Glicko-2. School 2,170 Not in cards
Crest Arcade Chess Models play standard chess and Chess960 against fixed engine opponents without a list of legal moves. The organizer fits separate ratings for each variant. School 1,758 Not in cards
Crest Arcade Elemental Armory Models equip four elemental balls with cards, then select and aim a ball each round. The score averages match win rates across three levels. Everyday 67.3% Not in cards
Crest Arcade FreeCell Models move cards to the four foundations using a variable number of free cells. The score averages win rates across five tiers. Everyday 63.3% Not in cards
Crest Arcade Hustle Sim Models buy upgrades and hire helpers to reach earnings targets in an idle game. The score compares simulated completion times with reference strategies. Everyday 83.7 Not in cards
Crest Arcade Mines Models reveal or flag cells on Minesweeper boards that can be solved without guessing. The score averages win rates across four board sizes. Everyday 100% Not in cards
Crest Arcade Nations Royale Models manage one nation against 47 computer opponents on a simulated map. Wins earn full credit and unfinished games earn credit for territory held. Everyday 100.0 Not in cards
Crest Arcade Ravelin Models build towers to protect a keep through waves of attackers. The score compares wins with the reference time plus a penalty for lives lost. Everyday 106.8 Not in cards
Crest Arcade Starjar Models place pieces in a jar so matching pieces touch and merge. The score compares their total points with a reference bot on identical jars. Everyday 114.5 Not in cards
Crest Arcade Wildhaven Models select a team of animals and choose where each lands in a village. The score measures the share of village strength they destroy. Everyday 100% Not in cards
Game Arena 3 Check Chess (Generality) Models play both colors. A player can win by checking the opposing king three times. A double check counts once. School 27.0 Not in cards
Game Arena Anti Chess (Generality) Models must capture when they can. A player wins by losing every piece or by having no legal move. The king can be captured. School 25.0 Not in cards
Game Arena Atomic Chess (Generality) Models play chess with exploding captures. A capture removes both involved pieces and neighboring pieces other than pawns. Exploding the opposing king wins. School 27.0 Not in cards
Game Arena Crazyhouse Chess (Generality) Models play Crazyhouse chess in both colors. A player can spend a turn placing a captured piece back on the board under its new owner's control. School 26.0 Not in cards
Game Arena Four-in-a-Row (Generality) Models place pieces on any empty square of a four by nine grid. Four in a line wins. The tournaments use different board encodings. Everyday 92.0 Not in cards
Game Arena Horde Chess (Generality) Models take both sides in turn. White tries to checkmate with 36 pawns. Black starts with the usual pieces and must capture every white pawn. School 24.0 Not in cards
Game Arena King of the Hill (Generality) Models can win by moving their king onto any of the four central squares. The king must reach the square without moving into check. School 24.0 Not in cards
Game Arena Racing Kings (Generality) Models race their kings to the eighth rank without giving check. Black can draw by reaching that rank immediately after White. School 23.5 Not in cards
Kaggle Chess Suite Models play chess from the standard starting position and from selected openings. Kaggle publishes both ratings for each model. School 1,238 Not in cards
LLM Games Models play Connect Four against one another. Ramp publishes match scores and final Elo ratings. Everyday 1,728 Not in cards
Mana RP Bench Models write the next turn in a player’s story. The player sees two anonymous responses and chooses which one becomes part of the story. Everyday 1,280 Not in cards
Peer Arena (oddbit.ai) Five models debate for five rounds, then each privately votes for a survivor. The organizer calculates Bradley-Terry ratings from game outcomes. Everyday 1,799 Not in cards
PGG-Bench Agents decide how much to contribute to a public good and whether to punish other players. Everyday 14 Not in cards
Pokémon VGC Bench (Ricotoma) Agents play competitive Pokémon doubles with assigned teams. They choose four Pokémon, select two leads, and choose actions for both active Pokémon each turn. Competition 1,901 Not in cards