Step Race
Several models talk in public, then secretly pick moves in a race where colliding moves cancel. The game tests cooperation and deception.
- GPT-5 5
- o3 5
- GPT-5.1 5
-
Gemini 3 Pro 5
- o1 4
Official leaderboard
We track 16 benchmarks and 4 arenas that Lech Mazur made or helped make.
Top five in each of its 4 arenas with three or more rated models, the most rated first.
Several models talk in public, then secretly pick moves in a race where colliding moves cancel. The game tests cooperation and deception.
Official leaderboard
Models debate an opponent over multiple rounds. Judges compare the arguments.
Official leaderboard
Models play a multi-player game of public and private talk, alliances and votes to eliminate each other. A jury of eliminated players picks the winner.
Official leaderboard
Two models bargain over 20 rounds as buyer and seller, each with a hidden private value. Matches across many pairings rank the models.
Official leaderboard
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Extended NYT Connections Models solve New York Times Connections puzzles with up to four extra trick words added. Each puzzle scores by the square of the share of groups found. | Language reasoning | Oct 2024 | | Not in cards |
| LLM Confabulation Leaderboard Models answer misleading questions about recent articles whose answers are not in the text. The board ranks confabulation rate against non-response rate. | Faithful to sources | Oct 2024 | 10.3% | Not in cards |
| Lech Mazur Creative Writing Models write short stories that must use 10 assigned elements such as characters, objects, and motivations. Judge models compare matched stories pairwise. | Creative writing | Jan 2025 | 4.30 | Not in cards |
| Buyout Game Eight language models play a money-driven elimination game with private transfers and a buyout endgame. | Multi-agent games | Jan 2026 | 1975.4 | Not in cards |
| LLM Position Bias Benchmark Models judge pairs of responses twice, with the presentation order reversed. | Bias, fairness and politics | 19.8% | Not in cards | |
| BAZAAR Agents buy and sell goods in a simulated competitive market. | Multi-agent games | 7 | Not in cards | |
| Flash Fiction Stylistic Diversity Models write flash fiction from shared prompts. A reviewing model describes each story, and weighted Gower distance measures stylistic diversity within a writer’s work. | Taste and style | Sep 2025 | 0.22 | Not in cards |
| LLM Sycophancy Benchmark Models judge the same dispute from opposite narrators’ perspectives. Contradictory judgments count toward the sycophancy rate. | Deception and alignment | 0% | Not in cards | |
| LLM Public Goods Game Agents choose how many tokens to contribute to a common pool. The evaluation reports the fraction each model contributes across repeated games. | Multi-agent games | Feb 2025 | | Not in cards |
| LLM Thematic Generalization Benchmark v2 Models rank eight candidates using three positive examples and three counterexamples. Version 2 filters ambiguous prompts and reports a harder subset of 703 items. | Language reasoning | Mar 2026 | 80.6 | Not in cards |
| LLM Divergent Thinking Creativity Benchmark Models generate sets of words that differ in meaning. | Creative writing | 4.79 | Not in cards | |
| LLM Deceptiveness and Gullibility Models try to deceive other models and recognize deceptive claims. | Multi-agent games | 1.10 | Not in cards | |
| PGG-Bench Agents decide how much to contribute to a public good and whether to punish other players. | Multi-agent games | | Not in cards | |
| Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. | Deception and alignment | Jul 2025 | 19% | Not in cards |
| LLM Persuasion Benchmark Models try to change another model’s stated position over multiple conversation turns. | Multi-turn conversation | 1.71 | Not in cards | |
| LLM Round-Trip Translation Benchmark Models translate text into another language and back. The evaluation measures how well the original meaning survives. | Translation | 8.69 | Not in cards |