Benchmark Almanac

Lech Mazur

We track 16 benchmarks and 4 arenas that Lech Mazur made or helped make.

0 in use 20 with public code 20 with a leaderboard here

Arena leaderboards

Top five in each of its 4 arenas with three or more rated models, the most rated first.

Benchmarks

Name Kind Released Best score Status
Extended NYT Connections Models solve New York Times Connections puzzles with up to four extra trick words added. Each puzzle scores by the square of the share of groups found. Language reasoning Oct 2024 99.3% Not in cards
LLM Confabulation Leaderboard Models answer misleading questions about recent articles whose answers are not in the text. The board ranks confabulation rate against non-response rate. Faithful to sources Oct 2024 10.3% Not in cards
Lech Mazur Creative Writing Models write short stories that must use 10 assigned elements such as characters, objects, and motivations. Judge models compare matched stories pairwise. Creative writing Jan 2025 4.30 Not in cards
Buyout Game Eight language models play a money-driven elimination game with private transfers and a buyout endgame. Multi-agent games Jan 2026 1975.4 Not in cards
LLM Position Bias Benchmark Models judge pairs of responses twice, with the presentation order reversed. Bias, fairness and politics 19.8% Not in cards
BAZAAR Agents buy and sell goods in a simulated competitive market. Multi-agent games 7 Not in cards
Flash Fiction Stylistic Diversity Models write flash fiction from shared prompts. A reviewing model describes each story, and weighted Gower distance measures stylistic diversity within a writer’s work. Taste and style Sep 2025 0.22 Not in cards
LLM Sycophancy Benchmark Models judge the same dispute from opposite narrators’ perspectives. Contradictory judgments count toward the sycophancy rate. Deception and alignment 0% Not in cards
LLM Public Goods Game Agents choose how many tokens to contribute to a common pool. The evaluation reports the fraction each model contributes across repeated games. Multi-agent games Feb 2025 45.2% Not in cards
LLM Thematic Generalization Benchmark v2 Models rank eight candidates using three positive examples and three counterexamples. Version 2 filters ambiguous prompts and reports a harder subset of 703 items. Language reasoning Mar 2026 80.6 Not in cards
LLM Divergent Thinking Creativity Benchmark Models generate sets of words that differ in meaning. Creative writing 4.79 Not in cards
LLM Deceptiveness and Gullibility Models try to deceive other models and recognize deceptive claims. Multi-agent games 1.10 Not in cards
PGG-Bench Agents decide how much to contribute to a public good and whether to punish other players. Multi-agent games 14 Not in cards
Emergent Price-Fixing by LLM Auction Agents Models trade in simulated auctions and communicate with other participants. A judge examines game logs for evidence of price fixing. Deception and alignment Jul 2025 19% Not in cards
LLM Persuasion Benchmark Models try to change another model’s stated position over multiple conversation turns. Multi-turn conversation 1.71 Not in cards
LLM Round-Trip Translation Benchmark Models translate text into another language and back. The evaluation measures how well the original meaning survives. Translation 8.69 Not in cards