Benchmark Almanac

Arena (formerly LMArena)

We track 10 benchmarks and 19 arenas that Arena (formerly LMArena) made or helped make.

2 in use 7 with public code 24 with a leaderboard here

Arena leaderboards

Top five in each of its 16 arenas with three or more rated models, the most rated first. We don't have a leaderboard yet for 3 others, listed below.

Benchmarks

Name Kind Released Best score Status
Arena-Hard-Auto 500 hard arena prompts judged by an LLM against a baseline. Cheap proxy for arena rank. Overall chat quality Apr 2024 97.1% Fading
Arena-Hard v2.0 Models answer 500 hard real user prompts from Chatbot Arena and 250 creative writing prompts. GPT-4.1 or Gemini 2.5 judges each answer against a baseline answer from o3-mini, and the score is the win rate. Overall chat quality Apr 2025 90.2% Fading
Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. Deception and alignment Oct 2026 87.9 Not in cards
VisionArena-Bench Models answer visual questions from Arena users. A vision language model judges their answers to approximate human preference rankings. General visual questions Dec 2024 88 Not in cards
Arena-T2I Hard Models generate images from detailed user prompts. A judge checks whether the images satisfy individual requirements and their dependencies. Image generation and editing Jun 2026 0.85 Not in cards
Preference Proxy Evaluations Reward models and judges rank responses. The benchmark compares their decisions against user preferences and verifiable answers. Overall chat quality Oct 2024 68.5% Not in cards
Arena Search Factuality Arena combines user preferences with independent checks of verifiable claims in model responses. Long-form factuality Jul 2026 1,218 Not in cards
ArenaWebdevCorrectness Judges inspect generated applications and predict whether their features work. Human labels provide the reference answers. Web and app front ends Oct 2026 81.6% Not in cards
ArenaWebdevPreferences Judges compare pairs of generated applications and predict which application the user preferred. The score measures agreement with user votes. Web and app front ends Oct 2026 69.1% Not in cards
On Evaluating User Simulators for Interactive Agents Models play the user in replays of 351 real sessions. The evaluation compares simulated behavior with observed behavior and measures agreement with human rankings of nine agents. Multi-turn conversation Nov 2026 Not in cards

Arenas without a leaderboard here

Name Kind Released Best score Status
BiomedArena Researchers compare anonymous answers to biomedical research questions and vote for the response they prefer. Expert-written questions Aug 2025 Not in cards
RedTeam Arena Players get 60 seconds to make a model say a target word. An extended Elo ranks players, models and target prompts, with models ranked by resistance. Jailbreaks and prompt injection Sep 2024 Not in cards
RepoChat Arena Users give a GitHub repository link and a question. A retriever model picks files, two answer models reply, and users vote for the better answer. Fixing real repositories Jan 2025 Not in cards