Benchmark Almanac

CodeElo

o1-mini 1,578 leads since January 2025
Not in model cards

Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles.

The automated submission service requires maker approval. The public rating code supports alternative submission systems.

Made by
Qwen team, Alibaba
Released
January 2025
Kind
Competitive programming
Level
Competition
Scored
Checked automatically
Answers
Code
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Code → Code
Interaction
Single turn
Environment
Code sandbox
Items from
Exams and contests
Refresh
One fixed set
Run by
Anyone
Strongest model tested
o1-mini #142 on the capability index
Last model added
November 2024
Scores refreshed
November 20th, 2026

In the authors’ words

With the increasing code reasoning capabilities of existing large language models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo, we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 25 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies.

Abstract of CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings (arXiv:2501.01257, CC0)
33
Entries in its paper, 9 models we track

o1-mini leads with 1,578 among models released by November 2024 o1-mini leads with 1,578 among models released by November 2024

Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in November 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 o1-mini 1,578
  2. 2 QwQ-32B-Preview 1,261
  3. 3 Claude 3.5 Sonnet 710
  4. 4 ChatGPT-4o-latest-2024-11-20 668
  5. 5 Qwen2.5-72B 634
  6. 6 Mistral-Large-Instruct-2411 631
  7. 7 DS-V2.5 629
  8. 8 Qwen2.5-Coder-32B-Instruct 575
  9. 9 DS-Coder-V2-Instruct 532
  10. 10 Qwen2.5-32B-Instruct 513
  11. 11 Llama-3.1-70B-Instruct 478
  12. 12 Qwen2.5-Coder-14B-Instruct 424
  13. 13 Qwen2.5-14B-Instruct 414
  14. 14 Qwen2.5-Coder-7B-Instruct 397
  15. 15 Codestral-22B-v0.1 385
  16. 16 DS-V2-Chat 318
  17. 17 Qwen2.5-7B-Instruct 315
  18. 18 Yi-Coder-9B-Chat 296
  19. 19 Mixtral-8x22B-Instruct-v0.1 295
  20. 20 DS-V2-Lite-Instruct 254
  21. 21 Llama-3.1-8B-Instruct 223
  22. 22 Ministral-8B-Instruct-2410 219
  23. 23 DS-Coder-33B-Instruct 207
  24. 24 Code Llama 70B Instruct 200
  25. 25 Qwen2.5-Coder-3B-Instruct 160
  26. 26 DS-Coder-6.7B-Instruct 155
  27. 27 OpenCoder-8B-Instruct 152
  28. 28 Starcoder2-15B-Instruct-v0.1 129
  29. 29 Mixtral-8x7B-Instruct-v0.1 98
  30. 30 Qwen2.5-Coder-1.5B-Instruct 93
  31. 31 DS-V2-Lite-Chat 60
  32. 32 Mistral-7B-Instruct-v0.2 49
  33. 33 DS-Coder-1.3B-Instruct 21

Source: The CodeElo paper

#EntryScoreSettingDateSource
1 o1-mini 1,578 Sep 2024 Official leaderboard
2 QwQ-32B-Preview 1,261 Nov 2024 Official leaderboard
3 Claude 3.5 Sonnet 710 Jun 2024 Official leaderboard
4 ChatGPT-4o-latest-2024-11-20 closed weights 668 Official leaderboard
5 Qwen2.5-72B 634 Sep 2024 Official leaderboard
6 Mistral-Large-Instruct-2411 open weights 631 Official leaderboard
7 DS-V2.5 open weights 629 Official leaderboard
8 Qwen2.5-Coder-32B-Instruct 575 Nov 2024 Official leaderboard
9 DS-Coder-V2-Instruct open weights 532 Official leaderboard
10 Qwen2.5-32B-Instruct open weights 513 Official leaderboard
11 Llama-3.1-70B-Instruct open weights 478 Official leaderboard
12 Qwen2.5-Coder-14B-Instruct 424 Nov 2024 Official leaderboard
13 Qwen2.5-14B-Instruct open weights 414 Official leaderboard
14 Qwen2.5-Coder-7B-Instruct open weights 397 Official leaderboard
15 Codestral-22B-v0.1 open weights 385 Official leaderboard
16 DS-V2-Chat open weights 318 Official leaderboard
17 Qwen2.5-7B-Instruct open weights 315 Official leaderboard
18 Yi-Coder-9B-Chat open weights 296 Official leaderboard
19 Mixtral-8x22B-Instruct-v0.1 open weights 295 Official leaderboard
20 DS-V2-Lite-Instruct open weights 254 Official leaderboard
21 Llama-3.1-8B-Instruct open weights 223 Official leaderboard
22 Ministral-8B-Instruct-2410 open weights 219 Official leaderboard
23 DS-Coder-33B-Instruct open weights 207 Official leaderboard
24 Code Llama 70B Instruct 200 Jan 2024 Official leaderboard
25 Qwen2.5-Coder-3B-Instruct 160 Nov 2024 Official leaderboard
26 DS-Coder-6.7B-Instruct open weights 155 Official leaderboard
27 OpenCoder-8B-Instruct open weights 152 Official leaderboard
28 Starcoder2-15B-Instruct-v0.1 open weights 129 Official leaderboard
29 Mixtral-8x7B-Instruct-v0.1 open weights 98 Official leaderboard
30 Qwen2.5-Coder-1.5B-Instruct open weights 93 Official leaderboard
31 DS-V2-Lite-Chat open weights 60 Official leaderboard
32 Mistral-7B-Instruct-v0.2 49 Dec 2023 Official leaderboard
33 DS-Coder-1.3B-Instruct open weights 21 Official leaderboard
#ModelScoreSettingDateSource
1 o1-mini 1,578 Sep 2024 Official leaderboard
2 QwQ-32B-Preview 1,261 Nov 2024 Official leaderboard
3 Claude 3.5 Sonnet 710 Jun 2024 Official leaderboard
4 Qwen2.5-72B 634 Sep 2024 Official leaderboard
5 Qwen2.5-Coder-32B-Instruct 575 Nov 2024 Official leaderboard
6 Qwen2.5-Coder-14B-Instruct 424 Nov 2024 Official leaderboard
7 Code Llama 70B Instruct 200 Jan 2024 Official leaderboard
8 Qwen2.5-Coder-3B-Instruct 160 Nov 2024 Official leaderboard
9 Mistral-7B-Instruct-v0.2 49 Dec 2023 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Benchmark paper Not tested since Jan 2025
o1-mini 1,578

Source: The CodeElo paper

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
In the benchmark’s paper
0 500 1,000 1,500 2,000 2024 2025 0 500 1,000 1,500 2,000 2024 2025

Source: Official CodeElo leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Coding

All 523 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in November 2024, for QwQ-32B-Preview. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the benchmark’s paper (33 entries).
  • Pass Models scored 9 models we track, plus 24 other entries on its leaderboard.
  • Warning Newest score The newest came 2 years ago, in November 2024, for QwQ-32B-Preview.
  • Note Frontier models None of the top 10 has a score. The strongest tested is o1-mini (#142).
  • Note Headroom Scored as an Elo rating, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on arxiv.org: 9 rows for models we track and 24 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check