CodeElo
Models solve Codeforces contest problems through the platform's judging system. The harness converts contest outcomes into human-comparable Elo ratings and percentiles.
The automated submission service requires maker approval. The public rating code supports alternative submission systems.
- Released
- January 2025
- Kind
- Competitive programming
- Level
- Competition
- Scored
- Checked automatically
- Answers
- Code
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Environment
- Code sandbox
- Items from
- Exams and contests
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
- o1-mini #142 on the capability index
- Last model added
- November 2024
- Scores refreshed
- November 20th, 2026
In the authors’ words
With the increasing code reasoning capabilities of existing large language models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo, we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 25 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies.
o1-mini leads with 1,578 among models released by November 2024 o1-mini leads with 1,578 among models released by November 2024
Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
The newest model with a score here came out in November 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 1,578 o1-mini
- 1,261
QwQ-32B-Preview
- 710 Claude 3.5 Sonnet
- 668 ChatGPT-4o-latest-2024-11-20
- 634
Qwen2.5-72B
- 631 Mistral-Large-Instruct-2411
- 629 DS-V2.5
- 575
Qwen2.5-Coder-32B-Instruct
- 532 DS-Coder-V2-Instruct
- 513 Qwen2.5-32B-Instruct
- 478 Llama-3.1-70B-Instruct
- 424 Qwen2.5-Coder-14B-Instruct
- 414 Qwen2.5-14B-Instruct
- 397 Qwen2.5-Coder-7B-Instruct
- 385 Codestral-22B-v0.1
- 318 DS-V2-Chat
- 315 Qwen2.5-7B-Instruct
- 296 Yi-Coder-9B-Chat
- 295 Mixtral-8x22B-Instruct-v0.1
- 254 DS-V2-Lite-Instruct
- 223 Llama-3.1-8B-Instruct
- 219 Ministral-8B-Instruct-2410
- 207 DS-Coder-33B-Instruct
- 200
Code Llama 70B Instruct
- 160 Qwen2.5-Coder-3B-Instruct
- 155 DS-Coder-6.7B-Instruct
- 152 OpenCoder-8B-Instruct
- 129 Starcoder2-15B-Instruct-v0.1
- 98 Mixtral-8x7B-Instruct-v0.1
- 93 Qwen2.5-Coder-1.5B-Instruct
- 60 DS-V2-Lite-Chat
- 49
Mistral-7B-Instruct-v0.2
- 21 DS-Coder-1.3B-Instruct
- 1 o1-mini 1,578
- 2
QwQ-32B-Preview 1,261
- 3 Claude 3.5 Sonnet 710
- 4 ChatGPT-4o-latest-2024-11-20 668
- 5
Qwen2.5-72B 634
- 6 Mistral-Large-Instruct-2411 631
- 7 DS-V2.5 629
- 8
Qwen2.5-Coder-32B-Instruct 575
- 9 DS-Coder-V2-Instruct 532
- 10 Qwen2.5-32B-Instruct 513
- 11 Llama-3.1-70B-Instruct 478
- 12 Qwen2.5-Coder-14B-Instruct 424
- 13 Qwen2.5-14B-Instruct 414
- 14 Qwen2.5-Coder-7B-Instruct 397
- 15 Codestral-22B-v0.1 385
- 16 DS-V2-Chat 318
- 17 Qwen2.5-7B-Instruct 315
- 18 Yi-Coder-9B-Chat 296
- 19 Mixtral-8x22B-Instruct-v0.1 295
- 20 DS-V2-Lite-Instruct 254
- 21 Llama-3.1-8B-Instruct 223
- 22 Ministral-8B-Instruct-2410 219
- 23 DS-Coder-33B-Instruct 207
- 24
Code Llama 70B Instruct 200
- 25 Qwen2.5-Coder-3B-Instruct 160
- 26 DS-Coder-6.7B-Instruct 155
- 27 OpenCoder-8B-Instruct 152
- 28 Starcoder2-15B-Instruct-v0.1 129
- 29 Mixtral-8x7B-Instruct-v0.1 98
- 30 Qwen2.5-Coder-1.5B-Instruct 93
- 31 DS-V2-Lite-Chat 60
- 32
Mistral-7B-Instruct-v0.2 49
- 33 DS-Coder-1.3B-Instruct 21
Source: The CodeElo paper
- 1 o1-mini 1,578
- 2
QwQ-32B-Preview 1,261
- 3 Claude 3.5 Sonnet 710
- 4
Qwen2.5-72B 634
- 5
Qwen2.5-Coder-32B-Instruct 575
- 6 Qwen2.5-Coder-14B-Instruct 424
- 7
Code Llama 70B Instruct 200
- 8 Qwen2.5-Coder-3B-Instruct 160
- 9
Mistral-7B-Instruct-v0.2 49
Source: The CodeElo paper
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | o1-mini | 1,578 | Sep 2024 | Official leaderboard | |
| 2 | | 1,261 | Nov 2024 | Official leaderboard | |
| 3 | Claude 3.5 Sonnet | 710 | Jun 2024 | Official leaderboard | |
| 4 | ChatGPT-4o-latest-2024-11-20 closed weights | 668 | Official leaderboard | ||
| 5 | | 634 | Sep 2024 | Official leaderboard | |
| 6 | Mistral-Large-Instruct-2411 open weights | 631 | Official leaderboard | ||
| 7 | DS-V2.5 open weights | 629 | Official leaderboard | ||
| 8 | | 575 | Nov 2024 | Official leaderboard | |
| 9 | DS-Coder-V2-Instruct open weights | 532 | Official leaderboard | ||
| 10 | Qwen2.5-32B-Instruct open weights | 513 | Official leaderboard | ||
| 11 | Llama-3.1-70B-Instruct open weights | 478 | Official leaderboard | ||
| 12 | Qwen2.5-Coder-14B-Instruct | 424 | Nov 2024 | Official leaderboard | |
| 13 | Qwen2.5-14B-Instruct open weights | 414 | Official leaderboard | ||
| 14 | Qwen2.5-Coder-7B-Instruct open weights | 397 | Official leaderboard | ||
| 15 | Codestral-22B-v0.1 open weights | 385 | Official leaderboard | ||
| 16 | DS-V2-Chat open weights | 318 | Official leaderboard | ||
| 17 | Qwen2.5-7B-Instruct open weights | 315 | Official leaderboard | ||
| 18 | Yi-Coder-9B-Chat open weights | 296 | Official leaderboard | ||
| 19 | Mixtral-8x22B-Instruct-v0.1 open weights | 295 | Official leaderboard | ||
| 20 | DS-V2-Lite-Instruct open weights | 254 | Official leaderboard | ||
| 21 | Llama-3.1-8B-Instruct open weights | 223 | Official leaderboard | ||
| 22 | Ministral-8B-Instruct-2410 open weights | 219 | Official leaderboard | ||
| 23 | DS-Coder-33B-Instruct open weights | 207 | Official leaderboard | ||
| 24 | | 200 | Jan 2024 | Official leaderboard | |
| 25 | Qwen2.5-Coder-3B-Instruct | 160 | Nov 2024 | Official leaderboard | |
| 26 | DS-Coder-6.7B-Instruct open weights | 155 | Official leaderboard | ||
| 27 | OpenCoder-8B-Instruct open weights | 152 | Official leaderboard | ||
| 28 | Starcoder2-15B-Instruct-v0.1 open weights | 129 | Official leaderboard | ||
| 29 | Mixtral-8x7B-Instruct-v0.1 open weights | 98 | Official leaderboard | ||
| 30 | Qwen2.5-Coder-1.5B-Instruct open weights | 93 | Official leaderboard | ||
| 31 | DS-V2-Lite-Chat open weights | 60 | Official leaderboard | ||
| 32 | | 49 | Dec 2023 | Official leaderboard | |
| 33 | DS-Coder-1.3B-Instruct open weights | 21 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | o1-mini | 1,578 | Sep 2024 | Official leaderboard | |
| 2 | | 1,261 | Nov 2024 | Official leaderboard | |
| 3 | Claude 3.5 Sonnet | 710 | Jun 2024 | Official leaderboard | |
| 4 | | 634 | Sep 2024 | Official leaderboard | |
| 5 | | 575 | Nov 2024 | Official leaderboard | |
| 6 | Qwen2.5-Coder-14B-Instruct | 424 | Nov 2024 | Official leaderboard | |
| 7 | | 200 | Jan 2024 | Official leaderboard | |
| 8 | Qwen2.5-Coder-3B-Instruct | 160 | Nov 2024 | Official leaderboard | |
| 9 | | 49 | Dec 2023 | Official leaderboard |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: The CodeElo paper
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official CodeElo leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Coding
- SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
- LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.
DeepSeek-V4-Pro 93.5%
- SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
- DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
- SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases.
Seed2.1 Pro 59.8%
- Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance.
DeepSeek V4.1 Flash 3,471
Data audit
8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in November 2024, for QwQ-32B-Preview. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
- Pass Independent runs Scores from the benchmark’s paper (33 entries).
- Pass Models scored 9 models we track, plus 24 other entries on its leaderboard.
- Warning Newest score The newest came 2 years ago, in November 2024, for QwQ-32B-Preview.
- Note Frontier models None of the top 10 has a score. The strongest tested is o1-mini (#142).
- Note Headroom Scored as an Elo rating, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on arxiv.org: 9 rows for models we track and 24 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.