xCodeEval
Models understand, generate, translate, and retrieve contest code across seven tasks and up to 17 programming languages. Generated programs run against test cases through ExecEval.
The corpus contains about 7,500 unique problems. Each task has its own data split and metric.
- Made by
- Islamic University of Technology, Nanyang Technological University, Qatar Computing Research Institute and others
- Released
- March 2023
- Kind
- Competitive programming
- Level
- Competition
- Scored
- Checked automatically
- Answers
- Code
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Exams and contests
- Refresh
- One fixed set
- Run by
- Anyone
- Strongest model tested
-
StarCoder 2 3B #252 on the capability index
- Last model added
- February 2024
- Scores refreshed
- November 20th, 2026
In the authors’ words
Recently, pre-trained large language models (LLMs) have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. However, the evaluation of these models has often been performed in a scattered way on only one or two specific tasks, in a few languages, at a partial granularity (e.g., function) level, and in many cases without proper training data. Even more concerning is that in most cases the evaluation of generated codes has been done in terms of mere lexical overlap with a reference code rather than actual execution. We introduce xCodeEval, the largest executable multilingual multitask benchmark to date consisting of 25M document-level coding examples (16.5B tokens) from about 7.5K unique problems covering up to 11 programming languages with execution-level parallelism. It features a total of 7 tasks involving code understanding, generation, translation and retrieval. xCodeEval adopts an execution-based evaluation and offers a multilingual code execution engine, ExecEval that supports unit test based execution in all the 11 languages. To address the challenge of balancing the distributions of text-code samples over multiple attributes in validation/test sets, we propose a novel data splitting and a data selection schema based on the geometric mean and graph-theoretic principle. Our experiments with OpenAI's LLMs (zero-shot) and open-LLMs (zero-shot and fine-tuned) on the tasks and languages demonstrate **xCodeEval** to be quite challenging as per the current advancements in language models.
Code Llama 34B Instruct leads with 3.81%
Best score per model from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. The paper compares models on program synthesis using average pass@5 on the Compact split. Where a model ran with several settings or harnesses, we keep its best.
- 1
Code Llama 34B Instruct Zero-shot 3.81%
- 2
StarCoder 2 3B Fine-tuned 2.25%
Source: The xCodeEval paper
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 3.81% | Zero-shot | Aug 2023 | Official leaderboard |
| 2 | | 2.25% | Fine-tuned | Feb 2024 | Official leaderboard |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: The xCodeEval paper
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official xCodeEval leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Coding
- SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
- LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.
DeepSeek-V4-Pro 93.5%
- SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
- DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
- SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases.
Seed2.1 Pro 59.8%
- Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance.
DeepSeek V4.1 Flash 3,471
Data audit
8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in February 2024, for StarCoder 2 3B. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
- Pass Independent runs Scores from the benchmark’s paper (2 entries).
- Note Models scored Only 2 models we track.
- Warning Newest score The newest came 2 years ago, in February 2024, for StarCoder 2 3B.
- Note Frontier models None of the top 10 has a score. The strongest tested is StarCoder 2 3B (#252).
- Pass Headroom The best is 3.81%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on aclanthology.org: 3 rows for models we track at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.