RegexEval
Models turn natural-language descriptions into regular expressions. The dataset contains 762 patterns with matching and nonmatching examples. The hub port holds out 62 patterns.
In the 2024 paper, GPT-3.5 Turbo led exact match at 33.9% with original prompts. The authors also tested text-davinci-003, Phi-1.5, and T5.
- Made by
- Mohammed Latif Siddiq and colleagues
- Released
- November 2023
- Kind
- Function-level problems
- Level
- University
- Scored
- Checked automatically
- Answers
- Code
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
GPT-3.5-Turbo (snapshot unspecified) tops the leaderboard with 33.9%
Best score per entry from the benchmark's paper, read on November 21st, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 1 GPT-3.5-Turbo (snapshot unspecified) original 33.9%
- 2 Text-DaVinci-003 refined 23.4%
- 3 Pre-trained Phi-1.5 refined 1.80%
- 4 Fine-tuned T5 original 0%
Source: The RegexEval paper
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-3.5-Turbo (snapshot unspecified) | 33.9% | original | Official leaderboard | |
| 2 | Text-DaVinci-003 | 23.4% | refined | Official leaderboard | |
| 3 | Pre-trained Phi-1.5 | 1.80% | refined | Official leaderboard | |
| 4 | Fine-tuned T5 | 0% | original | Official leaderboard |
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Coding
- SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
- LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.
DeepSeek-V4-Pro 93.5%
- SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
- DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
- SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases.
Seed2.1 Pro 59.8%
- Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance.
DeepSeek V4.1 Flash 3,471
Data audit
All 7 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 pass 4 notes
- Pass Independent runs Scores from the benchmark’s paper (4 entries).
- Note Models scored 4 entries on its leaderboard, none a model we track.
- Note Newest score No scores yet.
- Note Frontier models No models scored yet.
- Note Headroom No scores yet.
Leaderboard import 3 pass
- Pass Importer We read the board on s2e-lab.github.io: 4 entries at the last read, none a model we track.
- Pass Last read November 21st, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.