VerilogEval
Models complete Verilog modules from human or machine descriptions. A simulator checks their outputs against the reference behavior.
RTLCoder ran both suites. The original release contains 156 human descriptions and 143 machine descriptions.
- Made by
- NVIDIA
- Released
- September 2023
- Kind
- Electronics
- Level
- University
- Scored
- Checked automatically
- Answers
- Code
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Generated
- Refresh
- One fixed set
- Run by
- Anyone
- Based on
- HDLBits
- Strongest model tested
- o1-preview #149 on the capability index
- Last model added
- September 2024
- Scores refreshed
- November 20th, 2026
VFlow tops the leaderboard with 83.5% o1-preview leads with 77.1%
Best score per entry from Chip Design LLM Zoo’s leaderboard, read on November 20th, 2026. The leaderboard reports no overall v1 pass@1, so the headline is the plain mean of the machine and human suite scores. Where a model ran on its own with several settings, we keep its best. Agents other companies built around a model get a pale shade of that model's lab color. Gray marks blends and entries not in our model list, such as fine-tunes and older models. None of these count toward leaders. The best model we track is o1-preview, at 77.1%. Agents built around a model, such as VeriMind + Code Llama 70B Instruct, are their own entries. We don't count them as the model's score. This view shows only models we track, each at its best.
- 83.5 VFlow
- 81.8 ScaleRTL†-32B
- 78.2 ScaleRTL-32B
- 77.1 o1-preview
- 73.2 ChipSeek-R1
- 68.0 PyraNet-DeepSeek
- 65.6 Claude3.5-Sonnet (snapshot unspecified)
- 65.4 CodeV-CodeQwen
- 65.3 CodeV-DeepSeek
- 64.3 OriGen-DeepSeek-Coder-7B-Instruct
- 61.6 CodeV-CodeLlama
- 59.0 GPT-4 (snapshot unspecified)
- 58.9 VeriReason-Qwen2.5-7B
- 58.6 AutoVCoder-CodeQwen
- 58.4 AutoVCoder-CodeQwen-ER
- 58.0 AutoVCoder-DeepSeek
- 57.6 ITERTL-DeepSeekV2-7k
- 57.5 AutoVCoder-CodeQwen-KR
- 57.4 AutoVCoder-DeepSeek-ER
- 57.1 BetterV-CodeQwen
- 56.9 BetterV-DeepSeek
- 56.5 AutoVCoder-CodeQwen-1&2
- 56.2 AutoVCoder-DeepSeek-KR
- 56.1 AutoVCoder-DeepSeek-1&2
- 55.9 Deep-Seek-Coder-V2-Instruct
- 55.5 AutoVCoder-DeepSeek-2
- 55.4 AutoVCoder-CodeQwen-2
- 54.1 AutoVCoder-Codellama
- 54.1 CodeGen-16B MEV-LLM
- 53.8 AutoVCoder-Codellama-ER
- 53.1 ITERTL-DeepSeek-22k
- 52.7 Claude-3
- 52.5 BetterV-CodeLlama
- 52.1 GPT-4o-mini (snapshot unspecified)
- 51.9 ITERTL-DeepSeek-10k
- 51.4 RTLCoder-DeepSeek
- 51.3 ITERTL-DeepSeek-8k
- 50.9 AutoVCoder-Codellama-1&2
- 50.7 AutoVCoder-Codellama-KR
- 50.1 CodeGen-6B MEV-LLM
- 49.6 RTLCoder-Mistral
- 49.5 RTLCoder-DeepSeek-Direct
- 47.5 VeriMind + Code Llama 70B Instruct
- 47.1 AutoVCoder-Codellama-2
- 47.0 RTLCoder-DeepSeek-4bit
- 47.0 VeriCoder-Qwen2.5-14B-Instruct
- 46.6 RTLCoder-Mistral-4bit
- 46.6 RTLCoder-Mistral-Direct
- 46.0 AutoVCoder-DeepSeek-1
- 46.0 ChipSeek-SFT
- 46.0 RTLCoder-DeepSeek-10k
- 45.8 AutoVCoder-CodeQwen-1
- 44.9 GPT-3.5 (snapshot unspecified)
- 44.5 VeriReason-Qwen2.5-3B
- 44.1 RTLCoder-Mistral-10k
- 41.2 Deepseek-Coder-6.7b-Instruct
- 40.7 ChipNeMo-70B
- 40.1 CodeGen-16B fine-grained labels
- 39.9 GEMMA-7B MEV-LLM
- 39.4 VeriReason-codeLlama-7B
- 39.3 CodeGen-2B MEV-LLM
- 37.5 VerilogEval-Codegen-16B-verilog-sft
- 37.1 VeriGen-Codegen-16B
- 34.8 CodeGen-6B fine-grained labels
- 34.5 CodeQwen1.5-7B-Chat
- 34.1 VeriReason-Qwen2.5-1.5B
- 33.9 AutoVCoder-Codellama-1
- 33.3 VerilogEval-Codegen-6B-verilog-sft
- 32.9 ChipNeMo-13B
- 32.5 StarCoder
- 30.6 CodeLlama-7b-Instruct
- 29.5 VerilogEval-Codegen-2B-verilog-sft
- 29.4 GEMMA-2B MEV-LLM
- 27.1 VerilogEval-Codegen-16B-multi-sft
- 24.1 CodeGen-2B fine-grained labels
- 23.2 VerilogEval-Codegen-6B-multi-sft
- 21.7 VerilogEval-Codegen-2B-multi-sft
- 21.1 GEMMA-7B fine-grained labels
- 20.7
Mistral 7B v0.1
- 18.5 ChipNeMo-7B
- 14.3 GEMMA-2B fine-grained labels
- 6.70 VerilogEval-Codegen-350M-multi-sft
- 2.95 Codegen2
- 1 VFlow 83.5%
- 2 ScaleRTL†-32B 81.8%
- 3 ScaleRTL-32B 78.2%
- 4 o1-preview 77.1%
- 5 ChipSeek-R1 73.2%
- 6 PyraNet-DeepSeek 68%
- 7 Claude3.5-Sonnet (snapshot unspecified) 65.6%
- 8 CodeV-CodeQwen 65.4%
- 9 CodeV-DeepSeek 65.3%
- 10 OriGen-DeepSeek-Coder-7B-Instruct 64.3%
- 11 CodeV-CodeLlama 61.6%
- 12 GPT-4 (snapshot unspecified) 59%
- 13 VeriReason-Qwen2.5-7B 58.9%
- 14 AutoVCoder-CodeQwen 58.6%
- 15 AutoVCoder-CodeQwen-ER 58.4%
- 16 AutoVCoder-DeepSeek 58%
- 17 ITERTL-DeepSeekV2-7k 57.6%
- 18 AutoVCoder-CodeQwen-KR 57.5%
- 19 AutoVCoder-DeepSeek-ER 57.4%
- 20 BetterV-CodeQwen 57.1%
- 21 BetterV-DeepSeek 56.9%
- 22 AutoVCoder-CodeQwen-1&2 56.5%
- 23 AutoVCoder-DeepSeek-KR 56.2%
- 24 AutoVCoder-DeepSeek-1&2 56.1%
- 25 Deep-Seek-Coder-V2-Instruct 55.9%
- 26 AutoVCoder-DeepSeek-2 55.5%
- 27 AutoVCoder-CodeQwen-2 55.4%
- 28 AutoVCoder-Codellama 54.1%
- 29 CodeGen-16B MEV-LLM 54.1%
- 30 AutoVCoder-Codellama-ER 53.8%
- 31 ITERTL-DeepSeek-22k 53.1%
- 32 Claude-3 52.7%
- 33 BetterV-CodeLlama 52.5%
- 34 GPT-4o-mini (snapshot unspecified) 52.1%
- 35 ITERTL-DeepSeek-10k 51.9%
- 36 RTLCoder-DeepSeek 51.4%
- 37 ITERTL-DeepSeek-8k 51.3%
- 38 AutoVCoder-Codellama-1&2 50.9%
- 39 AutoVCoder-Codellama-KR 50.7%
- 40 CodeGen-6B MEV-LLM 50.1%
- 41 RTLCoder-Mistral 49.6%
- 42 RTLCoder-DeepSeek-Direct 49.5%
- 43 VeriMind + Code Llama 70B Instruct 47.5%
- 44 AutoVCoder-Codellama-2 47.1%
- 45 RTLCoder-DeepSeek-4bit 47%
- 46 VeriCoder-Qwen2.5-14B-Instruct 47%
- 47 RTLCoder-Mistral-4bit 46.6%
- 48 RTLCoder-Mistral-Direct 46.6%
- 49 AutoVCoder-DeepSeek-1 46%
- 50 ChipSeek-SFT 46%
- 51 RTLCoder-DeepSeek-10k 46%
- 52 AutoVCoder-CodeQwen-1 45.8%
- 53 GPT-3.5 (snapshot unspecified) 44.9%
- 54 VeriReason-Qwen2.5-3B 44.5%
- 55 RTLCoder-Mistral-10k 44.1%
- 56 Deepseek-Coder-6.7b-Instruct 41.2%
- 57 ChipNeMo-70B 40.7%
- 58 CodeGen-16B fine-grained labels 40.1%
- 59 GEMMA-7B MEV-LLM 39.9%
- 60 VeriReason-codeLlama-7B 39.4%
- 61 CodeGen-2B MEV-LLM 39.3%
- 62 VerilogEval-Codegen-16B-verilog-sft 37.5%
- 63 VeriGen-Codegen-16B 37.1%
- 64 CodeGen-6B fine-grained labels 34.8%
- 65 CodeQwen1.5-7B-Chat 34.5%
- 66 VeriReason-Qwen2.5-1.5B 34.1%
- 67 AutoVCoder-Codellama-1 33.9%
- 68 VerilogEval-Codegen-6B-verilog-sft 33.3%
- 69 ChipNeMo-13B 32.9%
- 70 StarCoder 32.5%
- 71 CodeLlama-7b-Instruct 30.6%
- 72 VerilogEval-Codegen-2B-verilog-sft 29.5%
- 73 GEMMA-2B MEV-LLM 29.4%
- 74 VerilogEval-Codegen-16B-multi-sft 27.1%
- 75 CodeGen-2B fine-grained labels 24.1%
- 76 VerilogEval-Codegen-6B-multi-sft 23.2%
- 77 VerilogEval-Codegen-2B-multi-sft 21.7%
- 78 GEMMA-7B fine-grained labels 21.1%
- 79
Mistral 7B v0.1 20.7%
- 80 ChipNeMo-7B 18.5%
- 81 GEMMA-2B fine-grained labels 14.3%
- 82 VerilogEval-Codegen-350M-multi-sft 6.70%
- 83 Codegen2 2.95%
Source: Chip Design LLM Zoo’s leaderboard
- 1 o1-preview 77.1%
- 2
Mistral 7B v0.1 20.7%
Source: Chip Design LLM Zoo’s leaderboard
| # | Entry | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | VFlow | 83.5% | arxiv.org | Official leaderboard | ||
| 2 | ScaleRTL†-32B 32B | 81.8% | arxiv.org | Official leaderboard | ||
| 3 | ScaleRTL-32B 32B | 78.2% | arxiv.org | Official leaderboard | ||
| 4 | o1-preview | 77.1% | Sep 2024 | Official leaderboard | ||
| 5 | ChipSeek-R1 7B | 73.2% | arxiv.org | Official leaderboard | ||
| 6 | PyraNet-DeepSeek 7B | 68% | VeriMind | arxiv.org | Official leaderboard | |
| 7 | Claude3.5-Sonnet (snapshot unspecified) | 65.6% | VeriMind | arxiv.org | Official leaderboard | |
| 8 | CodeV-CodeQwen 7B | 65.4% | arxiv.org | Official leaderboard | ||
| 9 | CodeV-DeepSeek 6.7B | 65.3% | arxiv.org | Official leaderboard | ||
| 10 | OriGen-DeepSeek-Coder-7B-Instruct 7B | 64.3% | arxiv.org | Official leaderboard | ||
| 11 | CodeV-CodeLlama 7B | 61.6% | arxiv.org | Official leaderboard | ||
| 12 | GPT-4 (snapshot unspecified) | 59% | VeriAssist | arxiv.org | Official leaderboard | |
| 13 | VeriReason-Qwen2.5-7B 7B | 58.9% | arxiv.org | Official leaderboard | ||
| 14 | AutoVCoder-CodeQwen 7B | 58.6% | arxiv.org | Official leaderboard | ||
| 15 | AutoVCoder-CodeQwen-ER 7B | 58.4% | arxiv.org | Official leaderboard | ||
| 16 | AutoVCoder-DeepSeek 6.7B | 58% | arxiv.org | Official leaderboard | ||
| 17 | ITERTL-DeepSeekV2-7k 16B | 57.6% | arxiv.org | Official leaderboard | ||
| 18 | AutoVCoder-CodeQwen-KR 7B | 57.5% | arxiv.org | Official leaderboard | ||
| 19 | AutoVCoder-DeepSeek-ER 6.7B | 57.4% | arxiv.org | Official leaderboard | ||
| 20 | BetterV-CodeQwen 7B | 57.1% | arxiv.org | Official leaderboard | ||
| 21 | BetterV-DeepSeek 6.7B | 56.9% | arxiv.org | Official leaderboard | ||
| 22 | AutoVCoder-CodeQwen-1&2 7B | 56.5% | arxiv.org | Official leaderboard | ||
| 23 | AutoVCoder-DeepSeek-KR 6.7B | 56.2% | arxiv.org | Official leaderboard | ||
| 24 | AutoVCoder-DeepSeek-1&2 6.7B | 56.1% | arxiv.org | Official leaderboard | ||
| 25 | Deep-Seek-Coder-V2-Instruct 16B | 55.9% | VeriMind | arxiv.org | Official leaderboard | |
| 26 | AutoVCoder-DeepSeek-2 6.7B | 55.5% | arxiv.org | Official leaderboard | ||
| 27 | AutoVCoder-CodeQwen-2 7B | 55.4% | arxiv.org | Official leaderboard | ||
| 28 | AutoVCoder-Codellama 7B | 54.1% | arxiv.org | Official leaderboard | ||
| 29 | CodeGen-16B MEV-LLM 16B | 54.1% | arxiv.org | Official leaderboard | ||
| 30 | AutoVCoder-Codellama-ER 7B | 53.8% | arxiv.org | Official leaderboard | ||
| 31 | ITERTL-DeepSeek-22k 6.7B | 53.1% | arxiv.org | Official leaderboard | ||
| 32 | Claude-3 | 52.7% | VeriAssist | arxiv.org | Official leaderboard | |
| 33 | BetterV-CodeLlama 7B | 52.5% | arxiv.org | Official leaderboard | ||
| 34 | GPT-4o-mini (snapshot unspecified) | 52.1% | VeriMind | arxiv.org | Official leaderboard | |
| 35 | ITERTL-DeepSeek-10k 6.7B | 51.9% | arxiv.org | Official leaderboard | ||
| 36 | RTLCoder-DeepSeek 6.7B | 51.4% | arxiv.org | Official leaderboard | ||
| 37 | ITERTL-DeepSeek-8k 6.7B | 51.3% | arxiv.org | Official leaderboard | ||
| 38 | AutoVCoder-Codellama-1&2 7B | 50.9% | arxiv.org | Official leaderboard | ||
| 39 | AutoVCoder-Codellama-KR 7B | 50.7% | arxiv.org | Official leaderboard | ||
| 40 | CodeGen-6B MEV-LLM 6B | 50.1% | arxiv.org | Official leaderboard | ||
| 41 | RTLCoder-Mistral 7B | 49.6% | arxiv.org | Official leaderboard | ||
| 42 | RTLCoder-DeepSeek-Direct 6.7B | 49.5% | arxiv.org | Official leaderboard | ||
| 43 | VeriMind + Code Llama 70B Instruct | 47.5% | Jan 2024 | arxiv.org | Official leaderboard | |
| 44 | AutoVCoder-Codellama-2 7B | 47.1% | arxiv.org | Official leaderboard | ||
| 45 | RTLCoder-DeepSeek-4bit 6.7B | 47% | arxiv.org | Official leaderboard | ||
| 46 | VeriCoder-Qwen2.5-14B-Instruct 14B | 47% | arxiv.org | Official leaderboard | ||
| 47 | RTLCoder-Mistral-4bit 7B | 46.6% | arxiv.org | Official leaderboard | ||
| 48 | RTLCoder-Mistral-Direct 7B | 46.6% | arxiv.org | Official leaderboard | ||
| 49 | AutoVCoder-DeepSeek-1 6.7B | 46% | arxiv.org | Official leaderboard | ||
| 50 | ChipSeek-SFT 7B | 46% | arxiv.org | Official leaderboard | ||
| 51 | RTLCoder-DeepSeek-10k 6.7B | 46% | arxiv.org | Official leaderboard | ||
| 52 | AutoVCoder-CodeQwen-1 7B | 45.8% | arxiv.org | Official leaderboard | ||
| 53 | GPT-3.5 (snapshot unspecified) | 44.9% | VeriAssist | arxiv.org | Official leaderboard | |
| 54 | VeriReason-Qwen2.5-3B 3B | 44.5% | arxiv.org | Official leaderboard | ||
| 55 | RTLCoder-Mistral-10k 7B | 44.1% | arxiv.org | Official leaderboard | ||
| 56 | Deepseek-Coder-6.7b-Instruct 6.7B | 41.2% | arxiv.org | Official leaderboard | ||
| 57 | ChipNeMo-70B 70B | 40.7% | arxiv.org | Official leaderboard | ||
| 58 | CodeGen-16B fine-grained labels 16B | 40.1% | arxiv.org | Official leaderboard | ||
| 59 | GEMMA-7B MEV-LLM 7B | 39.9% | arxiv.org | Official leaderboard | ||
| 60 | VeriReason-codeLlama-7B 7B | 39.4% | arxiv.org | Official leaderboard | ||
| 61 | CodeGen-2B MEV-LLM 2B | 39.3% | arxiv.org | Official leaderboard | ||
| 62 | VerilogEval-Codegen-16B-verilog-sft 16B | 37.5% | ieeexplore.ieee.org | Official leaderboard | ||
| 63 | VeriGen-Codegen-16B 16B | 37.1% | ieeexplore.ieee.org | Official leaderboard | ||
| 64 | CodeGen-6B fine-grained labels 6B | 34.8% | arxiv.org | Official leaderboard | ||
| 65 | CodeQwen1.5-7B-Chat 7B | 34.5% | arxiv.org | Official leaderboard | ||
| 66 | VeriReason-Qwen2.5-1.5B 1.5B | 34.1% | arxiv.org | Official leaderboard | ||
| 67 | AutoVCoder-Codellama-1 7B | 33.9% | arxiv.org | Official leaderboard | ||
| 68 | VerilogEval-Codegen-6B-verilog-sft 6B | 33.3% | ieeexplore.ieee.org | Official leaderboard | ||
| 69 | ChipNeMo-13B 13B | 32.9% | arxiv.org | Official leaderboard | ||
| 70 | StarCoder 15B | 32.5% | arxiv.org | Official leaderboard | ||
| 71 | CodeLlama-7b-Instruct 7B | 30.6% | arxiv.org | Official leaderboard | ||
| 72 | VerilogEval-Codegen-2B-verilog-sft 2B | 29.5% | ieeexplore.ieee.org | Official leaderboard | ||
| 73 | GEMMA-2B MEV-LLM 2B | 29.4% | arxiv.org | Official leaderboard | ||
| 74 | VerilogEval-Codegen-16B-multi-sft 16B | 27.1% | ieeexplore.ieee.org | Official leaderboard | ||
| 75 | CodeGen-2B fine-grained labels 2B | 24.1% | arxiv.org | Official leaderboard | ||
| 76 | VerilogEval-Codegen-6B-multi-sft 6B | 23.2% | ieeexplore.ieee.org | Official leaderboard | ||
| 77 | VerilogEval-Codegen-2B-multi-sft 2B | 21.7% | ieeexplore.ieee.org | Official leaderboard | ||
| 78 | GEMMA-7B fine-grained labels 7B | 21.1% | arxiv.org | Official leaderboard | ||
| 79 | | 20.7% | Sep 2023 | arxiv.org | Official leaderboard | |
| 80 | ChipNeMo-7B 7B | 18.5% | arxiv.org | Official leaderboard | ||
| 81 | GEMMA-2B fine-grained labels 2B | 14.3% | arxiv.org | Official leaderboard | ||
| 82 | VerilogEval-Codegen-350M-multi-sft 0.4B | 6.70% | ieeexplore.ieee.org | Official leaderboard | ||
| 83 | Codegen2 16B | 2.95% | arxiv.org | Official leaderboard |
| # | Model | Score | Setting | Date | Link | Source |
|---|---|---|---|---|---|---|
| 1 | o1-preview | 77.1% | Sep 2024 | Official leaderboard | ||
| 2 | | 20.7% | Sep 2023 | arxiv.org | Official leaderboard |
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Chip Design LLM Zoo’s leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Engineering and CAD
- EEBench In use 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge.
Gemini 4 Carbon 83.9%
- 3DCodeBench In use Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity.
LongCat-2.5 55.4%
- CADGenBench Only in system cards Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. Claude Fable 5 45.1%
- PartBench Only in system cards Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. Grok 4.6 58.5%
- Razavi-Bench Not in model cards Models explain what happens in analog circuits and why. A language model awards up to four points per answer against a reference and a rubric.
Gemini 4 Carbon 98.3%
- CircuitRubric Not in model cards Models write analog circuit netlists from design requests. A program compares transistor connections and relative device sizes with accepted reference circuits. Claude Opus 4.8 96%
Data audit
9 of 10 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in September 2024, for o1-preview. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 1 warning 3 pass 1 note
- Pass Independent runs Scores from the official leaderboard (83 entries).
- Pass Models scored 2 models we track, plus 81 other entries on its leaderboard.
- Warning Newest score The newest came 2 years ago, in September 2024, for o1-preview.
- Note Frontier models None of the top 10 has a score. The strongest tested is o1-preview (#149).
- Pass Headroom The best is 77.1%, under 85%, so it still separates models.
Leaderboard import 3 pass
- Pass Importer We read the board on Chip Design LLM Zoo: 3 rows for models we track and 80 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.