Benchmark Almanac

VerilogEval

VFlow 83.5% tops its leaderboard o1-preview 77.1% leads the models we track
Not in model cards

Models complete Verilog modules from human or machine descriptions. A simulator checks their outputs against the reference behavior.

RTLCoder ran both suites. The original release contains 156 human descriptions and 143 machine descriptions.

Made by
NVIDIA
Released
September 2023
Kind
Electronics
Level
University
Scored
Checked automatically
Answers
Code
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Code → Code
Interaction
Single turn
Items from
Generated
Refresh
One fixed set
Run by
Anyone
Based on
HDLBits
Strongest model tested
o1-preview #149 on the capability index
Last model added
September 2024
Scores refreshed
November 20th, 2026
83
Entries on Chip Design LLM Zoo’s leaderboard, 2 models we track

VFlow tops the leaderboard with 83.5% o1-preview leads with 77.1%

Best score per entry from Chip Design LLM Zoo’s leaderboard, read on November 20th, 2026. The leaderboard reports no overall v1 pass@1, so the headline is the plain mean of the machine and human suite scores. Where a model ran on its own with several settings, we keep its best. Agents other companies built around a model get a pale shade of that model's lab color. Gray marks blends and entries not in our model list, such as fine-tunes and older models. None of these count toward leaders. The best model we track is o1-preview, at 77.1%. Agents built around a model, such as VeriMind + Code Llama 70B Instruct, are their own entries. We don't count them as the model's score. This view shows only models we track, each at its best.

83 entries. Scroll sideways for the rest.

  1. 83.5 VFlow
  2. 81.8 ScaleRTL†-32B
  3. 78.2 ScaleRTL-32B
  4. 77.1 o1-preview
  5. 73.2 ChipSeek-R1
  6. 68.0 PyraNet-DeepSeek
  7. 65.6 Claude3.5-Sonnet (snapshot unspecified)
  8. 65.4 CodeV-CodeQwen
  9. 65.3 CodeV-DeepSeek
  10. 64.3 OriGen-DeepSeek-Coder-7B-Instruct
  11. 61.6 CodeV-CodeLlama
  12. 59.0 GPT-4 (snapshot unspecified)
  13. 58.9 VeriReason-Qwen2.5-7B
  14. 58.6 AutoVCoder-CodeQwen
  15. 58.4 AutoVCoder-CodeQwen-ER
  16. 58.0 AutoVCoder-DeepSeek
  17. 57.6 ITERTL-DeepSeekV2-7k
  18. 57.5 AutoVCoder-CodeQwen-KR
  19. 57.4 AutoVCoder-DeepSeek-ER
  20. 57.1 BetterV-CodeQwen
  21. 56.9 BetterV-DeepSeek
  22. 56.5 AutoVCoder-CodeQwen-1&2
  23. 56.2 AutoVCoder-DeepSeek-KR
  24. 56.1 AutoVCoder-DeepSeek-1&2
  25. 55.9 Deep-Seek-Coder-V2-Instruct
  26. 55.5 AutoVCoder-DeepSeek-2
  27. 55.4 AutoVCoder-CodeQwen-2
  28. 54.1 AutoVCoder-Codellama
  29. 54.1 CodeGen-16B MEV-LLM
  30. 53.8 AutoVCoder-Codellama-ER
  31. 53.1 ITERTL-DeepSeek-22k
  32. 52.7 Claude-3
  33. 52.5 BetterV-CodeLlama
  34. 52.1 GPT-4o-mini (snapshot unspecified)
  35. 51.9 ITERTL-DeepSeek-10k
  36. 51.4 RTLCoder-DeepSeek
  37. 51.3 ITERTL-DeepSeek-8k
  38. 50.9 AutoVCoder-Codellama-1&2
  39. 50.7 AutoVCoder-Codellama-KR
  40. 50.1 CodeGen-6B MEV-LLM
  41. 49.6 RTLCoder-Mistral
  42. 49.5 RTLCoder-DeepSeek-Direct
  43. 47.5 VeriMind + Code Llama 70B Instruct
  44. 47.1 AutoVCoder-Codellama-2
  45. 47.0 RTLCoder-DeepSeek-4bit
  46. 47.0 VeriCoder-Qwen2.5-14B-Instruct
  47. 46.6 RTLCoder-Mistral-4bit
  48. 46.6 RTLCoder-Mistral-Direct
  49. 46.0 AutoVCoder-DeepSeek-1
  50. 46.0 ChipSeek-SFT
  51. 46.0 RTLCoder-DeepSeek-10k
  52. 45.8 AutoVCoder-CodeQwen-1
  53. 44.9 GPT-3.5 (snapshot unspecified)
  54. 44.5 VeriReason-Qwen2.5-3B
  55. 44.1 RTLCoder-Mistral-10k
  56. 41.2 Deepseek-Coder-6.7b-Instruct
  57. 40.7 ChipNeMo-70B
  58. 40.1 CodeGen-16B fine-grained labels
  59. 39.9 GEMMA-7B MEV-LLM
  60. 39.4 VeriReason-codeLlama-7B
  61. 39.3 CodeGen-2B MEV-LLM
  62. 37.5 VerilogEval-Codegen-16B-verilog-sft
  63. 37.1 VeriGen-Codegen-16B
  64. 34.8 CodeGen-6B fine-grained labels
  65. 34.5 CodeQwen1.5-7B-Chat
  66. 34.1 VeriReason-Qwen2.5-1.5B
  67. 33.9 AutoVCoder-Codellama-1
  68. 33.3 VerilogEval-Codegen-6B-verilog-sft
  69. 32.9 ChipNeMo-13B
  70. 32.5 StarCoder
  71. 30.6 CodeLlama-7b-Instruct
  72. 29.5 VerilogEval-Codegen-2B-verilog-sft
  73. 29.4 GEMMA-2B MEV-LLM
  74. 27.1 VerilogEval-Codegen-16B-multi-sft
  75. 24.1 CodeGen-2B fine-grained labels
  76. 23.2 VerilogEval-Codegen-6B-multi-sft
  77. 21.7 VerilogEval-Codegen-2B-multi-sft
  78. 21.1 GEMMA-7B fine-grained labels
  79. 20.7 Mistral 7B v0.1
  80. 18.5 ChipNeMo-7B
  81. 14.3 GEMMA-2B fine-grained labels
  82. 6.70 VerilogEval-Codegen-350M-multi-sft
  83. 2.95 Codegen2
  1. 1 VFlow 83.5%
  2. 2 ScaleRTL†-32B 81.8%
  3. 3 ScaleRTL-32B 78.2%
  4. 4 o1-preview 77.1%
  5. 5 ChipSeek-R1 73.2%
  6. 6 PyraNet-DeepSeek 68%
  7. 7 Claude3.5-Sonnet (snapshot unspecified) 65.6%
  8. 8 CodeV-CodeQwen 65.4%
  9. 9 CodeV-DeepSeek 65.3%
  10. 10 OriGen-DeepSeek-Coder-7B-Instruct 64.3%
  11. 11 CodeV-CodeLlama 61.6%
  12. 12 GPT-4 (snapshot unspecified) 59%
  13. 13 VeriReason-Qwen2.5-7B 58.9%
  14. 14 AutoVCoder-CodeQwen 58.6%
  15. 15 AutoVCoder-CodeQwen-ER 58.4%
  16. 16 AutoVCoder-DeepSeek 58%
  17. 17 ITERTL-DeepSeekV2-7k 57.6%
  18. 18 AutoVCoder-CodeQwen-KR 57.5%
  19. 19 AutoVCoder-DeepSeek-ER 57.4%
  20. 20 BetterV-CodeQwen 57.1%
  21. 21 BetterV-DeepSeek 56.9%
  22. 22 AutoVCoder-CodeQwen-1&2 56.5%
  23. 23 AutoVCoder-DeepSeek-KR 56.2%
  24. 24 AutoVCoder-DeepSeek-1&2 56.1%
  25. 25 Deep-Seek-Coder-V2-Instruct 55.9%
  26. 26 AutoVCoder-DeepSeek-2 55.5%
  27. 27 AutoVCoder-CodeQwen-2 55.4%
  28. 28 AutoVCoder-Codellama 54.1%
  29. 29 CodeGen-16B MEV-LLM 54.1%
  30. 30 AutoVCoder-Codellama-ER 53.8%
  31. 31 ITERTL-DeepSeek-22k 53.1%
  32. 32 Claude-3 52.7%
  33. 33 BetterV-CodeLlama 52.5%
  34. 34 GPT-4o-mini (snapshot unspecified) 52.1%
  35. 35 ITERTL-DeepSeek-10k 51.9%
  36. 36 RTLCoder-DeepSeek 51.4%
  37. 37 ITERTL-DeepSeek-8k 51.3%
  38. 38 AutoVCoder-Codellama-1&2 50.9%
  39. 39 AutoVCoder-Codellama-KR 50.7%
  40. 40 CodeGen-6B MEV-LLM 50.1%
  41. 41 RTLCoder-Mistral 49.6%
  42. 42 RTLCoder-DeepSeek-Direct 49.5%
  43. 43 VeriMind + Code Llama 70B Instruct 47.5%
  44. 44 AutoVCoder-Codellama-2 47.1%
  45. 45 RTLCoder-DeepSeek-4bit 47%
  46. 46 VeriCoder-Qwen2.5-14B-Instruct 47%
  47. 47 RTLCoder-Mistral-4bit 46.6%
  48. 48 RTLCoder-Mistral-Direct 46.6%
  49. 49 AutoVCoder-DeepSeek-1 46%
  50. 50 ChipSeek-SFT 46%
  51. 51 RTLCoder-DeepSeek-10k 46%
  52. 52 AutoVCoder-CodeQwen-1 45.8%
  53. 53 GPT-3.5 (snapshot unspecified) 44.9%
  54. 54 VeriReason-Qwen2.5-3B 44.5%
  55. 55 RTLCoder-Mistral-10k 44.1%
  56. 56 Deepseek-Coder-6.7b-Instruct 41.2%
  57. 57 ChipNeMo-70B 40.7%
  58. 58 CodeGen-16B fine-grained labels 40.1%
  59. 59 GEMMA-7B MEV-LLM 39.9%
  60. 60 VeriReason-codeLlama-7B 39.4%
  61. 61 CodeGen-2B MEV-LLM 39.3%
  62. 62 VerilogEval-Codegen-16B-verilog-sft 37.5%
  63. 63 VeriGen-Codegen-16B 37.1%
  64. 64 CodeGen-6B fine-grained labels 34.8%
  65. 65 CodeQwen1.5-7B-Chat 34.5%
  66. 66 VeriReason-Qwen2.5-1.5B 34.1%
  67. 67 AutoVCoder-Codellama-1 33.9%
  68. 68 VerilogEval-Codegen-6B-verilog-sft 33.3%
  69. 69 ChipNeMo-13B 32.9%
  70. 70 StarCoder 32.5%
  71. 71 CodeLlama-7b-Instruct 30.6%
  72. 72 VerilogEval-Codegen-2B-verilog-sft 29.5%
  73. 73 GEMMA-2B MEV-LLM 29.4%
  74. 74 VerilogEval-Codegen-16B-multi-sft 27.1%
  75. 75 CodeGen-2B fine-grained labels 24.1%
  76. 76 VerilogEval-Codegen-6B-multi-sft 23.2%
  77. 77 VerilogEval-Codegen-2B-multi-sft 21.7%
  78. 78 GEMMA-7B fine-grained labels 21.1%
  79. 79 Mistral 7B v0.1 20.7%
  80. 80 ChipNeMo-7B 18.5%
  81. 81 GEMMA-2B fine-grained labels 14.3%
  82. 82 VerilogEval-Codegen-350M-multi-sft 6.70%
  83. 83 Codegen2 2.95%

Source: Chip Design LLM Zoo’s leaderboard

Source: Chip Design LLM Zoo’s leaderboard

#EntryScoreSettingDateLinkSource
1 VFlow 83.5% arxiv.org Official leaderboard
2 ScaleRTL†-32B 32B 81.8% arxiv.org Official leaderboard
3 ScaleRTL-32B 32B 78.2% arxiv.org Official leaderboard
4 o1-preview 77.1% Sep 2024 Official leaderboard
5 ChipSeek-R1 7B 73.2% arxiv.org Official leaderboard
6 PyraNet-DeepSeek 7B 68% VeriMind arxiv.org Official leaderboard
7 Claude3.5-Sonnet (snapshot unspecified) 65.6% VeriMind arxiv.org Official leaderboard
8 CodeV-CodeQwen 7B 65.4% arxiv.org Official leaderboard
9 CodeV-DeepSeek 6.7B 65.3% arxiv.org Official leaderboard
10 OriGen-DeepSeek-Coder-7B-Instruct 7B 64.3% arxiv.org Official leaderboard
11 CodeV-CodeLlama 7B 61.6% arxiv.org Official leaderboard
12 GPT-4 (snapshot unspecified) 59% VeriAssist arxiv.org Official leaderboard
13 VeriReason-Qwen2.5-7B 7B 58.9% arxiv.org Official leaderboard
14 AutoVCoder-CodeQwen 7B 58.6% arxiv.org Official leaderboard
15 AutoVCoder-CodeQwen-ER 7B 58.4% arxiv.org Official leaderboard
16 AutoVCoder-DeepSeek 6.7B 58% arxiv.org Official leaderboard
17 ITERTL-DeepSeekV2-7k 16B 57.6% arxiv.org Official leaderboard
18 AutoVCoder-CodeQwen-KR 7B 57.5% arxiv.org Official leaderboard
19 AutoVCoder-DeepSeek-ER 6.7B 57.4% arxiv.org Official leaderboard
20 BetterV-CodeQwen 7B 57.1% arxiv.org Official leaderboard
21 BetterV-DeepSeek 6.7B 56.9% arxiv.org Official leaderboard
22 AutoVCoder-CodeQwen-1&2 7B 56.5% arxiv.org Official leaderboard
23 AutoVCoder-DeepSeek-KR 6.7B 56.2% arxiv.org Official leaderboard
24 AutoVCoder-DeepSeek-1&2 6.7B 56.1% arxiv.org Official leaderboard
25 Deep-Seek-Coder-V2-Instruct 16B 55.9% VeriMind arxiv.org Official leaderboard
26 AutoVCoder-DeepSeek-2 6.7B 55.5% arxiv.org Official leaderboard
27 AutoVCoder-CodeQwen-2 7B 55.4% arxiv.org Official leaderboard
28 AutoVCoder-Codellama 7B 54.1% arxiv.org Official leaderboard
29 CodeGen-16B MEV-LLM 16B 54.1% arxiv.org Official leaderboard
30 AutoVCoder-Codellama-ER 7B 53.8% arxiv.org Official leaderboard
31 ITERTL-DeepSeek-22k 6.7B 53.1% arxiv.org Official leaderboard
32 Claude-3 52.7% VeriAssist arxiv.org Official leaderboard
33 BetterV-CodeLlama 7B 52.5% arxiv.org Official leaderboard
34 GPT-4o-mini (snapshot unspecified) 52.1% VeriMind arxiv.org Official leaderboard
35 ITERTL-DeepSeek-10k 6.7B 51.9% arxiv.org Official leaderboard
36 RTLCoder-DeepSeek 6.7B 51.4% arxiv.org Official leaderboard
37 ITERTL-DeepSeek-8k 6.7B 51.3% arxiv.org Official leaderboard
38 AutoVCoder-Codellama-1&2 7B 50.9% arxiv.org Official leaderboard
39 AutoVCoder-Codellama-KR 7B 50.7% arxiv.org Official leaderboard
40 CodeGen-6B MEV-LLM 6B 50.1% arxiv.org Official leaderboard
41 RTLCoder-Mistral 7B 49.6% arxiv.org Official leaderboard
42 RTLCoder-DeepSeek-Direct 6.7B 49.5% arxiv.org Official leaderboard
43 VeriMind + Code Llama 70B Instruct 47.5% Jan 2024 arxiv.org Official leaderboard
44 AutoVCoder-Codellama-2 7B 47.1% arxiv.org Official leaderboard
45 RTLCoder-DeepSeek-4bit 6.7B 47% arxiv.org Official leaderboard
46 VeriCoder-Qwen2.5-14B-Instruct 14B 47% arxiv.org Official leaderboard
47 RTLCoder-Mistral-4bit 7B 46.6% arxiv.org Official leaderboard
48 RTLCoder-Mistral-Direct 7B 46.6% arxiv.org Official leaderboard
49 AutoVCoder-DeepSeek-1 6.7B 46% arxiv.org Official leaderboard
50 ChipSeek-SFT 7B 46% arxiv.org Official leaderboard
51 RTLCoder-DeepSeek-10k 6.7B 46% arxiv.org Official leaderboard
52 AutoVCoder-CodeQwen-1 7B 45.8% arxiv.org Official leaderboard
53 GPT-3.5 (snapshot unspecified) 44.9% VeriAssist arxiv.org Official leaderboard
54 VeriReason-Qwen2.5-3B 3B 44.5% arxiv.org Official leaderboard
55 RTLCoder-Mistral-10k 7B 44.1% arxiv.org Official leaderboard
56 Deepseek-Coder-6.7b-Instruct 6.7B 41.2% arxiv.org Official leaderboard
57 ChipNeMo-70B 70B 40.7% arxiv.org Official leaderboard
58 CodeGen-16B fine-grained labels 16B 40.1% arxiv.org Official leaderboard
59 GEMMA-7B MEV-LLM 7B 39.9% arxiv.org Official leaderboard
60 VeriReason-codeLlama-7B 7B 39.4% arxiv.org Official leaderboard
61 CodeGen-2B MEV-LLM 2B 39.3% arxiv.org Official leaderboard
62 VerilogEval-Codegen-16B-verilog-sft 16B 37.5% ieeexplore.ieee.org Official leaderboard
63 VeriGen-Codegen-16B 16B 37.1% ieeexplore.ieee.org Official leaderboard
64 CodeGen-6B fine-grained labels 6B 34.8% arxiv.org Official leaderboard
65 CodeQwen1.5-7B-Chat 7B 34.5% arxiv.org Official leaderboard
66 VeriReason-Qwen2.5-1.5B 1.5B 34.1% arxiv.org Official leaderboard
67 AutoVCoder-Codellama-1 7B 33.9% arxiv.org Official leaderboard
68 VerilogEval-Codegen-6B-verilog-sft 6B 33.3% ieeexplore.ieee.org Official leaderboard
69 ChipNeMo-13B 13B 32.9% arxiv.org Official leaderboard
70 StarCoder 15B 32.5% arxiv.org Official leaderboard
71 CodeLlama-7b-Instruct 7B 30.6% arxiv.org Official leaderboard
72 VerilogEval-Codegen-2B-verilog-sft 2B 29.5% ieeexplore.ieee.org Official leaderboard
73 GEMMA-2B MEV-LLM 2B 29.4% arxiv.org Official leaderboard
74 VerilogEval-Codegen-16B-multi-sft 16B 27.1% ieeexplore.ieee.org Official leaderboard
75 CodeGen-2B fine-grained labels 2B 24.1% arxiv.org Official leaderboard
76 VerilogEval-Codegen-6B-multi-sft 6B 23.2% ieeexplore.ieee.org Official leaderboard
77 VerilogEval-Codegen-2B-multi-sft 2B 21.7% ieeexplore.ieee.org Official leaderboard
78 GEMMA-7B fine-grained labels 7B 21.1% arxiv.org Official leaderboard
79 Mistral 7B v0.1 20.7% Sep 2023 arxiv.org Official leaderboard
80 ChipNeMo-7B 7B 18.5% arxiv.org Official leaderboard
81 GEMMA-2B fine-grained labels 2B 14.3% arxiv.org Official leaderboard
82 VerilogEval-Codegen-350M-multi-sft 0.4B 6.70% ieeexplore.ieee.org Official leaderboard
83 Codegen2 16B 2.95% arxiv.org Official leaderboard
#ModelScoreSettingDateLinkSource
1 o1-preview 77.1% Sep 2024 Official leaderboard
2 Mistral 7B v0.1 20.7% Sep 2023 arxiv.org Official leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On Chip Design LLM Zoo’s leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) o1-preview Mistral 7B v0.1 0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) o1-preview Mistral 7B v0.1

Source: Chip Design LLM Zoo’s leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Engineering and CAD

All 112 in Engineering and CAD
  • EEBench In use 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. Gemini 4 Carbon 83.9%
  • 3DCodeBench In use Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. LongCat-2.5 55.4%
  • CADGenBench Only in system cards Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. Claude Fable 5 45.1%
  • PartBench Only in system cards Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. Grok 4.6 58.5%
  • Razavi-Bench Not in model cards Models explain what happens in analog circuits and why. A language model awards up to four points per answer against a reference and a rubric. Gemini 4 Carbon 98.3%
  • CircuitRubric Not in model cards Models write analog circuit netlists from design requests. A program compares transistor connections and relative device sizes with accepted reference circuits. Claude Opus 4.8 96%

Data audit

9 of 10 checks pass, 1 warning. Look first at the Newest score check. The newest came 2 years ago, in September 2024, for o1-preview. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 3 pass 1 note
  • Pass Independent runs Scores from the official leaderboard (83 entries).
  • Pass Models scored 2 models we track, plus 81 other entries on its leaderboard.
  • Warning Newest score The newest came 2 years ago, in September 2024, for o1-preview.
  • Note Frontier models None of the top 10 has a score. The strongest tested is o1-preview (#149).
  • Pass Headroom The best is 77.1%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on Chip Design LLM Zoo: 3 rows for models we track and 80 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check