Benchmark Almanac

CompileBench

Claude Opus 4.1 100% leads since August 2025
Not in model cards

Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code.

About 60 GitHub stars. Quesma also contributed to Terminal-Bench 3.0.

Made by
Quesma
Released
August 2025
Kind
Terminal tasks
Level
Expert
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
GPT-5 #70 on the capability index
Last model added
October 2025
Scores refreshed
November 12th, 2026
20
Models scored on its own leaderboard

Claude Opus 4.1 leads with 100% among models released by October 2025

Best score per model from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

The newest model with a score here came out in October 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

20 models. Scroll sideways for the rest.

  1. 100.0 Claude Opus 4.1
  2. 93.3 Claude Sonnet 4
  3. 93.3 GPT-5
  4. 93.3 GPT-5-Codex
  5. 86.7 Grok 4
  6. 86.7 Claude Sonnet 4.5
  7. 86.7 Claude Haiku 4.5
  8. 80.0 GPT-5 mini
  9. 80.0 DeepSeek-V3.1
  10. 80.0 Kimi K2
  11. 73.3 GLM-4.5
  12. 73.3 grok-code-fast-1
  13. 73.3 Grok 4 Fast
  14. 73.3 DeepSeek-V3.1-Terminus
  15. 66.7 GPT-4.1 mini
  16. 66.7 Qwen3-Max
  17. 60.0 GPT-4.1
  18. 60.0 Gemini 2.5 Pro
  19. 60.0 gpt-oss-120b
  20. 60.0 Gemini 2.5 Flash
  1. 1 Claude Opus 4.1 thinking 16k 100%
  2. 2 Claude Sonnet 4 thinking 16k 93.3%
  3. 3 GPT-5 high 93.3%
  4. 4 GPT-5-Codex high 93.3%
  5. 5 Grok 4 86.7%
  6. 6 Claude Sonnet 4.5 86.7%
  7. 7 Claude Haiku 4.5 thinking 16k 86.7%
  8. 8 GPT-5 mini high 80%
  9. 9 DeepSeek-V3.1 80%
  10. 10 Kimi K2 Sep 2025 80%
  11. 11 GLM-4.5 73.3%
  12. 12 grok-code-fast-1 73.3%
  13. 13 Grok 4 Fast 73.3%
  14. 14 DeepSeek-V3.1-Terminus 73.3%
  15. 15 GPT-4.1 mini 66.7%
  16. 16 Qwen3-Max 66.7%
  17. 17 GPT-4.1 60%
  18. 18 Gemini 2.5 Pro Jun 2025 60%
  19. 19 gpt-oss-120b high 60%
  20. 20 Gemini 2.5 Flash Sep 2025 60%

Source: Official CompileBench leaderboard

#ModelScoreSettingDateSource
1 Claude Opus 4.1 100% thinking 16k Aug 2025 Official leaderboard
2 Claude Sonnet 4 93.3% thinking 16k May 2025 Official leaderboard
3 GPT-5 93.3% high Aug 2025 Official leaderboard
4 GPT-5-Codex 93.3% high Sep 2025 Official leaderboard
5 Grok 4 86.7% Jul 2025 Official leaderboard
6 Claude Sonnet 4.5 86.7% Sep 2025 Official leaderboard
7 Claude Haiku 4.5 86.7% thinking 16k Oct 2025 Official leaderboard
8 GPT-5 mini 80% high Aug 2025 Official leaderboard
9 DeepSeek-V3.1 80% Aug 2025 Official leaderboard
10 Kimi K2 Sep 2025 80% Sep 2025 Official leaderboard
11 GLM-4.5 73.3% Aug 2025 Official leaderboard
12 grok-code-fast-1 73.3% Aug 2025 Official leaderboard
13 Grok 4 Fast 73.3% Sep 2025 Official leaderboard
14 DeepSeek-V3.1-Terminus 73.3% Sep 2025 Official leaderboard
15 GPT-4.1 mini 66.7% Apr 2025 Official leaderboard
16 Qwen3-Max 66.7% Sep 2025 Official leaderboard
17 GPT-4.1 60% Apr 2025 Official leaderboard
18 Gemini 2.5 Pro Jun 2025 60% Jun 2025 Official leaderboard
19 gpt-oss-120b 60% high Aug 2025 Official leaderboard
20 Gemini 2.5 Flash Sep 2025 60% Sep 2025 Official leaderboard

CompileBench Speed

Share of tasks passed within three attempts against the total time across tasks, with the Pareto frontier. The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

40% 60% 80% 100% 1001,00010K Total time across tasks (s), log scale, lower is better Tasks passed (pass@3) GPT-4.1 mini: Tasks passed (pass@3) 66.7%, total time across tasks 1,415 s Claude Sonnet 4: Tasks passed (pass@3) 80%, total time across tasks 1,409 s Gemini 2.5 Pro: Tasks passed (pass@3) 60%, total time across tasks 942 s Grok 4: Tasks passed (pass@3) 86.7%, total time across tasks 3,194 s GLM-4.5: Tasks passed (pass@3) 73.3%, total time across tasks 1,264 s gpt-oss-120b (high): Tasks passed (pass@3) 60%, total time across tasks 673 s GPT-5 (high): Tasks passed (pass@3) 93.3%, total time across tasks 3,431 s GPT-5 mini (high): Tasks passed (pass@3) 80%, total time across tasks 2,410 s GPT-5 mini (minimal): Tasks passed (pass@3) 46.7%, total time across tasks 527 s DeepSeek-V3.1: Tasks passed (pass@3) 80%, total time across tasks 1,095 s grok-code-fast-1: Tasks passed (pass@3) 73.3%, total time across tasks 1,238 s Kimi K2: Tasks passed (pass@3) 80%, total time across tasks 1,153 s GPT-5-Codex (high): Tasks passed (pass@3) 93.3%, total time across tasks 3,211 s Grok 4 Fast: Tasks passed (pass@3) 73.3%, total time across tasks 1,658 s DeepSeek-V3.1-Terminus: Tasks passed (pass@3) 73.3%, total time across tasks 1,449 s Gemini 2.5 Flash: Tasks passed (pass@3) 60%, total time across tasks 648 s Gemini 2.5 Flash (thinking): Tasks passed (pass@3) 53.3%, total time across tasks 693 s Claude Sonnet 4.5: Tasks passed (pass@3) 86.7%, total time across tasks 1,754 s Claude Sonnet 4.5 (thinking 16k): Tasks passed (pass@3) 86.7%, total time across tasks 2,210 s Claude Haiku 4.5 (thinking 16k): Tasks passed (pass@3) 86.7%, total time across tasks 2,882 s GPT-4.1: Tasks passed (pass@3) 60%, total time across tasks 490 s Claude Sonnet 4 (thinking 16k): Tasks passed (pass@3) 93.3%, total time across tasks 2,740 s Claude Opus 4.1 (thinking 16k): Tasks passed (pass@3) 100%, total time across tasks 3,528 s GPT-5 (minimal): Tasks passed (pass@3) 80%, total time across tasks 1,036 s Qwen3-Max: Tasks passed (pass@3) 66.7%, total time across tasks 678 s Claude Haiku 4.5: Tasks passed (pass@3) 86.7%, total time across tasks 1,597 s Claude Opus 4.1 (thinking 16k)Claude Sonnet 4 (thinking 16k)Claude Haiku 4.5GPT-5 (minimal)Qwen3-MaxGPT-4.1 40% 60% 80% 100% 1001,00010K Total time across tasks (s), log scale, lower is better Tasks passed (pass@3) GPT-4.1 mini: Tasks passed (pass@3) 66.7%, total time across tasks 1,415 s Claude Sonnet 4: Tasks passed (pass@3) 80%, total time across tasks 1,409 s Gemini 2.5 Pro: Tasks passed (pass@3) 60%, total time across tasks 942 s Grok 4: Tasks passed (pass@3) 86.7%, total time across tasks 3,194 s GLM-4.5: Tasks passed (pass@3) 73.3%, total time across tasks 1,264 s gpt-oss-120b (high): Tasks passed (pass@3) 60%, total time across tasks 673 s GPT-5 (high): Tasks passed (pass@3) 93.3%, total time across tasks 3,431 s GPT-5 mini (high): Tasks passed (pass@3) 80%, total time across tasks 2,410 s GPT-5 mini (minimal): Tasks passed (pass@3) 46.7%, total time across tasks 527 s DeepSeek-V3.1: Tasks passed (pass@3) 80%, total time across tasks 1,095 s grok-code-fast-1: Tasks passed (pass@3) 73.3%, total time across tasks 1,238 s Kimi K2: Tasks passed (pass@3) 80%, total time across tasks 1,153 s GPT-5-Codex (high): Tasks passed (pass@3) 93.3%, total time across tasks 3,211 s Grok 4 Fast: Tasks passed (pass@3) 73.3%, total time across tasks 1,658 s DeepSeek-V3.1-Terminus: Tasks passed (pass@3) 73.3%, total time across tasks 1,449 s Gemini 2.5 Flash: Tasks passed (pass@3) 60%, total time across tasks 648 s Gemini 2.5 Flash (thinking): Tasks passed (pass@3) 53.3%, total time across tasks 693 s Claude Sonnet 4.5: Tasks passed (pass@3) 86.7%, total time across tasks 1,754 s Claude Sonnet 4.5 (thinking 16k): Tasks passed (pass@3) 86.7%, total time across tasks 2,210 s Claude Haiku 4.5 (thinking 16k): Tasks passed (pass@3) 86.7%, total time across tasks 2,882 s GPT-4.1: Tasks passed (pass@3) 60%, total time across tasks 490 s Claude Sonnet 4 (thinking 16k): Tasks passed (pass@3) 93.3%, total time across tasks 2,740 s Claude Opus 4.1 (thinking 16k): Tasks passed (pass@3) 100%, total time across tasks 3,528 s GPT-5 (minimal): Tasks passed (pass@3) 80%, total time across tasks 1,036 s Qwen3-Max: Tasks passed (pass@3) 66.7%, total time across tasks 678 s Claude Haiku 4.5: Tasks passed (pass@3) 86.7%, total time across tasks 1,597 s Claude Opus 4.1 (thinking 16k)Claude Sonnet 4 (thinking 16k)Claude Haiku 4.5GPT-5 (minimal)Qwen3-MaxGPT-4.1
ModelSettingTasks passed (pass@3)Total time across tasks
Claude Opus 4.1 thinking 16k 100% 3,528 s
Claude Sonnet 4 thinking 16k 93.3% 2,740 s
GPT-5-Codex high 93.3% 3,211 s
GPT-5 high 93.3% 3,431 s
Claude Haiku 4.5 86.7% 1,597 s
Claude Sonnet 4.5 86.7% 1,754 s
Claude Sonnet 4.5 thinking 16k 86.7% 2,210 s
Claude Haiku 4.5 thinking 16k 86.7% 2,882 s
Grok 4 86.7% 3,194 s
GPT-5 minimal 80% 1,036 s
DeepSeek-V3.1 80% 1,095 s
Kimi K2 Sep 2025 80% 1,153 s
Claude Sonnet 4 80% 1,409 s
GPT-5 mini high 80% 2,410 s
grok-code-fast-1 73.3% 1,238 s
GLM-4.5 73.3% 1,264 s
DeepSeek-V3.1-Terminus 73.3% 1,449 s
Grok 4 Fast 73.3% 1,658 s
Qwen3-Max 66.7% 678.05 s
GPT-4.1 mini 66.7% 1,415 s
GPT-4.1 60% 489.63 s
Gemini 2.5 Flash Sep 2025 60% 647.88 s
gpt-oss-120b high 60% 672.51 s
Gemini 2.5 Pro Jun 2025 60% 941.85 s
Gemini 2.5 Flash Sep 2025 thinking 53.3% 693.42 s
GPT-5 mini minimal 46.7% 526.59 s

The best score, 100%, costs $22 for the full run

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Anthropic OpenAI Other labs Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 $100 Cost of the full run, log scale Claude Opus 4.1, thinking 16k GPT-5, high Claude Haiku 4.5 GPT-5 mini, high grok-code-fast-1 GPT-5 mini, minimal 0% 20% 40% 60% 80% 100% $0.01 $0.10 $1.00 $10 $100 Cost of the full run, log scale Claude Opus 4.1, thinking 16k GPT-5, high

Source: Official CompileBench leaderboard

ModelEffortHarnessScoreCost of the full runOutput tokens
Claude Opus 4.1 thinking 16k 100% $22
Claude Sonnet 4 thinking 16k 93.3% $5.75
GPT-5 high 93.3% $1.99
GPT-5-Codex high 93.3% $3.29
Grok 4 86.7% $3.11
Claude Sonnet 4.5 86.7% $3.61
Claude Sonnet 4.5 thinking 16k 86.7% $3.79
Claude Haiku 4.5 thinking 16k 86.7% $1.85
Claude Haiku 4.5 86.7% $1.08
Claude Sonnet 4 80% $2.65
GPT-5 minimal 80% $0.40
GPT-5 mini high 80% $0.27
DeepSeek-V3.1 80% $1.57
Kimi K2 Sep 2025 80% $2.75
GLM-4.5 73.3% $1.05
grok-code-fast-1 73.3% $0.10
Grok 4 Fast 73.3% $0.15
DeepSeek-V3.1-Terminus 73.3% $0.67
GPT-4.1 mini 66.7% $0.13
Qwen3-Max 66.7% $0.59
GPT-4.1 60% $0.47
Gemini 2.5 Pro Jun 2025 60% $0.58
gpt-oss-120b high 60% $0.22
Gemini 2.5 Flash Sep 2025 60% $0.10
Gemini 2.5 Flash Sep 2025 thinking 53.3% $0.23
GPT-5 mini minimal 46.7% $0.03

2 models have held number one on the official leaderboard since its release in August 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Passed 85% in Aug 2025, by Claude Sonnet 4
Claude Opus 4.1 100%

Source: Official CompileBench leaderboard

Claude Sonnet 4 on the official leaderboard passed 85% in May 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude Opus 4.1 Claude Sonnet 4 GPT-4.1 mini 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude Opus 4.1 Claude Sonnet 4

Source: Official CompileBench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Terminal agents

All 19 in Terminal agents
  • SkillsBench In use 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. GPT-5.5 67.3%
  • Terminal-Bench Hard In use Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. Gemma 4 31B IT 36%
  • Harbor-Index In use Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. GPT-5.5 28.1%
  • Terminal-Bench 2.1 Extended Not in model cards Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores. Muse Wonder 54%
  • Terminal-Bench 4.1 Not in model cards The Terminal-Bench 4.0 tasks with hardened verifiers: agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Claude Opus 5.5 65.8%
  • o11y-bench Not in model cards Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment. Claude Opus 5.5 93.8%