o11y-bench
Claude Opus 5.5 93.8% leads since September 2026
Not in model cards Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment.
64 tasks built on Harbor, scored pass^3. Version 1.1 (October 28th, 2026) made grading more grounded and reproducible, so check comparability with 1.0 scores.
- Made by
- Grafana Labs
- Released
- April 2026
- Kind
- Terminal tasks
- Level
- Expert
- Scored
- Checked automatically
- Answers
- Agent task
- Tasks
- All public
- Strongest model tested
- GPT-6 Astra #3 on the capability index
- Last model added
- October 2026
- Scores refreshed
- November 12th, 2026
23
Entries on its own leaderboard, 22 models we track
Claude Opus 5.5 leads with 93.8%
Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 93.8 Claude Opus 5.5
- 90.6 Claude Sonnet 5.5
- 87.5 GPT-6 Astra
- 87.5 GPT-6.1 Sol
- 87.5
Gemini 4 Argon - 87.5 Claude Haiku 5.5
- 85.9
Muse Spark Open - 84.4 GPT-6 Luna
- 81.3 Kimi K3
- 81.3 GLM-5.3
- 81.3
DeepSeek V4.1 Flash 1017 - 81.3 mistral-large-4-0
- 78.1
Qwen 3.8 Max - 76.6 Grok 4.7
- 65.6 GPT-5.6 Luna
- 64.1 Qwen 3.8 27B
- 64.1
MiniMax-M3.1-Flash - 56.3 Gemma 4 31B IT
- 54.7 GPT-5.4 Mini
- 51.6 Qwen3.6 27B
- 43.8 Qwen 3.6 35B-A3B
- 26.6 Claude Haiku 4.5
- 10.9 GPT-5.4 Nano
- 1 Claude Opus 5.5 medium 93.8%
- 2 Claude Sonnet 5.5 xhigh 90.6%
- 3 GPT-6 Astra low 87.5%
- 4 GPT-6.1 Sol medium 87.5%
- 5 Gemini 4 Argon low 87.5%
- 6 Claude Haiku 5.5 xhigh 87.5%
- 7 Muse Spark Open high 85.9%
- 8 GPT-6 Luna xhigh 84.4%
- 9 Kimi K3 max 81.3%
- 10 GLM-5.3 max 81.3%
- 11 DeepSeek V4.1 Flash 1017 max 81.3%
- 12 mistral-large-4-0 high 81.3%
- 13 Qwen 3.8 Max xhigh 78.1%
- 14 Grok 4.7 high 76.6%
- 15 GPT-5.6 Luna low 65.6%
- 16 Qwen 3.8 27B low 64.1%
- 17 MiniMax-M3.1-Flash high 64.1%
- 18 Gemma 4 31B IT low 56.3%
- 19 GPT-5.4 Mini low 54.7%
- 20 Qwen3.6 27B high 51.6%
- 21 Qwen 3.6 35B-A3B high 43.8%
- 22 Claude Haiku 4.5 off 26.6%
- 23 GPT-5.4 Nano off 10.9%
Source: Official o11y-bench leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | 93.8% | medium | Sep 2026 | Official leaderboard |
| 2 | Claude Sonnet 5.5 | 90.6% | xhigh | Sep 2026 | Official leaderboard |
| 3 | GPT-6 Astra | 87.5% | low | Sep 2026 | Official leaderboard |
| 4 | GPT-6.1 Sol | 87.5% | medium | Sep 2026 | Official leaderboard |
| 5 | Gemini 4 Argon | 87.5% | low | Sep 2026 | Official leaderboard |
| 6 | Claude Haiku 5.5 | 87.5% | xhigh | Oct 2026 | Official leaderboard |
| 7 | Muse Spark Open | 85.9% | high | Oct 2026 | Official leaderboard |
| 8 | GPT-6 Luna | 84.4% | xhigh | Sep 2026 | Official leaderboard |
| 9 | Kimi K3 | 81.3% | max | Jul 2026 | Official leaderboard |
| 10 | GLM-5.3 | 81.3% | max | Aug 2026 | Official leaderboard |
| 11 | DeepSeek V4.1 Flash 1017 | 81.3% | max | Oct 2026 | Official leaderboard |
| 12 | mistral-large-4-0 | 81.3% | high | Oct 2026 | Official leaderboard |
| 13 | Qwen 3.8 Max | 78.1% | xhigh | Aug 2026 | Official leaderboard |
| 14 | Grok 4.7 | 76.6% | high | Sep 2026 | Official leaderboard |
| 15 | GPT-5.6 Luna | 65.6% | low | Jul 2026 | Official leaderboard |
| 16 | Qwen 3.8 27B | 64.1% | low | Aug 2026 | Official leaderboard |
| 17 | MiniMax-M3.1-Flash | 64.1% | high | Oct 2026 | Official leaderboard |
| 18 | Gemma 4 31B IT | 56.3% | low | Apr 2026 | Official leaderboard |
| 19 | GPT-5.4 Mini | 54.7% | low | Mar 2026 | Official leaderboard |
| 20 | Qwen3.6 27B | 51.6% | high | Apr 2026 | Official leaderboard |
| 21 | Qwen 3.6 35B-A3B | 43.8% | high | Apr 2026 | Official leaderboard |
| 22 | Claude Haiku 4.5 | 26.6% | off | Oct 2025 | Official leaderboard |
| 23 | GPT-5.4 Nano | 10.9% | off | Mar 2026 | Official leaderboard |
Pass^3 vs. cost
The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.
| Model | Setting | Pass^3 | Cost per trial |
|---|---|---|---|
| Claude Opus 5.5 | medium | 93.8% | $0.26 |
| Claude Sonnet 5.5 | xhigh | 90.6% | $0.29 |
| Claude Sonnet 5.5 | high | 89.1% | $0.11 |
| GPT-6.1 Sol | medium | 87.5% | $0.04 |
| Claude Haiku 5.5 | xhigh | 87.5% | $0.06 |
| GPT-6 Astra | low | 87.5% | $0.2 |
| Gemini 4 Argon | low | 87.5% | $0.27 |
| Claude Opus 5.5 | max | 87.5% | $1.69 |
| Claude Haiku 5.5 | high | 85.9% | $0.02 |
| GPT-6.1 Sol | high | 85.9% | $0.05 |
| GPT-6.1 Sol | xhigh | 85.9% | $0.06 |
| Muse Spark Open | high | 85.9% | $0.12 |
| Claude Opus 5.5 | low | 85.9% | $0.13 |
| GPT-6 Astra | medium | 85.9% | $0.24 |
| GPT-6 Luna | xhigh | 84.4% | $0.01 |
| GPT-6.1 Sol | low | 84.4% | $0.03 |
| Gemini 4 Argon | medium | 84.4% | $0.22 |
| Claude Opus 5.5 | high | 84.4% | $0.34 |
| GPT-6 Astra | xhigh | 84.4% | $0.36 |
| Claude Opus 5.5 | xhigh | 84.4% | $0.66 |
| Claude Sonnet 5.5 | max | 84.4% | $1.34 |
| Claude Haiku 5.5 | max | 82.8% | $0.22 |
| GPT-6 Astra | high | 82.8% | $0.31 |
| GLM-5.3 | max | 81.3% | $0.03 |
| DeepSeek V4.1 Flash 1017 | max | 81.3% | $0.03 |
| mistral-large-4-0 | high | 81.3% | $0.05 |
| Kimi K3 | max | 81.3% | $0.09 |
| Gemini 4 Argon | high | 81.3% | $0.37 |
| DeepSeek V4.1 Flash 1017 | low | 79.7% | $0.01 |
| Muse Spark Open | xhigh | 79.7% | $0.12 |
| Claude Haiku 5.5 | medium | 78.1% | $0.01 |
| mistral-large-4-0 | low | 78.1% | $0.05 |
| Qwen 3.8 Max | xhigh | 78.1% | $0.25 |
| GPT-6 Luna | high | 76.6% | $0.01 |
| GLM-5.3 | high | 76.6% | $0.02 |
| Kimi K3 | high | 76.6% | $0.07 |
| Grok 4.7 | high | 76.6% | $0.17 |
| GPT-6 Luna | medium | 75% | $0 |
| DeepSeek V4.1 Flash 1017 | high | 75% | $0.02 |
| Claude Sonnet 5.5 | medium | 75% | $0.06 |
| Grok 4.7 | xhigh | 75% | $0.19 |
| Qwen 3.8 Max | low | 73.4% | $0.14 |
| Qwen 3.8 Max | medium | 73.4% | $0.14 |
| Claude Sonnet 5.5 | low | 71.9% | $0.06 |
| Muse Spark Open | medium | 70.3% | $0.09 |
| Grok 4.7 | medium | 70.3% | $0.14 |
| Claude Haiku 5.5 | low | 68.8% | $0.01 |
| Kimi K3 | low | 68.8% | $0.04 |
| Grok 4.7 | low | 68.8% | $0.09 |
| GPT-5.6 Luna | low | 65.6% | $0.01 |
| Muse Spark Open | low | 65.6% | $0.06 |
| Qwen 3.8 Max | off | 65.6% | $0.1 |
| GLM-5.3 | low | 64.1% | $0.01 |
| MiniMax-M3.1-Flash | high | 64.1% | $0.03 |
| Qwen 3.8 27B | low | 64.1% | $0.06 |
| Claude Haiku 5.5 | off | 62.5% | $0.01 |
| Qwen 3.8 27B | high | 60.9% | $0.07 |
| GPT-6 Luna | low | 56.3% | $0 |
| Gemma 4 31B IT | low | 56.3% | $0.02 |
| DeepSeek V4.1 Flash 1017 | off | 54.7% | $0.01 |
| GPT-5.4 Mini | low | 54.7% | $0.02 |
| Gemma 4 31B IT | high | 53.1% | $0.01 |
| MiniMax-M3.1-Flash | low | 51.6% | $0.02 |
| Qwen3.6 27B | high | 51.6% | $0.05 |
| GPT-6 Luna | off | 48.4% | $0.01 |
| Qwen 3.6 35B-A3B | high | 43.8% | $0.02 |
| Qwen3.6 27B | low | 40.6% | $0.04 |
| Qwen 3.6 35B-A3B | low | 29.7% | $0.03 |
| Claude Haiku 4.5 | off | 26.6% | $0.08 |
| Grok 4.7 | max | 26.6% | $0.14 |
| GPT-5.4 Nano | off | 10.9% | $0.01 |
The best score, 93.8%, costs $0.26 a task
Runs on the cost frontier score higher than any run that's cheaper. Claude Sonnet 5.5 at high comes within 4.7 points of the best for $0.11 a task, about half the cost. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Anthropic OpenAI
Cost frontier
One model at each effort level
Source: Official o11y-bench leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude Opus 5.5 | medium | 93.8% | $0.26 | ||
| Claude Sonnet 5.5 | xhigh | 90.6% | $0.29 | ||
| Claude Sonnet 5.5 | high | 89.1% | $0.11 | ||
| GPT-6 Astra | low | 87.5% | $0.20 | ||
| Claude Opus 5.5 | max | 87.5% | $1.69 | ||
| GPT-6.1 Sol | medium | 87.5% | $0.04 | ||
| Gemini 4 Argon | low | 87.5% | $0.27 | ||
| Claude Haiku 5.5 | xhigh | 87.5% | $0.06 | ||
| GPT-6 Astra | medium | 85.9% | $0.24 | ||
| Claude Opus 5.5 | low | 85.9% | $0.13 | ||
| GPT-6.1 Sol | xhigh | 85.9% | $0.06 | ||
| GPT-6.1 Sol | high | 85.9% | $0.05 | ||
| Claude Haiku 5.5 | high | 85.9% | $0.02 | ||
| Muse Spark Open | high | 85.9% | $0.12 | ||
| GPT-6 Astra | xhigh | 84.4% | $0.36 | ||
| Claude Opus 5.5 | high | 84.4% | $0.34 | ||
| Claude Opus 5.5 | xhigh | 84.4% | $0.66 | ||
| GPT-6 Luna | xhigh | 84.4% | $0.007 | ||
| Claude Sonnet 5.5 | max | 84.4% | $1.34 | ||
| GPT-6.1 Sol | low | 84.4% | $0.03 | ||
| Gemini 4 Argon | medium | 84.4% | $0.22 | ||
| GPT-6 Astra | high | 82.8% | $0.31 | ||
| Claude Haiku 5.5 | max | 82.8% | $0.22 | ||
| Kimi K3 | max | 81.3% | $0.09 | ||
| GLM-5.3 | max | 81.3% | $0.03 | ||
| Gemini 4 Argon | high | 81.3% | $0.37 | ||
| DeepSeek V4.1 Flash 1017 | max | 81.3% | $0.03 | ||
| Muse Spark Open | xhigh | 79.7% | $0.12 | ||
| DeepSeek V4.1 Flash 1017 | low | 79.7% | $0.01 | ||
| Qwen 3.8 Max | xhigh | 78.1% | $0.25 | ||
| Claude Haiku 5.5 | medium | 78.1% | $0.01 | ||
| Kimi K3 | high | 76.6% | $0.07 | ||
| GLM-5.3 | high | 76.6% | $0.02 | ||
| Grok 4.7 | high | 76.6% | $0.17 | ||
| GPT-6 Luna | high | 76.6% | $0.005 | ||
| Grok 4.7 | xhigh | 75% | $0.19 | ||
| GPT-6 Luna | medium | 75% | $0.004 | ||
| Claude Sonnet 5.5 | medium | 75% | $0.06 | ||
| DeepSeek V4.1 Flash 1017 | high | 75% | $0.01 | ||
| Qwen 3.8 Max | low | 73.4% | $0.14 | ||
| Qwen 3.8 Max | medium | 73.4% | $0.14 | ||
| Claude Sonnet 5.5 | low | 71.9% | $0.06 | ||
| Grok 4.7 | medium | 70.3% | $0.14 | ||
| Muse Spark Open | medium | 70.3% | $0.09 | ||
| Kimi K3 | low | 68.8% | $0.04 | ||
| Grok 4.7 | low | 68.8% | $0.09 | ||
| Claude Haiku 5.5 | low | 68.8% | $0.007 | ||
| GPT-5.6 Luna | low | 65.6% | $0.005 | ||
| Qwen 3.8 Max | off | 65.6% | $0.10 | ||
| Muse Spark Open | low | 65.6% | $0.06 | ||
| GLM-5.3 | low | 64.1% | $0.01 | ||
| Qwen 3.8 27B | low | 64.1% | $0.06 | ||
| MiniMax-M3.1-Flash | high | 64.1% | $0.03 | ||
| Claude Haiku 5.5 | off | 62.5% | $0.008 | ||
| Qwen 3.8 27B | high | 60.9% | $0.07 | ||
| Gemma 4 31B IT | low | 56.3% | $0.01 | ||
| GPT-6 Luna | low | 56.3% | $0.002 | ||
| GPT-5.4 Mini | low | 54.7% | $0.02 | ||
| DeepSeek V4.1 Flash 1017 | off | 54.7% | $0.009 | ||
| Gemma 4 31B IT | high | 53.1% | $0.01 | ||
| Qwen3.6 27B | high | 51.6% | $0.05 | ||
| MiniMax-M3.1-Flash | low | 51.6% | $0.01 | ||
| GPT-6 Luna | off | 48.4% | $0.005 | ||
| Qwen 3.6 35B-A3B | high | 43.8% | $0.02 | ||
| Qwen3.6 27B | low | 40.6% | $0.04 | ||
| Qwen 3.6 35B-A3B | low | 29.7% | $0.03 | ||
| Claude Haiku 4.5 | off | 26.6% | $0.08 | ||
| Grok 4.7 | max | 26.6% | $0.14 | ||
| GPT-5.4 Nano | off | 10.9% | $0.005 |
6 models have held number one on the official leaderboard since its release in April 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
OpenAI Google DeepMind Moonshot AI Anthropic
Source: Official o11y-bench leaderboard
GPT-6 Astra on the official leaderboard passed 85% in September 2026
All the scores we have, one dot each, so a model tested at several settings appears more than once.
On the benchmark’s own leaderboard
Best so far
Source: Official o11y-bench leaderboard
No lab model card reports it yet
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.
More in Terminal agents
- SkillsBench In use 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. GPT-5.5 67.3%
- Terminal-Bench Hard In use Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. Gemma 4 31B IT 36%
- Harbor-Index In use Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. GPT-5.5 28.1%
- Terminal-Bench 2.1 Extended Not in model cards Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores. Muse Wonder 54%
- Terminal-Bench 4.1 Not in model cards The Terminal-Bench 4.0 tasks with hardened verifiers: agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Claude Opus 5.5 65.8%
- CompileBench Not in model cards Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. Claude Opus 4.1 100%