FretBench
Models read guitar tablature in four tunings and name notes from string positions and fret numbers. The grader compares their answers with an answer key.
The maker ranks 18 models in 21 configurations on a 239-case dataset. The March release used 182 cases and had a grading error with sharp notes.
- Made by
- Jaiden Capra
- Released
- March 2026
- Kind
- Language reasoning
- Level
- School
- Scored
- Checked automatically
- Answers
- Short answer
- Tasks
- Public, plus a private test set
- Measures
- Capability
- Subject
- Arts
- Language
- English
- Interaction
- Single turn
- Refresh
- New versions replace it
- Run by
- A host
- Strongest model tested
- GPT-5.4 #28 on the capability index
- Last model added
- April 2026
- Scores refreshed
- November 20th, 2026
Gemini 3.1 Pro leads with 100% among models released by April 2026
Best score per model from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
The newest model with a score here came out in April 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6.2 Astra
- Claude Fable 5.2
-
Gemini 4 Carbon
- GPT-6 Astra
- and 16 more
- 100.0
Gemini 3.1 Pro
- 99.6
DeepSeek-V3.2-Speciale
- 99.6 Kimi K2.5
- 99.6
Qwen3.5-Plus
- 98.3
Qwen3.5-Flash
- 87.2 Claude Opus 4.7
- 84.9
MiniMax-M2.5
- 63.2 GPT-5.4
- 59.8 Claude Opus 4.6
- 51.7
Gemma 4 31B IT
- 38.5
Gemini 3.1 Flash-Lite
- 36.2
Gemma 4 26B A4B
- 24.7
Mistral Large
- 24.7
Llama 3.3 70B
- 20.5 Claude Sonnet 4.6
- 20.1
DeepSeek-V3.2
- 17.6 Claude Haiku 4.5
- 13.8
Llama 4 Scout
- 1
Gemini 3.1 Pro 100%
- 2
DeepSeek-V3.2-Speciale 99.6%
- 3 Kimi K2.5 Reasoning 99.6%
- 4
Qwen3.5-Plus 99.6%
- 5
Qwen3.5-Flash 98.3%
- 6 Claude Opus 4.7 87.2%
- 7
MiniMax-M2.5 84.9%
- 8 GPT-5.4 63.2%
- 9 Claude Opus 4.6 59.8%
- 10
Gemma 4 31B IT 51.7%
- 11
Gemini 3.1 Flash-Lite 38.5%
- 12
Gemma 4 26B A4B 36.2%
- 13
Mistral Large 24.7%
- 14
Llama 3.3 70B 24.7%
- 15 Claude Sonnet 4.6 20.5%
- 16
DeepSeek-V3.2 20.1%
- 17 Claude Haiku 4.5 17.6%
- 18
Llama 4 Scout 13.8%
Source: Official FretBench leaderboard
| # | Model | Score | Setting | Released | Run on | Source |
|---|---|---|---|---|---|---|
| 1 | | 100% | Feb 2026 | Apr 2026 | Official leaderboard | |
| 2 | | 99.6% | Dec 2025 | Mar 2026 | Official leaderboard | |
| 3 | Kimi K2.5 | 99.6% | Reasoning | Jan 2026 | Mar 2026 | Official leaderboard |
| 4 | | 99.6% | Feb 2026 | Mar 2026 | Official leaderboard | |
| 5 | | 98.3% | Feb 2026 | Mar 2026 | Official leaderboard | |
| 6 | Claude Opus 4.7 | 87.2% | Apr 2026 | Apr 2026 | Official leaderboard | |
| 7 | | 84.9% | Feb 2026 | Mar 2026 | Official leaderboard | |
| 8 | GPT-5.4 | 63.2% | Mar 2026 | Mar 2026 | Official leaderboard | |
| 9 | Claude Opus 4.6 | 59.8% | Feb 2026 | Mar 2026 | Official leaderboard | |
| 10 | | 51.7% | Apr 2026 | Apr 2026 | Official leaderboard | |
| 11 | | 38.5% | Mar 2026 | Mar 2026 | Official leaderboard | |
| 12 | | 36.2% | Apr 2026 | Apr 2026 | Official leaderboard | |
| 13 | | 24.7% | Feb 2024 | Mar 2026 | Official leaderboard | |
| 14 | | 24.7% | Dec 2024 | Mar 2026 | Official leaderboard | |
| 15 | Claude Sonnet 4.6 | 20.5% | Feb 2026 | Mar 2026 | Official leaderboard | |
| 16 | | 20.1% | Dec 2025 | Mar 2026 | Official leaderboard | |
| 17 | Claude Haiku 4.5 | 17.6% | Oct 2025 | Mar 2026 | Official leaderboard | |
| 18 | | 13.8% | Apr 2025 | Mar 2026 | Official leaderboard |
Rankings by part
We read 4 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Drop D, Drop Db and Half-Step Down. DeepSeek-V3.2-Speciale leads 3 of them, and Gemini 3.1 Pro leads the other one. Arrows show how far a model moved from its overall rank among the models we track.
Drop D
18 models we track have a score here.
- 1
DeepSeek-V3.2-Speciale ↑1 100%
- 2 Kimi K2.5 ↑1 100%
- 3
Qwen3.5-Plus ↑1 100%
- 4
Gemini 3.1 Pro ↓3 100%
- 5
Qwen3.5-Flash 100%
- 6 Claude Opus 4.7 87.8%
- 7
MiniMax-M2.5 80%
- 8 GPT-5.4 68.9%
- 9
Gemma 4 31B IT ↑1 58.4%
- 10
Gemini 3.1 Flash-Lite ↑1 53.3%
Drop Db
18 models we track have a score here.
- 1
DeepSeek-V3.2-Speciale ↑1 100%
- 2 Kimi K2.5 ↑1 100%
- 3
Qwen3.5-Plus ↑1 100%
- 4
Gemini 3.1 Pro ↓3 100%
- 5
Qwen3.5-Flash 93.9%
- 6
MiniMax-M2.5 ↑1 87.9%
- 7 Claude Opus 4.7 ↓1 83.3%
- 8 Claude Opus 4.6 ↑1 57.6%
- 9 GPT-5.4 ↓1 57.6%
- 10
Gemma 4 31B IT 27.9%
Half-Step Down
18 models we track have a score here.
- 1
DeepSeek-V3.2-Speciale ↑1 100%
- 2 Kimi K2.5 ↑1 100%
- 3
Qwen3.5-Plus ↑1 100%
- 4
Gemini 3.1 Pro ↓3 100%
- 5
Qwen3.5-Flash 98.6%
- 6 Claude Opus 4.7 89.3%
- 7
MiniMax-M2.5 87.1%
- 8 Claude Opus 4.6 ↑1 54.3%
- 9 GPT-5.4 ↓1 48.6%
- 10
Gemma 4 31B IT 42.9%
Standard
18 models we track have a score here.
- 1
Gemini 3.1 Pro 100%
- 2
DeepSeek-V3.2-Speciale 98.9%
- 3 Kimi K2.5 98.9%
- 4
Qwen3.5-Plus 98.9%
- 5
Qwen3.5-Flash 98.9%
- 6 Claude Opus 4.7 86.8%
- 7
MiniMax-M2.5 84.6%
- 8
Gemma 4 31B IT ↑2 73.9%
- 9 GPT-5.4 ↓1 73.6%
- 10 Claude Opus 4.6 ↓1 69.2%
The best score, 100%, costs $1.30 for the full run
Runs on the cost frontier score higher than any run that's cheaper. Qwen3.5-Flash comes within 1.7 points of the best for $0.23 for the full run, 6 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
Source: Official FretBench leaderboard
| Model | Effort | Harness | Score | Cost of the full run | Output tokens |
|---|---|---|---|---|---|
| 100% | $1.30 | ||||
| 99.6% | $0.42 | ||||
| Reasoning | 99.6% | $0.44 | |||
| Kimi K2.5 | Reasoning | 99.6% | $0.63 | ||
| Kimi K2.5 | 99.6% | $0.65 | |||
| 99.6% | $0.61 | ||||
| 98.3% | $0.23 | ||||
| Claude Opus 4.7 | 87.2% | $2.38 | |||
| 84.9% | $0.16 | ||||
| Reasoning | 82.8% | $0.16 | |||
| GPT-5.4 | 63.2% | $0.32 | |||
| Claude Opus 4.6 | 59.8% | $0.75 | |||
| 51.7% | $0.03 | ||||
| 38.5% | $0.03 | ||||
| 36.2% | $0.02 | ||||
| 24.7% | $0.05 | ||||
| 24.7% | $0.03 | ||||
| Claude Sonnet 4.6 | 20.5% | $0.59 | |||
| 20.1% | $0.03 | ||||
| Claude Haiku 4.5 | 17.6% | $0.17 | |||
| 13.8% | $0.02 |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official FretBench leaderboard
DeepSeek-V3.2-Speciale on the official leaderboard passed 85% in December 2025
Models released before it came out already scored that high. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official FretBench leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access Public tasks, plus a private test set.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (18 entries).
- Pass Models scored 18 models we track.
- Note Newest score The newest came 7 months ago, in April 2026, for Claude Opus 4.7. That’s expected for a saturated test.
- Note Frontier models None of the top 10 has a score, which is fine for a retired test. The strongest tested is GPT-5.4 (#28).
- Note Headroom Saturated. The best is 100%, past 85% since December 2025.
Leaderboard import 3 pass
- Pass Importer We read the board on fretbench.tymo.ai: 21 rows for models we track at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.