Benchmark Almanac

Okapi Bench: API Freshness

GPT-6 Astra 66.7% leads since November 2026
Not in model cards

Models write integrations for changing APIs. The grader checks whether their calls still work with the current interfaces.

Kaggle publishes scores for 34 distinct model versions.

Made by
Okapi
Released
November 2026
Kind
Function-level problems
Level
University
Scored
Checked automatically
Answers
Code
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Code → Code
Interaction
Single turn
Items from
Real work and use
Run by
Anyone
Strongest model tested
GPT-6 Astra #4 on the capability index
Last model added
October 2026
Scores refreshed
November 20th, 2026
34
Models scored on its own leaderboard

GPT-6 Astra leads with 66.7%

Best score per model from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best.

  1. 1 GPT-6 Astra 66.7%
  2. 2 GPT-6 Sol 62.5%
  3. 3 GPT-6.1 Sol 62.5%
  4. 4 GPT-5.6 Sol 60.4%
  5. 5 Claude Sonnet 5.5 60.4%
  6. 6 GPT-5.6 Terra 58.3%
  7. 7 Claude Opus 5.5 58.3%
  8. 8 GPT-5.6 Luna 56.3%
  9. 9 GPT-6 Luna 56.3%
  10. 10 Claude Haiku 5.5 54.2%
  11. 11 Gemini 4 Argon 52.1%
  12. 12 Claude Opus 5 50%
  13. 13 GPT-5.5 41.7%
  14. 14 Claude Opus 4.7 37.5%
  15. 15 Claude Opus 4.8 33.3%
  16. 16 GPT-5.4 31.3%
  17. 17 Gemini 3.5 Flash-Lite 31.3%
  18. 18 Gemini 3.8 Flash 31.3%
  19. 19 Gemini 3.1 Pro 29.2%
  20. 20 Claude Sonnet 5 29.2%
  21. 21 Claude Opus 4.5 27.1%
  22. 22 Claude Sonnet 4.6 27.1%
  23. 23 Gemini 3.6 Flash 27.1%
  24. 24 Gemini 3 Flash 25%
  25. 25 Claude Haiku 4.5 22.9%
  26. 26 Claude Opus 4.6 22.9%
  27. 27 GPT-5.4 Nano 20.8%
  28. 28 Gemma 4 26B A4B 20.8%
  29. 29 Gemma 4 31B IT 18.8%
  30. 30 Gemini 2.5 Pro Jun 2025 16.7%
  31. 31 GPT-5.4 Mini 16.7%
  32. 32 Gemini 2.5 Flash Sep 2025 14.6%
  33. 33 Gemini 3.1 Flash-Lite 14.6%
  34. 34 gpt-oss-120b 10.4%

Source: Official Okapi Bench: API Freshness leaderboard

#ModelScoreSettingReleasedRun onSource
1 GPT-6 Astra 66.7% Sep 2026 Nov 2026 Official leaderboard
2 GPT-6 Sol 62.5% Sep 2026 Nov 2026 Official leaderboard
3 GPT-6.1 Sol 62.5% Sep 2026 Nov 2026 Official leaderboard
4 GPT-5.6 Sol 60.4% Jul 2026 Nov 2026 Official leaderboard
5 Claude Sonnet 5.5 60.4% Sep 2026 Nov 2026 Official leaderboard
6 GPT-5.6 Terra 58.3% Jul 2026 Nov 2026 Official leaderboard
7 Claude Opus 5.5 58.3% Sep 2026 Nov 2026 Official leaderboard
8 GPT-5.6 Luna 56.3% Jul 2026 Nov 2026 Official leaderboard
9 GPT-6 Luna 56.3% Sep 2026 Nov 2026 Official leaderboard
10 Claude Haiku 5.5 54.2% Oct 2026 Nov 2026 Official leaderboard
11 Gemini 4 Argon 52.1% Sep 2026 Nov 2026 Official leaderboard
12 Claude Opus 5 50% Jul 2026 Nov 2026 Official leaderboard
13 GPT-5.5 41.7% Apr 2026 Nov 2026 Official leaderboard
14 Claude Opus 4.7 37.5% Apr 2026 Nov 2026 Official leaderboard
15 Claude Opus 4.8 33.3% May 2026 Nov 2026 Official leaderboard
16 GPT-5.4 31.3% Mar 2026 Nov 2026 Official leaderboard
17 Gemini 3.5 Flash-Lite 31.3% Jul 2026 Nov 2026 Official leaderboard
18 Gemini 3.8 Flash 31.3% Sep 2026 Nov 2026 Official leaderboard
19 Gemini 3.1 Pro 29.2% Feb 2026 Nov 2026 Official leaderboard
20 Claude Sonnet 5 29.2% Jun 2026 Nov 2026 Official leaderboard
21 Claude Opus 4.5 27.1% Nov 2025 Nov 2026 Official leaderboard
22 Claude Sonnet 4.6 27.1% Feb 2026 Nov 2026 Official leaderboard
23 Gemini 3.6 Flash 27.1% Jul 2026 Nov 2026 Official leaderboard
24 Gemini 3 Flash 25% Dec 2025 Nov 2026 Official leaderboard
25 Claude Haiku 4.5 22.9% Oct 2025 Nov 2026 Official leaderboard
26 Claude Opus 4.6 22.9% Feb 2026 Nov 2026 Official leaderboard
27 GPT-5.4 Nano 20.8% Mar 2026 Nov 2026 Official leaderboard
28 Gemma 4 26B A4B 20.8% Apr 2026 Nov 2026 Official leaderboard
29 Gemma 4 31B IT 18.8% Apr 2026 Nov 2026 Official leaderboard
30 Gemini 2.5 Pro Jun 2025 16.7% Jun 2025 Nov 2026 Official leaderboard
31 GPT-5.4 Mini 16.7% Mar 2026 Nov 2026 Official leaderboard
32 Gemini 2.5 Flash Sep 2025 14.6% Sep 2025 Nov 2026 Official leaderboard
33 Gemini 3.1 Flash-Lite 14.6% Mar 2026 Nov 2026 Official leaderboard
34 gpt-oss-120b 10.4% Aug 2025 Nov 2026 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard 1 leader
GPT-6 Astra 66.7%

Source: Official Okapi Bench: API Freshness leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) GPT-6 Astra GPT-5.6 Sol GPT-5.5 Claude Opus 4.7 GPT-5.4 Claude Opus 4.5 Claude Haiku 4.5 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) GPT-6 Astra GPT-5.6 Sol

Source: Official Okapi Bench: API Freshness leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Coding

All 523 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Fable 5.2 96.6%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471

Data audit

All 11 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 5 pass
  • Pass Independent runs Scores from the official leaderboard (34 entries).
  • Pass Models scored 34 models we track.
  • Pass Newest score The newest is from October 2026, for Claude Haiku 5.5.
  • Pass Frontier models 5 of the top 10 have a score, led by GPT-6 Astra (#4).
  • Pass Headroom The best is 66.7%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on kaggle.com: 34 rows for models we track at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check