Benchmark Almanac

SnorkelSequences

GPT-5 77.6% leads since August 2025
Not in model cards

Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math.

Answers are checked programmatically with no judge. Announced with SnorkelFinance and SnorkelUnderwrite as preview benchmarks. The board is archived.

Made by
Snorkel AI
Released
June 2025
Kind
Word problems
Level
School
Scored
Checked automatically
Answers
Short answer
Tasks
Kept private
Strongest model tested
GPT-5.4 #27 on the capability index
Last model added
March 2026
Scores refreshed
November 12th, 2026
32
Entries on its own leaderboard, 25 models we track

2 models share the top score of 77.6% among models released by March 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in March 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

32 entries. Scroll sideways for the rest.

  1. 77.6 GPT-5
  2. 77.6 GPT-5 mini
  3. 72.0 GPT-5 nano
  4. 71.6 GPT-5.4
  5. 71.2 o3-mini
  6. 70.8 Gemini 2.5 Flash [snapshot unspecified]
  7. 70.4 Claude Sonnet 4
  8. 70.2 Grok 4 Fast
  9. 68.8 o4-mini
  10. 66.8 Llama-3.3-Nemotron-Super-49B-v1.5
  11. 66.0 Gemini 2.5 Pro [snapshot unspecified]
  12. 65.6 Claude Opus 4
  13. 65.2 o3
  14. 63.2 Grok 4
  15. 62.0 Llama 4 Maverick
  16. 51.8 Amazon Nova Premier
  17. 48.4 Llama 4 Scout
  18. 47.6 Magistral Medium [snapshot unspecified]
  19. 47.6 Claude 3.7 Sonnet
  20. 44.8 Llama-3.3-Nemotron-Super-49B-v1
  21. 41.2 Amazon Nova Pro
  22. 40.0 Amazon Nova Lite
  23. 39.2 Grok 3
  24. 38.8 Mistral Large [snapshot unspecified]
  25. 38.8 Llama 3.3 70B
  26. 38.4 Codestral [snapshot unspecified]
  27. 36.8 GPT-4.1
  28. 36.4 Nvidia 70B Instruct [snapshot unspecified]
  29. 36.0 Kimi K2 Thinking
  30. 35.2 Llama 3.1-405B
  31. 33.6 Amazon Nova Micro
  32. 28.0 Qwen 3 235B [snapshot unspecified]
  1. 1 GPT-5 77.6%
  2. 2 GPT-5 mini 77.6%
  3. 3 GPT-5 nano 72%
  4. 4 GPT-5.4 71.6%
  5. 5 o3-mini 71.2%
  6. 6 Gemini 2.5 Flash [snapshot unspecified] 70.8%
  7. 7 Claude Sonnet 4 70.4%
  8. 8 Grok 4 Fast reasoning 70.2%
  9. 9 o4-mini 68.8%
  10. 10 Llama-3.3-Nemotron-Super-49B-v1.5 66.8%
  11. 11 Gemini 2.5 Pro [snapshot unspecified] 66%
  12. 12 Claude Opus 4 65.6%
  13. 13 o3 65.2%
  14. 14 Grok 4 63.2%
  15. 15 Llama 4 Maverick 62%
  16. 16 Amazon Nova Premier 51.8%
  17. 17 Llama 4 Scout 48.4%
  18. 18 Magistral Medium [snapshot unspecified] 47.6%
  19. 19 Claude 3.7 Sonnet 47.6%
  20. 20 Llama-3.3-Nemotron-Super-49B-v1 44.8%
  21. 21 Amazon Nova Pro 41.2%
  22. 22 Amazon Nova Lite 40%
  23. 23 Grok 3 39.2%
  24. 24 Mistral Large [snapshot unspecified] 38.8%
  25. 25 Llama 3.3 70B 38.8%
  26. 26 Codestral [snapshot unspecified] 38.4%
  27. 27 GPT-4.1 36.8%
  28. 28 Nvidia 70B Instruct [snapshot unspecified] 36.4%
  29. 29 Kimi K2 Thinking 36%
  30. 30 Llama 3.1-405B 35.2%
  31. 31 Amazon Nova Micro 33.6%
  32. 32 Qwen 3 235B [snapshot unspecified] 28%

Source: Official SnorkelSequences leaderboard

#EntryScoreSettingDateSource
1 GPT-5 77.6% Aug 2025 Official leaderboard
2 GPT-5 mini 77.6% Aug 2025 Official leaderboard
3 GPT-5 nano 72% Aug 2025 Official leaderboard
4 GPT-5.4 71.6% Mar 2026 Official leaderboard
5 o3-mini 71.2% Jan 2025 Official leaderboard
6 Gemini 2.5 Flash [snapshot unspecified] 70.8% Official leaderboard
7 Claude Sonnet 4 70.4% May 2025 Official leaderboard
8 Grok 4 Fast 70.2% reasoning Sep 2025 Official leaderboard
9 o4-mini 68.8% Apr 2025 Official leaderboard
10 Llama-3.3-Nemotron-Super-49B-v1.5 66.8% Jul 2025 Official leaderboard
11 Gemini 2.5 Pro [snapshot unspecified] 66% Official leaderboard
12 Claude Opus 4 65.6% May 2025 Official leaderboard
13 o3 65.2% Apr 2025 Official leaderboard
14 Grok 4 63.2% Jul 2025 Official leaderboard
15 Llama 4 Maverick 62% Apr 2025 Official leaderboard
16 Amazon Nova Premier 51.8% Apr 2025 Official leaderboard
17 Llama 4 Scout 48.4% Apr 2025 Official leaderboard
18 Magistral Medium [snapshot unspecified] 47.6% Official leaderboard
19 Claude 3.7 Sonnet 47.6% Feb 2025 Official leaderboard
20 Llama-3.3-Nemotron-Super-49B-v1 44.8% Mar 2025 Official leaderboard
21 Amazon Nova Pro 41.2% Dec 2024 Official leaderboard
22 Amazon Nova Lite 40% Dec 2024 Official leaderboard
23 Grok 3 39.2% Apr 2025 Official leaderboard
24 Mistral Large [snapshot unspecified] 38.8% Official leaderboard
25 Llama 3.3 70B 38.8% Dec 2024 Official leaderboard
26 Codestral [snapshot unspecified] 38.4% Official leaderboard
27 GPT-4.1 36.8% Apr 2025 Official leaderboard
28 Nvidia 70B Instruct [snapshot unspecified] 36.4% Official leaderboard
29 Kimi K2 Thinking 36% Nov 2025 Official leaderboard
30 Llama 3.1-405B 35.2% Jul 2024 Official leaderboard
31 Amazon Nova Micro 33.6% Dec 2024 Official leaderboard
32 Qwen 3 235B [snapshot unspecified] 28% Official leaderboard

2 models have held number one on the official leaderboard since its release in June 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Not tested since Mar 2026
GPT-5 77.6%

Source: Official SnorkelSequences leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) GPT-5 o3-mini Amazon Nova Pro Llama 3.1-405B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) GPT-5 o3-mini

Source: Official SnorkelSequences leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Math

All 113 in Math
  • AIME 2026 In use Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. GPT-5.5 100%
  • IMO-AnswerBench In use 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. GLM-5.2 91%
  • HMMT 2026 In use Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. GPT-5.5 98.5%
  • PolyMath In use Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Qwen3.7-Max 86.5%
  • MathArena (Apex, IMO, USAMO) In use Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. Claude Opus 4.8 81.3%
  • MathArena Apex Shortlist In use Models solve 47 hard final-answer problems that MathArena selected from 2025 competitions, where the best models scored about 50% at selection. Scored by accuracy over repeated attempts. GPT-5.5 98.4%