Benchmark Almanac

Judgemark v2.1

Claude Sonnet 4.6 94.1 leads since February 2026
Not in model cards

The model under test acts as a judge and scores creative writing on several literary criteria. The benchmark measures how well its scores separate stronger writers from weaker ones.

v2.1 swapped in stronger writer models to fight saturation and added judge ensembling. Now listed as a legacy board.

Released
January 2025
Kind
Taste and style
Level
Expert
Scored
Checked automatically
Answers
Short answer
Tasks
All public
Strongest model tested
GPT-5.4 #27 on the capability index
Last model added
March 2026
Scores refreshed
November 12th, 2026
47
Entries on its own leaderboard, 42 models we track

Claude Sonnet 4.6 leads with 94.1 among models released by March 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in March 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

47 entries. Scroll sideways for the rest.

  1. 94.1 Claude Sonnet 4.6
  2. 92.9 Claude Opus 4.6
  3. 86.6 Gemini 3 Pro
  4. 85.6 GLM-5
  5. 85.5 Qwen3.5 397B-A17B
  6. 85.4 Gemini 3 Flash
  7. 85.0 GPT-5.4
  8. 84.6 Claude Sonnet 4.5
  9. 84.3 Qwen3.5-27B
  10. 82.0 GPT-5.2
  11. 82.0 Claude Sonnet 4
  12. 81.8 qwen3.5-flash-02-23
  13. 81.4 Qwen3.5-35B-A3B
  14. 81.2 gpt-5-2025-08-07:minimal-reasoning
  15. 81.1 GPT-5.4 Mini
  16. 81.0 Qwen3.5-122B-A10B
  17. 80.9 Gemini 3.1 Flash-Lite
  18. 79.7 Claude Haiku 4.5
  19. 77.5 GLM-4.7
  20. 76.3 GPT-4.1
  21. 75.8 Grok 4.1 Fast
  22. 75.1 o3
  23. 75.0 Qwen3.5-9B
  24. 74.5 gpt-5-mini-2025-08-07:minimal-reasoning
  25. 73.1 Kimi K2
  26. 72.1 Gemini 2.5 Pro
  27. 68.2 Mistral Large 3
  28. 66.3 DeepSeek-V3.2
  29. 66.2 glm-4.7-flash
  30. 64.5 GPT-5.4 Nano
  31. 64.5 MiniMax-M2.5
  32. 63.4 GLM-4.5
  33. 63.0 Gemini 2.5 Flash
  34. 61.1 gpt-5-nano-2025-08-07__5x-ensemble
  35. 59.7 Qwen3-235B-A22B-Instruct
  36. 58.5 Mistral Medium 3
  37. 57.3 GPT-4.1 mini
  38. 56.2 Qwen3-235B-A22B
  39. 56.1 o4-mini
  40. 56.0 Gemini 2.5 Flash-Lite
  41. 54.8 gpt-oss-120b
  42. 52.2 gpt-5-nano-2025-08-07:minimal-reasoning
  43. 51.6 Mistral Small 4
  44. 50.2 Mistral Small 3.2
  45. 42.8 gpt-oss-20b
  46. 34.6 Qwen3-30B-A3B-Instruct
  47. 19.2 Llama 3.1-8B
  1. 1 Claude Sonnet 4.6 94.1
  2. 2 Claude Opus 4.6 92.9
  3. 3 Gemini 3 Pro 86.6
  4. 4 GLM-5 85.6
  5. 5 Qwen3.5 397B-A17B 85.5
  6. 6 Gemini 3 Flash 85.4
  7. 7 GPT-5.4 85.0
  8. 8 Claude Sonnet 4.5 84.6
  9. 9 Qwen3.5-27B 84.3
  10. 10 GPT-5.2 82.0
  11. 11 Claude Sonnet 4 82.0
  12. 12 qwen3.5-flash-02-23 81.8
  13. 13 Qwen3.5-35B-A3B 81.4
  14. 14 gpt-5-2025-08-07:minimal-reasoning 81.2
  15. 15 GPT-5.4 Mini 81.1
  16. 16 Qwen3.5-122B-A10B 81.0
  17. 17 Gemini 3.1 Flash-Lite 80.9
  18. 18 Claude Haiku 4.5 79.7
  19. 19 GLM-4.7 77.5
  20. 20 GPT-4.1 76.3
  21. 21 Grok 4.1 Fast 75.8
  22. 22 o3 75.1
  23. 23 Qwen3.5-9B 75.0
  24. 24 gpt-5-mini-2025-08-07:minimal-reasoning 74.5
  25. 25 Kimi K2 Jul 2025 73.1
  26. 26 Gemini 2.5 Pro Jun 2025 72.1
  27. 27 Mistral Large 3 68.2
  28. 28 DeepSeek-V3.2 66.3
  29. 29 glm-4.7-flash 66.2
  30. 30 GPT-5.4 Nano 64.5
  31. 31 MiniMax-M2.5 64.5
  32. 32 GLM-4.5 63.4
  33. 33 Gemini 2.5 Flash Sep 2025 63.0
  34. 34 gpt-5-nano-2025-08-07__5x-ensemble 61.1
  35. 35 Qwen3-235B-A22B-Instruct Jul 2025 59.7
  36. 36 Mistral Medium 3 58.5
  37. 37 GPT-4.1 mini 57.3
  38. 38 Qwen3-235B-A22B 56.2
  39. 39 o4-mini 56.1
  40. 40 Gemini 2.5 Flash-Lite Sep 2025 56.0
  41. 41 gpt-oss-120b 54.8
  42. 42 gpt-5-nano-2025-08-07:minimal-reasoning 52.2
  43. 43 Mistral Small 4 51.6
  44. 44 Mistral Small 3.2 50.2
  45. 45 gpt-oss-20b 42.8
  46. 46 Qwen3-30B-A3B-Instruct Jul 2025 34.6
  47. 47 Llama 3.1-8B 19.2

Source: Official Judgemark v2.1 leaderboard

#EntryScoreSettingDateSource
1 Claude Sonnet 4.6 94.1 Feb 2026 Official leaderboard
2 Claude Opus 4.6 92.9 Feb 2026 Official leaderboard
3 Gemini 3 Pro 86.6 Nov 2025 Official leaderboard
4 GLM-5 85.6 Feb 2026 Official leaderboard
5 Qwen3.5 397B-A17B 85.5 Feb 2026 Official leaderboard
6 Gemini 3 Flash 85.4 Dec 2025 Official leaderboard
7 GPT-5.4 85.0 Mar 2026 Official leaderboard
8 Claude Sonnet 4.5 84.6 Sep 2025 Official leaderboard
9 Qwen3.5-27B 84.3 Feb 2026 Official leaderboard
10 GPT-5.2 82.0 Dec 2025 Official leaderboard
11 Claude Sonnet 4 82.0 May 2025 Official leaderboard
12 qwen3.5-flash-02-23 81.8 Official leaderboard
13 Qwen3.5-35B-A3B 81.4 Feb 2026 Official leaderboard
14 gpt-5-2025-08-07:minimal-reasoning 81.2 Official leaderboard
15 GPT-5.4 Mini 81.1 Mar 2026 Official leaderboard
16 Qwen3.5-122B-A10B 81.0 Feb 2026 Official leaderboard
17 Gemini 3.1 Flash-Lite 80.9 Mar 2026 Official leaderboard
18 Claude Haiku 4.5 79.7 Oct 2025 Official leaderboard
19 GLM-4.7 77.5 Dec 2025 Official leaderboard
20 GPT-4.1 76.3 Apr 2025 Official leaderboard
21 Grok 4.1 Fast 75.8 Nov 2025 Official leaderboard
22 o3 75.1 Apr 2025 Official leaderboard
23 Qwen3.5-9B 75.0 Feb 2026 Official leaderboard
24 gpt-5-mini-2025-08-07:minimal-reasoning 74.5 Official leaderboard
25 Kimi K2 Jul 2025 73.1 Jul 2025 Official leaderboard
26 Gemini 2.5 Pro Jun 2025 72.1 Jun 2025 Official leaderboard
27 Mistral Large 3 68.2 Dec 2025 Official leaderboard
28 DeepSeek-V3.2 66.3 Dec 2025 Official leaderboard
29 glm-4.7-flash 66.2 Jan 2026 Official leaderboard
30 GPT-5.4 Nano 64.5 Mar 2026 Official leaderboard
31 MiniMax-M2.5 64.5 Feb 2026 Official leaderboard
32 GLM-4.5 63.4 Aug 2025 Official leaderboard
33 Gemini 2.5 Flash Sep 2025 63.0 Sep 2025 Official leaderboard
34 gpt-5-nano-2025-08-07__5x-ensemble 61.1 Official leaderboard
35 Qwen3-235B-A22B-Instruct Jul 2025 59.7 Jul 2025 Official leaderboard
36 Mistral Medium 3 58.5 May 2025 Official leaderboard
37 GPT-4.1 mini 57.3 Apr 2025 Official leaderboard
38 Qwen3-235B-A22B 56.2 Apr 2025 Official leaderboard
39 o4-mini 56.1 Apr 2025 Official leaderboard
40 Gemini 2.5 Flash-Lite Sep 2025 56.0 Sep 2025 Official leaderboard
41 gpt-oss-120b 54.8 Aug 2025 Official leaderboard
42 gpt-5-nano-2025-08-07:minimal-reasoning 52.2 Official leaderboard
43 Mistral Small 4 51.6 Mar 2026 Official leaderboard
44 Mistral Small 3.2 50.2 Jun 2025 Official leaderboard
45 gpt-oss-20b 42.8 Aug 2025 Official leaderboard
46 Qwen3-30B-A3B-Instruct Jul 2025 34.6 Jul 2025 Official leaderboard
47 Llama 3.1-8B 19.2 Jul 2024 Official leaderboard

The best score, 94.1, costs $18 for the full run

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Meta Google DeepMind Z.ai Qwen Anthropic Cost frontier One model at each effort level
0.00 20.0 40.0 60.0 80.0 100.0 $0.10 $1.00 $10 $100 Cost of the full run, log scale Claude Sonnet 4.6 Gemini 3 Pro GLM-5 Gemini 3 Flash Qwen3.5-35B-A3B Qwen3.5-9B Llama 3.1-8B 0.00 20.0 40.0 60.0 80.0 100.0 $0.10 $1.00 $10 $100 Cost of the full run, log scale Claude Sonnet 4.6 Gemini 3 Pro

Source: Official Judgemark v2.1 leaderboard

ModelEffortHarnessScoreCost of the full runOutput tokens
Claude Sonnet 4.6 94.1 $18
Claude Opus 4.6 92.9 $30
Gemini 3 Pro 86.6 $11
GLM-5 85.6 $3.45
Qwen3.5 397B-A17B 85.5 $3.39
Gemini 3 Flash 85.4 $2.74
GPT-5.4 85.0 $14
Claude Sonnet 4.5 84.6 $18
Qwen3.5-27B 84.3 $1.63
GPT-5.2 82.0 $10
Claude Sonnet 4 82.0 $19
Qwen3.5-35B-A3B 81.4 $1.07
GPT-5.4 Mini 81.1 $4.06
Qwen3.5-122B-A10B 81.0 $1.96
Gemini 3.1 Flash-Lite 80.9 $1.37
Claude Haiku 4.5 79.7 $6.05
GLM-4.7 77.5 $10
GPT-4.1 76.3 $10
Grok 4.1 Fast 75.8 $1.65
o3 75.1 $16
Qwen3.5-9B 75.0 $0.25
Kimi K2 Jul 2025 73.1 $3.70
Gemini 2.5 Pro Jun 2025 72.1 $53
Mistral Large 3 68.2 $2.51
DeepSeek-V3.2 66.3 $1.25
glm-4.7-flash 66.2 $1.95
GPT-5.4 Nano 64.5 $1.10
MiniMax-M2.5 64.5 $3.50
GLM-4.5 63.4 $4.40
Gemini 2.5 Flash Sep 2025 63.0 $1.87
Qwen3-235B-A22B-Instruct Jul 2025 59.7 $0.94
Mistral Medium 3 58.5 $2.31
GPT-4.1 mini 57.3 $2.09
Qwen3-235B-A22B 56.2 $1.00
o4-mini 56.1 $12
Gemini 2.5 Flash-Lite Sep 2025 56.0 $0.50
gpt-oss-120b 54.8 $1.14
Mistral Small 4 51.6 $0.79
Mistral Small 3.2 50.2 $0.49
gpt-oss-20b 42.8 $0.45
Qwen3-30B-A3B-Instruct Jul 2025 34.6 $0.98
Llama 3.1-8B 19.2 $0.11

7 models have held number one on the official leaderboard since its release in January 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta OpenAI Anthropic Google DeepMind
Own leaderboard Replaced by 4, May 2026
Claude Sonnet 4.6 94.1

Source: Official Judgemark v2.1 leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0.00 20.0 40.0 60.0 80.0 100.0 2025 2026 0.00 20.0 40.0 60.0 80.0 100.0 2025 2026

Source: Official Judgemark v2.1 leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

Replaced by Judgemark v4

More in Writing and taste

All 30 in Writing and taste
  • Design Bench (Moonshot) In use Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. Kimi K3.1 Pro 957
  • WritingBench Fading Models complete 1,000 real-world writing requests across 6 domains and 100 subdomains. A judge model scores each reply on five query-specific criteria. GPT-5 83.9
  • Creative Writing (MiniMax in-house) Only in system cards Models write creative pieces for real Hailuo AI user requests. Automatic assessments score the writing. MiniMax-Text-01 81.3%
  • DeepSeek Chinese functional writing evaluation Only in system cards Models answer 3,170 everyday Chinese writing prompts such as reports, emails and speeches. Pairwise judging against a baseline model gives a win rate. DeepSeek-V4-Pro 62.6%
  • Slide generation human evaluation (Z.ai) Only in system cards People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. GLM-5 67.5%
  • Task-dependent output homogenization evaluation Only in system cards Models answer prompts several times. A judge scores functional diversity where it is wanted and consistency where one answer is right. Muse Spark 0.21