Benchmark Almanac

CLEF JOKER 2026 Task 3: Onomastic Wordplay Translation

Gemini 3 Flash 71.9% leads since December 2025
Not in model cards

Models translate English names containing wordplay into French using a short description. The official score accepts reference matches and alternatives approved by translators.

Four teams submitted 55 valid runs. The official score covers 313 names. The full reference comparison covers 2,598 names.

Released
October 2025
Kind
Translation
Level
University
Scored
Graded by people
Answers
Open-ended
Tasks
Public, plus a private test set
Measures
Capability
Language
English, French
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
A host
Strongest model tested
Gemini 3 Flash #60 on the capability index
Last model added
December 2025
Scores refreshed
November 20th, 2026
16
Entries in its paper, 3 models we track

Gemini 3 Flash leads with 71.9% among models released by December 2025 Gemini 3 Flash leads with 71.9% among models released by December 2025

Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in December 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Gemini 3 Flash 71.9%
  2. 2 V14_OFFICIAL_B04_1700_2299 43.1%
  3. 3 V18_FRLEAD_STRICT_COPY 43.1%
  4. 4 V18_FRLEAD_WD_STRICT_COPY 43.1%
  5. 5 V21_ASTERIX_COMBO 43.1%
  6. 6 V21_ASTERIX_DIRECT_ONLY 43.1%
  7. 7 Llama 3.1-8B 17.3%
  8. 8 cascade 13.7%
  9. 9 marian-ft_context 13.7%
  10. 10 extraction 13.1%
  11. 11 identity_run 13.1%
  12. 12 marian-ft_barename 7.35%
  13. 13 Qwen3-4B 4.47%
  14. 14 baseline_sub 2.88%
  15. 15 mistral 2.24%
  16. 16 gemma 1.92%

Source: The CLEF JOKER 2026 Task 3: Onomastic Wordplay Translation paper

Source: The CLEF JOKER 2026 Task 3: Onomastic Wordplay Translation paper

#EntryScoreSettingSubmitted byDateSource
1 Gemini 3 Flash 71.9% UBO Dec 2025 Official leaderboard
2 V14_OFFICIAL_B04_1700_2299 43.1% arampageos Official leaderboard
3 V18_FRLEAD_STRICT_COPY 43.1% arampageos Official leaderboard
4 V18_FRLEAD_WD_STRICT_COPY 43.1% arampageos Official leaderboard
5 V21_ASTERIX_COMBO 43.1% arampageos Official leaderboard
6 V21_ASTERIX_DIRECT_ONLY 43.1% arampageos Official leaderboard
7 Llama 3.1-8B 17.3% UBO Jul 2024 Official leaderboard
8 cascade 13.7% ezrasmink Official leaderboard
9 marian-ft_context 13.7% ezrasmink Official leaderboard
10 extraction 13.1% ezrasmink Official leaderboard
11 identity_run 13.1% Baseline Official leaderboard
12 marian-ft_barename 7.35% ezrasmink Official leaderboard
13 Qwen3-4B 4.47% UBO Apr 2025 Official leaderboard
14 baseline_sub 2.88% pjmathematician Official leaderboard
15 mistral 2.24% UBO Official leaderboard
16 gemma 1.92% UBO Official leaderboard
#ModelScoreSettingSubmitted byDateSource
1 Gemini 3 Flash 71.9% UBO Dec 2025 Official leaderboard
2 Llama 3.1-8B 17.3% UBO Jul 2024 Official leaderboard
3 Qwen3-4B 4.47% UBO Apr 2025 Official leaderboard

2 models have held number one in the benchmark’s paper since its release in October 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta Google DeepMind
Benchmark paper Not tested since Dec 2025

Source: The CLEF JOKER 2026 Task 3: Onomastic Wordplay Translation paper

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
In the benchmark’s paper Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Gemini 3 Flash Llama 3.1-8B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Gemini 3 Flash Llama 3.1-8B

Source: Official CLEF JOKER 2026 Task 3: Onomastic Wordplay Translation leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

Versions

More in Multilingual

All 545 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%

Data audit

9 of 10 checks pass, 1 warning. Look first at the Newest score check. The newest came 11 months ago, in December 2025, for Gemini 3 Flash. Every benchmark

Catalog record 2 pass
  • Pass Task access Public tasks, plus a private test set.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 3 pass 1 note
  • Pass Independent runs Scores from the benchmark’s paper (16 entries).
  • Pass Models scored 3 models we track, plus 13 other entries on its leaderboard.
  • Warning Newest score The newest came 11 months ago, in December 2025, for Gemini 3 Flash.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Gemini 3 Flash (#60).
  • Pass Headroom The best is 71.9%, under 85%, so it still separates models.
Leaderboard import 3 pass
  • Pass Importer We read the board on ceur-ws.org: 3 rows for models we track and 13 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check