CLEF JOKER 2025 Task 3: Onomastic Wordplay Translation
Models translate English names containing wordplay into French using a description. The organizers check exact matches and have translators assess alternatives.
The organizers tested 2,333 names and received 20 runs from four teams. Human review covered a sample of 203 source names.
- Made by
- CLEF JOKER organizers
- Released
- October 2024
- Kind
- Translation
- Level
- University
- Scored
- Graded by people
- Answers
- Open-ended
- Tasks
- Public, plus a private test set
- Measures
- Capability
- Subject
- Languages and literature
- Interaction
- Single turn
- Refresh
- New versions replace it
- Run by
- A host
VerbaNex_gpt4o tops the leaderboard with 39%
Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 39.0 VerbaNex_gpt4o
- 22.9 pjmathematician_Q332
- 21.1 pjmathematician_Q314
- 14.8 duth_Helsinki
- 14.7 duth_xanthi_Helsinki-NLP-opus-mt-tc-big-en-fr
- 14.5 Cryptix_flanT5
- 11.8 copy
- 10.7 duth_xanthi_facebook-nllb-200-1.3B
- 10.7 duth_xanthi_facebook-nllb-200-distilled-600M
- 10.4 duth_xanthi_MarianMT_BLOOM
- 10.3 duth_xanthi_Helsinki-NLP-opus-mt-en-fr
- 8.57 duth_xanthi_t5-base
- 8.53 duth_xanthi_t5-small
- 4.71 duth_xanthi_facebook-m2m100_1.2B
- 4.37 duth_xanthi_facebook-m2m100_418M
- 4.20 team1_gemma2b_v2
- 0.04 duth_hybrid_v1
- 0.0 duth_xanthi_MarianMT_LLM_Prompting
- 0.0 Skommarkhos_Lucie-7B-Instruct_SFT_Q8B_LoRA
- 1 VerbaNex_gpt4o 39%
- 2 pjmathematician_Q332 22.9%
- 3 pjmathematician_Q314 21.1%
- 4 duth_Helsinki 14.8%
- 5 duth_xanthi_Helsinki-NLP-opus-mt-tc-big-en-fr 14.7%
- 6 Cryptix_flanT5 14.5%
- 7 copy 11.8%
- 8 duth_xanthi_facebook-nllb-200-1.3B 10.7%
- 9 duth_xanthi_facebook-nllb-200-distilled-600M 10.7%
- 10 duth_xanthi_MarianMT_BLOOM 10.4%
- 11 duth_xanthi_Helsinki-NLP-opus-mt-en-fr 10.3%
- 12 duth_xanthi_t5-base 8.57%
- 13 duth_xanthi_t5-small 8.53%
- 14 duth_xanthi_facebook-m2m100_1.2B 4.71%
- 15 duth_xanthi_facebook-m2m100_418M 4.37%
- 16 team1_gemma2b_v2 4.20%
- 17 duth_hybrid_v1 0.04%
- 18 duth_xanthi_MarianMT_LLM_Prompting 0%
- 19 Skommarkhos_Lucie-7B-Instruct_SFT_Q8B_LoRA 0%
Source: The CLEF JOKER 2025 Task 3: Onomastic Wordplay Translation paper
| # | Entry | Score | Setting | Submitted by | Date | Source |
|---|---|---|---|---|---|---|
| 1 | VerbaNex_gpt4o | 39% | VerbaNex | Official leaderboard | ||
| 2 | pjmathematician_Q332 | 22.9% | pjmathematician | Official leaderboard | ||
| 3 | pjmathematician_Q314 | 21.1% | pjmathematician | Official leaderboard | ||
| 4 | duth_Helsinki | 14.8% | duth | Official leaderboard | ||
| 5 | duth_xanthi_Helsinki-NLP-opus-mt-tc-big-en-fr | 14.7% | duth | Official leaderboard | ||
| 6 | Cryptix_flanT5 | 14.5% | Cryptix | Official leaderboard | ||
| 7 | copy | 11.8% | Official leaderboard | |||
| 8 | duth_xanthi_facebook-nllb-200-1.3B | 10.7% | duth | Official leaderboard | ||
| 9 | duth_xanthi_facebook-nllb-200-distilled-600M | 10.7% | duth | Official leaderboard | ||
| 10 | duth_xanthi_MarianMT_BLOOM | 10.4% | duth | Official leaderboard | ||
| 11 | duth_xanthi_Helsinki-NLP-opus-mt-en-fr | 10.3% | duth | Official leaderboard | ||
| 12 | duth_xanthi_t5-base | 8.57% | duth | Official leaderboard | ||
| 13 | duth_xanthi_t5-small | 8.53% | duth | Official leaderboard | ||
| 14 | duth_xanthi_facebook-m2m100_1.2B | 4.71% | duth | Official leaderboard | ||
| 15 | duth_xanthi_facebook-m2m100_418M | 4.37% | duth | Official leaderboard | ||
| 16 | team1_gemma2b_v2 | 4.20% | team1 | Official leaderboard | ||
| 17 | duth_hybrid_v1 | 0.04% | duth | Official leaderboard | ||
| 18 | duth_xanthi_MarianMT_LLM_Prompting | 0% | duth | Official leaderboard | ||
| 19 | Skommarkhos_Lucie-7B-Instruct_SFT_Q8B_LoRA | 0% | Skommarkhos | Official leaderboard |
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
Versions
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5%
- MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language.
Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7%
- Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Claude Opus 5 95.7%
- MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access Public tasks, plus a private test set.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the benchmark’s paper (19 entries).
- Pass Models scored 19 entries on its leaderboard, none a model we track.
- Note Newest score No scores yet.
- Note Frontier models No models scored yet.
- Note Headroom No scores yet.
Leaderboard import 3 pass
- Pass Importer We read the board on ceur-ws.org: 19 entries at the last read, none a model we track.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.