CLEF JOKER 2024 Task 2: Humour Classification According to Genre and Technique
Models assign one of six humor techniques to an English text. Organizers rank runs by accuracy and report F1 for each technique.
Eighteen teams submitted 54 runs, including GPT-4o and Mistral-7B systems. The organizers scored 722 of the 6,642 unlabeled test inputs.
- Made by
- CLEF JOKER organizers
- Released
- September 2023
- Kind
- Reading comprehension
- Level
- School
- Scored
- Checked automatically
- Answers
- Multiple choice
- Measures
- Capability
- Subject
- Languages and literature
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- A host
ORPAILLEUR_mistral-7b-ens tops the leaderboard with 76%
Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 76.0 ORPAILLEUR_mistral-7b-ens
- 74.0 ORPAILLEUR_llama2-ens
- 73.0 ORPAILLEUR_llama3-8b-ens
- 72.0 ORPAILLEUR_mistral-7b-high
- 71.0 ORPAILLEUR_llama2-high
- 70.0 Code Rangers_roberta
- 70.0 CYUT_llama3-fine-tuning
- 70.0 ORPAILLEUR_llama2-low
- 70.0 ORPAILLEUR_llama3-8b-high
- 70.0 ORPAILLEUR_llama3-8b-low
- 69.0 PunDerstand_DeBERTa
- 68.0 Arampatzis_BERT
- 68.0 Arampatzis_deberta
- 68.0 Arampatzis_DistilBertTokenizer
- 68.0 PunDerstand_DeBERTaSampled
- 67.0 CodingRangers_bert_uncased
- 67.0 DadJokers_bert_base_uncased
- 67.0 NLPalma_BERTd
- 67.0 ORPAILLEUR_mistral-7b-low
- 67.0 PunDerstand_GuidedAnnotation
- 66.0 Arampatzis_MLP
- 66.0 Arampatzis_Roberta
- 66.0 Demonteam_BERTM
- 65.0 Arampatzis_Albert
- 65.0 Arampatzis_XLNet
- 63.0 UAms_BERT_ft
- 60.0 Arampatzis_electra
- 60.0 NLPalma_PREDCNN
- 60.0 VayamSolveKurmaha_BERT
- 59.0 NaiveNeuron_fastText
- 56.0 DadJokers_RandomForest_MLP_Ensemble
- 55.0 HumourInsights_Random Forest
- 53.0 Dajana&Kathy_Joker_LogisticRegression
- 53.0 NaiveNeuron_llama3:70b_rag
- 53.0 NaiveNeuron_llama3:70b_rag-uae
- 53.0 team1_FRANE_AND_ANDREA_LogisticRegression
- 53.0 team1_Petra_and_Regina_LogisticRegression
- 53.0 UBO_RubyAiYoungTeam
- 52.0 DadJokers_RandomForest
- 51.0 Tomislav&Rowan_SVM
- 48.0 AB&DPV_MLP3000params
- 48.0 Tomislav&Rowan_LogisticRegression
- 47.0 PunDerstand_GPT4oFewShot
- 44.0 Tomislav&Rowan_NaiveBayes
- 38.0 AB&DPV_RandomForestClassifier250
- 38.0 AB&DPV_RandomForestClassifier500
- 37.0 AB&DPV_MLP2000
- 37.0 AB&DPV_MLP3000
- 36.0 CYUT_GPT-4
- 29.0 AB&DPV_DecisionTreeClassifier
- 27.0 AB&DPV_GaussianNB
- 19.0 CYUT_roBERTa-fine-tuning
- 1 ORPAILLEUR_mistral-7b-ens 76%
- 2 ORPAILLEUR_llama2-ens 74%
- 3 ORPAILLEUR_llama3-8b-ens 73%
- 4 ORPAILLEUR_mistral-7b-high 72%
- 5 ORPAILLEUR_llama2-high 71%
- 6 Code Rangers_roberta 70%
- 7 CYUT_llama3-fine-tuning 70%
- 8 ORPAILLEUR_llama2-low 70%
- 9 ORPAILLEUR_llama3-8b-high 70%
- 10 ORPAILLEUR_llama3-8b-low 70%
- 11 PunDerstand_DeBERTa 69%
- 12 Arampatzis_BERT 68%
- 13 Arampatzis_deberta 68%
- 14 Arampatzis_DistilBertTokenizer 68%
- 15 PunDerstand_DeBERTaSampled 68%
- 16 CodingRangers_bert_uncased 67%
- 17 DadJokers_bert_base_uncased 67%
- 18 NLPalma_BERTd 67%
- 19 ORPAILLEUR_mistral-7b-low 67%
- 20 PunDerstand_GuidedAnnotation 67%
- 21 Arampatzis_MLP 66%
- 22 Arampatzis_Roberta 66%
- 23 Demonteam_BERTM 66%
- 24 Arampatzis_Albert 65%
- 25 Arampatzis_XLNet 65%
- 26 UAms_BERT_ft 63%
- 27 Arampatzis_electra 60%
- 28 NLPalma_PREDCNN 60%
- 29 VayamSolveKurmaha_BERT 60%
- 30 NaiveNeuron_fastText 59%
- 31 DadJokers_RandomForest_MLP_Ensemble 56%
- 32 HumourInsights_Random Forest 55%
- 33 Dajana&Kathy_Joker_LogisticRegression 53%
- 34 NaiveNeuron_llama3:70b_rag 53%
- 35 NaiveNeuron_llama3:70b_rag-uae 53%
- 36 team1_FRANE_AND_ANDREA_LogisticRegression 53%
- 37 team1_Petra_and_Regina_LogisticRegression 53%
- 38 UBO_RubyAiYoungTeam 53%
- 39 DadJokers_RandomForest 52%
- 40 Tomislav&Rowan_SVM 51%
- 41 AB&DPV_MLP3000params 48%
- 42 Tomislav&Rowan_LogisticRegression 48%
- 43 PunDerstand_GPT4oFewShot 47%
- 44 Tomislav&Rowan_NaiveBayes 44%
- 45 AB&DPV_RandomForestClassifier250 38%
- 46 AB&DPV_RandomForestClassifier500 38%
- 47 AB&DPV_MLP2000 37%
- 48 AB&DPV_MLP3000 37%
- 49 CYUT_GPT-4 36%
- 50 AB&DPV_DecisionTreeClassifier 29%
- 51 AB&DPV_GaussianNB 27%
- 52 CYUT_roBERTa-fine-tuning 19%
Source: The CLEF JOKER 2024 Task 2: Humour Classification According to Genre and Technique paper
| # | Entry | Score | Setting | Submitted by | Date | Source |
|---|---|---|---|---|---|---|
| 1 | ORPAILLEUR_mistral-7b-ens | 76% | ORPAILLEUR | Official leaderboard | ||
| 2 | ORPAILLEUR_llama2-ens | 74% | ORPAILLEUR | Official leaderboard | ||
| 3 | ORPAILLEUR_llama3-8b-ens | 73% | ORPAILLEUR | Official leaderboard | ||
| 4 | ORPAILLEUR_mistral-7b-high | 72% | ORPAILLEUR | Official leaderboard | ||
| 5 | ORPAILLEUR_llama2-high | 71% | ORPAILLEUR | Official leaderboard | ||
| 6 | Code Rangers_roberta | 70% | Code Rangers | Official leaderboard | ||
| 7 | CYUT_llama3-fine-tuning | 70% | CYUT | Official leaderboard | ||
| 8 | ORPAILLEUR_llama2-low | 70% | ORPAILLEUR | Official leaderboard | ||
| 9 | ORPAILLEUR_llama3-8b-high | 70% | ORPAILLEUR | Official leaderboard | ||
| 10 | ORPAILLEUR_llama3-8b-low | 70% | ORPAILLEUR | Official leaderboard | ||
| 11 | PunDerstand_DeBERTa | 69% | PunDerstand | Official leaderboard | ||
| 12 | Arampatzis_BERT | 68% | Arampatzis | Official leaderboard | ||
| 13 | Arampatzis_deberta | 68% | Arampatzis | Official leaderboard | ||
| 14 | Arampatzis_DistilBertTokenizer | 68% | Arampatzis | Official leaderboard | ||
| 15 | PunDerstand_DeBERTaSampled | 68% | PunDerstand | Official leaderboard | ||
| 16 | CodingRangers_bert_uncased | 67% | CodingRangers | Official leaderboard | ||
| 17 | DadJokers_bert_base_uncased | 67% | DadJokers | Official leaderboard | ||
| 18 | NLPalma_BERTd | 67% | NLPalma | Official leaderboard | ||
| 19 | ORPAILLEUR_mistral-7b-low | 67% | ORPAILLEUR | Official leaderboard | ||
| 20 | PunDerstand_GuidedAnnotation | 67% | PunDerstand | Official leaderboard | ||
| 21 | Arampatzis_MLP | 66% | Arampatzis | Official leaderboard | ||
| 22 | Arampatzis_Roberta | 66% | Arampatzis | Official leaderboard | ||
| 23 | Demonteam_BERTM | 66% | Demonteam | Official leaderboard | ||
| 24 | Arampatzis_Albert | 65% | Arampatzis | Official leaderboard | ||
| 25 | Arampatzis_XLNet | 65% | Arampatzis | Official leaderboard | ||
| 26 | UAms_BERT_ft | 63% | UAms | Official leaderboard | ||
| 27 | Arampatzis_electra | 60% | Arampatzis | Official leaderboard | ||
| 28 | NLPalma_PREDCNN | 60% | NLPalma | Official leaderboard | ||
| 29 | VayamSolveKurmaha_BERT | 60% | VayamSolveKurmaha | Official leaderboard | ||
| 30 | NaiveNeuron_fastText | 59% | NaiveNeuron | Official leaderboard | ||
| 31 | DadJokers_RandomForest_MLP_Ensemble | 56% | DadJokers | Official leaderboard | ||
| 32 | HumourInsights_Random Forest | 55% | HumourInsights | Official leaderboard | ||
| 33 | Dajana&Kathy_Joker_LogisticRegression | 53% | Dajana&Kathy | Official leaderboard | ||
| 34 | NaiveNeuron_llama3:70b_rag | 53% | NaiveNeuron | Official leaderboard | ||
| 35 | NaiveNeuron_llama3:70b_rag-uae | 53% | NaiveNeuron | Official leaderboard | ||
| 36 | team1_FRANE_AND_ANDREA_LogisticRegression | 53% | team1 | Official leaderboard | ||
| 37 | team1_Petra_and_Regina_LogisticRegression | 53% | team1 | Official leaderboard | ||
| 38 | UBO_RubyAiYoungTeam | 53% | UBO | Official leaderboard | ||
| 39 | DadJokers_RandomForest | 52% | DadJokers | Official leaderboard | ||
| 40 | Tomislav&Rowan_SVM | 51% | Tomislav&Rowan | Official leaderboard | ||
| 41 | AB&DPV_MLP3000params | 48% | AB&DPV | Official leaderboard | ||
| 42 | Tomislav&Rowan_LogisticRegression | 48% | Tomislav&Rowan | Official leaderboard | ||
| 43 | PunDerstand_GPT4oFewShot | 47% | PunDerstand | Official leaderboard | ||
| 44 | Tomislav&Rowan_NaiveBayes | 44% | Tomislav&Rowan | Official leaderboard | ||
| 45 | AB&DPV_RandomForestClassifier250 | 38% | AB&DPV | Official leaderboard | ||
| 46 | AB&DPV_RandomForestClassifier500 | 38% | AB&DPV | Official leaderboard | ||
| 47 | AB&DPV_MLP2000 | 37% | AB&DPV | Official leaderboard | ||
| 48 | AB&DPV_MLP3000 | 37% | AB&DPV | Official leaderboard | ||
| 49 | CYUT_GPT-4 | 36% | CYUT | Official leaderboard | ||
| 50 | AB&DPV_DecisionTreeClassifier | 29% | AB&DPV | Official leaderboard | ||
| 51 | AB&DPV_GaussianNB | 27% | AB&DPV | Official leaderboard | ||
| 52 | CYUT_roBERTa-fine-tuning | 19% | CYUT | Official leaderboard |
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
7 of 8 checks pass, 1 warning. Look first at the Task access check. We don’t know whether the tasks are public. Every benchmark
Catalog record 1 warning 1 pass
- Warning Task access We don’t know whether the tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the benchmark’s paper (52 entries).
- Pass Models scored 52 entries on its leaderboard, none a model we track.
- Note Newest score No scores yet.
- Note Frontier models No models scored yet.
- Note Headroom No scores yet.
Leaderboard import 3 pass
- Pass Importer We read the board on ceur-ws.org: 52 entries at the last read, none a model we track.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.