Benchmark Almanac

CLEF JOKER 2024 Task 2: Humour Classification According to Genre and Technique

ORPAILLEUR_mistral-7b-ens 76% tops its leaderboard
Not in model cards

Models assign one of six humor techniques to an English text. Organizers rank runs by accuracy and report F1 for each technique.

Eighteen teams submitted 54 runs, including GPT-4o and Mistral-7B systems. The organizers scored 722 of the 6,642 unlabeled test inputs.

Released
September 2023
Kind
Reading comprehension
Level
School
Scored
Checked automatically
Answers
Multiple choice
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
A host
52
Entries in its paper, 0 models we track

ORPAILLEUR_mistral-7b-ens tops the leaderboard with 76%

Best score per entry from the benchmark's paper, read on November 20th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

52 entries. Scroll sideways for the rest.

  1. 76.0 ORPAILLEUR_mistral-7b-ens
  2. 74.0 ORPAILLEUR_llama2-ens
  3. 73.0 ORPAILLEUR_llama3-8b-ens
  4. 72.0 ORPAILLEUR_mistral-7b-high
  5. 71.0 ORPAILLEUR_llama2-high
  6. 70.0 Code Rangers_roberta
  7. 70.0 CYUT_llama3-fine-tuning
  8. 70.0 ORPAILLEUR_llama2-low
  9. 70.0 ORPAILLEUR_llama3-8b-high
  10. 70.0 ORPAILLEUR_llama3-8b-low
  11. 69.0 PunDerstand_DeBERTa
  12. 68.0 Arampatzis_BERT
  13. 68.0 Arampatzis_deberta
  14. 68.0 Arampatzis_DistilBertTokenizer
  15. 68.0 PunDerstand_DeBERTaSampled
  16. 67.0 CodingRangers_bert_uncased
  17. 67.0 DadJokers_bert_base_uncased
  18. 67.0 NLPalma_BERTd
  19. 67.0 ORPAILLEUR_mistral-7b-low
  20. 67.0 PunDerstand_GuidedAnnotation
  21. 66.0 Arampatzis_MLP
  22. 66.0 Arampatzis_Roberta
  23. 66.0 Demonteam_BERTM
  24. 65.0 Arampatzis_Albert
  25. 65.0 Arampatzis_XLNet
  26. 63.0 UAms_BERT_ft
  27. 60.0 Arampatzis_electra
  28. 60.0 NLPalma_PREDCNN
  29. 60.0 VayamSolveKurmaha_BERT
  30. 59.0 NaiveNeuron_fastText
  31. 56.0 DadJokers_RandomForest_MLP_Ensemble
  32. 55.0 HumourInsights_Random Forest
  33. 53.0 Dajana&Kathy_Joker_LogisticRegression
  34. 53.0 NaiveNeuron_llama3:70b_rag
  35. 53.0 NaiveNeuron_llama3:70b_rag-uae
  36. 53.0 team1_FRANE_AND_ANDREA_LogisticRegression
  37. 53.0 team1_Petra_and_Regina_LogisticRegression
  38. 53.0 UBO_RubyAiYoungTeam
  39. 52.0 DadJokers_RandomForest
  40. 51.0 Tomislav&Rowan_SVM
  41. 48.0 AB&DPV_MLP3000params
  42. 48.0 Tomislav&Rowan_LogisticRegression
  43. 47.0 PunDerstand_GPT4oFewShot
  44. 44.0 Tomislav&Rowan_NaiveBayes
  45. 38.0 AB&DPV_RandomForestClassifier250
  46. 38.0 AB&DPV_RandomForestClassifier500
  47. 37.0 AB&DPV_MLP2000
  48. 37.0 AB&DPV_MLP3000
  49. 36.0 CYUT_GPT-4
  50. 29.0 AB&DPV_DecisionTreeClassifier
  51. 27.0 AB&DPV_GaussianNB
  52. 19.0 CYUT_roBERTa-fine-tuning
  1. 1 ORPAILLEUR_mistral-7b-ens 76%
  2. 2 ORPAILLEUR_llama2-ens 74%
  3. 3 ORPAILLEUR_llama3-8b-ens 73%
  4. 4 ORPAILLEUR_mistral-7b-high 72%
  5. 5 ORPAILLEUR_llama2-high 71%
  6. 6 Code Rangers_roberta 70%
  7. 7 CYUT_llama3-fine-tuning 70%
  8. 8 ORPAILLEUR_llama2-low 70%
  9. 9 ORPAILLEUR_llama3-8b-high 70%
  10. 10 ORPAILLEUR_llama3-8b-low 70%
  11. 11 PunDerstand_DeBERTa 69%
  12. 12 Arampatzis_BERT 68%
  13. 13 Arampatzis_deberta 68%
  14. 14 Arampatzis_DistilBertTokenizer 68%
  15. 15 PunDerstand_DeBERTaSampled 68%
  16. 16 CodingRangers_bert_uncased 67%
  17. 17 DadJokers_bert_base_uncased 67%
  18. 18 NLPalma_BERTd 67%
  19. 19 ORPAILLEUR_mistral-7b-low 67%
  20. 20 PunDerstand_GuidedAnnotation 67%
  21. 21 Arampatzis_MLP 66%
  22. 22 Arampatzis_Roberta 66%
  23. 23 Demonteam_BERTM 66%
  24. 24 Arampatzis_Albert 65%
  25. 25 Arampatzis_XLNet 65%
  26. 26 UAms_BERT_ft 63%
  27. 27 Arampatzis_electra 60%
  28. 28 NLPalma_PREDCNN 60%
  29. 29 VayamSolveKurmaha_BERT 60%
  30. 30 NaiveNeuron_fastText 59%
  31. 31 DadJokers_RandomForest_MLP_Ensemble 56%
  32. 32 HumourInsights_Random Forest 55%
  33. 33 Dajana&Kathy_Joker_LogisticRegression 53%
  34. 34 NaiveNeuron_llama3:70b_rag 53%
  35. 35 NaiveNeuron_llama3:70b_rag-uae 53%
  36. 36 team1_FRANE_AND_ANDREA_LogisticRegression 53%
  37. 37 team1_Petra_and_Regina_LogisticRegression 53%
  38. 38 UBO_RubyAiYoungTeam 53%
  39. 39 DadJokers_RandomForest 52%
  40. 40 Tomislav&Rowan_SVM 51%
  41. 41 AB&DPV_MLP3000params 48%
  42. 42 Tomislav&Rowan_LogisticRegression 48%
  43. 43 PunDerstand_GPT4oFewShot 47%
  44. 44 Tomislav&Rowan_NaiveBayes 44%
  45. 45 AB&DPV_RandomForestClassifier250 38%
  46. 46 AB&DPV_RandomForestClassifier500 38%
  47. 47 AB&DPV_MLP2000 37%
  48. 48 AB&DPV_MLP3000 37%
  49. 49 CYUT_GPT-4 36%
  50. 50 AB&DPV_DecisionTreeClassifier 29%
  51. 51 AB&DPV_GaussianNB 27%
  52. 52 CYUT_roBERTa-fine-tuning 19%

Source: The CLEF JOKER 2024 Task 2: Humour Classification According to Genre and Technique paper

#EntryScoreSettingSubmitted byDateSource
1 ORPAILLEUR_mistral-7b-ens 76% ORPAILLEUR Official leaderboard
2 ORPAILLEUR_llama2-ens 74% ORPAILLEUR Official leaderboard
3 ORPAILLEUR_llama3-8b-ens 73% ORPAILLEUR Official leaderboard
4 ORPAILLEUR_mistral-7b-high 72% ORPAILLEUR Official leaderboard
5 ORPAILLEUR_llama2-high 71% ORPAILLEUR Official leaderboard
6 Code Rangers_roberta 70% Code Rangers Official leaderboard
7 CYUT_llama3-fine-tuning 70% CYUT Official leaderboard
8 ORPAILLEUR_llama2-low 70% ORPAILLEUR Official leaderboard
9 ORPAILLEUR_llama3-8b-high 70% ORPAILLEUR Official leaderboard
10 ORPAILLEUR_llama3-8b-low 70% ORPAILLEUR Official leaderboard
11 PunDerstand_DeBERTa 69% PunDerstand Official leaderboard
12 Arampatzis_BERT 68% Arampatzis Official leaderboard
13 Arampatzis_deberta 68% Arampatzis Official leaderboard
14 Arampatzis_DistilBertTokenizer 68% Arampatzis Official leaderboard
15 PunDerstand_DeBERTaSampled 68% PunDerstand Official leaderboard
16 CodingRangers_bert_uncased 67% CodingRangers Official leaderboard
17 DadJokers_bert_base_uncased 67% DadJokers Official leaderboard
18 NLPalma_BERTd 67% NLPalma Official leaderboard
19 ORPAILLEUR_mistral-7b-low 67% ORPAILLEUR Official leaderboard
20 PunDerstand_GuidedAnnotation 67% PunDerstand Official leaderboard
21 Arampatzis_MLP 66% Arampatzis Official leaderboard
22 Arampatzis_Roberta 66% Arampatzis Official leaderboard
23 Demonteam_BERTM 66% Demonteam Official leaderboard
24 Arampatzis_Albert 65% Arampatzis Official leaderboard
25 Arampatzis_XLNet 65% Arampatzis Official leaderboard
26 UAms_BERT_ft 63% UAms Official leaderboard
27 Arampatzis_electra 60% Arampatzis Official leaderboard
28 NLPalma_PREDCNN 60% NLPalma Official leaderboard
29 VayamSolveKurmaha_BERT 60% VayamSolveKurmaha Official leaderboard
30 NaiveNeuron_fastText 59% NaiveNeuron Official leaderboard
31 DadJokers_RandomForest_MLP_Ensemble 56% DadJokers Official leaderboard
32 HumourInsights_Random Forest 55% HumourInsights Official leaderboard
33 Dajana&Kathy_Joker_LogisticRegression 53% Dajana&Kathy Official leaderboard
34 NaiveNeuron_llama3:70b_rag 53% NaiveNeuron Official leaderboard
35 NaiveNeuron_llama3:70b_rag-uae 53% NaiveNeuron Official leaderboard
36 team1_FRANE_AND_ANDREA_LogisticRegression 53% team1 Official leaderboard
37 team1_Petra_and_Regina_LogisticRegression 53% team1 Official leaderboard
38 UBO_RubyAiYoungTeam 53% UBO Official leaderboard
39 DadJokers_RandomForest 52% DadJokers Official leaderboard
40 Tomislav&Rowan_SVM 51% Tomislav&Rowan Official leaderboard
41 AB&DPV_MLP3000params 48% AB&DPV Official leaderboard
42 Tomislav&Rowan_LogisticRegression 48% Tomislav&Rowan Official leaderboard
43 PunDerstand_GPT4oFewShot 47% PunDerstand Official leaderboard
44 Tomislav&Rowan_NaiveBayes 44% Tomislav&Rowan Official leaderboard
45 AB&DPV_RandomForestClassifier250 38% AB&DPV Official leaderboard
46 AB&DPV_RandomForestClassifier500 38% AB&DPV Official leaderboard
47 AB&DPV_MLP2000 37% AB&DPV Official leaderboard
48 AB&DPV_MLP3000 37% AB&DPV Official leaderboard
49 CYUT_GPT-4 36% CYUT Official leaderboard
50 AB&DPV_DecisionTreeClassifier 29% AB&DPV Official leaderboard
51 AB&DPV_GaussianNB 27% AB&DPV Official leaderboard
52 CYUT_roBERTa-fine-tuning 19% CYUT Official leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

7 of 8 checks pass, 1 warning. Look first at the Task access check. We don’t know whether the tasks are public. Every benchmark

Catalog record 1 warning 1 pass
  • Warning Task access We don’t know whether the tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the benchmark’s paper (52 entries).
  • Pass Models scored 52 entries on its leaderboard, none a model we track.
  • Note Newest score No scores yet.
  • Note Frontier models No models scored yet.
  • Note Headroom No scores yet.
Leaderboard import 3 pass
  • Pass Importer We read the board on ceur-ws.org: 52 entries at the last read, none a model we track.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check