SemEval 2026 Task 4: Narrative Story Similarity and Narrative Representation Learning
Models choose which story is more similar to a reference story. A second track compares story embeddings. Each track scores accuracy on 400 triples.
The organizers ranked 46 teams across two tracks. COGNAC used GPT-4.1 mini and Gemini 2.5 Flash and led both tracks.
- Made by
- University of Hamburg, Toloka AI, German UDS and TU Darmstadt
- Released
- July 2025
- Kind
- Reading comprehension
- Level
- School
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Items from
- Written by crowdworkers
- Refresh
- One fixed set
- Run by
- Anyone
COGNAC tops the leaderboard with 78%
Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 78.0 COGNAC
- 75.8 FactUEP
- 75.0 AI-Monitors
- 74.8 TeleAI
- 74.3 YNU-HPCC
- 74.0 UTD-HLTRI
- 73.8 JCT
- 73.5 CuriosAI
- 73.0 Yam
- 72.8 CascadeMind
- 70.8 Team CV
- 70.5 hermeneutic_hools
- 70.5 hits_team
- 70.3 NarSiL
- 70.3 NCL&HKU-NarrSim
- 69.8 DUTIR
- 69.5 CophiWue
- 69.3 CITD@UIT
- 69.3 Narrative Team
- 69.3 ttda704
- 68.5 Narrative Nexus
- 68.3 Comhis
- 68.3 harapalb
- 68.0 SoloSemantics
- 67.8 schmerle
- 65.8 L3IRIT
- 65.5 MarSan
- 65.5 NLP-FSDM
- 65.3 Team UBSE
- 65.0 MoodMetric
- 64.5 PEU Lab
- 64.3 CUNI
- 63.5 ChulaNLP
- 62.5 blue
- 61.5 Team HausaNLP
- 61.5 TFB
- 60.3 Spinfo Cologne
- 59.8 LIAAD INESCTEC
- 59.5 Cryptix
- 59.3 VerbaNexAI
- 59.0 CICL26
- 58.5 Duluth
- 58.0 Mendel292
- 57.8 IIITH Boys
- 57.3 Lacuna Inc.
- 55.5 PLlama
- 1 COGNAC Track A 78%
- 2 FactUEP Track A 75.8%
- 3 AI-Monitors Track A 75%
- 4 TeleAI Track A 74.8%
- 5 YNU-HPCC Track A 74.3%
- 6 UTD-HLTRI Track A 74%
- 7 JCT Track A 73.8%
- 8 CuriosAI Track A 73.5%
- 9 Yam Track A 73%
- 10 CascadeMind Track A 72.8%
- 11 Team CV Track A 70.8%
- 12 hermeneutic_hools Track A 70.5%
- 13 hits_team Track B 70.5%
- 14 NarSiL Track A 70.3%
- 15 NCL&HKU-NarrSim Track A 70.3%
- 16 DUTIR Track A 69.8%
- 17 CophiWue Track A 69.5%
- 18 CITD@UIT Track A 69.3%
- 19 Narrative Team Track B 69.3%
- 20 ttda704 Track A 69.3%
- 21 Narrative Nexus Track A 68.5%
- 22 Comhis Track A 68.3%
- 23 harapalb Track A 68.3%
- 24 SoloSemantics Track A 68%
- 25 schmerle Track A 67.8%
- 26 L3IRIT Track A 65.8%
- 27 MarSan Track B 65.5%
- 28 NLP-FSDM Track A 65.5%
- 29 Team UBSE Track A 65.3%
- 30 MoodMetric Track A 65%
- 31 PEU Lab Track A 64.5%
- 32 CUNI Track B 64.3%
- 33 ChulaNLP Track A 63.5%
- 34 blue Track A 62.5%
- 35 Team HausaNLP Track A 61.5%
- 36 TFB Track B 61.5%
- 37 Spinfo Cologne Track A 60.3%
- 38 LIAAD INESCTEC Track A 59.8%
- 39 Cryptix Track A 59.5%
- 40 VerbaNexAI Track B 59.3%
- 41 CICL26 Track A 59%
- 42 Duluth Track A 58.5%
- 43 Mendel292 Track B 58%
- 44 IIITH Boys Track A 57.8%
- 45 Lacuna Inc. Track A 57.3%
- 46 PLlama Track A 55.5%
Source: Official SemEval 2026 Task 4: Narrative Story Similarity and Narrative Representation Learning leaderboard
| # | Entry | Score | Setting | Submitted by | Date | Link | Source |
|---|---|---|---|---|---|---|---|
| 1 | COGNAC | 78% | Track A | COGNAC | aclanthology.org | Official leaderboard | |
| 2 | FactUEP | 75.8% | Track A | FactUEP | aclanthology.org | Official leaderboard | |
| 3 | AI-Monitors | 75% | Track A | AI-Monitors | aclanthology.org | Official leaderboard | |
| 4 | TeleAI | 74.8% | Track A | TeleAI | aclanthology.org | Official leaderboard | |
| 5 | YNU-HPCC | 74.3% | Track A | YNU-HPCC | aclanthology.org | Official leaderboard | |
| 6 | UTD-HLTRI | 74% | Track A | UTD-HLTRI | aclanthology.org | Official leaderboard | |
| 7 | JCT | 73.8% | Track A | JCT | aclanthology.org | Official leaderboard | |
| 8 | CuriosAI | 73.5% | Track A | CuriosAI | aclanthology.org | Official leaderboard | |
| 9 | Yam | 73% | Track A | Yam | aclanthology.org | Official leaderboard | |
| 10 | CascadeMind | 72.8% | Track A | CascadeMind | aclanthology.org | Official leaderboard | |
| 11 | Team CV | 70.8% | Track A | Team CV | aclanthology.org | Official leaderboard | |
| 12 | hermeneutic_hools | 70.5% | Track A | hermeneutic_hools | aclanthology.org | Official leaderboard | |
| 13 | hits_team | 70.5% | Track B | hits_team | aclanthology.org | Official leaderboard | |
| 14 | NarSiL | 70.3% | Track A | NarSiL | aclanthology.org | Official leaderboard | |
| 15 | NCL&HKU-NarrSim | 70.3% | Track A | NCL&HKU-NarrSim | aclanthology.org | Official leaderboard | |
| 16 | DUTIR | 69.8% | Track A | DUTIR | aclanthology.org | Official leaderboard | |
| 17 | CophiWue | 69.5% | Track A | CophiWue | aclanthology.org | Official leaderboard | |
| 18 | CITD@UIT | 69.3% | Track A | CITD@UIT | aclanthology.org | Official leaderboard | |
| 19 | Narrative Team | 69.3% | Track B | Narrative Team | aclanthology.org | Official leaderboard | |
| 20 | ttda704 | 69.3% | Track A | ttda704 | aclanthology.org | Official leaderboard | |
| 21 | Narrative Nexus | 68.5% | Track A | Narrative Nexus | aclanthology.org | Official leaderboard | |
| 22 | Comhis | 68.3% | Track A | Comhis | aclanthology.org | Official leaderboard | |
| 23 | harapalb | 68.3% | Track A | harapalb | aclanthology.org | Official leaderboard | |
| 24 | SoloSemantics | 68% | Track A | SoloSemantics | aclanthology.org | Official leaderboard | |
| 25 | schmerle | 67.8% | Track A | schmerle | aclanthology.org | Official leaderboard | |
| 26 | L3IRIT | 65.8% | Track A | L3IRIT | aclanthology.org | Official leaderboard | |
| 27 | MarSan | 65.5% | Track B | MarSan | aclanthology.org | Official leaderboard | |
| 28 | NLP-FSDM | 65.5% | Track A | NLP-FSDM | aclanthology.org | Official leaderboard | |
| 29 | Team UBSE | 65.3% | Track A | Team UBSE | aclanthology.org | Official leaderboard | |
| 30 | MoodMetric | 65% | Track A | MoodMetric | aclanthology.org | Official leaderboard | |
| 31 | PEU Lab | 64.5% | Track A | PEU Lab | aclanthology.org | Official leaderboard | |
| 32 | CUNI | 64.3% | Track B | CUNI | aclanthology.org | Official leaderboard | |
| 33 | ChulaNLP | 63.5% | Track A | ChulaNLP | aclanthology.org | Official leaderboard | |
| 34 | blue | 62.5% | Track A | blue | aclanthology.org | Official leaderboard | |
| 35 | Team HausaNLP | 61.5% | Track A | Team HausaNLP | aclanthology.org | Official leaderboard | |
| 36 | TFB | 61.5% | Track B | TFB | aclanthology.org | Official leaderboard | |
| 37 | Spinfo Cologne | 60.3% | Track A | Spinfo Cologne | aclanthology.org | Official leaderboard | |
| 38 | LIAAD INESCTEC | 59.8% | Track A | LIAAD INESCTEC | aclanthology.org | Official leaderboard | |
| 39 | Cryptix | 59.5% | Track A | Cryptix | aclanthology.org | Official leaderboard | |
| 40 | VerbaNexAI | 59.3% | Track B | VerbaNexAI | aclanthology.org | Official leaderboard | |
| 41 | CICL26 | 59% | Track A | CICL26 | aclanthology.org | Official leaderboard | |
| 42 | Duluth | 58.5% | Track A | Duluth | aclanthology.org | Official leaderboard | |
| 43 | Mendel292 | 58% | Track B | Mendel292 | aclanthology.org | Official leaderboard | |
| 44 | IIITH Boys | 57.8% | Track A | IIITH Boys | aclanthology.org | Official leaderboard | |
| 45 | Lacuna Inc. | 57.3% | Track A | Lacuna Inc. | aclanthology.org | Official leaderboard | |
| 46 | PLlama | 55.5% | Track A | PLlama | aclanthology.org | Official leaderboard |
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8%
- MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.
Seed1.8 92.3%
- SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9%
- WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9%
- BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly.
Gemma 4 31B IT 74.4%
- HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (46 entries).
- Pass Models scored 46 entries on its leaderboard, none a model we track.
- Note Newest score No scores yet.
- Note Frontier models No models scored yet.
- Note Headroom No scores yet.
Leaderboard import 3 pass
- Pass Importer We read the board on narrative-similarity-task.github.io: 46 entries at the last read, none a model we track.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.