Task-dependent output homogenization evaluation
Muse Spark 0.21 Models answer prompts several times. A judge scores functional diversity where it is wanted and consistency where one answer is right.
Reports a homogenization score and a heterogeneity score on a 0 to 1 scale.
- Made by
- Jain et al. (Meta FAIR)
- Released
- September 2025
- Kind
- Creative writing
- Level
- Everyday
- Scored
- Graded by a model
- Answers
- Open-ended
- Tasks
- All public
- Strongest model tested
- Muse Spark #53 on the capability index
- Last in a system card
- June 2026
- Last model added
- June 2026
In the authors’ words
Large language models often generate homogeneous outputs, but whether this is problematic depends on the specific task. For objective math tasks, responses may vary in terms of problem-solving strategy but should maintain the same verifiable answer. Whereas, for creative writing tasks, we often expect variation in key narrative components (e.g. plot, setting, etc.) beyond mere vocabulary diversity. Prior work on homogenization rarely conceptualizes diversity in a task-dependent way. We address this gap with four contributions: (1) a task taxonomy with distinct notions of functional diversity – whether a user would perceive two responses as meaningfully different for a given task; (2) a small user study validating that the taxonomy aligns with human perception of functional diversity; (3) a task-dependent sampling technique that increases diversity only where homogenization is undesired; (4) evidence challenging the perceived diversity-quality trade-off, showing it may stem from mis-conceptualizing both diversity and quality in a task-agnostic way.
Muse Spark leads with 0.21
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 Muse Spark 0.21
Source: Labs’ model cards
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Muse Spark | 0.21 | Jun 2026 | Lab model card |
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Labs’ model cards
Labs report it only in system cards
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.
One system card reports it
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Jun 12, 2026 | Muse Spark system card | 0.21 | Response diversity heterogeneity score (Figure 49). Homogenization score 0.67 | Other |
More in Writing and taste
- Design Bench (Moonshot) In use Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. Kimi K3.1 Pro 957
- WritingBench Fading Models complete 1,000 real-world writing requests across 6 domains and 100 subdomains. A judge model scores each reply on five query-specific criteria. GPT-5 83.9
- Creative Writing (MiniMax in-house) Only in system cards Models write creative pieces for real Hailuo AI user requests. Automatic assessments score the writing.
MiniMax-Text-01 81.3% - DeepSeek Chinese functional writing evaluation Only in system cards Models answer 3,170 everyday Chinese writing prompts such as reports, emails and speeches. Pairwise judging against a baseline model gives a win rate.
DeepSeek-V4-Pro 62.6% - Slide generation human evaluation (Z.ai) Only in system cards People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. GLM-5 67.5%
- DeepSeek Chinese creative writing evaluation Only in system cards Models write Chinese fiction, prose, poetry and lyrics for 2,837 prompts. Pairwise judging gives separate win rates for instruction following and writing quality.