Benchmark Almanac

MMDM

Perplexity Decider-V1.3 76.2% tops its leaderboard
Not in model cards

Decision models assign probabilities to options in 5,448 questions across 16 categories. Hard choices use accuracy or exact set match, while soft targets use Brier error.

Inputs contain up to 1,024 options and about 64,000 tokens. Hard accuracy is higher-is-better, but Brier error is lower-is-better. Keep the tracks separate.

Released
November 2026
Kind
Language reasoning
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone

In the authors’ words

System-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization. Moreover, real-world decision making often involves long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together. Thus, we introduce MMDM (Massive Multitask Decision Making), a benchmark of 5,448 high-quality questions across 16 task categories, combining newly constructed tasks with source-grounded recompositions and rewrites for single-select, multi-select, and soft-label decision making in fresh contexts. It covers realistic scenarios such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs extending to 64,817 tokens. With MMDM, we conduct a comprehensive quantitative evaluation of 28 decision models, showing that many perform well on common knowledge yet struggle with interdependent options, constrained decisions, and multi-select tasks. For example, Perplexity Decider-V1.2 reaches 78.75% Hard Single-select Accuracy on world knowledge, but only 53.95% on option logic and 47.95% on constraint planning, while the best Hard Multi-select Accuracy across all models is 36.81%. These results highlight joint reasoning, constraint-aware selection, and complete multi-select recovery as key priorities for decision model development, with MMDM providing a testbed for measuring this progress.

Abstract of MMDM: Measuring Massive Multitask Decision Making (arXiv:2611.14309, CC0)
36
Entries on its own leaderboard, 0 models we track

Perplexity Decider-V1.3 tops the leaderboard with 76.2%

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

  1. 1 Perplexity Decider-V1.3 76.2%
  2. 2 WeMM-JEV 27B 76%
  3. 3 KnowLine 31B Gen1 75.1%
  4. 4 Liquid-d1.2-Ultra 75%
  5. 5 Perplexity Decider-V1.2 74.4%
  6. 6 Liquid-d1-Ultra 73.3%
  7. 7 HeyGen Decision-1 69.9%
  8. 8 HeyGen Decision-1 Turbo 69.7%
  9. 9 Liquid-d1.2 69.1%
  10. 10 Microsoft Decision-1.1 67.3%
  11. 11 Jev 1.14.0 67.2%
  12. 12 Autotrust JEV 27B-VL 67.1%
  13. 13 Gemini Decision 1 Partial coverage (4094 of 4112 single-select items) 65.3%
  14. 14 Perplexity Decider-Flash-V1.3 65%
  15. 15 Decisio-Gemma-4-31B 64.6%
  16. 16 Clef 64.2%
  17. 17 Perplexity Decider-V1 62.4%
  18. 18 Liquid-d1 62.3%
  19. 19 Perplexity Decider-Flash-V1.2 62.1%
  20. 20 Microsoft Decision-1 61.4%
  21. 21 H2O-Lightning 4B 58.3%
  22. 22 Decisio-Gemma-4-12B 57.7%
  23. 23 Quyet-1.0-Large 57.4%
  24. 24 OpenJev 4B 57.2%
  25. 25 Clef-Omni 56.9%
  26. 26 GPT-6 Luna Decisions 56.6%
  27. 27 Mapika Decider 4B 56.5%
  28. 28 Clef-Flash 55.7%
  29. 29 Decisio-Qwen3.6-35B-A3B 53.3%
  30. 30 Intern-Decision 4B 49%
  31. 31 JevK5 4B 48.7%
  32. 32 Strands-Decider 2B 44.6%
  33. 33 SemIf 4B 42.4%
  34. 34 Intern-Decision 2B 38.8%
  35. 35 Intern-Decision 0.8B 32.2%
  36. 36 NanoJev 0.6B 26.7%

Source: Official MMDM leaderboard

#EntryScoreSettingDateSource
1 Perplexity Decider-V1.3 27B 76.2% Official leaderboard
2 WeMM-JEV 27B 27B 76% Official leaderboard
3 KnowLine 31B Gen1 31B 75.1% Official leaderboard
4 Liquid-d1.2-Ultra 75% Official leaderboard
5 Perplexity Decider-V1.2 27B 74.4% Official leaderboard
6 Liquid-d1-Ultra 73.3% Official leaderboard
7 HeyGen Decision-1 69.9% Official leaderboard
8 HeyGen Decision-1 Turbo 69.7% Official leaderboard
9 Liquid-d1.2 69.1% Official leaderboard
10 Microsoft Decision-1.1 67.3% Official leaderboard
11 Jev 1.14.0 67.2% Official leaderboard
12 Autotrust JEV 27B-VL 27B 67.1% Official leaderboard
13 Gemini Decision 1 65.3% Partial coverage (4094 of 4112 single-select items) Official leaderboard
14 Perplexity Decider-Flash-V1.3 4B 65% Official leaderboard
15 Decisio-Gemma-4-31B 31B 64.6% Official leaderboard
16 Clef 27B 64.2% Official leaderboard
17 Perplexity Decider-V1 27B 62.4% Official leaderboard
18 Liquid-d1 62.3% Official leaderboard
19 Perplexity Decider-Flash-V1.2 4B 62.1% Official leaderboard
20 Microsoft Decision-1 9B 61.4% Official leaderboard
21 H2O-Lightning 4B 4B 58.3% Official leaderboard
22 Decisio-Gemma-4-12B 12B 57.7% Official leaderboard
23 Quyet-1.0-Large 31B 57.4% Official leaderboard
24 OpenJev 4B 4B 57.2% Official leaderboard
25 Clef-Omni 30B 56.9% Official leaderboard
26 GPT-6 Luna Decisions 56.6% Official leaderboard
27 Mapika Decider 4B 4B 56.5% Official leaderboard
28 Clef-Flash 9B 55.7% Official leaderboard
29 Decisio-Qwen3.6-35B-A3B 35B 53.3% Official leaderboard
30 Intern-Decision 4B 4B 49% Official leaderboard
31 JevK5 4B 4B 48.7% Official leaderboard
32 Strands-Decider 2B 2B 44.6% Official leaderboard
33 SemIf 4B 4B 42.4% Official leaderboard
34 Intern-Decision 2B 2B 38.8% Official leaderboard
35 Intern-Decision 0.8B 0.8B 32.2% Official leaderboard
36 NanoJev 0.6B 0.6B 26.7% Official leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Knowledge

All 332 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • MMLU In use 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. Seed1.8 92.3%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • BBEH micro average In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is the share of all examples answered correctly. Gemma 4 31B IT 74.4%
  • HLE-Diamond In use Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. Claude Fable 5.2 77.9%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (36 entries).
  • Pass Models scored 36 entries on its leaderboard, none a model we track.
  • Note Newest score No scores yet.
  • Note Frontier models No models scored yet.
  • Note Headroom No scores yet.
Leaderboard import 3 pass
  • Pass Importer We read the board on mmdm-benchmark.github.io: 36 entries at the last read, none a model we track.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check