IHEval
Nemotron 3.5 Bolt E4B 82.4% leads since October 2026 Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction.
Instruction hierarchy benchmark. Meta reports a composite across conflict types.
- Released
- February 2025
- Kind
- Jailbreaks and prompt injection
- Level
- Everyday
- Scored
- Checked automatically
- Answers
- Open-ended
- Tasks
- All public
- Strongest model tested
-
Muse Spark #53 on the capability index - Last in a launch post
- October 2026
- Last model added
- October 2026
- Scores refreshed
- November 12th, 2026
In the authors’ words
The instruction hierarchy, which establishes a priority order from system messages to user messages, conversation history, and tool outputs, is essential for ensuring consistent and safe behavior in language models (LMs). Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy. We bridge this gap by introducing IHEval, a novel benchmark comprising 3,538 examples across nine tasks, covering cases where instructions in different priorities either align or conflict. Our evaluation of popular LMs highlights their struggle to recognize instruction priorities. All evaluated models experience a sharp performance decline when facing conflicting instructions, compared to their original instruction-following performance. Moreover, the most competitive open-source model only achieves 48% accuracy in resolving such conflicts. Our results underscore the need for targeted optimization in the future development of LMs.
GPT-4o leads with 70% among models released by August 2024
Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
Nemotron 3.5 Bolt E4B leads with 82.4%, claims included
Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.
The newest model with a score here came out in August 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Fable 5.2
-
Gemini 4 Carbon - GPT-6 Astra
- Claude Opus 5.5
- and 16 more
- 1 GPT-4o Aug 2024 70%
- 2
Qwen2-72B 47.8% - 3 GPT-4o mini 44.3%
- 4 GPT-3.5 Turbo Jan 2024 33.9%
- 5 Claude 3 Sonnet 30.7%
- 6
Mistral Large 2 Jul 2024 29.4% - 7 Claude 3 Haiku 27.9%
- 8 Llama 3.1-70B instruction priority prompt 17.1%
- 9 Qwen2-7B 16.4%
- 10 Mistral-7B Instruct-v0.3 15%
- 11 Llama 3.1-8B 11.3%
Source: Official IHEval leaderboard
- 1 Nemotron 3.5 Bolt E4B 82.4%
- 2 Muse Spark 80.3%
- 3 GPT-4o Aug 2024 70%
- 4 Qwen2-72B 47.8%
- 5 GPT-4o mini 44.3%
- 6 GPT-3.5 Turbo Jan 2024 33.9%
- 7 Claude 3 Sonnet 30.7%
- 8 Mistral Large 2 Jul 2024 29.4%
- 9 Claude 3 Haiku 27.9%
- 10 Llama 3.1-70B instruction priority prompt 17.1%
- 11 Qwen2-7B 16.4%
- 12 Llama 3.1-8B 11.3%
Sources: Labs’ model cards and official IHEval leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-4o Aug 2024 | 70% | Aug 2024 | Official leaderboard | |
| 2 | Qwen2-72B | 47.8% | Jun 2024 | Official leaderboard | |
| 3 | GPT-4o mini | 44.3% | Jul 2024 | Official leaderboard | |
| 4 | GPT-3.5 Turbo Jan 2024 | 33.9% | Jan 2024 | Official leaderboard | |
| 5 | Claude 3 Sonnet | 30.7% | Feb 2024 | Official leaderboard | |
| 6 | Mistral Large 2 Jul 2024 | 29.4% | Jul 2024 | Official leaderboard | |
| 7 | Claude 3 Haiku | 27.9% | Mar 2024 | Official leaderboard | |
| 8 | Llama 3.1-70B | 17.1% | instruction priority prompt | Jul 2024 | Official leaderboard |
| 9 | Qwen2-7B | 16.4% | Jun 2024 | Official leaderboard | |
| 10 | Mistral-7B Instruct-v0.3 | 15% | Official leaderboard | ||
| 11 | Llama 3.1-8B | 11.3% | Jul 2024 | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Nemotron 3.5 Bolt E4B | 82.4% | Oct 2026 | Lab model card | |
| 2 | Muse Spark | 80.3% | Jun 2026 | Lab model card | |
| 3 | GPT-4o Aug 2024 | 70% | Aug 2024 | Official leaderboard | |
| 4 | Qwen2-72B | 47.8% | Jun 2024 | Official leaderboard | |
| 5 | GPT-4o mini | 44.3% | Jul 2024 | Official leaderboard | |
| 6 | GPT-3.5 Turbo Jan 2024 | 33.9% | Jan 2024 | Official leaderboard | |
| 7 | Claude 3 Sonnet | 30.7% | Feb 2024 | Official leaderboard | |
| 8 | Mistral Large 2 Jul 2024 | 29.4% | Jul 2024 | Official leaderboard | |
| 9 | Claude 3 Haiku | 27.9% | Mar 2024 | Official leaderboard | |
| 10 | Llama 3.1-70B | 17.1% | instruction priority prompt | Jul 2024 | Official leaderboard |
| 11 | Qwen2-7B | 16.4% | Jun 2024 | Official leaderboard | |
| 12 | Llama 3.1-8B | 11.3% | Jul 2024 | Official leaderboard |
Labs report scores for 2 models in their cards. Switch the scores above to rank them together.
Scores at each effort level
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Mistral Large 2 Llama 3.1-70B Source: Official IHEval leaderboard
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| GPT-4o Aug 2024 | 70% | ||||
| GPT-4o Aug 2024 | instruction priority prompt | 67.2% | |||
| Qwen2-72B | 47.8% | ||||
| GPT-4o mini | 44.3% | ||||
| GPT-3.5 Turbo Jan 2024 | 33.9% | ||||
| Claude 3 Sonnet | 30.7% | ||||
| Mistral Large 2 Jul 2024 | 29.4% | ||||
| Mistral Large 2 Jul 2024 | instruction priority prompt | 28.3% | |||
| Claude 3 Haiku | 27.9% | ||||
| Llama 3.1-70B | instruction priority prompt | 17.1% | |||
| Qwen2-7B | 16.4% | ||||
| Llama 3.1-70B | 14% | ||||
| Llama 3.1-8B | 11.3% |
Number one over time 3 models have held number one with claims included since its release in February 2025
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Sources: Official IHEval leaderboard and labs’ model cards
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Sources: Official IHEval leaderboard and labs’ model cards
One lab put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| Oct 23, 2026 | Nemotron 3.5 Bolt E4B | 82.4 | IHEval |
2 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| Oct 23, 2026 | Nemotron 3.5 Bolt E4B technical report | 82.4 | IHEval, released Q4_K_M imatrix checkpoint (Table 5). BF16 84.79 (Table 3) | Capabilities |
| Jun 12, 2026 | Muse Spark system card | 80.3 | IHEval composite score (Table 2, updated after the evaluation bug fix noted in the change log) | Alignment |
More in Safety and alignment
- FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
- StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is.
Inkling 98.6% - Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Gemini 4 Carbon 0%
- DistressBench In use Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Muse Spark 1.2 88.4%
- Lakera B3 (Backbone Breaker Benchmark) In use Crowdsourced prompt-injection attacks hit backbone LLMs inside agent threat snapshots at three defense levels. The score is attack resistance, 100 minus the board's vulnerability percentage. Claude Opus 4.5 89%
- Broken Search Tool In use Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. GPT-6 Cyber 0.33%