Benchmark Almanac

AlpacaEval

GPT-4 Turbo Nov 2023 97.7% leads since November 2023
Not in model cards

Models answer 805 open instructions, and a GPT-4 judge compares each reply with text-davinci-003's reply. The score is the win rate.

AlpacaEval 2.0 replaced the baseline and the judge on January 3rd, 2024, so 1.0 and 2.0 scores are not comparable.

Released
May 2023
Kind
Overall chat quality
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
GPT-4 Turbo Nov 2023 #175 on the capability index
Last model added
December 2023
Scores refreshed
November 12th, 2026
102
Entries on its own leaderboard, 19 models we track

GPT-4 Turbo leads with 97.7% among models released by December 2023

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in December 2023, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

102 entries. Scroll sideways for the rest.

  1. 97.7 GPT-4 Turbo
  2. 97.6 XwinLM 70b V0.3
  3. 96.8 Mistral Medium
  4. 95.6 XwinLM 70b V0.1
  5. 95.4 PairRM 0.4B+Tulu 2+DPO 70B (best-of-16)
  6. 95.3 GPT-4 (snapshot unspecified)
  7. 95.0 Tulu 2+DPO 70B
  8. 94.9 Mistral-7B+RAHF-DUAL+LoRA
  9. 94.8 GPT-4
  10. 94.8 Mixtral 8x7B
  11. 94.4 Mistral-7B-ReMax-v0.1
  12. 94.1 Yi-34B
  13. 93.8 GPT-4
  14. 93.4 GPT-3.5 Turbo
  15. 93.4 PairRM 0.4B+Zephyr 7B Beta (best-of-16)
  16. 92.8 UltraLM 13B V2.0 (best-of-16)
  17. 92.8 Mistral 7B v0.2
  18. 92.7 LLaMA2 Chat 70B
  19. 92.2 LMCocktail-10.7B-v1
  20. 91.8 XwinLM 13b V0.1
  21. 91.5 Claude (snapshot unspecified)
  22. 91.5 UltraLM 13B (best-of-16)
  23. 91.4 CUT 13B
  24. 91.4 Claude 2
  25. 91.1 PairRM 0.4B+Tulu 2+DPO 13B (best-of-16)
  26. 90.6 Cohere Command
  27. 90.6 Zephyr 7B Beta
  28. 90.1 DEITA 7B v1.0
  29. 89.5 OpenChat V3.1 13B
  30. 89.4 GPT-3.5 Turbo (Mar 2023)
  31. 89.3 Evo v2 7B
  32. 89.2 WizardLM 13B V1.2
  33. 89.0 Vicuna 33B v1.3
  34. 88.3 CausalLM-14B
  35. 88.1 Tulu 2+DPO 13B
  36. 87.9 Humpback LLaMa2 70B
  37. 87.8 XwinLM 7b V0.1
  38. 87.7 OpenBudddy-LLaMA2-70B-v10.1
  39. 87.1 OpenChat V2-W 13B
  40. 87.1 Claude 2.1
  41. 86.5 OpenBuddy-LLaMA-65B-v8
  42. 86.3 WizardLM 13B V1.1
  43. 86.3 UltraLM 13B V2.0
  44. 86.3 GPT-3.5 Turbo
  45. 85.8 Zephyr 7B Alpha
  46. 85.0 OpenChat V2 13B
  47. 84.2 Tulu 2+DPO 7B
  48. 83.7 Humpback LLaMa 65B
  49. 83.5 Recycled WizardLM 7B V2.0
  50. 82.3 Phi-2 DPO
  51. 82.1 Vicuna-13B-v1.3
  52. 81.9 PlatoLM 7B
  53. 81.9 LLaMA2 Chat 7B Evol70k-NEFT
  54. 81.7 GPT-3.5 (snapshot unspecified)
  55. 81.5 OpenBuddy-LLaMA-30B-v7.1
  56. 81.1 LLaMA2 Chat 13B
  57. 80.9 OpenChat-13B
  58. 80.7 OpenBuddy-Falcon-40B-v9
  59. 80.6 UltraLM 13B
  60. 79.7 Gemini 1.0 Pro
  61. 79.5 OpenChat8192-13B
  62. 79.2 Evo 7B
  63. 78.9 Claude2 Alpaca 13B
  64. 78.9 Recycled WizardLM 7B V1.0
  65. 78.7 OpenCoderPlus-15B
  66. 78.5 MiniChat 1.5 3B
  67. 77.5 OpenBudddy-LLaMA2-13B-v11.1
  68. 76.8 Vicuna 7B v1.3
  69. 75.3 WizardLM 13B
  70. 74.1 JinaChat
  71. 73.9 airoboros 65B
  72. 73.3 airoboros 33B
  73. 71.8 Guanaco 65B
  74. 71.4 LLaMA2 Chat 7B
  75. 70.4 Ghost 7B Alpha
  76. 70.4 Vicuna 13B
  77. 70.4 OpenBuddy-Falcon-7b-v6
  78. 68.5 Phi-2 SFT
  79. 67.0 Baize-v2 13B
  80. 66.5 LLaMA 33B OASST RLHF
  81. 66.0 Minotaur 13B
  82. 66.0 Guanaco 33B
  83. 65.5 Nous Hermes 13B
  84. 64.4 Vicuna 7B
  85. 63.9 Baize-v2 7B
  86. 61.6 Alpaca-7B-NEFT
  87. 55.0 LLaMA 33B OASST SFT
  88. 52.6 Guanaco 13B
  89. 50.0 davinci-003
  90. 48.8 MiniChat 3B
  91. 47.1 chatglm2-6b
  92. 46.6 Guanaco 7B
  93. 45.7 Falcon 40B Instruct
  94. 45.7 Falcon 7B Instruct
  95. 44.1 Alpaca Farm PPO Sim (GPT-4) 7B
  96. 41.9 Pythia 12B SFT
  97. 41.2 Alpaca Farm PPO Human 7B
  98. 30.7 Phi-2
  99. 26.5 Alpaca 7B
  100. 26.0 Pythia 12B OASST SFT
  101. 21.8 Baichuan-13B-Chat
  102. 15.2 Davinci001
  1. 1 GPT-4 Turbo Nov 2023 97.7%
  2. 2 XwinLM 70b V0.3 97.6%
  3. 3 Mistral Medium 96.8%
  4. 4 XwinLM 70b V0.1 95.6%
  5. 5 PairRM 0.4B+Tulu 2+DPO 70B (best-of-16) 95.4%
  6. 6 GPT-4 (snapshot unspecified) 95.3%
  7. 7 Tulu 2+DPO 70B 95%
  8. 8 Mistral-7B+RAHF-DUAL+LoRA 94.9%
  9. 9 GPT-4 Mar 2023 94.8%
  10. 10 Mixtral 8x7B 94.8%
  11. 11 Mistral-7B-ReMax-v0.1 94.4%
  12. 12 Yi-34B 94.1%
  13. 13 GPT-4 Jun 2023 93.8%
  14. 14 GPT-3.5 Turbo Jun 2023 93.4%
  15. 15 PairRM 0.4B+Zephyr 7B Beta (best-of-16) 93.4%
  16. 16 UltraLM 13B V2.0 (best-of-16) 92.8%
  17. 17 Mistral 7B v0.2 92.8%
  18. 18 LLaMA2 Chat 70B 92.7%
  19. 19 LMCocktail-10.7B-v1 92.2%
  20. 20 XwinLM 13b V0.1 91.8%
  21. 21 Claude (snapshot unspecified) 91.5%
  22. 22 UltraLM 13B (best-of-16) 91.5%
  23. 23 CUT 13B 91.4%
  24. 24 Claude 2 91.4%
  25. 25 PairRM 0.4B+Tulu 2+DPO 13B (best-of-16) 91.1%
  26. 26 Cohere Command 90.6%
  27. 27 Zephyr 7B Beta 90.6%
  28. 28 DEITA 7B v1.0 90.1%
  29. 29 OpenChat V3.1 13B 89.5%
  30. 30 GPT-3.5 Turbo (Mar 2023) 89.4%
  31. 31 Evo v2 7B 89.3%
  32. 32 WizardLM 13B V1.2 89.2%
  33. 33 Vicuna 33B v1.3 89%
  34. 34 CausalLM-14B 88.3%
  35. 35 Tulu 2+DPO 13B 88.1%
  36. 36 Humpback LLaMa2 70B 87.9%
  37. 37 XwinLM 7b V0.1 87.8%
  38. 38 OpenBudddy-LLaMA2-70B-v10.1 87.7%
  39. 39 OpenChat V2-W 13B 87.1%
  40. 40 Claude 2.1 87.1%
  41. 41 OpenBuddy-LLaMA-65B-v8 86.5%
  42. 42 WizardLM 13B V1.1 86.3%
  43. 43 UltraLM 13B V2.0 86.3%
  44. 44 GPT-3.5 Turbo Nov 2023 86.3%
  45. 45 Zephyr 7B Alpha 85.8%
  46. 46 OpenChat V2 13B 85%
  47. 47 Tulu 2+DPO 7B 84.2%
  48. 48 Humpback LLaMa 65B 83.7%
  49. 49 Recycled WizardLM 7B V2.0 83.5%
  50. 50 Phi-2 DPO 82.3%
  51. 51 Vicuna-13B-v1.3 82.1%
  52. 52 PlatoLM 7B 81.9%
  53. 53 LLaMA2 Chat 7B Evol70k-NEFT 81.9%
  54. 54 GPT-3.5 (snapshot unspecified) 81.7%
  55. 55 OpenBuddy-LLaMA-30B-v7.1 81.5%
  56. 56 LLaMA2 Chat 13B 81.1%
  57. 57 OpenChat-13B 80.9%
  58. 58 OpenBuddy-Falcon-40B-v9 80.7%
  59. 59 UltraLM 13B 80.6%
  60. 60 Gemini 1.0 Pro 79.7%
  61. 61 OpenChat8192-13B 79.5%
  62. 62 Evo 7B 79.2%
  63. 63 Claude2 Alpaca 13B 78.9%
  64. 64 Recycled WizardLM 7B V1.0 78.9%
  65. 65 OpenCoderPlus-15B 78.7%
  66. 66 MiniChat 1.5 3B 78.5%
  67. 67 OpenBudddy-LLaMA2-13B-v11.1 77.5%
  68. 68 Vicuna 7B v1.3 76.8%
  69. 69 WizardLM 13B 75.3%
  70. 70 JinaChat 74.1%
  71. 71 airoboros 65B 73.9%
  72. 72 airoboros 33B 73.3%
  73. 73 Guanaco 65B 71.8%
  74. 74 LLaMA2 Chat 7B 71.4%
  75. 75 Ghost 7B Alpha 70.4%
  76. 76 Vicuna 13B 70.4%
  77. 77 OpenBuddy-Falcon-7b-v6 70.4%
  78. 78 Phi-2 SFT 68.5%
  79. 79 Baize-v2 13B 67%
  80. 80 LLaMA 33B OASST RLHF 66.5%
  81. 81 Minotaur 13B 66%
  82. 82 Guanaco 33B 66%
  83. 83 Nous Hermes 13B 65.5%
  84. 84 Vicuna 7B 64.4%
  85. 85 Baize-v2 7B 63.9%
  86. 86 Alpaca-7B-NEFT 61.6%
  87. 87 LLaMA 33B OASST SFT 55%
  88. 88 Guanaco 13B 52.6%
  89. 89 davinci-003 50%
  90. 90 MiniChat 3B 48.8%
  91. 91 chatglm2-6b 47.1%
  92. 92 Guanaco 7B 46.6%
  93. 93 Falcon 40B Instruct 45.7%
  94. 94 Falcon 7B Instruct 45.7%
  95. 95 Alpaca Farm PPO Sim (GPT-4) 7B 44.1%
  96. 96 Pythia 12B SFT 41.9%
  97. 97 Alpaca Farm PPO Human 7B 41.2%
  98. 98 Phi-2 30.7%
  99. 99 Alpaca 7B 26.5%
  100. 100 Pythia 12B OASST SFT 26%
  101. 101 Baichuan-13B-Chat 21.8%
  102. 102 Davinci001 15.2%

Source: Official AlpacaEval leaderboard

#EntryScoreSettingDateSource
1 GPT-4 Turbo Nov 2023 97.7% Nov 2023 Official leaderboard
2 XwinLM 70b V0.3 97.6% Official leaderboard
3 Mistral Medium 96.8% Dec 2023 Official leaderboard
4 XwinLM 70b V0.1 95.6% Official leaderboard
5 PairRM 0.4B+Tulu 2+DPO 70B (best-of-16) 95.4% Official leaderboard
6 GPT-4 (snapshot unspecified) 95.3% Official leaderboard
7 Tulu 2+DPO 70B 95% Official leaderboard
8 Mistral-7B+RAHF-DUAL+LoRA 94.9% Official leaderboard
9 GPT-4 Mar 2023 94.8% Mar 2023 Official leaderboard
10 Mixtral 8x7B 94.8% Dec 2023 Official leaderboard
11 Mistral-7B-ReMax-v0.1 94.4% Official leaderboard
12 Yi-34B 94.1% Nov 2023 Official leaderboard
13 GPT-4 Jun 2023 93.8% Jun 2023 Official leaderboard
14 GPT-3.5 Turbo Jun 2023 93.4% Jun 2023 Official leaderboard
15 PairRM 0.4B+Zephyr 7B Beta (best-of-16) 93.4% Official leaderboard
16 UltraLM 13B V2.0 (best-of-16) 92.8% Official leaderboard
17 Mistral 7B v0.2 92.8% Dec 2023 Official leaderboard
18 LLaMA2 Chat 70B 92.7% Official leaderboard
19 LMCocktail-10.7B-v1 92.2% Official leaderboard
20 XwinLM 13b V0.1 91.8% Official leaderboard
21 Claude (snapshot unspecified) 91.5% Official leaderboard
22 UltraLM 13B (best-of-16) 91.5% Official leaderboard
23 CUT 13B 91.4% Official leaderboard
24 Claude 2 91.4% Jul 2023 Official leaderboard
25 PairRM 0.4B+Tulu 2+DPO 13B (best-of-16) 91.1% Official leaderboard
26 Cohere Command 90.6% Official leaderboard
27 Zephyr 7B Beta 90.6% Oct 2023 Official leaderboard
28 DEITA 7B v1.0 90.1% Official leaderboard
29 OpenChat V3.1 13B 89.5% Official leaderboard
30 GPT-3.5 Turbo (Mar 2023) 89.4% Mar 2023 Official leaderboard
31 Evo v2 7B 89.3% Official leaderboard
32 WizardLM 13B V1.2 89.2% Official leaderboard
33 Vicuna 33B v1.3 89% Official leaderboard
34 CausalLM-14B 88.3% Official leaderboard
35 Tulu 2+DPO 13B 88.1% Official leaderboard
36 Humpback LLaMa2 70B 87.9% Official leaderboard
37 XwinLM 7b V0.1 87.8% Official leaderboard
38 OpenBudddy-LLaMA2-70B-v10.1 87.7% Official leaderboard
39 OpenChat V2-W 13B 87.1% Official leaderboard
40 Claude 2.1 87.1% Nov 2023 Official leaderboard
41 OpenBuddy-LLaMA-65B-v8 86.5% Official leaderboard
42 WizardLM 13B V1.1 86.3% Official leaderboard
43 UltraLM 13B V2.0 86.3% Official leaderboard
44 GPT-3.5 Turbo Nov 2023 86.3% Nov 2023 Official leaderboard
45 Zephyr 7B Alpha 85.8% Oct 2023 Official leaderboard
46 OpenChat V2 13B 85% Official leaderboard
47 Tulu 2+DPO 7B 84.2% Official leaderboard
48 Humpback LLaMa 65B 83.7% Official leaderboard
49 Recycled WizardLM 7B V2.0 83.5% Official leaderboard
50 Phi-2 DPO 82.3% Official leaderboard
51 Vicuna-13B-v1.3 82.1% Jun 2023 Official leaderboard
52 PlatoLM 7B 81.9% Official leaderboard
53 LLaMA2 Chat 7B Evol70k-NEFT 81.9% Official leaderboard
54 GPT-3.5 (snapshot unspecified) 81.7% Official leaderboard
55 OpenBuddy-LLaMA-30B-v7.1 81.5% Official leaderboard
56 LLaMA2 Chat 13B 81.1% Official leaderboard
57 OpenChat-13B 80.9% Official leaderboard
58 OpenBuddy-Falcon-40B-v9 80.7% Official leaderboard
59 UltraLM 13B 80.6% Official leaderboard
60 Gemini 1.0 Pro 79.7% Dec 2023 Official leaderboard
61 OpenChat8192-13B 79.5% Official leaderboard
62 Evo 7B 79.2% Official leaderboard
63 Claude2 Alpaca 13B 78.9% Official leaderboard
64 Recycled WizardLM 7B V1.0 78.9% Official leaderboard
65 OpenCoderPlus-15B 78.7% Official leaderboard
66 MiniChat 1.5 3B 78.5% Official leaderboard
67 OpenBudddy-LLaMA2-13B-v11.1 77.5% Official leaderboard
68 Vicuna 7B v1.3 76.8% Official leaderboard
69 WizardLM 13B 75.3% Official leaderboard
70 JinaChat 74.1% Official leaderboard
71 airoboros 65B 73.9% Official leaderboard
72 airoboros 33B 73.3% Official leaderboard
73 Guanaco 65B 71.8% Official leaderboard
74 LLaMA2 Chat 7B 71.4% Official leaderboard
75 Ghost 7B Alpha 70.4% Official leaderboard
76 Vicuna 13B 70.4% Official leaderboard
77 OpenBuddy-Falcon-7b-v6 70.4% Official leaderboard
78 Phi-2 SFT 68.5% Official leaderboard
79 Baize-v2 13B 67% Official leaderboard
80 LLaMA 33B OASST RLHF 66.5% Official leaderboard
81 Minotaur 13B 66% Official leaderboard
82 Guanaco 33B 66% Official leaderboard
83 Nous Hermes 13B 65.5% Official leaderboard
84 Vicuna 7B 64.4% Official leaderboard
85 Baize-v2 7B 63.9% Official leaderboard
86 Alpaca-7B-NEFT 61.6% Official leaderboard
87 LLaMA 33B OASST SFT 55% Official leaderboard
88 Guanaco 13B 52.6% Official leaderboard
89 davinci-003 50% Nov 2022 Official leaderboard
90 MiniChat 3B 48.8% Official leaderboard
91 chatglm2-6b 47.1% Jun 2023 Official leaderboard
92 Guanaco 7B 46.6% Official leaderboard
93 Falcon 40B Instruct 45.7% Official leaderboard
94 Falcon 7B Instruct 45.7% Official leaderboard
95 Alpaca Farm PPO Sim (GPT-4) 7B 44.1% Official leaderboard
96 Pythia 12B SFT 41.9% Official leaderboard
97 Alpaca Farm PPO Human 7B 41.2% Official leaderboard
98 Phi-2 30.7% Dec 2023 Official leaderboard
99 Alpaca 7B 26.5% Official leaderboard
100 Pythia 12B OASST SFT 26% Official leaderboard
101 Baichuan-13B-Chat 21.8% Official leaderboard
102 Davinci001 15.2% Official leaderboard

2 models have held number one on the official leaderboard since its release in May 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Passed 85% in May 2023, by GPT-4
GPT-4 Turbo 97.7%

Source: Official AlpacaEval leaderboard

GPT-3.5 Turbo (Mar 2023) on the official leaderboard passed 85% in March 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2023 2024 Near the ceiling (85%) GPT-4 Turbo GPT-4 GPT-3.5 Turbo (Mar 2023) davinci-003 0% 20% 40% 60% 80% 100% 2023 2024 Near the ceiling (85%) GPT-4 Turbo GPT-3.5 Turbo (Mar 2023)

Source: Official AlpacaEval leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI Qwen DeepSeek Ai2
AlpacaEval 2.0 Passed 85% in Jan 2025, by DeepSeek-R1
DeepSeek-R1 87.6%
AlpacaEval Passed 85% in May 2023, by GPT-4
GPT-4 Turbo 97.7%
AlpacaEval 2.0 Passed 85% in Jan 2025, by DeepSeek-R1
DeepSeek-R1 87.6%

Sources: Official leaderboards and labs’ model cards

Replaced by AlpacaEval 2.0

More in Instructions and chat

All 81 in Instructions and chat
  • IFEval In use Models answer about 500 prompts containing verifiable instructions such as word limits or required keywords. Code checks each instruction for strict and loose accuracy. Gemma 4 31B IT 98.9%
  • IFBench In use 58 new out-of-domain verifiable output constraints. Scored by programmatic constraint checks (strict or loose). Nemotron-Cascade 2 30B-A3B 82.9%
  • MultiChallenge In use Multi-turn conversations testing instruction retention, inference memory, versioned editing and self-coherence. Graded by LLM judge with rubrics. GPT-6.1 Sol 82%
  • Multi-IF In use Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. LFM3-1.7B 89.1%
  • AdvancedIF In use Models respond to over 1,600 prompts with complex, multi-turn, and system-level instructions. A judge model checks each response against expert-written rubrics. MAI-Thinking-1 85%
  • IFStruct In use Models generate JSON or YAML that must follow a requested schema and phrasing constraints, without constrained decoding. A response passes only if every structural constraint holds. Gemma 4 31B IT 95.9%