Benchmark Almanac

MedQA

GPT-4o Aug 2024 86.3% leads since August 2024
GPT-4o May 2024 89.4% leads since August 2024
Dropped

Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy.

The full dataset also has simplified and traditional Chinese exam sets.

Made by
MIT
Released
September 2020
Kind
Medicine
Level
Expert
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
DeepSeek-V3 Mar 2025 #141 on the capability index
Last in a launch post
April 2024, dropped before it saturated
Last model added
March 2025
Scores refreshed
November 12th, 2026

In the authors’ words

Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7%, 42.0%, and 70.1% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.

Abstract of What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams (arXiv:2009.13081, CC0)
91
Entries on Stanford HELM’s leaderboard, 71 models we track
1
Launch post lists it, from 1 lab
4
System cards report it, from 2 labs

GPT-4o leads with 86.3% among models released by March 2025

Best score per entry from Stanford HELM’s leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

GPT-4o leads with 89.4%, claims included

Best score per model from every source, including labs’ own numbers and claims Epoch AI collected from reports and papers. Labs pick their own settings, so a claim can beat an independent run of the same model.

The newest model with a score here came out in March 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

91 entries. Scroll sideways for the rest.

  1. 86.3 GPT-4o
  2. 85.9 Claude 3.5 Sonnet (October 2024)
  3. 85.7 GPT-4o
  4. 82.5 Claude 3.5 Sonnet
  5. 81.7 gpt-4-1106-preview
  6. 81.5 GPT-4
  7. 81.1 Amazon Nova Pro
  8. 80.9 DeepSeek-V3
  9. 80.5 Llama 3.1-405B
  10. 78.3 GPT-4 Turbo
  11. 77.7 Llama 3-70B
  12. 77.5 Claude 3 Opus
  13. 77.5 Mistral Large 2
  14. 77.5 Palmyra X4
  15. 77.1 Gemini 1.5 Pro
  16. 76.9 Llama 3.1-70B
  17. 76.9 Llama 3.2 90B
  18. 76.1 Llama 3.3 70B Instruct
  19. 75.3 Qwen2.5-72B
  20. 74.8 GPT-4o mini
  21. 74.5 Qwen2-72B
  22. 73.0 Gemini 2.0 Flash
  23. 72.2 Claude 3.5 Haiku
  24. 70.4 Mixtral 8x22B
  25. 70.2 Claude 3 Haiku
  26. 69.8 solar-pro-241126
  27. 69.8 Jamba 1.5 Large
  28. 69.6 phi-3-medium 14B
  29. 69.6 Amazon Nova Lite
  30. 69.4 DBRX
  31. 69.2 Gemini 1.5 Pro
  32. 68.4 palmyra-x-v3
  33. 68.4 text-unicorn@001
  34. 68.4 Claude 3 Sonnet
  35. 68.4 Gemma 2 27B
  36. 68.0 Gemini 1.5 Flash
  37. 67.2 phi-3-small 7.4B
  38. 67.0 Qwen1.5-72B
  39. 66.0 Yi-Large
  40. 65.6 Yi-34B
  41. 65.6 Qwen1.5-32B
  42. 65.6 Gemini 1.5 Flash
  43. 65.2 Claude 2
  44. 65.2 Mixtral 8x7B
  45. 64.4 Claude 2.1
  46. 64.0 Qwen1.5-110B
  47. 63.2 Jamba 1.5 Mini
  48. 63.0 Gemma 2 9B
  49. 62.8 DeepSeek LLM 67B
  50. 62.2 GPT-3.5 Turbo
  51. 61.8 Claude 1.3
  52. 61.8 Llama 2-70B
  53. 61.6 mistral-small-2402
  54. 61.0 mistral-medium-2312
  55. 60.8 Amazon Nova Micro
  56. 60.0 Qwen2.5-7B
  57. 59.8 palmyra-x-v2
  58. 59.0 open-mistral-nemo-2407
  59. 58.0 Llama 3-8B
  60. 58.0 Snowflake Arctic Instruct
  61. 56.7 Command R+
  62. 55.9 Claude Instant
  63. 55.5 Command R (03-2024)
  64. 54.7 text-bison@001
  65. 53.1 text-davinci-003
  66. 52.5 text-davinci-002
  67. 52.5 Mistral 7B v0.1
  68. 51.9 jamba-instruct
  69. 51.7 Mistral 7B v0.3
  70. 51.5 Qwen1.5-14B
  71. 51.3 Gemma 7B
  72. 50.7 LLaMA-65B
  73. 49.9 mistral-large-2402
  74. 49.7 Yi 6B
  75. 48.3 Gemini 1.0 Pro
  76. 47.9 Qwen1.5-7B
  77. 44.5 command
  78. 43.1 j2-jumbo
  79. 42.0 Falcon-40B
  80. 41.0 Phi-2
  81. 39.2 Llama 2-13B
  82. 39.2 Llama 2-7B
  83. 39.0 j2-grande
  84. 31.2 command-light
  85. 27.6 luminous-extended
  86. 27.6 luminous-supreme
  87. 27.0 Llama 3.2 11B
  88. 26.0 luminous-base
  89. 25.4 Falcon-7B
  90. 24.4 Llama 3.1-8B
  91. 22.9 olmo-7b
  1. 1 GPT-4o Aug 2024 86.3%
  2. 2 Claude 3.5 Sonnet (October 2024) 85.9%
  3. 3 GPT-4o May 2024 85.7%
  4. 4 Claude 3.5 Sonnet 82.5%
  5. 5 gpt-4-1106-preview 81.7%
  6. 6 GPT-4 Jun 2023 81.5%
  7. 7 Amazon Nova Pro 81.1%
  8. 8 DeepSeek-V3 Mar 2025 80.9%
  9. 9 Llama 3.1-405B 80.5%
  10. 10 GPT-4 Turbo Apr 2024 78.3%
  11. 11 Llama 3-70B 77.7%
  12. 12 Claude 3 Opus 77.5%
  13. 13 Mistral Large 2 Jul 2024 77.5%
  14. 14 Palmyra X4 77.5%
  15. 15 Gemini 1.5 Pro Sep 2024 77.1%
  16. 16 Llama 3.1-70B 76.9%
  17. 17 Llama 3.2 90B 76.9%
  18. 18 Llama 3.3 70B Instruct 76.1%
  19. 19 Qwen2.5-72B 75.3%
  20. 20 GPT-4o mini 74.8%
  21. 21 Qwen2-72B 74.5%
  22. 22 Gemini 2.0 Flash Feb 2025 73%
  23. 23 Claude 3.5 Haiku 72.2%
  24. 24 Mixtral 8x22B 70.4%
  25. 25 Claude 3 Haiku 70.2%
  26. 26 solar-pro-241126 69.8%
  27. 27 Jamba 1.5 Large 69.8%
  28. 28 phi-3-medium 14B 69.6%
  29. 29 Amazon Nova Lite 69.6%
  30. 30 DBRX 69.4%
  31. 31 Gemini 1.5 Pro May 2024 69.2%
  32. 32 palmyra-x-v3 68.4%
  33. 33 text-unicorn@001 68.4%
  34. 34 Claude 3 Sonnet 68.4%
  35. 35 Gemma 2 27B 68.4%
  36. 36 Gemini 1.5 Flash May 2024 68%
  37. 37 phi-3-small 7.4B 67.2%
  38. 38 Qwen1.5-72B 67%
  39. 39 Yi-Large 66%
  40. 40 Yi-34B 65.6%
  41. 41 Qwen1.5-32B 65.6%
  42. 42 Gemini 1.5 Flash Sep 2024 65.6%
  43. 43 Claude 2 65.2%
  44. 44 Mixtral 8x7B 65.2%
  45. 45 Claude 2.1 64.4%
  46. 46 Qwen1.5-110B 64%
  47. 47 Jamba 1.5 Mini 63.2%
  48. 48 Gemma 2 9B 63%
  49. 49 DeepSeek LLM 67B 62.8%
  50. 50 GPT-3.5 Turbo Jun 2023 62.2%
  51. 51 Claude 1.3 61.8%
  52. 52 Llama 2-70B 61.8%
  53. 53 mistral-small-2402 61.6%
  54. 54 mistral-medium-2312 61%
  55. 55 Amazon Nova Micro 60.8%
  56. 56 Qwen2.5-7B 60%
  57. 57 palmyra-x-v2 59.8%
  58. 58 open-mistral-nemo-2407 59%
  59. 59 Llama 3-8B 58%
  60. 60 Snowflake Arctic Instruct 58%
  61. 61 Command R+ 56.7%
  62. 62 Claude Instant 55.9%
  63. 63 Command R (03-2024) 55.5%
  64. 64 text-bison@001 54.7%
  65. 65 text-davinci-003 53.1%
  66. 66 text-davinci-002 52.5%
  67. 67 Mistral 7B v0.1 52.5%
  68. 68 jamba-instruct 51.9%
  69. 69 Mistral 7B v0.3 51.7%
  70. 70 Qwen1.5-14B 51.5%
  71. 71 Gemma 7B 51.3%
  72. 72 LLaMA-65B 50.7%
  73. 73 mistral-large-2402 49.9%
  74. 74 Yi 6B 49.7%
  75. 75 Gemini 1.0 Pro 48.3%
  76. 76 Qwen1.5-7B 47.9%
  77. 77 command 44.5%
  78. 78 j2-jumbo 43.1%
  79. 79 Falcon-40B 42%
  80. 80 Phi-2 41%
  81. 81 Llama 2-13B 39.2%
  82. 82 Llama 2-7B 39.2%
  83. 83 j2-grande 39%
  84. 84 command-light 31.2%
  85. 85 luminous-extended 27.6%
  86. 86 luminous-supreme 27.6%
  87. 87 Llama 3.2 11B 27%
  88. 88 luminous-base 26%
  89. 89 Falcon-7B 25.4%
  90. 90 Llama 3.1-8B 24.4%
  91. 91 olmo-7b 22.9%

Source: Stanford HELM’s leaderboard

72 models. Scroll sideways for the rest.

  1. 89.4 GPT-4o
  2. 86.3 GPT-4o
  3. 85.9 Claude 3.5 Sonnet (October 2024)
  4. 82.5 Claude 3.5 Sonnet
  5. 81.5 GPT-4
  6. 81.1 Amazon Nova Pro
  7. 80.9 DeepSeek-V3
  8. 80.5 Llama 3.1-405B
  9. 78.3 GPT-4 Turbo
  10. 77.7 Llama 3-70B
  11. 77.5 Claude 3 Opus
  12. 77.5 Mistral Large 2
  13. 77.5 Palmyra X4
  14. 77.1 Gemini 1.5 Pro
  15. 76.9 Llama 3.1-70B
  16. 76.9 Llama 3.2 90B
  17. 76.1 Llama 3.3 70B Instruct
  18. 75.3 Qwen2.5-72B
  19. 74.8 GPT-4o mini
  20. 74.5 Qwen2-72B
  21. 73.0 Gemini 2.0 Flash
  22. 72.2 Claude 3.5 Haiku
  23. 70.4 Mixtral 8x22B
  24. 70.2 Claude 3 Haiku
  25. 69.9 phi-3-medium 14B
  26. 69.8 Jamba 1.5 Large
  27. 69.6 Amazon Nova Lite
  28. 69.4 DBRX
  29. 69.2 Gemini 1.5 Pro
  30. 68.4 Claude 3 Sonnet
  31. 68.4 Gemma 2 27B
  32. 68.0 Gemini 1.5 Flash
  33. 67.2 phi-3-small 7.4B
  34. 67.0 Qwen1.5-72B
  35. 66.0 Yi-Large
  36. 65.6 Yi-34B
  37. 65.6 Qwen1.5-32B
  38. 65.6 Gemini 1.5 Flash
  39. 65.2 Claude 2
  40. 65.2 Mixtral 8x7B
  41. 64.4 Claude 2.1
  42. 64.0 Qwen1.5-110B
  43. 63.2 Jamba 1.5 Mini
  44. 63.0 Gemma 2 9B
  45. 62.8 DeepSeek LLM 67B
  46. 62.2 GPT-3.5 Turbo
  47. 61.8 Claude 1.3
  48. 61.8 Llama 2-70B
  49. 61.6 mistral-small-2402
  50. 60.8 Amazon Nova Micro
  51. 60.0 Qwen2.5-7B
  52. 58.0 Llama 3-8B
  53. 58.0 Snowflake Arctic Instruct
  54. 56.7 Command R+
  55. 55.9 Claude Instant
  56. 55.5 Command R (03-2024)
  57. 53.8 phi-3-mini 3.8B
  58. 52.5 Mistral 7B v0.1
  59. 51.7 Mistral 7B v0.3
  60. 51.5 Qwen1.5-14B
  61. 51.3 Gemma 7B
  62. 50.7 LLaMA-65B
  63. 49.7 Yi 6B
  64. 48.3 Gemini 1.0 Pro
  65. 47.9 Qwen1.5-7B
  66. 42.0 Falcon-40B
  67. 41.0 Phi-2
  68. 39.2 Llama 2-13B
  69. 39.2 Llama 2-7B
  70. 27.0 Llama 3.2 11B
  71. 25.4 Falcon-7B
  72. 24.4 Llama 3.1-8B
  1. 1 GPT-4o May 2024 89.4%
  2. 2 GPT-4o Aug 2024 86.3%
  3. 3 Claude 3.5 Sonnet (October 2024) 85.9%
  4. 4 Claude 3.5 Sonnet 82.5%
  5. 5 GPT-4 Jun 2023 81.5%
  6. 6 Amazon Nova Pro 81.1%
  7. 7 DeepSeek-V3 Mar 2025 80.9%
  8. 8 Llama 3.1-405B 80.5%
  9. 9 GPT-4 Turbo Apr 2024 78.3%
  10. 10 Llama 3-70B 77.7%
  11. 11 Claude 3 Opus 77.5%
  12. 12 Mistral Large 2 Jul 2024 77.5%
  13. 13 Palmyra X4 77.5%
  14. 14 Gemini 1.5 Pro Sep 2024 77.1%
  15. 15 Llama 3.1-70B 76.9%
  16. 16 Llama 3.2 90B 76.9%
  17. 17 Llama 3.3 70B Instruct 76.1%
  18. 18 Qwen2.5-72B 75.3%
  19. 19 GPT-4o mini 74.8%
  20. 20 Qwen2-72B 74.5%
  21. 21 Gemini 2.0 Flash Feb 2025 73%
  22. 22 Claude 3.5 Haiku 72.2%
  23. 23 Mixtral 8x22B 70.4%
  24. 24 Claude 3 Haiku 70.2%
  25. 25 phi-3-medium 14B 69.9%
  26. 26 Jamba 1.5 Large 69.8%
  27. 27 Amazon Nova Lite 69.6%
  28. 28 DBRX 69.4%
  29. 29 Gemini 1.5 Pro May 2024 69.2%
  30. 30 Claude 3 Sonnet 68.4%
  31. 31 Gemma 2 27B 68.4%
  32. 32 Gemini 1.5 Flash May 2024 68%
  33. 33 phi-3-small 7.4B 67.2%
  34. 34 Qwen1.5-72B 67%
  35. 35 Yi-Large 66%
  36. 36 Yi-34B 65.6%
  37. 37 Qwen1.5-32B 65.6%
  38. 38 Gemini 1.5 Flash Sep 2024 65.6%
  39. 39 Claude 2 65.2%
  40. 40 Mixtral 8x7B 65.2%
  41. 41 Claude 2.1 64.4%
  42. 42 Qwen1.5-110B 64%
  43. 43 Jamba 1.5 Mini 63.2%
  44. 44 Gemma 2 9B 63%
  45. 45 DeepSeek LLM 67B 62.8%
  46. 46 GPT-3.5 Turbo Jun 2023 62.2%
  47. 47 Claude 1.3 61.8%
  48. 48 Llama 2-70B 61.8%
  49. 49 mistral-small-2402 61.6%
  50. 50 Amazon Nova Micro 60.8%
  51. 51 Qwen2.5-7B 60%
  52. 52 Llama 3-8B 58%
  53. 53 Snowflake Arctic Instruct 58%
  54. 54 Command R+ 56.7%
  55. 55 Claude Instant 55.9%
  56. 56 Command R (03-2024) 55.5%
  57. 57 phi-3-mini 3.8B 53.8%
  58. 58 Mistral 7B v0.1 52.5%
  59. 59 Mistral 7B v0.3 51.7%
  60. 60 Qwen1.5-14B 51.5%
  61. 61 Gemma 7B 51.3%
  62. 62 LLaMA-65B 50.7%
  63. 63 Yi 6B 49.7%
  64. 64 Gemini 1.0 Pro 48.3%
  65. 65 Qwen1.5-7B 47.9%
  66. 66 Falcon-40B 42%
  67. 67 Phi-2 41%
  68. 68 Llama 2-13B 39.2%
  69. 69 Llama 2-7B 39.2%
  70. 70 Llama 3.2 11B 27%
  71. 71 Falcon-7B 25.4%
  72. 72 Llama 3.1-8B 24.4%

Sources: Labs’ model cards and Stanford HELM’s leaderboard

#EntryScoreSettingDateSource
1 GPT-4o Aug 2024 86.3% Aug 2024 Official leaderboard
2 Claude 3.5 Sonnet (October 2024) 85.9% Oct 2024 Official leaderboard
3 GPT-4o May 2024 85.7% May 2024 Official leaderboard
4 Claude 3.5 Sonnet 82.5% Jun 2024 Official leaderboard
5 gpt-4-1106-preview 81.7% Official leaderboard
6 GPT-4 Jun 2023 81.5% Jun 2023 Official leaderboard
7 Amazon Nova Pro 81.1% Dec 2024 Official leaderboard
8 DeepSeek-V3 Mar 2025 80.9% Mar 2025 Official leaderboard
9 Llama 3.1-405B 80.5% Jul 2024 Official leaderboard
10 GPT-4 Turbo Apr 2024 78.3% Apr 2024 Official leaderboard
11 Llama 3-70B 77.7% Apr 2024 Official leaderboard
12 Claude 3 Opus 77.5% Feb 2024 Official leaderboard
13 Mistral Large 2 Jul 2024 77.5% Jul 2024 Official leaderboard
14 Palmyra X4 77.5% Oct 2024 Official leaderboard
15 Gemini 1.5 Pro Sep 2024 77.1% Sep 2024 Official leaderboard
16 Llama 3.1-70B 76.9% Jul 2024 Official leaderboard
17 Llama 3.2 90B 76.9% Sep 2024 Official leaderboard
18 Llama 3.3 70B Instruct 76.1% Dec 2024 Official leaderboard
19 Qwen2.5-72B 75.3% Sep 2024 Official leaderboard
20 GPT-4o mini 74.8% Jul 2024 Official leaderboard
21 Qwen2-72B 74.5% Jun 2024 Official leaderboard
22 Gemini 2.0 Flash Feb 2025 73% Feb 2025 Official leaderboard
23 Claude 3.5 Haiku 72.2% Oct 2024 Official leaderboard
24 Mixtral 8x22B 70.4% Apr 2024 Official leaderboard
25 Claude 3 Haiku 70.2% Mar 2024 Official leaderboard
26 solar-pro-241126 69.8% Official leaderboard
27 Jamba 1.5 Large 69.8% Aug 2024 Official leaderboard
28 phi-3-medium 14B 69.6% Apr 2024 Official leaderboard
29 Amazon Nova Lite 69.6% Dec 2024 Official leaderboard
30 DBRX 69.4% Mar 2024 Official leaderboard
31 Gemini 1.5 Pro May 2024 69.2% May 2024 Official leaderboard
32 palmyra-x-v3 68.4% Official leaderboard
33 text-unicorn@001 68.4% Official leaderboard
34 Claude 3 Sonnet 68.4% Feb 2024 Official leaderboard
35 Gemma 2 27B 68.4% Jun 2024 Official leaderboard
36 Gemini 1.5 Flash May 2024 68% May 2024 Official leaderboard
37 phi-3-small 7.4B 67.2% Apr 2024 Official leaderboard
38 Qwen1.5-72B 67% Feb 2024 Official leaderboard
39 Yi-Large 66% May 2024 Official leaderboard
40 Yi-34B 65.6% Nov 2023 Official leaderboard
41 Qwen1.5-32B 65.6% Feb 2024 Official leaderboard
42 Gemini 1.5 Flash Sep 2024 65.6% Aug 2024 Official leaderboard
43 Claude 2 65.2% Jul 2023 Official leaderboard
44 Mixtral 8x7B 65.2% Dec 2023 Official leaderboard
45 Claude 2.1 64.4% Nov 2023 Official leaderboard
46 Qwen1.5-110B 64% Apr 2024 Official leaderboard
47 Jamba 1.5 Mini 63.2% Aug 2024 Official leaderboard
48 Gemma 2 9B 63% Jun 2024 Official leaderboard
49 DeepSeek LLM 67B 62.8% Nov 2023 Official leaderboard
50 GPT-3.5 Turbo Jun 2023 62.2% Jun 2023 Official leaderboard
51 Claude 1.3 61.8% Apr 2023 Official leaderboard
52 Llama 2-70B 61.8% Jul 2023 Official leaderboard
53 mistral-small-2402 61.6% Feb 2024 Official leaderboard
54 mistral-medium-2312 61% Official leaderboard
55 Amazon Nova Micro 60.8% Dec 2024 Official leaderboard
56 Qwen2.5-7B 60% Sep 2024 Official leaderboard
57 palmyra-x-v2 59.8% Official leaderboard
58 open-mistral-nemo-2407 59% Official leaderboard
59 Llama 3-8B 58% Apr 2024 Official leaderboard
60 Snowflake Arctic Instruct 58% Apr 2024 Official leaderboard
61 Command R+ 56.7% Aug 2024 Official leaderboard
62 Claude Instant 55.9% Aug 2023 Official leaderboard
63 Command R (03-2024) 55.5% Mar 2024 Official leaderboard
64 text-bison@001 54.7% Official leaderboard
65 text-davinci-003 53.1% Official leaderboard
66 text-davinci-002 52.5% Official leaderboard
67 Mistral 7B v0.1 52.5% Sep 2023 Official leaderboard
68 jamba-instruct 51.9% Official leaderboard
69 Mistral 7B v0.3 51.7% Sep 2023 Official leaderboard
70 Qwen1.5-14B 51.5% Feb 2024 Official leaderboard
71 Gemma 7B 51.3% Feb 2024 Official leaderboard
72 LLaMA-65B 50.7% Feb 2023 Official leaderboard
73 mistral-large-2402 49.9% Official leaderboard
74 Yi 6B 49.7% Nov 2023 Official leaderboard
75 Gemini 1.0 Pro 48.3% Dec 2023 Official leaderboard
76 Qwen1.5-7B 47.9% Feb 2024 Official leaderboard
77 command 44.5% Official leaderboard
78 j2-jumbo 43.1% Official leaderboard
79 Falcon-40B 42% Mar 2023 Official leaderboard
80 Phi-2 41% Dec 2023 Official leaderboard
81 Llama 2-13B 39.2% Jul 2023 Official leaderboard
82 Llama 2-7B 39.2% Jul 2023 Official leaderboard
83 j2-grande 39% Official leaderboard
84 command-light 31.2% Official leaderboard
85 luminous-extended 27.6% Official leaderboard
86 luminous-supreme 27.6% Official leaderboard
87 Llama 3.2 11B 27% Sep 2024 Official leaderboard
88 luminous-base 26% Official leaderboard
89 Falcon-7B 25.4% Apr 2023 Official leaderboard
90 Llama 3.1-8B 24.4% Jul 2024 Official leaderboard
91 olmo-7b 22.9% Official leaderboard
#ModelScoreSettingDateSource
1 GPT-4o May 2024 89.4% Aug 2024 Lab model card
2 GPT-4o Aug 2024 86.3% Aug 2024 Official leaderboard
3 Claude 3.5 Sonnet (October 2024) 85.9% Oct 2024 Official leaderboard
4 Claude 3.5 Sonnet 82.5% Jun 2024 Official leaderboard
5 GPT-4 Jun 2023 81.5% Jun 2023 Official leaderboard
6 Amazon Nova Pro 81.1% Dec 2024 Official leaderboard
7 DeepSeek-V3 Mar 2025 80.9% Mar 2025 Official leaderboard
8 Llama 3.1-405B 80.5% Jul 2024 Official leaderboard
9 GPT-4 Turbo Apr 2024 78.3% Apr 2024 Official leaderboard
10 Llama 3-70B 77.7% Apr 2024 Official leaderboard
11 Claude 3 Opus 77.5% Feb 2024 Official leaderboard
12 Mistral Large 2 Jul 2024 77.5% Jul 2024 Official leaderboard
13 Palmyra X4 77.5% Oct 2024 Official leaderboard
14 Gemini 1.5 Pro Sep 2024 77.1% Sep 2024 Official leaderboard
15 Llama 3.1-70B 76.9% Jul 2024 Official leaderboard
16 Llama 3.2 90B 76.9% Sep 2024 Official leaderboard
17 Llama 3.3 70B Instruct 76.1% Dec 2024 Official leaderboard
18 Qwen2.5-72B 75.3% Sep 2024 Official leaderboard
19 GPT-4o mini 74.8% Jul 2024 Official leaderboard
20 Qwen2-72B 74.5% Jun 2024 Official leaderboard
21 Gemini 2.0 Flash Feb 2025 73% Feb 2025 Official leaderboard
22 Claude 3.5 Haiku 72.2% Oct 2024 Official leaderboard
23 Mixtral 8x22B 70.4% Apr 2024 Official leaderboard
24 Claude 3 Haiku 70.2% Mar 2024 Official leaderboard
25 phi-3-medium 14B 69.9% Aug 2024 Lab model card
26 Jamba 1.5 Large 69.8% Aug 2024 Official leaderboard
27 Amazon Nova Lite 69.6% Dec 2024 Official leaderboard
28 DBRX 69.4% Mar 2024 Official leaderboard
29 Gemini 1.5 Pro May 2024 69.2% May 2024 Official leaderboard
30 Claude 3 Sonnet 68.4% Feb 2024 Official leaderboard
31 Gemma 2 27B 68.4% Jun 2024 Official leaderboard
32 Gemini 1.5 Flash May 2024 68% May 2024 Official leaderboard
33 phi-3-small 7.4B 67.2% Apr 2024 Official leaderboard
34 Qwen1.5-72B 67% Feb 2024 Official leaderboard
35 Yi-Large 66% May 2024 Official leaderboard
36 Yi-34B 65.6% Nov 2023 Official leaderboard
37 Qwen1.5-32B 65.6% Feb 2024 Official leaderboard
38 Gemini 1.5 Flash Sep 2024 65.6% Aug 2024 Official leaderboard
39 Claude 2 65.2% Jul 2023 Official leaderboard
40 Mixtral 8x7B 65.2% Dec 2023 Official leaderboard
41 Claude 2.1 64.4% Nov 2023 Official leaderboard
42 Qwen1.5-110B 64% Apr 2024 Official leaderboard
43 Jamba 1.5 Mini 63.2% Aug 2024 Official leaderboard
44 Gemma 2 9B 63% Jun 2024 Official leaderboard
45 DeepSeek LLM 67B 62.8% Nov 2023 Official leaderboard
46 GPT-3.5 Turbo Jun 2023 62.2% Jun 2023 Official leaderboard
47 Claude 1.3 61.8% Apr 2023 Official leaderboard
48 Llama 2-70B 61.8% Jul 2023 Official leaderboard
49 mistral-small-2402 61.6% Feb 2024 Official leaderboard
50 Amazon Nova Micro 60.8% Dec 2024 Official leaderboard
51 Qwen2.5-7B 60% Sep 2024 Official leaderboard
52 Llama 3-8B 58% Apr 2024 Official leaderboard
53 Snowflake Arctic Instruct 58% Apr 2024 Official leaderboard
54 Command R+ 56.7% Aug 2024 Official leaderboard
55 Claude Instant 55.9% Aug 2023 Official leaderboard
56 Command R (03-2024) 55.5% Mar 2024 Official leaderboard
57 phi-3-mini 3.8B 53.8% Aug 2024 Lab model card
58 Mistral 7B v0.1 52.5% Sep 2023 Official leaderboard
59 Mistral 7B v0.3 51.7% Sep 2023 Official leaderboard
60 Qwen1.5-14B 51.5% Feb 2024 Official leaderboard
61 Gemma 7B 51.3% Feb 2024 Official leaderboard
62 LLaMA-65B 50.7% Feb 2023 Official leaderboard
63 Yi 6B 49.7% Nov 2023 Official leaderboard
64 Gemini 1.0 Pro 48.3% Dec 2023 Official leaderboard
65 Qwen1.5-7B 47.9% Feb 2024 Official leaderboard
66 Falcon-40B 42% Mar 2023 Official leaderboard
67 Phi-2 41% Dec 2023 Official leaderboard
68 Llama 2-13B 39.2% Jul 2023 Official leaderboard
69 Llama 2-7B 39.2% Jul 2023 Official leaderboard
70 Llama 3.2 11B 27% Sep 2024 Official leaderboard
71 Falcon-7B 25.4% Apr 2023 Official leaderboard
72 Llama 3.1-8B 24.4% Jul 2024 Official leaderboard

Labs report scores for 4 models in their cards. Switch the scores above to rank them together.

5 models have held number one on Stanford HELM’s leaderboard since 2023 5 models have held number one with claims included since 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for Stanford HELM’s leaderboard and what labs claim in their cards.

Meta Anthropic OpenAI Microsoft
Stanford HELM leaderboard Passed 85% in May 2024, by GPT-4o
GPT-4o 86.3%
Lab claims Passed 85% in Aug 2024, by GPT-4o
GPT-4o 89.4%

Sources: Stanford HELM’s leaderboard and labs’ model cards

GPT-4o on the official leaderboard passed 85% in May 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On Stanford HELM’s leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 Near the ceiling (85%) GPT-4o GPT-4 Claude 1.3 LLaMA-65B 0% 20% 40% 60% 80% 100% 2023 2024 2025 Near the ceiling (85%) GPT-4o GPT-4

Sources: Stanford HELM’s leaderboard and labs’ model cards

No launch post has listed it since April 2024

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Microsoft
DateLaunch postScoreAs the post writes it
Apr 22, 2024 phi-3-medium 14B 69.4 MedQA (2-Shot)

4 system cards report it, from 2 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Aug 30, 2024 phi-3-small 7.4B technical report 65.4 MedQA (2-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-mini 3.8B technical report 53.8 MedQA (2-Shot) (Section 3 table) Capabilities
Aug 30, 2024 phi-3-medium 14B technical report 69.9 MedQA (2-Shot) (Section 3 table) Capabilities
Aug 8, 2024 GPT-4o system card 89.4 MedQA USMLE 4 Options (0-shot), 89.4% in text (Table 7 shows 0.89, 5-shot 0.89) Health

The lab's number comes out ahead in 2 of 3 pairs that differ

Each score a lab reported for its own model, beside the same model's best score on the leaderboard. Gaps mostly come from different harnesses, effort levels and task subsets. Every benchmark.

Model Lab says Board says Gap
GPT-4o May 2024 89.4% MedQA USMLE 4 Options (0-shot), 89.4% in text (Table 7 shows 0.89, 5-shot 0.89) 85.7% Leaderboard +3.7 pts
phi-3-small 7.4B 65.4% MedQA (2-Shot) (Section 3 table) 67.2% Leaderboard −1.8 pts
phi-3-medium 14B 69.9% MedQA (2-Shot) (Section 3 table) 69.6% Leaderboard +0.3 pts

More in Law, finance and medicine

All 179 in Law, finance and medicine
  • MedXpertQA-MM In use Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. Qwen 3.8 Max 80.4%
  • HealthBench Professional In use 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. Claude Sonnet 5.5 69.2%
  • HealthBench In use 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. GPT-5 67.2%
  • FinSearchComp In use Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. Ling-3.1-flash 81.7%
  • KBL In use Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items. Solar Pro 4 77.5%
  • KorMedMCQA In use Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. Solar Pro 4 93.2%