Phi-2

Phi-2.

Microsoft Released December 12th, 2023 Family Phi-2 The lab's model card Compare with others

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Microsoft Dec 2023

Phi-2

Open weights Small

96.4 #223 of 256

Epoch Capabilities Index

Self-Reported Index 104.5 #387 of 404

CyberMetric 52.1% #7 /10
MMLU 56.7% #121 /131
BIG-Bench Hard 59.2% #60 /73

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
OpenBookQA	73.6%	#9 of 41	Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
ARC-Challenge	75.9%	#20 of 75	Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
BoolQ	83.3%	#10 of 28	Lab model card
MuSR	41%	#53 of 98	Official leaderboard
Vectara Hallucination Leaderboard (HHEM)	6.70%	#110 of 148	Official leaderboard
ANLI	42.5%	#9 of 11	Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
BIG-Bench Hard	59.2%	#60 of 73	Lab model card
HellaSwag	53.6%	#58 of 63	Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
MMLU	56.7%	#121 of 131	Lab model card
AlpacaEval 2.0	4.40%	#45 of 47	Official leaderboard
WinoGrande	54.7%	#74 of 76	Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
EQ-Bench v2	27.6	#83 of 85	Official leaderboard
TriviaQA	45.2%	#38 of 38	Claim in Phi-3 Technical Report: A Highly Capable Language Model L…, via Epoch AI
AlpacaEval	30.7%	#19 of 19	Official leaderboard

Math and science

Benchmark	Score		Rank	Source
GSM8K	61.1%		#79 of 92	Lab model card

Coding

Benchmark	Score	Rank	Source
HumanEval	49.4%	#33 of 46	Official leaderboard
HumanEval+	45.1%	#33 of 46	Official leaderboard
MBPP+	54.2%	#26 of 35	Official leaderboard
MBPP	59.1%	#58 of 71	Lab model card
CRUXEval	33.5%	#10 of 11	Official leaderboard

Professional work

Benchmark	Score		Rank	Source
MedQA	41%		#66 of 71	Official leaderboard

Safety and security

Benchmark	Score		Rank	Source
CyberMetric	52.1%		#7 of 10	Official leaderboard

Other

Benchmark	Score		Rank	Source
Open LLM Leaderboard v2	15.5		#68 of 99	Official leaderboard
HELM Lite	16.9		#69 of 71	Official leaderboard

Its lab's numbers put it at 104.5, 8.1 points above its ECI

The Self-Reported Index fits one score to the 5 capability results Microsoft reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 100.9 to 108.2. How we compute it.

BoolQ 83.3
GSM8K 61.1
BIG-Bench Hard 59.2
MBPP 59.1
MMLU 56.7

Its lab's number next to a leaderboard

Where the benchmark's own leaderboard, or Epoch AI's run, also has this model. Gaps mostly come from different harnesses, effort levels and task subsets. Every pair.

Benchmark	Lab says	Board says	Gap
MBPP	59.1% MBPP, Table 2	64% Leaderboard, EvalPlus subset, original tests	−4.9 pts

Microsoft's models over time

Microsoft's models in color, against every other model on the Epoch Capabilities Index, by release date.

Source: Epoch Capabilities Index, Epoch AI (CC BY 4.0)

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Phi-2: The surprising power of small language models (Microsoft Research blog), Tables 1 and 2 Dec 12, 2023