Benchmark Almanac

ChatGLM3-6B

ChatGLM3-6B places in the top three on 2 benchmarks.

Z.ai Released October 27th, 2023 Family ChatGLM3 The lab's model card Compare with others
34
Benchmarks with a score
0
Ranked first, out of five or more models
2
In the top three
Z.ai Oct 2023

ChatGLM3-6B

Open weights Open model

107.9 #383 of 404
Self-Reported Index, from the lab's own numbers. No Epoch Capabilities Index score yet.
  • MathBench 19.9% #20 /21
  • CFLUE Knowledge 40.8% #4 /8
  • FinEval 43.2% #7 /7

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
BoolQ 87.9% #2 of 28 Lab model card
CommonSenseQA 86.5% #2 of 19 Lab model card
AGIEval 53.7% #24 of 40 Lab model card
BIG-Bench Hard 66.1% #52 of 73 Lab model card
HellaSwag 79.7% #40 of 53 Lab model card
PIQA 80.1% #30 of 37 Lab model card
MT-Bench-101 6.47 #5 of 6 Official leaderboard
DROP 70.9% #36 of 41 Lab model card
MMLU 61.4% #116 of 131 Lab model card
C-Eval 69% #41 of 46 Lab model card
CriticEval 3.95 #11 of 12 Official leaderboard
Chinese SimpleQA 12% #26 of 27 Official leaderboard
LMArena (Elo) 1,057 #329 of 335 Official leaderboard
CMMLU 67.5% #17 of 17 Lab model card
C3 73.9% #6 of 6 Lab model card
FollowBench 38.7% #3 of 3 Official leaderboard
GAOKAO-Bench 67.3% #2 of 2 Lab model card

Math and science

BenchmarkScoreRankSource
GSM8K 72.3% #72 of 92 Lab model card
MATH 25.7% #89 of 99 Lab model card
MathBench 19.9% #20 of 21 Official leaderboard
GPQA Diamond 26.8% #299 of 302 Lab model card
SuperCLUE-Math6 33.0 #4 of 4 Official leaderboard

Coding

BenchmarkScoreRankSource
HumanEval 58.5% #83 of 101 Lab model card
MBPP 52.4% #60 of 71 Lab model card
LeetCode Weekly Contest 2.40% #2 of 2 Official leaderboard

Agents and tools

BenchmarkScoreRankSource
T-Eval 51.4% #8 of 12 Official leaderboard
Berkeley Function Calling Leaderboard (BFCL) 57.9% #37 of 41 Lab model card
Code Interpreter Benchmark General 62.1% #2 of 2 Official leaderboard
Code Interpreter Benchmark Math 54.2% #2 of 2 Official leaderboard

Professional work

BenchmarkScoreRankSource
CFLUE Knowledge 40.8% #4 of 8 Official leaderboard
CFLUE Application 36.6 #3 of 4 Official leaderboard
FinEval 43.2% #7 of 7 Official leaderboard

Safety and security

BenchmarkScoreRankSource
SALAD-Bench 90.5% #8 of 13 Official leaderboard
SORRY-Bench 35.9% #17 of 23 Official leaderboard

Its lab's numbers put it at 107.9, number 383 of 404

The Self-Reported Index fits one score to the 16 capability results Z.ai reported for this model, on the scale of the Epoch Capabilities Index. Labs choose which benchmarks to show and how to run them, so read it as the lab's claim, not an independent measurement. The 90% interval runs from 105.5 to 110.3. How we compute it.

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Its system card

Every result in the longer documents the lab published with it. 10 of them aren't in the launch post.