Novel logical reasoning problems expert evaluation (Z.ai)

GLM-4.5 62.0

Only in system cards

Human experts score model answers to new multi-step logic problems with a fixed scoring standard for each question.

In-house Z.ai evaluation built to avoid contamination. The problems are not public.

Made by: Zhipu AI (Z.ai)
Released: August 2025
Kind: Abstract puzzles and logic
Level: University
Scored: Graded by people
Answers: Open-ended
Tasks: Kept private

Last in a system card: August 2025
Last model added: August 2025

Paper arxiv.org

Models scored in lab cards

System cards report it, from 1 lab

GLM-4.5 leads with 62.0

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

1 GLM-4.5 62.0
2 GLM-4.5-Air 53.4

Source: Labs’ model cards

#	Model	Score	Setting	Date	Source
1	GLM-4.5	62.0		Aug 2025	Lab model card
2	GLM-4.5-Air	53.4		Aug 2025	Lab model card

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card

Source: Labs’ model cards

Labs report it only in system cards

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark. Some system cards do, below.

2 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
Aug 8, 2025	GLM-4.5 technical report	62	Expert Evaluation Scores on Novel Logical Reasoning Problems (Table 11)	Capabilities
Aug 8, 2025	GLM-4.5-Air technical report	53.4	Expert Evaluation Scores on Novel Logical Reasoning Problems (Table 11)	Capabilities

GLM-4.5 leads with 62.0

Scores over time

Labs report it only in system cards

2 system cards report it, from 1 lab

More in Science and reasoning