Benchmark Almanac

MathTutorBench

Gemini 3.6 Flash 0.98 leads since July 2026
Not in model cards

Models solve math problems, diagnose student mistakes, ask Socratic questions, and give instructional support. Automatic answer checks, F1, and text similarity score the constrained tasks. A trained pedagogical reward model compares tutoring responses with teacher examples.

Seven tasks cover three teaching skills, with additional hard variants. Most models run without reasoning, while three Gemini configurations retain adaptive reasoning. The board reports separate task metrics rather than one accuracy.

Made by
ETH Zurich and TU Darmstadt
Released
February 2025
Kind
Word problems
Level
School
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Measures
Capability
Language
English
In and out
Text → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
Strongest model tested
Gemini 3.1 Pro #40 on the capability index
Last model added
July 2026
Scores refreshed
November 20th, 2026

In the authors’ words

Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.

Abstract of MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors (arXiv:2502.18940, CC0)
17
Entries on its own leaderboard, 12 models we track

Gemini 3.6 Flash leads with 0.98 among models released by July 2026 Gemini 3.6 Flash leads with 0.98 among models released by July 2026

Best score per entry from the benchmark's own leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

The newest model with a score here came out in July 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Gemini 3.6 Flash Adaptive reasoning 0.98
  2. 2 Gemini 3.1 Pro Adaptive reasoning 0.97
  3. 3 Qwen3.6 27B Reasoning disabled 0.97
  4. 4 LearnLM-1.5-Pro Reasoning disabled 0.94
  5. 5 Llama 3.1-70B Reasoning disabled 0.91
  6. 6 Claude Sonnet 4.6 Reasoning disabled 0.91
  7. 7 GPT-4o Mar 2025, Reasoning disabled 0.90
  8. 8 Qwen2.5-Math-7B-Instruct Reasoning disabled 0.88
  9. 9 Gemini 2.5 Pro Jun 2025, Adaptive reasoning 0.88
  10. 10 Qwen2.5-7B Reasoning disabled 0.85
  11. 11 glm-4.7-flash Reasoning disabled 0.78
  12. 12 eth-nlped/TutorRL-7B Reasoning disabled 0.77
  13. 13 Apertus 1.5 8B Reasoning disabled 0.76
  14. 14 Qwen2.5-7B-SocraticLM Reasoning disabled 0.73
  15. 15 Llama 3.1-8B Reasoning disabled 0.70
  16. 16 Llemma-7B-ScienceTutor Reasoning disabled 0.62
  17. 17 Llama 3.2 3B Reasoning disabled 0.60

Source: Official MathTutorBench leaderboard

#EntryScoreSettingDateSource
1 Gemini 3.6 Flash 0.98 Adaptive reasoning Jul 2026 Official leaderboard
2 Gemini 3.1 Pro 0.97 Adaptive reasoning Feb 2026 Official leaderboard
3 Qwen3.6 27B 0.97 Reasoning disabled Apr 2026 Official leaderboard
4 LearnLM-1.5-Pro 0.94 Reasoning disabled Official leaderboard
5 Llama 3.1-70B 0.91 Reasoning disabled Jul 2024 Official leaderboard
6 Claude Sonnet 4.6 0.91 Reasoning disabled Feb 2026 Official leaderboard
7 GPT-4o Mar 2025 0.90 Reasoning disabled Mar 2025 Official leaderboard
8 Qwen2.5-Math-7B-Instruct 0.88 Reasoning disabled Official leaderboard
9 Gemini 2.5 Pro Jun 2025 0.88 Adaptive reasoning Jun 2025 Official leaderboard
10 Qwen2.5-7B 0.85 Reasoning disabled Sep 2024 Official leaderboard
11 glm-4.7-flash 0.78 Reasoning disabled Jan 2026 Official leaderboard
12 eth-nlped/TutorRL-7B 0.77 Reasoning disabled Official leaderboard
13 Apertus 1.5 8B 0.76 Reasoning disabled Jul 2026 Official leaderboard
14 Qwen2.5-7B-SocraticLM 0.73 Reasoning disabled Official leaderboard
15 Llama 3.1-8B 0.70 Reasoning disabled Jul 2024 Official leaderboard
16 Llemma-7B-ScienceTutor 0.62 Reasoning disabled Official leaderboard
17 Llama 3.2 3B 0.60 Reasoning disabled Sep 2024 Official leaderboard
#ModelScoreSettingDateSource
1 Gemini 3.6 Flash 0.98 Adaptive reasoning Jul 2026 Official leaderboard
2 Gemini 3.1 Pro 0.97 Adaptive reasoning Feb 2026 Official leaderboard
3 Qwen3.6 27B 0.97 Reasoning disabled Apr 2026 Official leaderboard
4 Llama 3.1-70B 0.91 Reasoning disabled Jul 2024 Official leaderboard
5 Claude Sonnet 4.6 0.91 Reasoning disabled Feb 2026 Official leaderboard
6 GPT-4o Mar 2025 0.90 Reasoning disabled Mar 2025 Official leaderboard
7 Gemini 2.5 Pro Jun 2025 0.88 Adaptive reasoning Jun 2025 Official leaderboard
8 Qwen2.5-7B 0.85 Reasoning disabled Sep 2024 Official leaderboard
9 glm-4.7-flash 0.78 Reasoning disabled Jan 2026 Official leaderboard
10 Apertus 1.5 8B 0.76 Reasoning disabled Jul 2026 Official leaderboard
11 Llama 3.1-8B 0.70 Reasoning disabled Jul 2024 Official leaderboard
12 Llama 3.2 3B 0.60 Reasoning disabled Sep 2024 Official leaderboard

Rankings by part

We read 9 parts from the benchmark’s own leaderboard, each ranking models on one part of the test, such as Problem Solving, Socratic Questioning and Solution Correctness. Gemini 3.1 Pro leads 5 of them, and 3 other models lead the rest. Arrows show how far a model moved from its overall rank among the models we track.

Problem Solving

Score from 0 to 1 12 models we track have a score here.

  1. 1 Gemini 3.6 Flash 0.98
  2. 2 Gemini 3.1 Pro 0.97
  3. 3 Qwen3.6 27B 0.97
  4. 4 Llama 3.1-70B 0.91
  5. 5 Claude Sonnet 4.6 0.91
  6. 6 GPT-4o Mar 2025 0.90
  7. 7 Gemini 2.5 Pro Jun 2025 0.88
  8. 8 Qwen2.5-7B 0.85
  9. 9 glm-4.7-flash 0.78
  10. 10 Apertus 1.5 8B 0.76

3 models have held number one on the official leaderboard since its release in February 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta Google DeepMind
Own leaderboard Not tested since Jul 2026

Source: Official MathTutorBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
On the benchmark’s own leaderboard
0.60 0.70 0.80 0.90 1.00 2025 2026 0.60 0.70 0.80 0.90 1.00 2025 2026

Source: Official MathTutorBench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Math

All 140 in Math
  • AIME 2026 In use Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. GPT-5.5 100%
  • IMO-AnswerBench In use 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. GLM-5.2 91%
  • HMMT 2026 In use Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. GPT-5.5 98.5%
  • PolyMath In use Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Qwen3.7-Max 86.5%
  • MathArena (Apex, IMO, USAMO) In use Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. Claude Opus 4.8 81.3%
  • MathArena Apex Shortlist In use Models solve 47 hard final-answer problems that MathArena selected from 2025 competitions, where the best models scored about 50% at selection. Scored by accuracy over repeated attempts. GPT-5.5 98.4%

Data audit

All 9 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 3 pass 2 notes
  • Pass Independent runs Scores from the official leaderboard (17 entries).
  • Pass Models scored 12 models we track, plus 5 other entries on its leaderboard.
  • Pass Newest score The newest is from July 2026, for Apertus 1.5 8B.
  • Note Frontier models None of the top 10 has a score. The strongest tested is Gemini 3.1 Pro (#40).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on eth-lre.github.io: 12 rows for models we track and 5 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check