Benchmark Almanac

AutoMedBench

Claude Opus 4.6 66.5 leads since June 2026
Not in model cards

Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission.

The current full release has 48 tasks with two difficulty tiers, giving 96 task-tier combinations. These agents execute inference workflows rather than train models.

Released
June 2026
Kind
Machine learning engineering
Level
Expert
Scored
Graded by a model
Answers
Agent task
Tasks
Public, plus a private test set
Measures
Capability
Language
English
In and out
Text, Images, Code → Code, Actions
Interaction
Agentic
Environment
Terminal
Refresh
One fixed set
Run by
A host
Strongest model tested
GPT-5.4 #28 on the capability index
Last model added
March 2026
Scores refreshed
November 18th, 2026

In the authors’ words

Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average.

Abstract of AutoMedBench: Towards Medical AutoResearch with Agentic AI Models (arXiv:2606.01961, CC0)
6
Models scored in its paper

Claude Opus 4.6 leads with 66.5 among models released by March 2026

Best score per model from the benchmark's paper, read on November 18th, 2026. A paper’s table doesn’t add newer models. Where a model ran with several settings or harnesses, we keep its best.

The newest model with a score here came out in March 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 1 Claude Opus 4.6 paper release; 24 tasks; Lite and Standard 66.5
  2. 2 GLM-5 paper release; 24 tasks; Lite and Standard 61.6
  3. 3 Gemini 3.1 Pro paper release; 24 tasks; Lite and Standard 59.0
  4. 4 GPT-5.4 paper release; 24 tasks; Lite and Standard 55.3
  5. 5 MiniMax-M2.5 paper release; 24 tasks; Lite and Standard 51.6
  6. 6 Qwen3.5 397B-A17B paper release; 24 tasks; Lite and Standard 51.2

Source: The AutoMedBench paper

#ModelScoreSettingDateSource
1 Claude Opus 4.6 66.5 paper release; 24 tasks; Lite and Standard Feb 2026 Official leaderboard
2 GLM-5 61.6 paper release; 24 tasks; Lite and Standard Feb 2026 Official leaderboard
3 Gemini 3.1 Pro 59.0 paper release; 24 tasks; Lite and Standard Feb 2026 Official leaderboard
4 GPT-5.4 55.3 paper release; 24 tasks; Lite and Standard Mar 2026 Official leaderboard
5 MiniMax-M2.5 51.6 paper release; 24 tasks; Lite and Standard Feb 2026 Official leaderboard
6 Qwen3.5 397B-A17B 51.2 paper release; 24 tasks; Lite and Standard Feb 2026 Official leaderboard

Rankings by part

We read 5 parts from the benchmark’s paper, each ranking models on one part of the test, such as Segmentation, Enhancement and VQA. Claude Opus 4.6 leads 4 of them, and GLM-5 leads the other one. Arrows show how far a model moved from its overall rank among the models we track.

Segmentation

6 models we track have a score here.

  1. 1 Claude Opus 4.6 67.2
  2. 2 GPT-5.4 ↑2 59.4
  3. 3 GLM-5 ↓1 58.0
  4. 4 Gemini 3.1 Pro ↓1 54.7
  5. 5 MiniMax-M2.5 46.5
  6. 6 Qwen3.5 397B-A17B 42.8

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Benchmark paper Not tested since Jun 2026

Source: The AutoMedBench paper

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
In the benchmark’s paper
50.0 55.0 60.0 65.0 70.0 2026 50.0 55.0 60.0 65.0 70.0 2026

Source: Official AutoMedBench leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in AI research

All 164 in AI research
  • PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. Qwen 3.8 Max 93%
  • MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
  • PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
  • MLE-bench In use Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. GPT-6.2 Astra 96.1%
  • PMPP-Hard In use Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. Kimi K3 0.71
  • KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. MiniMax-M3 28.8%

Data audit

8 of 9 checks pass, 1 warning. Look first at the Newest score check. The newest came 8 months ago, in March 2026, for GPT-5.4. Every benchmark

Catalog record 2 pass
  • Pass Task access Public tasks, plus a private test set.
  • Pass Confidence Checked against a primary source.
Scores 1 warning 2 pass 2 notes
  • Pass Independent runs Scores from the benchmark’s paper (6 entries).
  • Pass Models scored 6 models we track.
  • Warning Newest score The newest came 8 months ago, in March 2026, for GPT-5.4.
  • Note Frontier models None of the top 10 has a score. The strongest tested is GPT-5.4 (#28).
  • Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
  • Pass Importer We read the board on arxiv.org: 6 rows for models we track at the last read.
  • Pass Last read November 18th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check