Benchmark Almanac

SimplerEnv

Praxis-1 80% leads since November 2026
In use

Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks.

Self-reported under mixed zero-shot and fine-tuned protocols. Papers quote conflicting baselines. The WidowX track is near ceiling. Google Robot tracks still separate models.

Released
May 2024
Kind
Simulated robots
Level
Everyday
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Images, Time series → Actions
Interaction
Agentic
Environment
A simulator
Items from
Generated
Refresh
One fixed set
Run by
Anyone
In views
Agents, Vision, Robotics
Last in a launch post
November 2026
Last model added
November 2026

In the authors’ words

The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks.

Abstract of Evaluating Real-World Robot Manipulation Policies in Simulation (arXiv:2405.05941, CC0)
4
Models scored in lab cards
3
Launch posts list it, from 2 labs
1
System card reports it, from 1 lab

Praxis-1 leads with 80%

Best score per model as labs report them in their own cards, drawn striped. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Praxis-1 80%
  2. 2 GR00T N1.5 63%
  3. 3 GR00T N1.7 62.3%
  4. 4 GR00T N1.6 62.1%

Source: Labs’ model cards

#ModelScoreSettingDateSource
1 Praxis-1 80% Nov 2026 Lab model card
2 GR00T N1.5 63% Sep 2025 Lab model card
3 GR00T N1.7 62.3% Jul 2026 Lab model card
4 GR00T N1.6 62.1% Dec 2025 Lab model card

2 models have held number one in labs’ own numbers since its release in May 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

NVIDIA Runway
Lab claims 2 leaders
Praxis-1 80%

Source: Labs’ model cards

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Plot with other benchmarks
Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Praxis-1 GR00T N1.5 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Praxis-1 GR00T N1.5

Source: Labs’ model cards

2 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

NVIDIA
Runway
DateLaunch postScoreAs the post writes it
Nov 11, 2026 Praxis-1 80 80.0% success on SimplerEnv
Jul 7, 2026 GR00T N1.7 62.3 SimplerEnv Bridge (WidowX) average 62.3%. Fractal (Google robot) average 72.5%. N1.6 re-run 56.6% and 52.0% (examples/SimplerEnv README)
Dec 16, 2025 GR00T N1.6 62.1 SimplerEnv Bridge (WidowX) average 62.07%, GR00T-N1.6-bridge checkpoint. Fractal (Google robot) average 67.66% (examples/SimplerEnv README)

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
Sep 26, 2025 GR00T N1.5 model card 63 SimplerEnv Bridge (WidowX) average 63% over 7 tasks with 300 trials each. Fractal (Google robot) average 41% (examples/SimplerEnv README) Capabilities

More in Robotics

All 172 in Robotics
  • ERQA In use 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. Gemini Robotics ER 2 78.5%
  • RoboLab In use Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. Praxis-1 46.2%
  • RoboDojo In use Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. GPT-6 Astra 27.6%
  • Progress classification (Google DeepMind) In use Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Gemini Robotics ER 2 57.4%
  • Human proximity (Google DeepMind robotics) In use Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Gemini Robotics ER 2 93%
  • Moment finding (Google DeepMind) In use Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Gemini Robotics ER 2 91.3%

Data audit

All 5 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Note Independent runs Only labs’ own numbers, so no one else has checked them. Scores from launch posts (3 scores) and system cards (1 score).
  • Note Models scored Only 4 models we track.
  • Pass Newest score The newest is from November 2026, for Praxis-1.
  • Note Frontier models None of its models are on the capabilities index.
  • Pass Headroom The best is 80%, under 85%, so it still separates models.
Leaderboard import 1 note
  • Note Importer No public leaderboard to read.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check