EBench

In the authors’ words

We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including π₀, π0.5, XVLA, and InternVLA-A1, and reveal that the models exhibit strikingly different capability profiles: π0.5 achieves the highest test success rate, the best train–test retention, and the strongest mobile manipulation performance; π₀ leads on dexterous fixed-base and high-precision tasks; XVLA and InternVLA-A1 exhibit complementary strengths across atomic skills and operating regimes. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.