Benchmark Almanac

CALVIN

FLOWER 4.53 tops its leaderboard
Not in model cards

Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length.

ABC-to-D scores are out of 5 and near saturation. Few current models report the standard split. Others use easier splits.

Released
December 2021
Kind
Simulated robots
Level
Everyday
Scored
Checked automatically
Answers
Agent task
Tasks
All public

In the authors’ words

General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this paper, we present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments and objects. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

Abstract of CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks (arXiv:2112.03227, CC0)
17
Entries on its own leaderboard, 0 models we track

FLOWER tops the leaderboard with 4.53

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

17 entries. Scroll sideways for the rest.

  1. 4.53 FLOWER
  2. 4.41 UniVLA
  3. 4.28 SeeR-Large
  4. 4.04 GR-MG
  5. 4.01 MoDE
  6. 3.69 GHIL-Glue
  7. 3.64 RoboUniView
  8. 3.61 Diffusion Transformer Policy
  9. 3.53 CLOVER
  10. 3.27 3D Diffuser Actor
  11. 3.06 GR-1
  12. 2.82 DeeR
  13. 2.69 SuSIE
  14. 2.47 RoboFlamingo
  15. 1.71 SPIL
  16. 0.67 HULC
  17. 0.31 Baseline
  1. 1 FLOWER Static RGB + Gripper RGB 4.53
  2. 2 UniVLA Static RGB + Gripper RGB 4.41
  3. 3 SeeR-Large Static RGB + Gripper RGB + Propiro 4.28
  4. 4 GR-MG Static RGB + Gripper RGB + Proprio 4.04
  5. 5 MoDE Static RGB + Gripper RGB 4.01
  6. 6 GHIL-Glue Static RGB 3.69
  7. 7 RoboUniView Static RGB-D + Gripper RGB-D + Cam params 3.64
  8. 8 Diffusion Transformer Policy Static RGB 3.61
  9. 9 CLOVER Static RGB-D 3.53
  10. 10 3D Diffuser Actor Static RGB-D + Gripper RGB-D + Proprio + Cam params 3.27
  11. 11 GR-1 Static RGB + Gripper RGB + Proprio 3.06
  12. 12 DeeR Static RGB + Gripper RGB 2.82
  13. 13 SuSIE Static RGB 2.69
  14. 14 RoboFlamingo Static RGB + Gripper RGB 2.47
  15. 15 SPIL Static RGB + Gripper RGB 1.71
  16. 16 HULC Static RGB + Gripper RGB 0.67
  17. 17 Baseline Static RGB + Gripper RGB 0.31

Source: Official CALVIN leaderboard

#EntryScoreSettingDateSource
1 FLOWER 4.53 Static RGB + Gripper RGB Official leaderboard
2 UniVLA 4.41 Static RGB + Gripper RGB Official leaderboard
3 SeeR-Large 4.28 Static RGB + Gripper RGB + Propiro Official leaderboard
4 GR-MG 4.04 Static RGB + Gripper RGB + Proprio Official leaderboard
5 MoDE 4.01 Static RGB + Gripper RGB Official leaderboard
6 GHIL-Glue 3.69 Static RGB Official leaderboard
7 RoboUniView 3.64 Static RGB-D + Gripper RGB-D + Cam params Official leaderboard
8 Diffusion Transformer Policy 3.61 Static RGB Official leaderboard
9 CLOVER 3.53 Static RGB-D Official leaderboard
10 3D Diffuser Actor 3.27 Static RGB-D + Gripper RGB-D + Proprio + Cam params Official leaderboard
11 GR-1 3.06 Static RGB + Gripper RGB + Proprio Official leaderboard
12 DeeR 2.82 Static RGB + Gripper RGB Official leaderboard
13 SuSIE 2.69 Static RGB Official leaderboard
14 RoboFlamingo 2.47 Static RGB + Gripper RGB Official leaderboard
15 SPIL 1.71 Static RGB + Gripper RGB Official leaderboard
16 HULC 0.67 Static RGB + Gripper RGB Official leaderboard
17 Baseline 0.31 Static RGB + Gripper RGB Official leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Robotics

All 104 in Robotics
  • ERQA In use 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. Gemini Robotics ER 2 78.5%
  • RoboDojo In use Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. GPT-6 Astra 29%
  • RoboLab In use Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. FLUX 3 Action 42.9%
  • Progress classification (Google DeepMind) In use Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. Gemini Robotics ER 2 57.4%
  • Human proximity (Google DeepMind robotics) In use Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. Gemini Robotics ER 2 93%
  • Moment finding (Google DeepMind) In use Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. Gemini Robotics ER 2 91.3%