Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance.
Models turn the same drawing into a three-dimensional figure. Voters compare two anonymous prints or renders and choose which better matches the drawing.
Vending-Bench 2Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs.
Running a business
Nov 2025
$16,843
In use
Blueprint-Bench 2Agents turn about 20 interior photos of each of 50 apartments into 2D floor plans. Scores compare room connectivity with ground truth, normalized against random.
Spatial reasoning and counting
May 2026
58.6%
In use
Vending-BenchAgents run a simulated vending machine business over long horizons, ordering stock, setting prices, and paying daily fees. Scoring is final net worth in dollars.
Running a business
Feb 2025
$4,694
Dropped
Drone-BenchAgent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline.
Real robots
Jul 2026
97.3%
Not in cards
Blueprint-BenchModels turn about 20 interior photos of each of 50 apartments into a 2D floor plan. Scored by similarity of room connectivity and size ranking to the true plan.
Spatial reasoning and counting
Sep 2025
0.43
Not in cards
Butter-BenchA language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task.