Benchmark Almanac

referee-lab

We track 3 benchmarks that referee-lab made or helped make.

0 in use 0 with public code 0 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Effective FPS Models decide whether video frames are consecutive or duplicated to simulate a higher capture rate. The evaluator checks each prediction against the construction of the clip. Video understanding Aug 2026 Not in cards
Escalation Router Models predict whether Qwen2.5-VL 7B correctly identifies visible hands or grippers in a clip. The reference label comes from two observed runs per clip. Video understanding Aug 2026 Not in cards
Hands Visible Models inspect 12 frames per clip and determine whether a human hand or robot gripper appears. The 45-clip evaluation uses provisional machine-generated labels. Video understanding Aug 2026 Not in cards