Benchmark Almanac

Sinatras

We track 3 benchmarks that Sinatras made or helped make.

1 in use 2 with public code 1 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
PMPP-Hard Agents write GPU kernels for 69 tasks that must be correct, and on 48 of them also run within 1.25 times the reference time. The score is the area under the solved-tasks curve over a 10-hour budget, out of 690 task-hours. GPU kernels Jul 2026 0.71 In use
CARLA Driving Tasks (sinatras) Agents navigate and explore simulated roads across 19 tasks using text or camera images. The simulator scores outcomes such as reaching destinations and avoiding collisions. Simulated robots Feb 2026 Not in cards
PMPP CUDA Evaluation Models implement 53 CUDA kernels and answer 146 questions about parallel programming. The tasks follow the textbook Programming Massively Parallel Processors. GPU kernels Oct 2025 Not in cards