Benchmark Almanac

Bespoke Labs

We track 3 benchmarks that Bespoke Labs made or helped make.

0 in use 3 with public code 3 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
AutoResearchExam Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed. Machine learning engineering Sep 2026 0.61 Not in cards
OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. Terminal tasks Feb 2026 75.1% Not in cards
Multi Agent Coordination Bench Teams of eight agents work for four hours on twelve tasks built so each agent depends on its teammates. Each model also runs alone and as a team that can't talk, which isolates the gain from coordination. Multi-agent games Nov 2026 0.53 Not in cards