Benchmark Almanac

Thoughtful Lab

We track 7 benchmarks that Thoughtful Lab made or helped make.

4 in use 5 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
PostTrainBench CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Machine learning engineering Mar 2026 34.1% In use
PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Machine learning engineering Sep 2026 44.6% In use
DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. Finance Jul 2026 58.1% In use
PostTrainBench 1.1 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Machine learning engineering Jul 2026 41.8% In use
FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. Machine learning engineering Apr 2026 34% Not in cards
BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. Real-world events Oct 2026 0.11 Not in cards
PitchBench Models listen to recordings and identify notes, intervals, chords, or melodies. The authors vary the instrument, timing, and background noise. Audio understanding and voice May 2026 19.8% Not in cards