Thoughtful Lab
We track 7 benchmarks that Thoughtful Lab made or helped make.
4 in use 5 with public code 5 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| PostTrainBench CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Machine learning engineering | Mar 2026 | 34.1% | In use |
| PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Machine learning engineering | Sep 2026 | 44.6% | In use |
| DiligenceBench 150 public-equity research tasks on large US companies across sectors. Agents answer open-ended research questions in their own harness, and a model judge scores each answer against a dense task rubric for factual accuracy, analysis and risk awareness. | Finance | Jul 2026 | | In use |
| PostTrainBench 1.1 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Machine learning engineering | Jul 2026 | 41.8% | In use |
| FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. | Machine learning engineering | Apr 2026 | 34% | Not in cards |
| BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. | Real-world events | Oct 2026 | | Not in cards |
| PitchBench Models listen to recordings and identify notes, intervals, chords, or melodies. The authors vary the instrument, timing, and background noise. | Audio understanding and voice | May 2026 | | Not in cards |