Benchmark Almanac

Vellum

We track 2 benchmarks that Vellum made or helped make.

0 in use 0 with public code 1 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Hardest SAT Problems (Vellum) Models solve 50 hard SAT math problems. A judge compares each final answer with a reference, and a person reviews the result. Word problems 94% Not in cards
Adaptive Reasoning (Vellum) Models solve 28 familiar logic puzzles after Vellum changes a premise. A judge checks the final answer, and a person reviews the result. Language reasoning Not in cards