Benchmark Almanac

BigCode project

We track 2 benchmarks and 1 arena that BigCode project made or helped make.

0 in use 3 with public code 3 with a leaderboard here

Arena leaderboards

Top five in each of its 1 arena with three or more rated models, the most rated first.

Benchmarks

Name Kind Released Best score Status
BigCodeBench 1,140 practical Python tasks requiring calls to 139 libraries. Scored pass@1 on unit tests. Function-level problems Jun 2024 60.6% Fading
BigCodeBench-Hard Models solve 148 harder BigCodeBench Python tasks that call many libraries, chosen to match real programming work. Scored pass@1 on unit tests. Function-level problems Jul 2024 35.8% Dropped