BigCodeBench1,140 practical Python tasks requiring calls to 139 libraries. Scored pass@1 on unit tests.
Function-level problems
Jun 2024
60.6%
Fading
BigCodeBench-HardModels solve 148 harder BigCodeBench Python tasks that call many libraries, chosen to match real programming work. Scored pass@1 on unit tests.