academic collaborators
We track 2 benchmarks that academic collaborators made or helped make.
0 in use 2 with public code 2 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. | Running a business | Oct 2024 | 72.6 | Not in cards |
| Long Code Arena Models use project-wide context to generate and complete code, repair builds, find bugs, write commit messages, and summarize modules. Task scores use build success, exact match, API recall, text similarity, localization F1, or a model-based documentation comparison. | Fixing real repositories | Jun 2024 | 0.96 | Not in cards |