Benchmark Almanac

academic collaborators

We track 2 benchmarks that academic collaborators made or helped make.

0 in use 2 with public code 2 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. Running a business Oct 2024 72.6 Not in cards
Long Code Arena Models use project-wide context to generate and complete code, repair builds, find bugs, write commit messages, and summarize modules. Task scores use build success, exact match, API recall, text similarity, localization F1, or a model-based documentation comparison. Fixing real repositories Jun 2024 0.96 Not in cards