Benchmark Almanac

Columbia University

We track 8 benchmarks that Columbia University made or helped make.

0 in use 7 with public code 6 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. Finding and exploiting real bugs Mar 2024 63% System cards only
CWEval Models write code for security-critical tasks in several languages. Outcome-driven test oracles check both functionality (func@k) and security (func-sec@k). Secure code and review Jan 2025 50.2% System cards only
MBABench Agents build complete financial models in Excel from a case prompt and documents. An LLM judge grades accuracy, formulas and formatting on a 0 to 100 scale. Finance May 2026 31.2 Not in cards
MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. Finance Oct 2026 15.8% Not in cards
Seeing What Execution Needs Agents choose camera views that help a robot complete a tabletop task. The evaluator runs the action and checks whether the requested state holds. Simulated robots Nov 2026 42.9% Not in cards
CREATE Models find a word that connects three photographs. A judge grades each answer against the intended connection. General visual questions Oct 2026 42.2% Not in cards
DyCodeEval Models solve dynamically rewritten HumanEval and MBPP problems with equivalent program behavior. The evaluator runs tests and computes DyPass@K across generated variants. Function-level problems Mar 2025 39% Not in cards
TailOR Models generate images and videos of regular, unconventional, and impossible tool interactions. Annotators score instruction adherence, interaction accuracy, physical realism, and perceptual quality. Physical prediction Jun 2026 Not in cards