Benchmark Almanac

Georgia Institute of Technology

We track 8 benchmarks that Georgia Institute of Technology made or helped make.

0 in use 8 with public code 4 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
SimulatorArena Models act as students or writing clients. Researchers compare their messages and assistant ratings with those from people who used the same assistants. Multi-turn conversation 1.01 Not in cards
K9-Bench Models answer 4,744 questions about dog behavior in 907 videos. The evaluator compares each free-form answer with five options that the model never sees. Video understanding Jun 2026 40.1% Not in cards
MESA Robot policies follow instructions as object locations, categories, and task sequences change. The authors compare seven methods on single-arm and two-arm tasks. Simulated robots Nov 2026 50.4% Not in cards
ToolQA Models answer questions that need outside information, using 13 tools such as database queries, text retrieval and a calculator over fixed data sources. The questions are built so that a model's internal knowledge cannot answer them. An easy and a hard set separate single lookups from multi-step tool use, and answers are checked by exact match. Function calling Jun 2023 24.1% Not in cards
HLS-Eval Models write or edit C++ for high-level synthesis. Tests check circuit behavior, and synthesis tools check whether the code can become hardware. Electronics 100% Not in cards
CAMeL Models fill gaps in Arabic sentences with names, foods, and other cultural references. The Cultural Bias Score measures how often models favor Western over Arab alternatives. Bias, fairness and politics Nov 2023 Not in cards
MedReadMe Models rate how easy a medical sentence is to understand. Researchers compare the ratings with judgments from readers. Medicine May 2024 Not in cards
MultiCochrane Models rewrite medical review sentences in plain language. The collection pairs technical and simplified wording in English, Spanish, French, and Farsi. Medicine May 2023 Not in cards