Benchmark Almanac

UC San Diego

We track 9 benchmarks that UC San Diego made or helped make.

1 in use 8 with public code 6 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. Simulated robots May 2024 80% In use
CoreCognition Models answer true or false and multiple choice questions on 12 core knowledge concepts from developmental cognitive science, such as object permanence and perspective taking. General visual questions Oct 2024 74.9% System cards only
RepoBench Models complete code using context retrieved from other files in the same Python or Java repository. Three tasks cover retrieval, completion and the full pipeline. Fixing real repositories Jun 2023 38% Dropped
Spider 2.0-AIFunc Agents generate SQL that uses AI functions to solve 465 tasks across 125 databases. The evaluator compares execution results with reference results. Data science and SQL Jul 2026 70.3% Not in cards
MonitorBench Models work through 1,540 instances across 19 tasks in 7 categories, each built around a decision-critical factor such as a cue, a hidden goal or a required step. A monitor LLM reads the chain of thought, and the monitorability score measures how often it surfaces that factor. Deception and alignment Mar 2026 91.9% Not in cards
CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. Web browsing Apr 2026 45.1% Not in cards
MemoryAgentBench Memory agents receive long inputs in chunks over many turns and are tested on accurate retrieval, test-time learning, long-range understanding and selective forgetting. It adds two new sets, EventQA and FactConsolidation. Learning from the context Jul 2025 60.6% Not in cards
CLIPSep Sound Separation Models isolate a requested sound from a mixture using a text or image query. The two tests mix music or everyday sounds with interfering audio. Audio understanding and voice Nov 2022 Not in cards
ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. Simulated robots Oct 2024 Not in cards