Benchmark Almanac

xbench

We track 10 benchmarks that xbench made or helped make.

2 in use 6 with public code 8 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
BabyVision Model solves 388 perception-heavy visual puzzles (discrimination, tracking, spatial, patterns) that young children solve. Accuracy versus 94% human baseline. Spatial reasoning and counting Jan 2026 91.3% In use
xbench-DeepSearch Agents search the web to answer 100 Chinese questions that need multi-step retrieval. Scored by accuracy against reference answers. Deep research May 2025 83.7% Fading
$OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. Tasks from real jobs Mar 2026 42.1% In use
xbench-DeepSearch-2510 Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key. Deep research Oct 2025 56.3% Fading
xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. Science questions May 2025 76.4% Not in cards
Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1. Fixing real repositories Mar 2026 76% Not in cards
BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. Genomics and life sciences May 2026 63.9 Not in cards
xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. Running a business Jun 2025 78.5 Not in cards
xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. Running a business Jun 2025 50.8 Not in cards
AgentIF-OneDay Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements. General assistants Dec 2025 62.6% Not in cards