Benchmark Almanac

Superintelligence Labs

We track 9 benchmarks that Superintelligence Labs made or helped make.

0 in use 9 with public code 0 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
AI Agent Privacy Web agents complete tasks while exposed to private information. The benchmark measures privacy leakage and unsafe disclosure during autonomous operation. Deception and alignment Mar 2025 Not in cards
CRAG-MM Multimodal retrieval agents answer single-turn and multi-turn questions grounded in images and changing web content. The benchmark includes wearable-style egocentric images. Questions that need search Oct 2025 Not in cards
E2E-SWE Agents build complete software repositories from natural language specifications in empty workspaces. Hidden tests require every generated project to install and run. Building whole projects Sep 2026 Not in cards
EgoToM Video models infer a camera wearer’s goals, beliefs, and future actions from egocentric clips. Multiple-choice questions test theory of mind. General visual questions Mar 2025 Not in cards
HalluLens Language models answer dynamically generated prompts that test factual and contextual hallucination. The evaluator builds fresh examples to reduce test leakage. Hallucination rates Apr 2025 Not in cards
KernelBench-Verified Language models generate CUDA kernels that must match PyTorch outputs and improve runtime. The verifier checks correctness with robust performance baselines. GPU kernels Jan 2026 Not in cards
PARTNR Agents follow natural language instructions in human-robot collaboration scenes. Tasks require planning and coordination among multiple embodied agents. Embodied reasoning Nov 2024 Not in cards
PHYRE Agents solve two-dimensional physics puzzles by placing new bodies into a simulated scene. The benchmark measures success on unseen puzzles and the number of attempts. Physical prediction May 2020 Not in cards
WorldPrediction Agents predict and plan in procedural environments with long horizons. Tasks test whether models maintain a useful world model while acting. Interactive world models Jun 2025 Not in cards