Benchmark Almanac

NVIDIA

We track 47 benchmarks that NVIDIA made or helped make. Its models are on its lab page.

6 in use 34 with public code 29 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
RULER Synthetic retrieval, multi-hop tracing, aggregation and QA tasks at configurable lengths. Measures effective context size. Finding facts in long inputs Apr 2024 99% In use
LiveBench Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. Other Jun 2024 87.4% Dropped
RoboSpatial-Home Models answer 350 spatial questions on home scenes for robots, in three types: object context, compatibility of placements and spatial configuration. Spatial reasoning and counting Nov 2024 73.9% Dropped
RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. Simulated robots Mar 2025 50% Fading
IOI 2025 Model solves the 2025 International Olympiad in Informatics tasks. Scored with official subtask points (max 600). Competitive programming Aug 2025 570.0 Fading
Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. Embodied reasoning Jun 2024 64% System cards only
RoboCasa Simulated mobile manipulators and humanoids do 100 kitchen tasks, 25 atomic and 75 composite, across 120 scenes. Scored by success rate. Simulated robots Jun 2024 70.8% Fading
ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. Tasks from real jobs Oct 2025 66.2% In use
RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. Simulated robots Apr 2026 46.2% In use
Aegis 2.0 Models classify human and LLM conversation turns as safe or unsafe under a taxonomy of 12 hazard categories. Scored by F1 against human and jury labels. Refusing harmful requests Jan 2025 85.8% System cards only
NV Chat Benchmark Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy. Overall chat quality Oct 2026 52.7 In use
When2Call Models decide when to call a tool, when to ask a follow-up question and when to say the tools cannot answer. Scored on multiple-choice and LLM-judged test sets. Function calling Apr 2025 64.6% In use
PAI-Bench Physical-AI video generation from 1,044 prompts in six domains, VLM-QA plus quality metrics. Companion control and understanding tracks. Video world models Dec 2025 83.7% System cards only
SPEED-Bench Measures speculative decoding across varied prompts and domains. Reports the average acceptance length of draft tokens, so higher means faster generation at the same output. Other Feb 2026 2.81 System cards only
Brevity Benchmark NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method. Overall chat quality Oct 2026 82.8% In use
MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. Video games Jun 2022 49% Fading
MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. Long videos Mar 2026 46.8% System cards only
Orak LLM agents play 12 popular video games across most genres through an MCP interface. Each game scores in-game progress or outcomes, and the board ranks models by their average rank across the games. Video games Jun 2025 3.90 Fading
ChatRAG Bench Models answer multi-turn conversational questions over documents or retrieved context. The suite is built from 10 existing datasets such as Doc2Dial, QuAC, QReCC and CoQA. Finding facts in long inputs Jan 2024 53.9 System cards only
CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. Electronics Jun 2025 64.1% Not in cards
MemLens Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths. Finding facts in long inputs May 2026 53.6% Not in cards
VerilogEval v2 Models write Verilog hardware descriptions from specifications. The hub port uses the specification-to-RTL tasks, where simulation checks generated designs against reference behavior. Electronics Aug 2024 88% Not in cards
VANTAGE-BENCH Models answer questions, describe events, and locate objects or actions in infrastructure-camera footage. Task-specific metrics measure answer accuracy, caption quality, localization, and tracking. Video understanding Sep 2026 75.3 Not in cards
Realtime-Bench Agents act in 50 live game and web tasks where a late action fails: 30 web tasks (travel, shopping, services and security) and 20 games. The environment keeps moving while the model thinks. Three clocks separate speed from skill: Wall Clock runs on real time, Paused Clock stops while the model thinks, and Token Clock advances with each generated token. Scores are task success rates over three runs. Web browsing Oct 2026 42.7% Not in cards
AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. Machine learning engineering Jun 2026 0.68 Not in cards
CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. Simulated robots Mar 2026 32.3% Not in cards
SWE-Serve Coding agents solve 53 production inference-engineering tasks derived from merged SGLang pull requests. Hidden tests measure task success, cost, agent steps and wall time. Fixing real repositories Oct 2026 75.5% Not in cards
AutoMedBench Agents set up and run medical imaging research workflows from a task brief through final submission. Scores combine task performance with rubric judgments of planning, setup, and validation, plus automatic checks of inference and submission. Machine learning engineering Jun 2026 66.5 Not in cards
VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. Speech recognition and translation May 2026 3.40 Not in cards
RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. Simulated robots Mar 2026 23.9% Not in cards
FVEval Models write SystemVerilog assertions from properties or circuit designs. A formal verifier checks syntax and tests whether assertions express the reference behavior. Electronics Jun 2024 24.8% Not in cards
RULERv2 Models solve synthetic multi-key and multi-value needle tasks and QA tasks at basic, easy, medium and hard levels. Contexts run from 8K to 1M tokens, and accuracy is averaged across lengths. Finding facts in long inputs Oct 2025 89.2% Not in cards
VerilogEval Models complete Verilog modules from human or machine descriptions. A simulator checks their outputs against the reference behavior. Electronics Sep 2023 77.1% Not in cards
ViDoRe V3 Retrieval and RAG pipelines answer queries over visually rich documents, such as reports with tables, charts and images. The benchmark has 10 datasets from professional domains, about 26,000 pages and 3,099 human-verified queries, each in 6 languages. It scores retrieval relevance, bounding-box grounding and the final answers. Documents, charts and text in images Jan 2026 0.58 Not in cards
Preference-ASR Models transcribe speech in a requested style. The evaluation checks preferences such as casing, number formatting, and removing disfluencies. Speech recognition and translation Jun 2026 13.8% Not in cards
AssertionForge design evaluation Models generate SystemVerilog assertions from design specifications and source code. Cadence Jasper counts proven assertions and measures their coverage. Electronics Mar 2025 Not in cards
ComputeEval Models write CUDA code for handcrafted problems on kernels, runtime APIs and GPU libraries. Held-out test harnesses check correctness, and optional benchmarks time the code. Speed and optimization Apr 2025 Not in cards
DreamGen Bench Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline. Video world models May 2025 Not in cards
GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. Simulated robots Jul 2026 Not in cards
LeanDojo Benchmark Models prove theorems from Lean's mathlib library inside the Lean proof assistant, which checks every proof. The benchmark has 98,734 theorems and proofs with premise annotations. A novel-premises split tests whether a prover can use lemmas it never saw in training. Formal proofs Jun 2023 Not in cards
Numb3rs Models transcribe spoken numbers, including addresses, amounts, and times. NVIDIA compares the output with spoken-form references. Speech recognition and translation Not in cards
PhyWorldBench Video generators create scenes that follow physical laws across object motion, energy conservation, rigid interactions and human or animal motion. Video generation Feb 2026 Not in cards
RefineBench Models refine answers to 1,002 problems across 11 domains using checklists that specify required content and errors. Following precise instructions Nov 2025 Not in cards
SCORE (Consistency and Robustness) Models answer the same questions under different option orders, prompt formats and decoding settings. The evaluator measures consistency as well as accuracy. Following precise instructions Nov 2024 Not in cards
SOL-ExecBench Systems write CUDA kernels for 235 workloads taken from real model architectures, run on B200 GPUs. The SOL score measures how much of the gap between a baseline and the analytic hardware speed limit a kernel closes. GPU kernels Mar 2026 Not in cards
VABAR Agents program robots to sort tools, mesh gears, and handle cables in simulation. The harness checks success as object shapes and positions vary. Simulated robots Nov 2026 Not in cards
VIMA-Bench Robot policies carry out tabletop manipulation tasks given as multimodal prompts that mix text with images of objects or goal scenes. Thousands of tasks are generated procedurally, with over 600,000 expert trajectories for imitation learning. A four-level protocol tests generalization from new object placements up to unseen tasks, and scores are success rates. Simulated robots Oct 2022 Not in cards