Benchmark Almanac

Nanyang Technological University

We track 13 benchmarks that Nanyang Technological University made or helped make.

1 in use 13 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
ChartQA Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Documents, charts and text in images Mar 2022 90.9% In use
R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. General visual questions Oct 2024 75.4% System cards only
General-Bench Models perform multimodal understanding and generation tasks. General-Level averages task scores and measures performance above specialist baselines at progressively broader levels. General visual questions May 2025 8.41 Not in cards
Q-Bench Multimodal models are tested on low-level vision: perceiving image distortions, describing them and predicting quality scores. General visual questions Sep 2023 73.9% Not in cards
HumanCLAW-Bench Models control a simulated human body from egocentric images to find objects, navigate, and sit. The benchmark has 1,218 episodes across 41 houses. Success requires geometric completion and the agent's acknowledgment. Simulated robots Jul 2026 11.1% Not in cards
AudioTrust Audio language models answer prompts that probe hallucination, robustness, speaker authentication, privacy, fairness, and safety. Automated evaluators, including language-model judges, compare responses with task-specific references and criteria. Audio understanding and voice May 2025 0.37 Not in cards
xCodeEval Models understand, generate, translate, and retrieve contest code across seven tasks and up to 17 programming languages. Generated programs run against test cases through ExecEval. Competitive programming Mar 2023 3.81% Not in cards
DeepResearchEval Deep research systems answer complex research tasks that require evidence from many web sources. A persona-driven pipeline writes the tasks and keeps only those that need outside search. A model judge scores each report on general and task-specific dimensions, and an agent fact-checks the report's statements with web search even when citations are missing. Deep research Jan 2026 Not in cards
DeepResearchEval v1 Research agents prepare reports. The score is average report quality on the original evaluation tasks. Deep research Jan 2026 Not in cards
MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. Video understanding Aug 2023 Not in cards
PST50 Models transfer a reference look to a photograph. The paired test compares 50 outputs with reference grades that professionals refined in DaVinci Resolve. Image generation and editing Jun 2025 Not in cards
Q-Bench Image-Pair Perception Models answer multiple-choice questions comparing low-level attributes across image pairs. The score is overall answer accuracy in the A1-Pair table. General visual questions Sep 2023 Not in cards
Q-Bench Single-Image Perception Models answer multiple-choice questions about low-level attributes of individual images. The score is overall answer accuracy in the A1-Single table. General visual questions Sep 2023 Not in cards