Benchmark Almanac

Shanghai Jiao Tong University

We track 26 benchmarks and 1 arena that Shanghai Jiao Tong University made or helped make.

0 in use 22 with public code 15 with a leaderboard here

Arena leaderboards

Top five in each of its 1 arena with three or more rated models, the most rated first.

Benchmarks

Name Kind Released Best score Status
PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. Medicine May 2023 66.2% Fading
SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. Web and app front ends Aug 2025 0.92 Fading
R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. General visual questions Oct 2024 75.4% System cards only
SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. Fixing real repositories May 2025 66.6% Dropped
MemGUI-Bench Agents remember details while working across Android apps. Tasks include recording audio and creating calendar events. A model checks the actions and results across repeated attempts. Phones Feb 2026 73.4% Not in cards
OBI-Bench Models locate inscriptions, rejoin bone fragments, classify characters, retrieve matches, and explain glyphs. The authors publish separate scores for each task. Documents, charts and text in images Oct 2024 0.61 Not in cards
CaLM Models answer causal discovery, intervention, and counterfactual reasoning tasks in English and Chinese. Evaluation compares answers with reference labels to measure accuracy and robustness across causal targets and prompting methods. Language reasoning May 2024 45.3% Not in cards
IG-Bench Models trace scientific idea lineages across 1,029 questions and propose research ideas grounded in those lineages. IG-Exam requires every answer field to match. Research science Jul 2026 27.3% Not in cards
OmniCAD Models choose parts from a component library and predict their assembly positions, orientations, and connections. The evaluator measures pose accuracy and how well connections match. CAD models Aug 2026 100% Not in cards
MM-AlignBench Models answer open-ended questions about 252 human-annotated images. GPT-4o compares their answers with Claude-Sonnet-3 reference answers to assess alignment with human preferences. General visual questions Feb 2025 82.5% Not in cards
ConceptEdit-Bench Models edit images according to 1,000 instructions across six categories. A vision-language judge scores instruction following, visual quality, and consistency in unedited regions from 0 to 10. The per-case score averages those three dimensions. Image generation and editing Aug 2026 66.2 Not in cards
HABIT-Bench Agent memory systems must infer user habits from weak evidence spread across many past conversations and apply them correctly later. Learning from the context Oct 2026 42% Not in cards
SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. Fixing real repositories Jun 2026 21.5% Not in cards
SWE-Game Agents build, complete, repair, or port games across 247 tasks derived from 41 reference games. Runtime probes and input replays check gameplay mechanics and preserved behavior. Vision-language rubrics separately grade presentation in construction tasks. Building whole projects Sep 2026 64.0 Not in cards
PICABench Models edit images while preserving physical effects in optics, mechanics, and state transitions. A vision-language judge answers region-specific questions to measure physical correctness. Image generation and editing Oct 2025 61.9% Not in cards
VidScribe Models write, copy, preserve, or edit visible text in videos. The evaluator tracks spelling, glyph shapes, placement, and stability across frames. Video generation Sep 2026 79.0 Not in cards
OS-Kairos Agents predict Android actions for tasks such as finding a product in Amazon. The task succeeds when every action matches the recorded trajectory. Phones Feb 2025 29.8% Not in cards
APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. Long videos Sep 2026 60.2% Not in cards
CMD-AD-Eval Models describe what happens on screen during gaps in movie dialogue. The authors compare the descriptions with timed human narration using text overlap and model judgments. Video captioning Apr 2024 Not in cards
doteBench Models edit English and Chinese recordings using a transcript that marks where each change belongs. The evaluator checks edit success and preservation outside those regions. Audio understanding and voice Aug 2026 Not in cards
M³AV Speech Recognition Models transcribe academic lectures. The authors measure word errors across the transcript and separately for rare words. Slide text can provide context. Speech recognition and translation Mar 2024 Not in cards
MAD-Eval Models write descriptions for scenes from ten movies. One set of references names characters. Another replaces their names with the word “SOMEONE.” Video captioning Mar 2023 Not in cards
META-GUI Models predict the next Android action and reply from screenshots and dialogue history. A turn succeeds when every predicted action matches the recorded interaction. Phones May 2022 Not in cards
TacMind-Bench Robots use touch to manipulate objects in simulation. The benchmark tests precision, recognition, and movement as visual and physical conditions change. Simulated robots Oct 2026 Not in cards
TV-AD Models describe what viewers can see during clips from television episodes. The test pairs each clip with a timed reference description. Video captioning Jul 2024 Not in cards
WebSRC Models answer questions about a web page from its screenshot, HTML or both. The set has 400K question and answer pairs from 6.4K web pages. Scored by exact match and F1. Documents, charts and text in images Jan 2021 Not in cards