Benchmark Almanac

Soochow University

We track 8 benchmarks that Soochow University made or helped make.

0 in use 7 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
ToolMaze Agents complete tool-use tasks while the runtime introduces explicit or hidden tool failures. A judge checks completion across four task structures and five perturbation settings. Metrics report task success, recovery rate, and recovery cost. Function calling Jun 2026 53% Not in cards
CFLUE Models answer Chinese financial questions in two parts. The knowledge part has more than 38,000 multiple-choice questions from financial qualification exams, with explanations to test reasoning. The application part has more than 16,000 test items for classification, translation, relation extraction, reading comprehension and text generation. Finance May 2024 53.8% Not in cards
CFLUE Knowledge Models answer Chinese financial qualification questions. The score is answer accuracy across the knowledge test. Finance May 2024 53.8% Not in cards
CFLUE Application Models complete Chinese financial language tasks. The score is the published average of the application task metrics. Finance May 2024 44.2 Not in cards
PRMBench Models identify errors and redundant steps in mathematical solutions. PRMScore combines positive and negative F1 scores across simplicity, soundness, and sensitivity tests. Word problems Jan 2025 68.8 Not in cards
UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. Real robots Jun 2026 55.8% Not in cards
UAQFact Models answer factual questions or decline when the facts do not support an answer. The benchmark also checks whether models know the facts needed to decide. Short factual questions May 2025 49.6% Not in cards
ChCatExt Models reconstruct headings and paragraphs from Chinese bid notices, financial announcements, and credit rating reports. The combined test contains 65 documents. Reasoning over long documents Apr 2023 Not in cards