Benchmark Almanac

IBM

We track 25 benchmarks that IBM made or helped make. Its models are on its lab page.

0 in use 16 with public code 12 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
AttaQ Models answer 1,402 adversarial questions in seven harm classes, such as deception, violence and PII. Responses are scored for harmlessness. Refusing harmful requests Sep 2023 88.5% Fading
ACPBench Models answer boolean and multiple-choice questions on 7 planning reasoning tasks over 13 formally described planning domains. Answers are provably correct. Language reasoning Oct 2024 62.9% System cards only
FailureSensorIQ Models answer multiple-choice questions on how industrial sensors relate to failure modes of assets. The Kaggle board shows it under IBM's AssetOpsBench citation. Other professions Jun 2025 69.5% Not in cards
Visual Aesthetic Benchmark Models see a set of images with the same subject and intent and must pick the best one, and in a harder setting both the best and the worst. The 400 sets hold 1,195 commissioned works across fine art, photography and illustration. Labels come from expert judgments, and only sets where experts largely agreed are kept. Taste and style May 2026 40.3% Not in cards
ToRR Models answer questions over tables given in several formats, and the board checks how much accuracy changes when the table format or prompt changes. Language reasoning Feb 2025 49.9 Not in cards
BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. General assistants Sep 2025 64.6 Not in cards
ITBench Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. Terminal tasks Feb 2025 58.3% Not in cards
AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. Machine learning engineering Jun 2026 0.68 Not in cards
VAREX-Bench Models extract structured fields from 1,777 government forms using a supplied JSON schema. Normalized exact match checks field values, while string similarity gives partial credit. Array entries use order-independent matching. Documents, charts and text in images Mar 2026 97.3% Not in cards
AssetOpsBench Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders. Other professions Jun 2025 65% Not in cards
VAKRA Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory. Function calling Mar 2026 38.1% Not in cards
Climate-Change-NER Models identify and label climate terms in scientific abstracts. The authors provide 13 categories and reserve 75 abstracts for testing. Lab and data work Apr 2024 0.44 Not in cards
τc-Bench Agents use tools while conversing with simulated customers in airline, retail, and telecom tasks. The evaluator compares the final environment state with the expected state. Generated scenarios add unusual tool combinations, ambiguous requests, and distracting database records. Customer service May 2026 87.7% Not in cards
PHMForge Agents analyze battery and aircraft engine data to diagnose faults, estimate remaining life, and make maintenance decisions. Harbor packages 99 industrial scenarios as 123 tasks. Other professions Apr 2026 80.8% Not in cards
Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. Medicine Jul 2021 Not in cards
FLOW-BENCH Models translate natural-language business requirements into executable enterprise workflows. The evaluator checks constrained workflow syntax and behavior. Apps and MCP servers Mar 2025 Not in cards
ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. Terminal tasks May 2026 Not in cards
MermaidSeqBench Models translate natural-language descriptions into Mermaid sequence diagrams. Judges score syntax, logic, completeness, activation handling and error tracking. Web and app front ends Nov 2025 Not in cards
NASA SDE IR Benchmark v5 Models retrieve NASA science documents from questions and search phrases. The authors generated 176,901 queries from Science Discovery Engine content. Questions that need search Nov 2025 Not in cards
NASA-IR Models retrieve the scientific passage that answers a question. The test measures how often the correct passage appears among the first ten results. Questions that need search Apr 2024 Not in cards
NASA-QA Models read passages from Earth science papers and extract the words that answer a question. Some questions have no answer in the passage. Reading comprehension Apr 2024 Not in cards
SandhiKosh Models join Sanskrit words under sound change rules or recover the original words from a joined form. The authors score accuracy separately for joining and splitting. Reading in many languages May 2018 Not in cards
ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. Fixing real repositories May 2026 Not in cards
SQL-API-Bench Agents answer questions that require both database queries and API calls. The evaluator checks answers across two mixed database and API benchmark settings. Function calling Oct 2025 Not in cards
WikiVQABench Vision-language models answer multiple-choice questions that require image evidence and linked Wikipedia and Wikidata knowledge. General visual questions May 2026 Not in cards