IBM
We track 25 benchmarks that IBM made or helped make. Its models are on its lab page.
0 in use 16 with public code 12 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| AttaQ Models answer 1,402 adversarial questions in seven harm classes, such as deception, violence and PII. Responses are scored for harmlessness. | Refusing harmful requests | Sep 2023 | 88.5% | Fading |
| ACPBench Models answer boolean and multiple-choice questions on 7 planning reasoning tasks over 13 formally described planning domains. Answers are provably correct. | Language reasoning | Oct 2024 | | System cards only |
| FailureSensorIQ Models answer multiple-choice questions on how industrial sensors relate to failure modes of assets. The Kaggle board shows it under IBM's AssetOpsBench citation. | Other professions | Jun 2025 | 69.5% | Not in cards |
| Visual Aesthetic Benchmark Models see a set of images with the same subject and intent and must pick the best one, and in a harder setting both the best and the worst. The 400 sets hold 1,195 commissioned works across fine art, photography and illustration. Labels come from expert judgments, and only sets where experts largely agreed are kept. | Taste and style | May 2026 | 40.3% | Not in cards |
| ToRR Models answer questions over tables given in several formats, and the board checks how much accuracy changes when the table format or prompt changes. | Language reasoning | Feb 2025 | 49.9 | Not in cards |
| BlueBench Agents solve enterprise tasks across knowledge, coding, security and business workflows. The benchmark aggregates capability scores from multiple public task suites. | General assistants | Sep 2025 | 64.6 | Not in cards |
| ITBench Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. | Terminal tasks | Feb 2025 | | Not in cards |
| AutoLab Agents start from a correct but weak baseline and improve it within a fixed wall-clock budget by measuring, editing and measuring again. The 36 tasks span system optimization, puzzles, model development and CUDA kernels, such as scaling-law fitting, radix sort and FlashAttention. Each task scores the measured improvement, averaged over three runs. | Machine learning engineering | Jun 2026 | 0.68 | Not in cards |
| VAREX-Bench Models extract structured fields from 1,777 government forms using a supplied JSON schema. Normalized exact match checks field values, while string similarity gives partial credit. Array entries use order-independent matching. | Documents, charts and text in images | Mar 2026 | | Not in cards |
| AssetOpsBench Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders. | Other professions | Jun 2025 | 65% | Not in cards |
| VAKRA Agents answer multi-step questions by calling more than 8,000 locally hosted APIs over real databases in 62 domains, along with matching document collections. Tasks need chains of 3 to 7 steps across APIs and documents. A deterministic evaluator replays tools and checks the full trajectory. | Function calling | Mar 2026 | | Not in cards |
| Climate-Change-NER Models identify and label climate terms in scientific abstracts. The authors provide 13 categories and reserve 75 abstracts for testing. | Lab and data work | Apr 2024 | 0.44 | Not in cards |
| τc-Bench Agents use tools while conversing with simulated customers in airline, retail, and telecom tasks. The evaluator compares the final environment state with the expected state. Generated scenarios add unusual tool combinations, ambiguous requests, and distracting database records. | Customer service | May 2026 | | Not in cards |
| PHMForge Agents analyze battery and aircraft engine data to diagnose faults, estimate remaining life, and make maintenance decisions. Harbor packages 99 industrial scenarios as 123 tasks. | Other professions | Apr 2026 | 80.8% | Not in cards |
| Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. | Medicine | Jul 2021 | Not in cards | |
| FLOW-BENCH Models translate natural-language business requirements into executable enterprise workflows. The evaluator checks constrained workflow syntax and behavior. | Apps and MCP servers | Mar 2025 | Not in cards | |
| ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. | Terminal tasks | May 2026 | Not in cards | |
| MermaidSeqBench Models translate natural-language descriptions into Mermaid sequence diagrams. Judges score syntax, logic, completeness, activation handling and error tracking. | Web and app front ends | Nov 2025 | Not in cards | |
| NASA SDE IR Benchmark v5 Models retrieve NASA science documents from questions and search phrases. The authors generated 176,901 queries from Science Discovery Engine content. | Questions that need search | Nov 2025 | Not in cards | |
| NASA-IR Models retrieve the scientific passage that answers a question. The test measures how often the correct passage appears among the first ten results. | Questions that need search | Apr 2024 | Not in cards | |
| NASA-QA Models read passages from Earth science papers and extract the words that answer a question. Some questions have no answer in the passage. | Reading comprehension | Apr 2024 | Not in cards | |
| SandhiKosh Models join Sanskrit words under sound change rules or recover the original words from a joined form. The authors score accuracy separately for joining and splitting. | Reading in many languages | May 2018 | Not in cards | |
| ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. | Fixing real repositories | May 2026 | Not in cards | |
| SQL-API-Bench Agents answer questions that require both database queries and API calls. The evaluator checks answers across two mixed database and API benchmark settings. | Function calling | Oct 2025 | Not in cards | |
| WikiVQABench Vision-language models answer multiple-choice questions that require image evidence and linked Wikipedia and Wikidata knowledge. | General visual questions | May 2026 | Not in cards |