Artificial Analysis
We track 39 benchmarks and 7 arenas that Artificial Analysis made or helped make.
25 in use 5 with public code 16 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| AA-LCR (Artificial Analysis Long Context Reasoning) Questions requiring reasoning across multiple real documents totaling about 100K tokens. Answers judged by an LLM. | Reasoning over long documents | Aug 2025 |  80% | In use |
| GDPval-AA v2 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Outputs are compared pairwise and ranked by Elo. | Tasks from real jobs | Jun 2026 | 1,861 | In use |
| AA-Omniscience 41-topic closed-book knowledge test. Net score rewards correct answers, penalizes wrong ones and gives zero for abstentions. | Short factual questions | Nov 2025 |  42.4 | In use |
| GDPval-AA Agents complete GDPval occupational tasks. Artificial Analysis compares the outputs in pairs and ranks models by Elo. | Tasks from real jobs | Dec 2025 | 1,932 | Fading |
| Terminal-Bench Hard Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. | Terminal tasks | Sep 2025 | 36% | In use |
| GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. | Tasks from real jobs | Sep 2026 | 1,846 | In use |
| AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. | Tasks from real jobs | Sep 2026 | 1,822 | In use |
| AA-Briefcase Artificial Analysis knowledge-work evaluation of professional deliverables. Models ranked by Elo. | Tasks from real jobs | 2026 | 1,720 | In use |
| Artificial Analysis Intelligence Index v4.1 Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents. | Composite indices | Jun 2026 | 61.0 | In use |
| Big Bench Audio Models answer 1,000 spoken questions from four Big Bench Hard tasks: formal fallacies, navigate, object counting and web of lies. Built to test reasoning over audio input. | Audio understanding and voice | Dec 2024 | 98.5% | In use |
| AA-LCR 1.1 Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. | Reasoning over long documents | Sep 2026 |  88.3% | In use |
| Artificial Analysis Intelligence Index v4.0 Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests. | Composite indices | Jan 2026 | 60.2 | Fading |
| Artificial Analysis Intelligence Index v4.3 Version 4.3 of Artificial Analysis's index, eleven weighted parts led by knowledge work, office tasks and Terminal-Bench 4.0. | Composite indices | Sep 2026 | 56.0 | In use |
| Harvey LAB-AA v1.1 Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. | Law | Oct 2026 | 13.1% | In use |
| Harvey LAB-AA v1.0 Artificial Analysis runs agents on Harvey's Legal Agent Benchmark tasks in an offline sandbox, where they write legal work products such as memos and redlines from case documents. Judges grade each deliverable against a task rubric, and the headline is the share of rubric criteria passed. | Law | 94.6% | In use | |
| Artificial Analysis Coding Agent Index v1.1 Version 1.1 of Artificial Analysis's coding agent index, which replaced SWE-Bench-Pro-Hard-AA with DeepSWE. | Fixing real repositories | Jun 2026 | 80.0 | In use |
| Artificial Analysis Intelligence Index v5 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. | Composite indices | Oct 2026 | 59.0 | In use |
| Artificial Analysis Intelligence Index v2 Artificial Analysis's index of seven public reasoning, knowledge, math and coding benchmarks, each run by Artificial Analysis. | Composite indices | Feb 2025 | 67.7 | Dropped |
| Artificial Analysis Coding Agent Index v2 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores. | Fixing real repositories | Oct 2026 | 46.0 | In use |
| Speech-to-Speech Index Artificial Analysis's combined score for speech-to-speech models. | Speech recognition and translation | Oct 2026 | 86.9 | In use |
| Artificial Analysis Intelligence Index v3 Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests. | Composite indices | Sep 2025 |  61.0 | Dropped |
| AA-Omniscience v1.1 Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. | Short factual questions | Oct 2026 |  16.1 | In use |
| Artificial Analysis average word error rate Speech transcription systems are scored by average word error rate across the Artificial Analysis evaluation set. | Speech recognition and translation | Aug 2026 | 4.00% | In use |
| Artificial Analysis Coding Agent Index v1.4 Version 1.4 of Artificial Analysis's coding agent index, which moved to the full Terminal-Bench 2.1 and began scoring reward-hacked runs as zero. | Fixing real repositories | Aug 2026 | 67.0 | In use |
| Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. | Finding and exploiting real bugs | Sep 2026 |  51% | In use |
| Code2Video-AA Artificial Analysis' implementation of HeyGen's Code2Video Bench: models write a HyperFrames HTML composition for each of 114 product and brand video briefs. A model judge compares two rendered videos at a time on five axes, and the comparisons become Elo ratings. | Web and app front ends | Oct 2026 | 1,020 | In use |
| MKA Agents complete professional tasks whose work is visual, 200 tasks across 10 fields and 83 occupations, each about three to eight hours of expert work. An agentic grader checks each deliverable against an expert rubric of about 70 yes or no checks, weighted 75% on factual accuracy and 25% on visual design. | Tasks from real jobs | Nov 2026 | 53.4% | In use |
| AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. | Running a business | Aug 2026 | Not in cards | |
| AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. | Simulated robots | Oct 2026 | Not in cards | |
| Artificial Analysis Coding Agent Index v1.0 Artificial Analysis's first index of coding agents, each a model in a harness, averaged over three equally weighted coding benchmarks. | Fixing real repositories | May 2026 | Not in cards | |
| Artificial Analysis Coding Agent Index v1.5 Version 1.5 of Artificial Analysis's coding agent index, which moved to the harder Terminal-Bench 4.0 and DeepSWE 1.1. | Fixing real repositories | Sep 2026 | Not in cards | |
| Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. | Composite indices | Oct 2026 | Not in cards | |
| Artificial Analysis Intelligence Index v1 Artificial Analysis's first composite score, called the Quality Index in 2024, which normalized and combined a few public benchmarks into one number. | Composite indices | Jan 2024 | Not in cards | |
| Artificial Analysis Intelligence Index v2.2 Version 2.2 of Artificial Analysis's index, which swapped the older math tests for AIME 2025 and added instruction following and long-context reasoning. | Composite indices | Aug 2025 | Not in cards | |
| Artificial Analysis Intelligence Index v4.2 Version 4.2 of Artificial Analysis's index, which added multi-hour office work and document tasks and dropped GPQA Diamond. | Composite indices | Sep 2026 | Not in cards | |
| Artificial Analysis Progress Index Artificial Analysis's index that tracks how model capability changes over time on one continuous scale, launched alongside Intelligence Index v5. | Composite indices | Oct 2026 | Not in cards | |
| Artificial Analysis Search Index One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores. | Deep research | Aug 2026 | Not in cards | |
| ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. | Terminal tasks | May 2026 | Not in cards | |
| MLCR-AA Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy. | Medicine | Aug 2026 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| AA-3D Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings. | Web and app front ends | Oct 2026 | 1,021 | In use |
| Artificial Analysis Speech Arena Voters listen to two speech clips made from the same text and pick the more natural one. Votes become Elo ratings. | Audio understanding and voice | 1,211 | Fading | |
| Speech Agent Arena People hold live conversations with two hidden speech-to-speech agents on the same scenario, some with tool calls, then pick the one they preferred. Votes are fit to an Elo rating, and the arena also tracks task success. | Speech recognition and translation | Aug 2026 | 1,070 | In use |
| AA-Image-Editing Validated private evaluators compare two image edits made from the same instruction and vote blind. Votes become Elo ratings. | Image generation and editing | Not in cards | ||
| AA-Video-Editing Voters compare two edits of the same video made from one instruction and pick the better one. Votes become Elo ratings. | Video generation | Not in cards | ||
| Artificial Analysis Image Arena Voters compare images from two text-to-image models for the same prompt and pick the better one. Votes become Elo ratings. | Image generation and editing | Not in cards | ||
| Artificial Analysis Video Arena Voters compare videos from two text-to-video models for the same prompt and pick the better one. Votes become Elo ratings. | Video generation | Not in cards |