Views
A view is a saved filter over the facets we tag on every benchmark, so anyone can see why a benchmark is in it. CAD grows fastest: 55% of its 40 benchmarks came out since July.
- Agents 2,306
Models take a sequence of actions and decide when to stop, alone or with other agents.
When interaction is agentic or multi-agent. 941 since July.
- Vision 1,392
Models work from images or video.
When input includes images or video. 289 since July.
- Biology and medicine 730
Tasks that need biology or medical knowledge.
When subject includes biological sciences, medical and health sciences, medical engineering, environmental biotechnology or industrial biotechnology. 126 since July.
- Lab internal evals 730
A developer’s own tests of its own models, reported only in its model releases. Its versions of public benchmarks are a separate view.
When it is run by a model developer, and it is its own tasks. 177 since July.
- Law and finance 592
Tasks that need legal, economic or business knowledge.
When subject includes law or economics and business. 225 since July.
- Audio and speech 533
Models listen or speak.
When input includes audio. Or output includes audio. 128 since July.
- Image, video and 3D generation 401
Models make images, video or 3D models.
When output includes images, video or 3D. 116 since July.
- Robotics 394
Models control a real robot or a simulated one.
When environment includes a physical robot or device. Or environment includes a simulator, and output includes actions. 174 since July.
- Kept fresh 369
Benchmarks that add new items on a schedule or draw new ones every time.
When refresh is new items on a schedule or live. 76 since July.
- Content safety 329
Tests of refusing harmful requests, by the MLCommons hazards.
When risk includes violent crimes, non-violent crimes, sex-related crimes, child sexual exploitation, defamation, specialized advice, privacy, intellectual property, indiscriminate weapons, hate, suicide and self-harm, sexual content or elections. 71 since July.
- Many languages 315
Benchmarks in five or more languages.
When language includes many languages. 35 since July.
- Games 205
Models play named games, alone or against each other.
When environment includes a named game. 74 since July.
- Dangerous capabilities 167
Tests of abilities that could enable CBRN or cyber harm, as the EU Code of Practice defines them.
When it measures capability, and risk includes CBRN or cyber offence. 36 since July.
- Loss of control 154
Tests of deceiving overseers, sandbagging, self-replication and other routes to losing control of a model.
When risk includes loss of control. 46 since July.
- Computer use 149
Models operate a computer or phone through its screen.
When environment includes desktop apps, browser or phone apps, and input includes screens. 31 since July.
- Chemistry and materials 147
Tasks that need chemistry or materials science.
When subject includes chemical sciences, chemical engineering or materials engineering. 33 since July.
- Deception and scheming 105
Tests of what models choose to do when they could deceive overseers, manipulate people or pursue their own goals.
When it measures propensity, and risk includes loss of control or harmful manipulation. 31 since July.
- Labs’ versions of public benchmarks 68
A developer’s own version of a public benchmark, or its own run of a public method, reported in its model releases: GDPval v2, a hard subset of SWE-bench Verified, a lab’s ECI fit.
When it is run by a model developer, and it is a version of a public benchmark or a run of another’s method. 13 since July.
- CAD 40
Models design mechanical parts and assemblies, in CAD software or as CAD code and 3D models.
When environment includes SolidWorks, AutoCAD, Autodesk Fusion, Fusion 360, FreeCAD, Onshape, CATIA, Siemens NX, OpenSCAD, CadQuery or Rhinoceros 3D. Or output includes 3D, and subject includes mechanical engineering. Or subject includes mechanical engineering, and output includes code, and environment includes none. 22 since July.