LifeSci Diamond

Models write regulatory affairs deliverables for drugs, biologics and medical devices, such as change assessment memos, dossier inconsistency logs and agency letters. The 55 tasks span five families: reporting categories, Module 2/3 dossier consistency, missing evidence, submissions and correspondence, and device vigilance. A model judge grades each answer against a task rubric, and a task passes only when every rubric line is met and no penalty fires.

Arca sells regulatory AI software and ranks four configurations of its own product next to 18 plain model APIs. Ten tasks are public (CC BY 4.0) and 45 are held out. The November 4th, 2026 edition regraded every answer with one judge, GPT-6.1 Sol at medium effort, so earlier numbers are not comparable. Each system gave one answer per task, and one task moves the pass rate by 1.8 points. Of the 55 answer keys, 26 had specialist review, 18 were written by contributors without a second read and 11 rest on primary sources only. The repository had 1 GitHub star on November 5th, 2026.