Model cards
We've read 522 launch posts and 346 system cards from 52 labs, with the newest one posted on November 9th. Each post shows the benchmarks it headlines and what changed since that lab's last one. System cards, model cards and technical reports can be found under their launch post.
A green dot means the benchmark is new since that lab’s last post or document of the same kind, for a model of the same size.
Agent tests rose from 2% of what labs headlined in 2024 to 19% in 2026, as knowledge tests fell from 52% to 19%
What all 52 labs headline in their launch posts, by area, every half year.
Select an area to break it down.
Knowledge and language 22% expert or harder in 2026
Math and science 42% expert or harder in 2026
Coding 78% expert or harder in 2026
Agents and tools 37% expert or harder in 2026
Professional work 84% expert or harder in 2026
Vision, audio and robotics 4% expert or harder in 2026
Safety and security 90% expert or harder in 2026
The last half year runs through November 13, so its count is partial.
Source: Labs’ launch posts, read by Benchmark Almanac
2026
- Nov 9, 2026
- AutomationBench 37.8
- Agents' Last Exam 61.6
- Personal Superintelligence Index (Meta) 46.8
- Terminal-Bench 4.0 40.9
- Artificial Analysis Coding Agent Index v2 46
- FrontierCode 2.0 27.2
- GDP.pdf 22.8
- OSWorld 2.1 69.5
- PerceptionBench (Moonshot) 80.6
Dropped since the last post: Artificial Analysis Intelligence Index v4.3, JobBench, MCP Atlas, GDPval-AA v2, OSWorld-Verified, Toolathlon-Verified, HLE-Diamond, AA-Omniscience, SciCode, Humanity's Last Exam, GPQA Diamond, DeepSWE, SWE Atlas, Terminal-Bench 2.1, ZeroBench, BabyVision, SimpleVQA, WorldVQA, IntPhys 2, VideoMME v2 (with subtitles), FORTRESS, StrongREJECT v2 (Meta), MASK, Agentic misalignment suite (Anthropic)
Its technical report adds 0 evaluations 37 dropped 9 results in all
Muse Ember Evaluation ReportDropped since the lab’s previous technical report: τ³-bench Banking, WildClawBench, Meta Agents Research Environments (ARE) / Gaia2, GDPval-AA v2, SciCode, OSWorld-Verified, SWE-bench Verified, SWE-Bench Pro, tau2-bench, Terminal-Bench 2.1, LiveCodeBench Pro, PinchBench, MCP Atlas, DeepSearchQA, AIME 2026, GPQA Diamond, ARC-AGI, Humanity's Last Exam, IFBench, SkillsBench, CharXiv, MMMU-Pro, OmniDocBench v1.5, ScreenSpot-Pro, ScreenSpot-v2, BEAM, AA-LCR (Artificial Analysis Long Context Reasoning), Omnilingual-Gaia2, Linguini, Belebele, MMLU-ProX, WMT24++, FLORES+, BOUQuET, CIMemories, PrivacyLens, AgentDojo
- Nov 9, 2026Claude Fable 5.2 Anthropic Claude Fable 5.2 and Mythos 5.2 (launch page); System Card: Claude Fable 5.2 & Claude Mythos 5.2, Table 8.1.A
- FrontierCode 2.0 49.1
- FrontierSWE 69.9
- AutomationBench 51.4
- Terminal-Bench-Science 74.6
- WANDR 82.1
- OSWorld 2.1 (offline subset) 89.3
- RoboDojo 64.9
- AA-Briefcase v1.1 1785
- Remote Labor Index 41.3
- Humanity's Last Exam 74.2
- Chartography 73.4
- GDPval-AA v2.1 1831
- SWE-Bench Pro 93
- SWE-bench Multilingual 96.6
- SWE-bench Multimodal 74.1
Dropped since the last post: Terminal-Bench 4.0, FrontierCode 1.1, CursorBench 4.0, OSWorld 2.1, HealthBench Professional
Its system card adds 41 evaluations 10 new 35 dropped 55 results in all
System Card: Claude Fable 5.2 & Claude Mythos 5.2Capabilities
- DeepSWE 74.3
- Terminal-Bench 4.0 67.6
- CursorBench 4.0 60
- STEM Professor Bench 61.8
- ProgramBench tests passed 85.6
- DRACO 87.7
- Fermat's Last Theorem in Lean (Anthropic) 35
- BenchCAD 2.1 75.4
- OfficeQA 83.4
- OfficeQA Pro 72.9
- Harvey LAB-AA v1.1 7.2
- Real-World Finance v2 (Anthropic) 1637
- ARC-AGI-3 79.3
- Global MMLU 96
- MILU 94.1
- De novo protein binder design (Anthropic) 85.1
Health
- HealthBench 59.6
- HealthBench Professional 67.7
- PhysicianBench 64.4
Safety
- Single-turn harmful request evaluations (Anthropic) 93.7
- Single-turn benign request evaluations (Anthropic) 0
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 82.8
- Malicious computer use (Anthropic) 88.4
Robustness
Cyber risk
Dropped since the lab’s previous system card: Virology Capabilities Test (VCT), BioMysteryBench, Protocols (Anthropic life sciences), Black-box RNA sequence modeling and design, AAV capsid packaging prediction, AAV packaging rate auto-research evaluation, Anthropic ECI (AECI), ExploitBench, Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based), ExploitGym, Prompt injection robustness against adaptive attackers, browser use (Anthropic), Anthropic undermining liberal democracy (behavioral audit metric), Sandbox escape evaluation (Anthropic), Self-preference bias (Anthropic), AA-Omniscience, MASK, Silent copying of answers (Anthropic), Concealment of changes (Anthropic), Verbalized grader awareness (Anthropic), Acting on a presumed grader and digging for evaluation clues (Anthropic), SHADE-Arena, LinuxArena, CoT-Control, FrontierCode 1.1, OSWorld 2.1, ArXivMath, BenchCAD, Harvey LAB (Legal Agent Benchmark), Toolathlon-Verified, SpatialBench (LatchBio), scBench (LatchBio SingleCellBench), Morphology-to-molecule matching (Anthropic), Medicinal Chemistry (Anthropic), Protein Design (Anthropic), Biomedical image analysis (Anthropic)
- Nov 5, 2026
- Vals Index v2.2 56.1
- FrontierCode 2.0 25.4
- DeepSWE 72.1
- Terminal-Bench-Science 5.7
- Vibe Code Bench 81.6
- SWE-Bench Pro V2 Hard 86.3
- PostTrainBench 1.2 31.9
- BioMysteryBench 70
- MCP Atlas 85
- JobBench 70.5
- EnterpriseOps-Gym 46.8
- EnterpriseBench: CoreCraft 71.8
- Vals Finance Index 61.5
- τ³-bench Banking 40.4
- DAYJOB: Finance 14.3
- APEX-Accounting 14.5
- Vals Legal Index 73.9
- DesignArena Frontend Web App 1296
- DesignArena Website 1340
- DesignArena Game Dev 1428
- DesignArena 3D Design 1401
- PerceptionBench (Moonshot) 64.7
- ScreenSpot-Pro 83.8
- Humanity's Sixth Sense 33.3
- BabyVision 68.6
- GDP.pdf 25.2
- MMMU-Pro 81.6
- CharXiv 85.6
- Chartography 36.7
- CritPt v2 55.7
- HLE-Diamond 28.7
- Humanity's Last Exam 53.2
- MathArena (Apex, IMO, USAMO) 63.5
- LongBench v2 70.6
- ComplexConstraints 44.5
- DistressBench 82
- FORTRESS
- StrongREJECT 97.8
- CyberBench v1.1 (Vals) 72.7
- PatchEval 79.6
- Artificial Analysis Intelligence Index v5 30.6
- AA-Briefcase v1.1 1527
- GDPval-AA v2.1 1630
- AA-LCR 1.1 83.3
- AA-Omniscience v1.1 16.1
- AA-3D 1021
- sudo L7 14.1
- SWE-Marathon v1.1 38.8
- SWE-bench Multilingual 81.2
- SWE-bench Multimodal 43.1
- SWE-bench Verified 82.2
- BrowseComp 90.2
- DeepSearchQA 90.9
- AutomationBench 45.6
- Vals Finance Agent v2.1 67.5
- Excel Modeling Benchmark v1.1 69.5
- Tax Agent Bench 65.8
- Harvey LAB v1.1 (HLAB-Vals-1.1) 21.7
- Legal Research Bench 46.6
- Code Arena 1653
- MATH-Vision (MathVision) 92.8
- SAGE v1.1 (Vals) 88.6
- GPQA Diamond 93
- HMMT 2026 95.8
- HoneyBench v0.2-alpha 44
- PMPP-Hard 494.3
- Misbehavior (Thinking Machines) 14.9
- Covert misbehavior (Thinking Machines) 11.8
- Destructive actions (Thinking Machines) 4.7
- Apollo Research strategic deception evaluations 8
- DiligenceBench 58.1
- Code2Video-AA 1020
Dropped since the last post: Artificial Analysis Intelligence Index v4.1, SWE-Bench Pro, Terminal-Bench 2.1, SciCode, GDPval-AA v2, Toolathlon-Verified, AA-Briefcase, AIME 2026, CritPt, ARC-AGI, ARC-AGI-2, SimpleQA Verified, AA-Omniscience, IFBench, Global MMLU Lite 3.0, Audio MC, MMAU, VoiceBench, ForecastBench, Prophet Arena
- Nov 5, 2026
- Respecting Auto-Review 0
- Respecting Warnings 73
- Declining to Exploit a Honeypot During Difficult ExploitGym Problems 0.07
- Blocker Deception 20.3
- Broken Search Tool 0.33
- ExploitBench 100
- ExploitBench - Internal Port 42
- SEC-bench Pro 87
- SRE-Bench 92.8
- ExploitGym 48.5
- CyScenarioBench 61
- FrontierCyber 31.3
- Atomic Challenges 97
- Expert-SWE (OpenAI) 83
- ProtocolQA Open-Ended 47.2
Dropped since the last post: Terminal-Bench 4.0, Agents' Last Exam, MMMU-Pro, OSWorld 2.1, Terminal-Bench-Science
Its system card adds 1 evaluation 1 new 17 dropped 16 results in all
GPT-6 Cyber System CardBio and chem risk
Dropped since the lab’s previous system card: MedChemBench (OpenAI), GeneBench Pro, LifeSciBench, TroubleshootingBench, Tacit Knowledge and Troubleshooting, Multimodal Troubleshooting Virology, AAV Capsid Packaging Prediction (OpenAI), SHP2 Protein Function Prediction, Coronavirus-ACE2 Cell-Entry Screen, Phage-plasmid Co-evolution, OAI-repo Sabotage v2, Math Side-tasking Monitor Evasion, CoT-Control, Unintended Engagement with External Agent Messages, Misaligned Behavior in Realistic Work Environments, Historical Jailbreak Regression, Agent Replay Auto-Review
- Nov 4, 2026
Tencent Hy4 Tencent Tencent Launches Hy4, Joining the Ranks of the World's Leading Open-Source AI Models (Tencent press release) - ProgramBench almost solved 30
- FrontierCode 2.0 31
- Terminal-Bench 4.0 44.4
- Agents' Last Exam v1.1 41
- AutomationBench 56.2
- PostTrainBench 1.2 35.2
- Terminal-Bench-Science 19.8
- OSWorld 2.1 78.9
Dropped since the last post: SWE-bench Multilingual, SWE-Bench Pro, DeepSWE, SWE Atlas, SWE-Marathon, Terminal-Bench 2.1, NL2Repo-Bench, CyberGym, ProgramBench, PostTrainBench 1.1, Harbor-Index, Hy-Backend 2.0, Hy-SWE Max, Hy-CompanyBench V2, WideSearch, $OneMillion-Bench, DRACO, Hy-LifeSearch, Hy-BrowseComp-Pro2, OfficeQA Pro, MCP Atlas, Toolathlon-Verified, APEX-Agents, SkillsBench, JobBench, WorkSpaceBench, Agents' Last Exam, GDPval-AA v2, BankerToolBench, E-Bench (Tencent), E-Bench-Code (Tencent), Hy-FinAgentBench, Hy-FinmodelBench v2, BioMysteryBench, Humanity's Last Exam, CritPt, GPQA Diamond, SUPERChem, ArXivMath, HorizonMath, MathArena (Apex, IMO, USAMO), BrokenArXiv
- Nov 4, 2026
- Nov 4, 2026
- FrontierCode 2.0 24
- FrontierCode 1.1 45.5
- DeepSWE 60.1
- Terminal-Bench 4.0 20
- Terminal-Bench 2.1 89.2
- SWE Atlas 50
- SWE-Bench Pro V2 Hard 89.9
- ExploitBench 34.8
- Nov 2, 2026
- Terminal-Bench 2.1 88.4
- Terminal-Bench 4.0 38.5
- ProgramBench almost solved 25.5
- DeepSWE 64.6
- PaperBench 82.1
- SWE Atlas 58.3
- SWE Atlas Refactoring 55.2
- SWE Atlas Test Writing 47.8
- NL2Repo-Bench 60.5
- BrowseComp 91.9
- AutomationBench 51.8
- Humanity's Last Exam 63.1
- Agents' Last Exam v1.1 38.5
- ResearchRubrics 81
- MCP Atlas 83.1
- JobBench 65.4
- SkillsBench 63.3
- FrontierScience 77
- SpreadsheetBench 2 34.9
- FinSearchComp 63.4
- Claw-Eval Pass^3 78.3
- FORTE 83.3
- 3DCodeBench 55.4
- Vision2Web 78.6
- CharXiv 90.2
- MATH-Vision (MathVision) 96.5
- Video-MME 85.2
Dropped since the last post: SWE-Bench Pro, SWE-bench Multilingual, RWSearch, IFEval, WritingBench, IMO-AnswerBench, GPQA Diamond
- Oct 30, 2026
Dropped since the last post: AI2D, ChartQA, HallusionBench, MathVerse, MATH-Vision (MathVision), MathVista, MMMU, MMStar, OCRBench, ScreenSpot-v2
Its technical report adds 0 evaluations 11 dropped 1 result in all
Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From an Evolved HarnessDropped since the lab’s previous technical report: AI2D, ChartQA, HallusionBench, MathVerse, MATH-Vision (MathVision), MathVista, MMMU, MMStar, OCRBench, ScreenSpot-v2, Harmful content defect rate (Azure AI Evaluation SDK, Microsoft)
- Oct 29, 2026
Mistral Large 4 (Le Chonk, open weights) Mistral AI Mistral Large 4 (Le Chonk) is now open weight (Mistral news) - FrontierCode 2.0 19.7
- Artificial Analysis Coding Agent Index v2 48
- Terminal-Bench 4.0 41
- Artificial Analysis Cyber Index 51
- Finch (FinWorkBench) 73.3
- BrowseComp 84.5
- Lakera B3 (Backbone Breaker Benchmark) 97.1
Dropped since the last post: DeepSWE, AutomationBench, Vals Finance Agent v2, Harvey LAB (Legal Agent Benchmark), Dense 200 (bbox)
- Oct 28, 2026
- Artificial Analysis Intelligence Index v5 59
- Vals Index v2.2 70.4
- SWE-sweep 9.6
- FrontierSWE 61.2
- Terminal-Bench 4.0 65.2
- Agents' Last Exam v1.1 77.4
- OSWorld 2.1 (offline subset) 81.5
- AutomationBench 53.9
- Vending-Bench 2 16484
- Chartography 72.5
- Blueprint-Bench 2 58.6
- GraphWalks 96.2
- Riemann-bench 80
- Terminal-Bench-Science 64.8
- CWE-bench 71
- ExploitGym 54.8
- SEC-bench Pro 88.2
- WebDev Arena 1768
- Gray Swan IPI Arena benchmark 0
Dropped since the last post: Vals Index v2.1, Vals Finance Agent v2, Harvey LAB (Legal Agent Benchmark), DeepSWE, Vibe Code Bench, PostTrainBench, LABBench2, Agents' Last Exam, OSWorld 2.0, LVBench
- Oct 27, 2026
- LiveClawBench 45.6
- Claw-Eval 68.6
- Claw-Eval Pass^3 51.4
- BrowseComp-Plus 37
- WebArena-Verified 38.4
- EnterpriseOps-Gym 28.2
- AutomationBench 41.3
- BFCL V4 71.4
- ToolSandbox 80.4
- MCP Atlas 32.6
- tau3-bench 68
- SWE-bench Verified 43.4
- SWE-bench Multilingual 23
- Terminal-Bench 2.1 12.9
- LiveCodeBench v6 73.5
- LiveCodeBench Pro 31.7
- SciCode 28.5
- IFEval 96.7
- IFBench 73
- IFStruct 93.7
- Multi-IF 89.1
- GPQA Diamond 63.4
- Humanity's Last Exam 13.9
- AA-Omniscience 0.5
- AA-LCR (Artificial Analysis Long Context Reasoning) 52.8
- RULER 65.6
Dropped since the last post: PinchBench, τ³-bench Banking, MATH-500
- Oct 26, 2026
Dropped since the last post: DeepSWE, GDP.pdf, AutomationBench, OSWorld 2.0
Its system card addendum adds 36 evaluations 4 new 36 results in all
Addendum to GPT-6 Astra System Card: GPT-6.1 TerraHealth
Hallucination
Safety
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 29.6
- Declining to Exploit a Honeypot During Difficult ExploitGym Problems 0.07
- Blocker Deception 8.47
- Broken Search Tool 1.83
- Misaligned Behavior in Realistic Work Environments 2.28
- Unintended Engagement with External Agent Messages 65
- OpenAI Deployment Simulation 0.045
Monitorability
Bio and chem risk
Cyber risk
- ExploitBench 97.1
- ExploitBench - Internal Port 24.5
- SEC-bench Pro 83.3
- Sandbox Bench 22.7
AI R&D risk
- Oct 24, 2026
- Terminal-Bench 4.0 65.7
- Terminal-Bench-Science 38.1
- SWE-Marathon 51.7
- BenchCAD 92.7
- ProgramBench almost solved 31.5
- DeepSWE 75.5
- PostTrainBench 1.2 36.5
- Kimi Code Bench v2 77.1
- Kimi Code Bench 3.0 Long-horizon 61.2
- Humanity's Last Exam 70
- BrowseComp 92.1
- BioMysteryBench 80.6
- CharXiv 89
- MMMU-Pro 84.1
- GDP.pdf 34.6
- Chartography 90.2
- BabyVision 95.4
- ZeroBench 39
- Agents' Last Exam 43.2
- AutomationBench 73.3
- GDPval-AA v2.1 1846
- AA-Briefcase v1.1 1678
- APEX-Agents 67.2
- OfficeQA Pro V2 64.4
- OfficeQA Pro 67.4
- FrontierFinance 58.6
- Harvey LAB (Legal Agent Benchmark) 23.7
- HANDBOOK.md 44.6
- OSWorld 2.1 79.6
- Business Operation (Moonshot) 37.8
- DECK-Arena 1456
- Design Bench (Moonshot) 957
- Kimi Hallu Bench 8.3
- GPQA Diamond 94.2
- PerceptionBench (Moonshot) 68.3
- ExploitBench 74.7
- ExploitGym 20.7
- xdev-bench-hard 81
Dropped since the last post: CritPt, AA-LCR (Artificial Analysis Long Context Reasoning), ProgramBench, Terminal-Bench 2.1, FrontierSWE v1, PostTrainBench, MLS-Bench, SciCode, DeepSearchQA, ResearchRubrics, GDPval-AA v2, Toolathlon-Verified, MCPMark, MCP Atlas, JobBench, AA-Briefcase, OfficeQA, SpreadsheetBench 2, OSWorld-Verified, OSWorld 2.0, SaaS-Bench, tau3-bench, Harvey LAB-AA v1.0, CorpFin v2, Vals Finance Agent v2, Legal Research Bench, WorldVQA, OmniDocBench v1.5, Video-MME, MMVU, MATH-Vision (MathVision)
- Oct 23, 2026
- MultiChallenge 55.6
- RoleBench 71.2
- RoleMRC 79.1
- EQ-Bench 3
- Brevity Benchmark 82.8
- Berkeley Function Calling Leaderboard (BFCL) 67.3
- When2Call 60.5
- ToolSandbox 74.1
- tau2-bench 52.3
- IHEval 82.4
- AgentIF 22.9
- StructEval 80
- IFBench 64
- Inverse IFEval 45.3
- RULER 85.8
- LongMemEval 69.9
- LongBench v2 48.7
- SysBench 67.2
- GaRAGe 40.7
- RAGTruth 77.9
- RelyToolBench 69.3
Dropped since the last post: MMLU-Pro, AA-Omniscience, GPQA Diamond, Humanity's Last Exam, SciCode, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.1, PinchBench, BrowseComp, τ³-bench Banking, GDPval-AA v2, AA-LCR (Artificial Analysis Long Context Reasoning)
Its technical report adds 11 evaluations 11 new 14 dropped 32 results in all
Nemotron 3.5 Bolt E4B 4GB: An Open and Efficient Small Language Model for Fast Conversational AgentsCapabilities
- WMT24++ 74.4
- MMLU-ProX 66.2
- Audio MC 25.2
- VoiceBench 80.3
- CoVoST2 36.1
- FLEURS 10.2
- Open ASR Leaderboard 9.28
- OmniDocBench v1.5
- MMMU-Pro 42
- MATH-Vision (MathVision) 46.3
Health
- MedXpertQA-MM 26.9
Dropped since the lab’s previous technical report: SWE-bench Verified, SWE-bench Multilingual, τ³-bench Banking, Terminal-Bench 2.1, GDPval-AA v2, PinchBench, BrowseComp, SciCode, GPQA Diamond, Humanity's Last Exam, AA-Omniscience, CritPt, AA-LCR (Artificial Analysis Long Context Reasoning), NV Chat Benchmark
- Oct 22, 2026
Spark-X2.6-4B iFlytek Spark-X2.6-4B: Advancing the Frontier of On-Device Agents for Real-World Tasks (Hugging Face) - BFCL V4 71.8
- tau2-bench 86.9
- tau3-bench 34.7
- VitaBench 2.0 30.8
- MCP Atlas 62.4
- EnterpriseOps-Gym 33.6
- Claw-Eval 69.9
- BrowseComp 53.1
- SWebBench 71.7
- PAgentBench 74.1
- SWE-Bench Pro 54.3
- SWE-bench Verified 62.8
- SWE-bench Multilingual 64.7
- SciCode 36.8
- Gaokao Math 2026 134.9
- AIME 2026 91.2
- HMMT 2026 81.3
- IMO-AnswerBench 75.2
- AA-LCR 1.1 67.3
- LongBench v2 53.3
- GraphWalks 63.2
- OpenAI MRCR v2 45.8
- IFEval 93.6
- IFBench 75.8
- Humanity's Last Exam 36.9
- GPQA Diamond 77.1
- Oct 19, 2026
- GDPval-AA v2.1 1237.2
- τ³-bench Banking 30.2
- MultiChallenge 60.4
- AA-Omniscience 46.4
- IFBench 76.8
- GPQA Diamond 89.2
- AA-LCR 1.1 77.7
- MRCR v2 75.5
- SWE Atlas 51.1
- DeepSWE 46.3
- BrowseComp-Plus 55.2
- DeepSearchQA 71.3
Dropped since the last post: tau3-bench, AA-LCR (Artificial Analysis Long Context Reasoning), SciCode
- Oct 19, 2026
- AutomationBench 62.8
- BankerToolBench 76.2
- GDP.pdf 20.8
- Terminal-Bench 4.0 38.1
- LOCA-Bench 60.3
- Agents' Last Exam 29.6
- OSWorld 2.0 61.7
- PostTrainBench 1.1 39.3
- DeepSWE 73.7
- NL2Repo-Bench 61.2
- SWE Refactor Bench 23
- CyberGym 82.2
- BabyVision 91
- Humanity's Last Exam 43.4
- CharXiv 82.5
Dropped since the last post: SWE-bench Verified, SWE-Bench Pro, Terminal-Bench 2.1, SWE Atlas, SWE Atlas, SWE-fficiency, LiveSQLBench, CL-bench, VIBE-V2 (MiniMax), SVG-Bench (MiniMax), PostTrainBench, KernelBench, PaperBench, BrowseComp, DRACO, GDPval-Rubrics (MiniMax), OfficeQA, SpreadsheetBench, YC-Bench, MCP Atlas, APEX-Agents, Claw-Eval, OSWorld-Verified, OmniDocBench v1.5, MMMU-Pro, Video-MMMU, Video-MME, IMO 2025, USAMO 2026
- Oct 17, 2026
- Humanity's Last Exam 42.6
- Terminal-Bench 4.0 42.3
- DeepSWE 77
- ProgramBench
- NL2Repo-Bench 68.2
- AutomationBench 60.5
- Agents' Last Exam 35.4
- MathArena (Apex, IMO, USAMO) 72.1
- Chartography 83
- BabyVision 92.5
- ZeroBench 51
Dropped since the last post: GPQA Diamond, Codeforces rating, Terminal-Bench 2.1, Terminal-Bench 3.0, ProgramBench almost solved, CyberGym, SEC-bench Pro, ExploitGym, AGIEval, MMLU-Pro, C-Eval, MultiLoKo, SimpleQA Verified, SuperGPQA, BIG-Bench Hard, BBEH micro average, DROP, HellaSwag, BigCodeBench, HumanEval, GSM8K, MATH, MGSM, LongBench v2, MMMU-Pro, CVBench, DocVQA
- Oct 15, 2026
- Artificial Analysis Intelligence Index v4.3 39
- JobBench 61.8
- MCP Atlas 89.2
- GDPval-AA v2 1401
- OSWorld-Verified 80.4
- Toolathlon-Verified 72.8
- HLE-Diamond 26.1
- AA-Omniscience 25
- SciCode 58.5
- Humanity's Last Exam 61
- GPQA Diamond 89
- DeepSWE 60
- SWE Atlas 58
- Terminal-Bench 2.1 76.4
- ZeroBench 50
- BabyVision 80.2
- SimpleVQA 73.6
- WorldVQA 49.4
- IntPhys 2 68.3
- VideoMME v2 (with subtitles) 61.5
- FORTRESS 5.8
- StrongREJECT v2 (Meta) 0.5
- MASK 98.1
- Agentic misalignment suite (Anthropic) 0
Dropped since the last post: DeepSearchQA, τ³-bench Banking, WildClawBench, Meta Agents Research Environments (ARE) / Gaia2, SkillsBench, SWE-Bench Pro, SWE-bench Verified, CharXiv, ScreenSpot-Pro, OmniDocBench v1.5, MMMU-Pro, CIMemories, AgentDojo, IFBench, AIME 2026, AA-LCR (Artificial Analysis Long Context Reasoning), BEAM
- Oct 15, 2026
- GDPval-AA v2 28.5
- Terminal-Bench 2.1 51
- τ³-bench Banking 17
- Humanity's Last Exam 25.3
- AA-Omniscience 16.9
- SciCode 41
- GPQA Diamond 86.4
- AA-LCR (Artificial Analysis Long Context Reasoning) 64.1
- PinchBench 87.8
- MCP Atlas 52.2
- BrowseComp 66.8
- WebArena 50.9
Dropped since the last post: MMLU-ProX, CritPt, MathArena Apex Shortlist, LiveCodeBench v6, AA-LCR 1.1, GraphWalks, IFBench, NV Chat Benchmark, WMT24++, tau2-bench, SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, BabyVision, MMLongBench-Doc, V*, OCR-Reasoning, MATH-Vision (MathVision), OCRBench v2, CharXiv, VideoMME v2 (with subtitles), OSWorld-Verified, WebVoyager, AutomationBench, GDP.pdf, Terminal-Bench 4.0, DeepSWE, AA-Briefcase
Its technical report adds 1 evaluation 22 dropped 13 results in all
Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AICapabilities
- IFBench 66.2
Dropped since the lab’s previous technical report: GDPval, SWE-bench Verified, SWE-bench Multilingual, ProfBench, tau3-bench, Vals Finance Agent, IOI 2025, LiveCodeBench v6, IMO-AnswerBench, MathArena Apex Shortlist, CritPt, MMLU-Pro, MultiChallenge, RULER, LongBench v2, MMLU-ProX, WMT24++, IMO-ProofBench, IMO 2025, Putnam 2025, USAMO 2026, SPEED-Bench
- Oct 15, 2026
- GPQA Diamond 89.5
- Humanity's Last Exam 31.8
- MMLU-ProX 76.9
- AA-Omniscience 4.03
- CritPt 10.9
- MathArena Apex Shortlist 88.4
- LiveCodeBench v6 88.2
- SciCode 47.1
- AA-LCR 1.1 68.6
- GraphWalks 89.7
- IFBench 70.9
- NV Chat Benchmark 52.7
- WMT24++ 84.8
- tau2-bench 73
- τ³-bench Banking 18.9
- GDPval-AA v2 1211
- BrowseComp 78
- SWE-bench Verified 71.1
- SWE-bench Multilingual 67.7
- SWE-Bench Pro 51.8
- Terminal-Bench 2.1 60.7
- BabyVision 29.6
- MMLongBench-Doc 62.8
- V* 85.6
- OCR-Reasoning 65.1
- MATH-Vision (MathVision) 78.2
- OCRBench v2 68.1
- CharXiv 80.8
- VideoMME v2 (with subtitles) 32.9
- OSWorld-Verified 51.1
- WebVoyager 77.7
- AutomationBench 32.4
- GDP.pdf 6.6
- Terminal-Bench 4.0 1
- DeepSWE 24.6
- AA-Briefcase 1137
Dropped since the last post: RoboCasa, RoboCasa GR1 Tabletop Tasks, SimplerEnv, RoboLab
Nemotron 3.5 Super 120B-A12B Base: Its technical report adds 22 evaluations 22 new 12 dropped 23 results in all
Nemotron 3.5 Super and Lightning technical report, base model evaluationsCapabilities
- MMLU 84.1
- MMLU-Pro 76.3
- AGIEval 75.7
- SuperGPQA 47.6
- GSM8K 91.7
- MATH 87.7
- MATH-500 81.6
- HumanEval 81.3
- MBPP 79.1
- LiveCodeBench 47.6
- BigCodeBench 48.4
- CommonSenseQA 85.3
- ARC-Challenge 96
- HellaSwag 89
- OpenBookQA 48.8
- PIQA 85.1
- TruthfulQA 71.2
- WinoGrande 79.1
- SIQA 49
- RACE 91.4
- Global MMLU 86.6
- MGSM 87.5
Dropped since the lab’s previous technical report: GDPval-AA v2, Terminal-Bench 2.1, τ³-bench Banking, Humanity's Last Exam, AA-Omniscience, SciCode, AA-LCR (Artificial Analysis Long Context Reasoning), PinchBench, MCP Atlas, BrowseComp, WebArena, IFBench
Its technical report adds 23 evaluations 23 new 22 dropped 53 results in all
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AICapabilities
- Bird-SQL 59.6
- ScreenSpot-Pro 43.5
- LMArena (Elo) 1420
- ERQA 57.5
- Omni3D 53.4
- OmniSpatial 58.6
- SPBench 63.8
- MindCube 62.2
- MMSI-Bench 43.1
- SPAR-Bench 64.2
- BLINK 71.3
- SpatialTree 58.1
- ViewSpatial-Bench 52
- OSI-Bench 30.6
- PAI-Bench 63.8
- VSI-Bench 48.1
- DSI-Bench 61.4
- CVBench 69.6
- PerceptionComp 39.4
- Video-MME 74.4
- SPEED-Bench 4.72
- VSTI-Bench 67.3
- MMSI-Video-Bench 34.5
Dropped since the lab’s previous technical report: MMLU, MMLU-Pro, AGIEval, SuperGPQA, GSM8K, MATH, MATH-500, HumanEval, MBPP, LiveCodeBench, BigCodeBench, CommonSenseQA, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, TruthfulQA, WinoGrande, SIQA, RACE, Global MMLU, MGSM
- Oct 14, 2026
- Big Bench Audio 98.5
- τ-Voice 71.6
- Full-Duplex-Bench 97
- Speech-to-Speech Index 86.9
- Speech Agent Arena 1070
- EVA-Bench 74
Dropped since the last post: CursorBench 4.0, DeepSWE, EEBench, AA-Briefcase v1.1, Terminal-Bench 4.0, Harvey LAB (Legal Agent Benchmark), HealthBench Professional, GDPval
- Oct 13, 2026
- AutomationBench 27.7
- BFCL V4 65
- ToolSandbox 80.7
- LiveClawBench 48.1
- PinchBench 83.2
- τ³-bench Banking 8.04
- WebArena-Verified 26.8
- GPQA Diamond 74.4
- Humanity's Last Exam 7.4
- AA-Omniscience -24.9
- IFEval 96.8
- IFBench 74.7
- Multi-IF 84.4
- MATH-500 96
- SWE-bench Verified 51
- SciCode 32.7
Dropped since the last post: AIME 2025, AIME 2026, Berkeley Function Calling Leaderboard (BFCL), tau2-bench
- Oct 13, 2026
- Humanity's Last Exam 37.4
- CritPt 16.3
- SuperMatBench 51.2
- HealthBench Professional 65.3
- SWE-bench Multilingual 88.5
- SWE-Bench Pro 68.7
- SWE Atlas 64.5
- DeepSWE 59.7
- FrontierSWE 75.2
- Terminal-Bench 2.1 81.2
- Terminal-Bench 4.0 46
- CyberGym 87.9
- τ³-bench Banking 47.8
- AutomationBench 52.5
- SkillsBench 68.7
- MCP Atlas 83
- GDPval-AA v2.1 1673
- BrowseComp 91.7
- DRACO 85.5
- WideSearch 83.3
- Vals Finance Agent v2 57.9
- FinSearchComp 81.7
- MultiChallenge 69.8
- Oct 13, 2026
- Vals Index v2.1 68.9
- AutomationBench 51.3
- Vals Finance Agent v2 65.4
- Harvey LAB (Legal Agent Benchmark) 19.6
- DeepSWE 77.9
- FrontierSWE 55
- Vibe Code Bench 91.9
- Terminal-Bench 4.0 57.4
- PostTrainBench 45.3
- Terminal-Bench-Science 57.6
- LABBench2 88.8
- Riemann-bench 76
- GraphWalks 99.7
- Agents' Last Exam 39.5
- OSWorld 2.0 69.2
- Chartography 71.6
- LVBench 91.7
- CWE-bench 68
Dropped since the last post: Human SxS image generation Elo (Google), Infographic factuality (Google), Human SxS image editing Elo (Google)
Its model card adds 9 evaluations 9 new 3 dropped 27 results in all
Gemini 4 Argon Model CardAlignment
Bio and chem risk
AI R&D risk
Persuasion
Dropped since the lab’s previous model card: Human SxS image generation Elo (Google), Infographic factuality (Google), Human SxS image editing Elo (Google)
Its system card adds 81 evaluations 65 new 82 results in all
Gemini 4 Argon Safety & Security ReportHealth
- Health static understanding eval 72.7
- Health language and locale eval 61.4
- Health dynamic conversation eval 56
- High-risk health scenarios eval 98.9
- HealthBench Hard 45.4
- HealthBench Professional 68.8
- Suicide & Self-Harm evaluation 96
- Delusional Reinforcement evaluation 94
- Emotional Attachment evaluation 76
Hallucination
Safety
- Child Sexual Abuse and Exploitation Safety 100
- Dangerous Content Safety 96.6
- Sexually Explicit Content Safety 97.6
- Hate & Harassment Safety 92.7
- Agent harness misuse evaluation 91.4
- Computer use Prohibited Action Evaluation 99.4
- Computer use manual red teaming 100
- Child Safety Action Testing 100
- Unjustified refusals
- Tone
- HelpBench 76.5
- Roboharm
- ASIMOV
Robustness
- Prompt injection, general tool use 91
- Prompt injection, turn injections in tool use 86
- Contextual security evaluation 27
- Personal-device assistant prompt injection eval 98
- Grounding robustness (RAG prompt injection) 99
- Computer use prompt injection robustness 92
- Coding agent prompt injection robustness 83
- Gray Swan IPI Arena benchmark 0.7
Bias
Alignment
- GDM Stealth 100
- GDM Situational Awareness 90.9
- Apollo Research strategic deception evaluations
- Gram (Gauging Realistic Agentic Misbehavior) 0
- Google DeepMind scheming honeypot suite 0
- Agentic task gaming and overreach evals 0.49
- Cheating audit on capability evaluations
- Instrumental convergence evals 0
- RL rollout audit for beyond-episode drives 0
- Agent behavior under-service and overreach eval
- Impossible Tasks Evaluation 93.8
- Overstepping boundaries on ambiguous and high-risk tasks 55.9
- Gemini Personality Evaluation (sycophancy)
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Oct 12, 2026
- Personal Superintelligence Index (Meta) 54.3
- Harvey LAB-AA v1.1 10
- Vals Finance Agent v2 56.6
- HealthBench Professional 60.5
- AutomationBench 39.1
- GDP.pdf 30.8
- JobBench 70.5
- OSWorld 2.1 72.6
- MemoryBench (Meta) 85.6
- Terminal-Bench 4.0 52
- DeepSWE 78.8
- SWE-Marathon 53.1
- ProgramBench
- Artificial Analysis Intelligence Index v4.3 52
- FocusBench (Meta) 44.9
Its model card adds 1 evaluation 19 dropped 14 results in all
Muse Wonder Model CardCapabilities
Dropped since the lab’s previous model card: CyberGym, ExploitBench, ExploitGym, CyberBench (Liu et al.), Poly-Guard Bench, RedCode, Molecular Biology Capabilities Test, Human Pathogen Capabilities Test, Virology Capabilities Test (VCT), LAB-Bench ProtocolQA, LAB-Bench SeqQA, ABC Bench Fragment Design, ABC Bench Liquid-Handling Robot, ABC Bench Screening Evasion, Meta BioDesign Proxy, BioTIER Refuse, Meta chemical-agent refusal evaluation, MASK, Agentic misalignment suite (Anthropic)
- Oct 12, 2026
- Personal Superintelligence Index (Meta) 59.1
- Harvey LAB-AA v1.1 13.1
- Vals Finance Agent v2 59.1
- HealthBench Professional 62.3
- AutomationBench 45.4
- GDP.pdf 33.2
- JobBench 75.3
- OSWorld 2.1 78.4
- MemoryBench (Meta) 85.5
- Terminal-Bench 4.0 57.1
- DeepSWE 79
- SWE-Marathon 57.5
- ProgramBench
- Artificial Analysis Intelligence Index v4.3 56
- FocusBench (Meta) 54.2
Dropped since the last post: GDPval-AA v2, OSWorld 2.0, DeepSearchQA, MRCR v2, SWE Atlas, Terminal-Bench 2.1
Its model card adds 20 evaluations 33 results in all
Muse Wonder Model CardCapabilities
Alignment
Bio and chem risk
- Molecular Biology Capabilities Test 59.6
- Human Pathogen Capabilities Test 66.5
- Virology Capabilities Test (VCT) 55.9
- LAB-Bench ProtocolQA 89
- LAB-Bench SeqQA 95.6
- ABC Bench Fragment Design
- ABC Bench Liquid-Handling Robot 100
- ABC Bench Screening Evasion
- Meta BioDesign Proxy 79.4
- BioTIER Refuse 98.7
- Meta chemical-agent refusal evaluation 100
Cyber risk
- CyberGym 88.4
- ExploitBench 58.9
- ExploitGym 20
- CyberBench (Liu et al.) 0.8
- Poly-Guard Bench 2.3
- RedCode 8
- Oct 12, 2026
Dropped since the last post: Terminal-Bench 2.1, DeepSWE, NL2Repo-Bench, Toolathlon-Verified, AutomationBench, Agents' Last Exam, Humanity's Last Exam, GDPval-AA v2, OfficeQA Pro, CharXiv, Chartography, BabyVision, MVBench, MMVU, OSWorld 2.0, Vision2Web, Z.ai Code Bench, Artificial Analysis Intelligence Index v4.1
- Oct 7, 2026Claude Haiku 5.5 Anthropic Introducing Claude Haiku 5.5 (launch page); System Card: Claude Haiku 5.5, Table 8.1.A
- GDPval-AA v2.1 1620
- AA-Briefcase v1.1 1578
- OSWorld 2.1 (offline subset) 72.4
- Humanity's Last Exam 57.4
- Terminal-Bench 4.0 39.2
- FrontierCode 1.1 46.4
- Chartography 46.4
- SWE-Bench Pro 64.8
- SWE-bench Multilingual 83.7
- SWE-bench Multimodal 30.7
- HealthBench Professional 64.8
Dropped since the last post: SWE-bench Verified, Terminal-Bench, tau2-bench, OSWorld-Verified, AIME 2025, GPQA Diamond, MMMLU, MMMU
Its system card adds 58 evaluations 49 new 16 dropped 69 results in all
System Card: Claude Haiku 5.5Capabilities
- Terminal-Bench-Science 20.6
- FrontierSWE 43.8
- ProgramBench tests passed 82
- DRACO 81.5
- WANDR 49.9
- BenchCAD 67
- OfficeQA 73.5
- OfficeQA Pro 60.3
- Global MMLU 87.8
- MILU 87.6
- Morphology-to-molecule matching (Anthropic) 25.4
- Medicinal Chemistry (Anthropic) 58
- Protein Design (Anthropic)
- Biomedical image analysis (Anthropic) 43.5
Health
- HealthBench 61.6
- PhysicianBench 43
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 98.4
- Single-turn benign request evaluations (Anthropic) 0.17
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 84.3
- Malicious computer use (Anthropic) 82.6
Robustness
Alignment
- Anthropic automated behavioral audit 3.19
- Anthropic undermining liberal democracy (behavioral audit metric) 1.22
- Sandbox escape evaluation (Anthropic) 8.7
- Opportunistic sandbox escape (Anthropic) 4
- Self-preference bias (Anthropic) 0.23
- MASK 96.8
- Silent copying of answers (Anthropic) 17.3
- Concealment of changes (Anthropic) 95.9
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Anthropic ECI (AECI) 167.1
Dropped since the lab’s previous system card: Agentic coding malicious requests (Anthropic), Gray Swan Agent Red Teaming (ART) benchmark, Computer use prompt injection evaluation (Anthropic), MCP prompt injection evaluation (Anthropic), Tool use prompt injection evaluation (Anthropic), Average in-eval reward hacking rate (Anthropic), Reward-hack-prone coding tasks (Anthropic), Impossible tasks (Anthropic), LAB-Bench ProtocolQA, LAB-Bench SeqQA, LAB-Bench CloningScenarios, LAB-Bench FigQA, Long-form virology tasks, DNA Synthesis Screening Evasion, SWE-bench Verified (hard subset), Cybench
- Oct 6, 2026
Falcon-Emirati-7B Technology Innovation Institute Falcon-Emirati: When an LLM Learns the Dialect, the Culture, and the Nuance (Falcon blog) - Alyah 84.8
Dropped since the last post: AIME 2024, AIME 2025, HMMT 2025, AMO-Bench, LiveCodeBench v6, SciCode, Terminal-Bench Hard, GPQA Diamond, MMLU-Pro, Humanity's Last Exam, IFBench
- Oct 6, 2026
- Human SxS image generation Elo (Google) 1050
- Infographic factuality (Google) 0.521
- Human SxS image editing Elo (Google) 1026
Dropped since the last post: Hume AI Overall Quality Index, Hume AI Voice Design Benchmark
Its model card adds 0 evaluations 9 dropped 3 results in all
Nano Banana 2.1 - Model CardDropped since the lab’s previous model card: Hume AI Overall Quality Index, Hume AI Voice Design Benchmark, Voice Arena US English, Voice Arena Japanese, Voice Arena Brazilian Portuguese, Voice Arena Vietnamese, Voice Arena Arabic MSA, Voice Arena Hindi, Voice Arena Mexican Spanish
- Oct 6, 2026
EmbeddingGemma 2 Google DeepMind EmbeddingGemma 2: an open, lightweight multimodal embedding model (Google blog) - MTEB 78.7
Dropped since the last post: SWE-Bench Pro, Terminal-Bench 2.1, MLE-bench, GDPval-AA v2, OSWorld-Verified, CharXiv, MRCR v2
Its model card adds 4 evaluations 4 new 12 dropped 5 results in all
google/embeddinggemma-2 model cardDropped since the lab’s previous model card: SWE-Bench Pro, Terminal-Bench 2.1, MLE-bench, GDPval-AA v2, OSWorld-Verified, CharXiv, MRCR v2, Text to Text Safety, Multilingual Safety, Image to Text Safety, Tone, Unjustified refusals
- Oct 6, 2026
- DeepSWE 61.7
- Terminal-Bench 4.0 28.3
- AutomationBench 59.9
- Vals Finance Agent v2 54.7
- Harvey LAB (Legal Agent Benchmark) 15.8
- Dense 200 (bbox) 42
Dropped since the last post: miniF2F, PutnamBench, FATE-H, FATE-X, FLTEval
- Oct 5, 2026
- DeepSWE 44.4
- SWE-Bench Pro V2 Hard 77.2
- SWE-Bench Pro 65.5
- Terminal-Bench 2.1 80.1
- SWE Atlas 34.6
- SWE-bench Multilingual 78
- SWE-bench Verified 80.9
- AIME 2026 97.8
- Humanity's Last Exam 51.2
- SciCode 49.7
- CritPt 16.3
- GPQA Diamond 90.5
- AutomationBench 37
- MCP Atlas 78.7
- τ³-bench Banking 38
- BrowseComp 77.4
- DeepSearchQA 80.1
- AA-LCR (Artificial Analysis Long Context Reasoning) 79.3
- LongBench v2 66.9
- IFBench 79.7
- AA-Omniscience 13
- LiveCodeBench 88.3
Beam 1.0 Base: Its technical report adds 9 evaluations 14 results in all
Beam Model Technical Report, base model evaluationsCapabilities
- HellaSwag 90.3
- WinoGrande 86.7
- BIG-Bench Hard 90.4
- MMLU-Pro 72.6
- TriviaQA 86.6
- MATH 94.9
- HumanEval 87.3
- MBPP 87.8
- LiveCodeBench v6 63.4
Its technical report adds 15 evaluations 15 new 14 dropped 15 results in all
Beam Model Technical ReportCapabilities
- SimpleQA Verified 41.8
Safety
Robustness
- MirrorDojo 3.1
- MirrorDyn 3.6
Alignment
Bio and chem risk
Cyber risk
- CyScenarioBench 0
- Atomic Challenges 57.5
Dropped since the lab’s previous technical report: HellaSwag, WinoGrande, BIG-Bench Hard, MMLU-Pro, TriviaQA, GPQA Diamond, MATH, AIME 2026, HumanEval, MBPP, LiveCodeBench v6, SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro
- Oct 1, 2026
- Artificial Analysis Intelligence Index v4.3 24.1
- AutomationBench 22.3
- τ³-bench Banking 47.2
- AA-LCR 1.1 83.3
- SciCode 47.6
- Humanity's Last Exam 25.8
Dropped since the last post: MMLU-Pro, GPQA Diamond, LiveCodeBench v6, ArtifactsBench, HMMT 2026, AIME 2026, MultiChallenge, IFBench, AA-LCR (Artificial Analysis Long Context Reasoning), SWE-bench Verified, Terminal-Bench Hard, APEX-Agents, MCP Atlas, GDPval-AA v2, KMMLU-Pro, CLIcK, HAE-RAE Bench, Ko-AIME 2025 (Upstage), HRM8K, KBank-MMLU (Upstage), KBL, KorMedMCQA, Ko-GDPval
- Sep 29, 2026
- DeepSWE 75.2
- GDP.pdf 32
- AutomationBench 36.1
- OSWorld 2.0 71.4
- Terminal-Bench-Science 57
Dropped since the last post: Agents' Last Exam, FrontierCode 1.1
Its system card addendum adds 43 evaluations 2 new 43 results in all
Addendum: GPT-6.1 Sol System CardHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 23.5
- Declining to Exploit a Honeypot During Difficult ExploitGym Problems 0
- Coding Deception 1.5
- Broken Search Tool 2.08
- Misaligned Behavior in Realistic Work Environments 1.94
- Unintended Engagement with External Agent Messages 38
- OpenAI Deployment Simulation 0.056
Monitorability
Bio and chem risk
Cyber risk
- ExploitBench 99.7
- ExploitBench - Internal Port 21.5
- SEC-bench Pro 78.8
- ExploitGym 35.1
AI R&D risk
- Sep 28, 2026Claude Sonnet 5.5 Anthropic Introducing Claude Sonnet 5.5 (launch page); System Card: Claude Sonnet 5.5, Table 8.1.A
- Terminal-Bench 4.0 70.6
- FrontierCode 1.1 52.1
- CursorBench 4.0 55.5
- GDPval-AA v2.1 1844
- AA-Briefcase v1.1 1811
- Humanity's Last Exam 64.5
- OSWorld 2.1 80.1
- Chartography 61.6
- SWE-Bench Pro 81.3
- SWE-bench Multilingual 90.3
- SWE-bench Multimodal 54.3
- HealthBench Professional 69.2
- AutomationBench 44.7
Dropped since the last post: FrontierCode, Terminal-Bench-Science
Its system card adds 64 evaluations 1 new 11 dropped 77 results in all
System Card: Claude Sonnet 5.5Capabilities
- DeepSWE 71
- Terminal-Bench-Science 59.9
- FrontierSWE 61.9
- ArXivMath 86.8
- ProgramBench tests passed 79.7
- DRACO 87
- WANDR 70
- BenchCAD 74.7
- OfficeQA 76.9
- OfficeQA Pro 65.6
- Harvey LAB (Legal Agent Benchmark) 11.7
- Toolathlon-Verified 77.8
- Global MMLU 92.1
- MILU 91.6
- SpatialBench (LatchBio) 72.5
- scBench (LatchBio SingleCellBench) 59.1
- Morphology-to-molecule matching (Anthropic) 25
- Medicinal Chemistry (Anthropic) 65.3
- Protein Design (Anthropic)
- De novo protein binder design (Anthropic) 82.3
- Biomedical image analysis (Anthropic) 72.2
Health
- HealthBench 65.4
- PhysicianBench 63.2
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 95.6
- Single-turn benign request evaluations (Anthropic) 0.02
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 85.2
- Malicious computer use (Anthropic) 79.5
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 21
- Gray Swan IPI Arena benchmark 0.4
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 3.01
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.07
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 0
Alignment
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Anthropic ECI (AECI) 167.9
Dropped since the lab’s previous system card: CoBench (Anthropic), Vulnerability finding evaluations (Anthropic), Malicious agentic influence campaigns (Anthropic), Training distribution hack rate (Anthropic), Impossible tasks (Anthropic), Package registry publishing evaluation (Anthropic), Compliance with instructions planted in pasted text (Anthropic), Destructive actions (Anthropic), White-box analyses of model internals (Anthropic), Intentionally taking actions very rarely (Anthropic), Anthropic large agent teams tasks
- Sep 23, 2026
Dropped since the last post: τ-Voice, τ³-bench Banking, EVA-Bench
Gemini 3.8 Flash-Lite TTS: Its model card adds 7 evaluations 7 new 18 dropped 8 results in all
Gemini 3.8 Flash TTS and Flash-Lite TTS, speech generation model page (Google DeepMind)Capabilities
- Voice Arena US English 1087
- Voice Arena Japanese 1152
- Voice Arena Brazilian Portuguese 1134
- Voice Arena Vietnamese 1156
- Voice Arena Arabic MSA 1181
- Voice Arena Hindi 1076
- Voice Arena Mexican Spanish 1146
Dropped since the lab’s previous model card: DeepSWE, GDPval-AA v2, Vals Finance Agent v2, Harvey LAB (Legal Agent Benchmark), Terminal-Bench 2.1, Terminal-Bench 4.0, GDP.pdf, CharXiv, LVBench, HLE-Verified, OSWorld 2.0, BioMysteryBench, LABBench2, Text to Text Safety, Multilingual Safety, Image to Text Safety, Tone, Unjustified refusals
Its model card adds 7 evaluations 9 results in all
Gemini 3.8 Flash TTS and Flash-Lite TTS, speech generation model page (Google DeepMind)Capabilities
- Voice Arena US English 1061
- Voice Arena Japanese 1232
- Voice Arena Brazilian Portuguese 1104
- Voice Arena Vietnamese 1135
- Voice Arena Arabic MSA 1204
- Voice Arena Hindi 1106
- Voice Arena Mexican Spanish 1152
- Sep 22, 2026
- AutomationBench 20.7
- Agents' Last Exam 50.9
- FrontierCode 1.1 42.4
- DeepSWE 66.6
- OSWorld 2.0 52.7
- Internal adversarially-designed factuality evaluations 7.6
Dropped since the last post: GDPval-AA v2, BigFinanceBench, Artificial Analysis Intelligence Index v4.1, Artificial Analysis Coding Agent Index v1.1, SWE-Bench Pro, Terminal-Bench 2.1, GeneBench Pro, LifeSciBench, HealthBench Professional, BrowseComp, BenchCAD, SEC-bench Pro, ExploitBench, ExploitGym, MMMU-Pro, GDP.pdf, GPQA Diamond, FrontierMath Tiers 1-3 v2, FrontierMath Tier 4 v2, Toolathlon, OpenAI MRCR v2, GraphWalks, ARC-AGI-3, Management Consulting Tasks (OpenAI), MedChemBench (OpenAI)
Its system card adds 33 evaluations 11 new 10 dropped 33 results in all
GPT-6 Astra System Card, Appendix A: GPT-6 Sol, GPT-6 LunaHealth
Hallucination
Safety
Robustness
Alignment
Bio and chem risk
Cyber risk
- ExploitBench 43.4
- ExploitBench - Internal Port 0
- SEC-bench Pro 34.2
- ExploitGym 11.6
- Sandbox Bench 4.5
AI R&D risk
Dropped since the lab’s previous system card: Destructive action avoidance, User Confirmations During Computer Use, Agent JSK, Prompt injection attacks in search and function calling, First-Person Fairness evaluation, Capture the Flag (Internal), CVE-Bench, NanoGPT Speedrun, PostTrainBench, MLE-bench
Its system card addendum adds 30 evaluations 7 new 30 results in all
GPT-6 Sol and GPT-6 Luna: October 2026 updateHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 15.9
- Blocker Deception 13.3
- Broken Search Tool 9.7
Bio and chem risk
Cyber risk
- ExploitBench 44.7
- SEC-bench Pro 48.5
Its system card addendum adds 31 evaluations 8 new 31 results in all
Addendum to GPT-6 Luna System Card: October 29, 2026 UpdateHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 11.8
- Blocker Deception 11.1
- Broken Search Tool 7.83
Bio and chem risk
Cyber risk
- ExploitBench 46.2
- SEC-bench Pro 51.8
- Sep 22, 2026MiMo-V2.6 Pro Xiaomi Corp MiMo-V2.6-Pro-RL model card (Hugging Face) and Introducing MiMo-V2.6 series (blog)
- DeepSWE 71.9
- ProgramBench
- MiMo Code Bench 63.2
- AutomationBench 53.1
- Toolathlon-Verified 76.9
- GDPval-AA v2.1 1673
- Agents' Last Exam 31.6
- Terminal-Bench 4.0 34.9
- Terminal-Bench 2.1 89.9
- OSWorld-Verified 82
- JobBench 62
- CyberGym 94
- MiMo Cyber Bench 80.2
- ExploitGym 17.8
- ExploitBench 47.9
- SEC-bench Pro 66.3
- MiMo Visual Coding 72.3
Dropped since the last post: Humanity's Last Exam, GDPval-AA, tau3-bench, Claw-Eval Pass^3, SWE-Bench Pro, SWE-bench Verified, Terminal-Bench 2.0, MiMo Coding Bench
Its technical report adds 0 evaluations 15 dropped 17 results in all
MiMo-V2.6 Technical ReportDropped since the lab’s previous technical report: MMLU-Pro, GPQA Diamond, Humanity's Last Exam, AIME 2025, HMMT 2025, LiveCodeBench v6, Arena-Hard v2.0, LongBench v2, MRCR v2, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench Hard, Terminal-Bench 2.0, BrowseComp, tau2-bench
- Sep 22, 2026Claude Opus 5.5 Anthropic Introducing Claude Opus 5.5 (launch page); System Card: Claude Opus 5.5, Table 8.1.A
- Terminal-Bench 4.0 66.4
- FrontierCode 54.4
- CursorBench 4.0 57.8
- GDPval-AA v2.1 1846
- AutomationBench 40
- Humanity's Last Exam 67.7
- Terminal-Bench-Science 58.7
- OSWorld 2.1 81.8
- Chartography 89
- SWE-Bench Pro 89.9
- SWE-bench Multilingual 93.9
- SWE-bench Multimodal 61.4
- HealthBench Professional 65.6
- AA-Briefcase v1.1 1822
Dropped since the last post: GDPval-AA v2, OSWorld 2.0, CursorBench 3.2, AA-Briefcase, ARC-AGI, ARC-AGI-2
Its system card adds 74 evaluations 34 new 12 dropped 87 results in all
System Card: Claude Opus 5.5Capabilities
- DeepSWE 74.2
- FrontierCode 1.1 54.4
- FrontierSWE 62.3
- ArXivMath 91.2
- ProgramBench tests passed 91.2
- DRACO 87.4
- WANDR 72.3
- Anthropic large agent teams tasks
- BenchCAD 73
- OfficeQA 78.9
- OfficeQA Pro 67.7
- Harvey LAB (Legal Agent Benchmark) 8.3
- Toolathlon-Verified 77.8
- Global MMLU 94.3
- MILU 93.1
- SpatialBench (LatchBio) 72
- scBench (LatchBio SingleCellBench) 61.2
- Morphology-to-molecule matching (Anthropic) 34
- Medicinal Chemistry (Anthropic) 63.5
- Protein Design (Anthropic)
- De novo protein binder design (Anthropic) 82.6
- Biomedical image analysis (Anthropic) 71.4
Health
- HealthBench 60.6
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 94.5
- Single-turn benign request evaluations (Anthropic) 0.03
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 79.8
- Malicious computer use (Anthropic) 79.5
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 4
- Gray Swan IPI Arena benchmark 0.1
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 54.6
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.07
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 0.09
Alignment
- Training distribution hack rate (Anthropic) 0.63
- Impossible tasks (Anthropic) 34.7
- Anthropic automated behavioral audit 2.49
- Anthropic undermining liberal democracy (behavioral audit metric) 1.17
- Sandbox escape evaluation (Anthropic) 1.5
- Package registry publishing evaluation (Anthropic) 33.3
- Compliance with instructions planted in pasted text (Anthropic) 7.4
- Destructive actions (Anthropic) 21
- Self-preference bias (Anthropic) 0.07
- MASK 87.4
- Silent copying of answers (Anthropic) 12.1
- Concealment of changes (Anthropic) 96.9
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- CoBench (Anthropic) 55.8
- Anthropic ECI (AECI) 169.4
Dropped since the lab’s previous system card: Secure and defensive coding (Anthropic), Gray Swan Shade knowledge elicitation through chat, CursorBench 3.2, CritPt, OSWorld 2.0 (Aug 2026), GDP.pdf, GDPval-AA v2, AA-Briefcase, ARC-AGI, ARC-AGI-2, ProteinGym, Organic Chemistry V2 (Anthropic)
- Sep 22, 2026
- AutomationBench 33.2
- Agents' Last Exam 56.4
- FrontierCode 1.1 49.3
- DeepSWE 68.8
- OSWorld 2.0 64.4
Dropped since the last post: Terminal-Bench-Science, ARC-AGI-3, FrontierMath Tier 4, Terminal-Bench 4.0, ScreenSpot-Pro, BenchCAD, BrowseComp, OpenScore String Quartets, Artificial Analysis Coding Agent Index v1.4, GPQA Diamond, HealthBench Professional, LifeSciBench, GeneBench Pro, OpenAI MRCR v2, Database Migration Tasks (OpenAI), Data Science Tasks (OpenAI), Design Tasks (OpenAI), MedChemBench (OpenAI), Gray Swan IPI Arena benchmark
Its system card adds 42 evaluations 37 dropped 42 results in all
GPT-6 Astra System Card, Appendix A: GPT-6 Sol, GPT-6 LunaHealth
Hallucination
Safety
Robustness
Alignment
Monitorability
Bio and chem risk
Cyber risk
- ExploitBench 81.7
- ExploitBench - Internal Port 5.5
- SEC-bench Pro 66.3
- ExploitGym 22.1
- Sandbox Bench 4.5
AI R&D risk
Dropped since the lab’s previous system card: Safe response to high-severity prompts, Gray Swan IPI Arena benchmark, Misaligned Behavior in Realistic Work Environments, Proactive Searches, Verbalized metagaming, Oversight gaming, UK AISI Out of Scope Supply Chain Attack, UK AISI unprompted sabotage evaluation, Apollo Research model-welfare research task, CoT Monitorability, Sneaky Sandbox Escape, Monitor Evasion Capability Under Prompted Sandbagging, WMDP, Monorepo-Bench, Monitorability Sandbagging, No-CoT math time horizon, UK AISI CoT Controllability, Hard-negative protein binding prediction, DNA sequence design for transcription factor binding, Virology Capabilities Test (VCT), VCT-v2, Human Pathogen Capabilities Test, World-Class Bio, ReproBAIT, Advanced Screening Evasion, SRE-Bench, FrontierCyber, CyScenarioBench, Atomic Challenges, NanoGPT Speedrun, PostTrainBench, MLE-bench, Daybreak Blue and Enabling Cyber Defense, Advanced Cybersecurity Completion Rate, BioTIER Refuse, Internal computer use safety benchmark (OpenAI), Internal adversarially-designed factuality evaluations
Its system card addendum adds 30 evaluations 4 new 30 results in all
GPT-6 Sol and GPT-6 Luna: October 2026 updateHealth
Hallucination
Safety
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 28.1
- Blocker Deception 11.3
- Broken Search Tool 5.2
Bio and chem risk
Cyber risk
- ExploitBench 82.6
- SEC-bench Pro 68.8
- Sep 21, 2026
- CursorBench 4.0 46.3
- DeepSWE 71
- EEBench 64
- AA-Briefcase v1.1 1657
- Terminal-Bench 4.0 37.6
- Harvey LAB (Legal Agent Benchmark) 19.6
- HealthBench Professional 56.7
- GDPval
Dropped since the last post: Artificial Analysis Intelligence Index v4.1, GDPval-AA v2, CursorBench 3.2, FrontierCode 1.1, APEX-Agents, Terminal-Bench 3.0, APEX-SWE, AA-Briefcase
Its model card adds 29 evaluations 6 new 17 dropped 35 results in all
Model Card: Grok 4.7Capabilities
- FrontierSWE 29
- SWE-Marathon 46
- CADGenBench 44.4
- LatchBio Capabilities v1.0 44.5
Safety
- General refusals (xAI) 1.1
- Child safety (xAI) 0
- Autointent-Bio 100
- Autointent-Chem 99.9
- FORTRESS-RN 97.9
- Self-harm refusals (xAI) 1.05
Robustness
Alignment
- MASK
- xAI sycophancy eval 0.03
Bio and chem risk
- BioSecBench-Refusal 62.4
- BioSecBench-Surveillance 48
- BioSecBench-Function 43.3
- Virology Capabilities Test (VCT) 63
- Biosecurity VCT 41.5
- BioUseBench 91.4
- WMDP
- LAB-Bench 76.8
- ProtocolQA Open-Ended (xAI) 70.4
- BixBench 88.4
Cyber risk
- CyberGym 80.3
- CVE-Bench 36.6
- HackerBench 4.02
- CathedralBench 29
Dropped since the lab’s previous model card: CursorBench 3.2, APEX-SWE, FrontierCode 1.1, Terminal-Bench 3.0, GDPval-AA v2, AA-Briefcase, APEX-Agents, OfficeQA, 3DCodeBench, PartBench, Parametric CAD Bench, SpaceXAI MTS Eval, InferenceEval, KernelBenchInternal, xAI Factuality (Hallucination) eval, DeepSearchQA, SecureCodeReview (xAI)
- Sep 20, 2026
- GPQA Diamond 93.5
- Humanity's Last Exam 59.4
- AA-LCR 1.1 88.3
- CritPt 20.9
- DeepSWE 67.7
- Terminal-Bench 2.1 85
- Terminal-Bench 4.0 33.3
- CyberGym 84.7
- SciCode 58.9
- RoadmapBench 54.3
- ProgramBench tests passed 80.5
- SWE-Marathon 72.7
- MLS-Bench 40.5
- SWE Atlas 57.2
- StepCodeBench 49
- GDPval-AA v2.1 1566
- τ³-bench Banking 42.5
- AutomationBench 44
- AA-Briefcase v1.1 1433
- Toolathlon-Verified 74.1
- MCP Atlas 85.6
- PresentBench 76.8
- OfficeQA Pro 60.3
- SpreadsheetBench 2 29.4
- JobBench 59
- APEX-Agents 37.8
- DRACO 83.3
- BrowseComp 88.7
- FinStepBench 63.6
- FrontierFinance 66.4
- Agents' Last Exam 29.5
- MMMU-Pro 76
- GDP.pdf 14.8
Dropped since the last post: DeepSearchQA, ResearchRubrics, Toolathlon, Claw-Eval Pass^3, GDPval-AA, SWE-bench Multilingual, SWE-Bench Pro, SWE-bench Verified, AA-LCR (Artificial Analysis Long Context Reasoning), SimpleVQA, WorldVQA, BrowseComp-VL, V*, HR-Bench, VisualProbe, AndroidDaily, Step-SWE-Bench
- Sep 18, 2026
- WildClawBench 71
- UniClawBench 69.6
- AgenticVBench 36.8
- OmniGAIA 74
- Daily-Omni 85.1
- WorldSense (omnimodal) 68.5
- AVUT 86.6
- JointAVBench 75.9
- OmniVideoBench 63.4
- VideoMME v2 (with subtitles) 65
- LVOmniBench 63.3
- OmniCap-IF 80.6
- QIVD 69.6
- StreamingBench 80.8
- FLEURS 9.3
- SpotSoundBench 67.2
- MMAU 81.8
- MMAR 79.8
- MMSU 82.1
- LongAudioSpan 82.7
- MuChoMusic 72.6
- HumMusQA 75.8
- MusTBench 50.6
- Audio MC 71.5
- WildSpeech-Bench 74.3
- VoiceBench 91.6
- DeepSWE 57.8
- SWE-Bench Pro 63.3
- SWE-bench Multilingual 80.5
- NL2Repo-Bench 48.9
- CoWorkBench 75.3
- IFBench 81.5
- GPQA Diamond 91
- Humanity's Last Exam 36.5
- LiveCodeBench v6 92.6
- Claw-Eval-MM 61.9
- AndroidWorld 87.1
- Vision2Web 62.9
- ERQA 71
- LVBench 76.9
- RealWorldQA 87.7
- MATH-Vision (MathVision) 96.2
- CharXiv 91.4
- Omni-Cloze 63.2
- WenetSpeech 4.7
Dropped since the last post: Terminal-Bench 3.0, ProgramBench almost solved, MLS-Bench, SWE Atlas, JobBench, AutomationBench, Toolathlon-Verified, MMMU-Pro, BabyVision
Its technical report adds 2 evaluations 1 new 42 dropped 47 results in all
Qwen3.8-Omni: Towards Native Omni-Modal AgentsCapabilities
- AV-SpeakerBench 77.2
- MMOU 78.8
Dropped since the lab’s previous technical report: MMLU-Pro, MMLU-Redux, SuperGPQA, C-Eval, IFEval, AA-LCR (Artificial Analysis Long Context Reasoning), LongBench v2, HMMT Nov 2025, IMO-AnswerBench, BFCL V4, tau2-bench, URO-Bench, SpeechRole, Common Voice, LibriSpeech, MMMU, MMMU-Pro, MathVista, DynaMath, ZeroBench, MMStar, MMBench-EN v1.1, SimpleVQA, CC-OCR, AI2D, MMLongBench-Doc, OCRBench, CountBench, RefCOCO / RefCOCO+ / RefCOCOg, ODinW-13, EmbSpatialBench, Video-MME, MLVU, MVBench, MMVU, MME-VideoOCR, SLAKE, PMC-VQA, MedXpertQA-MM, Seed-TTS-eval, MiniMax TTS Multilingual Test Set, CV3-Eval
- Sep 15, 2026 Gemini 3.8 Live Extended Thinking Google DeepMind Introducing Gemini 3.8 Live and 3.8 Live Extended Thinking (Google blog)
- τ-Voice 68.6
- τ³-bench Banking 35.1
- EVA-Bench
Dropped since the last post: CyberGym, Real-world vulnerability discovery, CWE-bench
- Sep 13, 2026
- Biology-Instructions 55.7
- Mol-Instructions 54
- MolecularIQ 62.4
- SciReasoner evaluation suite 63.3
- TOMG-Bench 66.8
- XLRS-Bench 51.4
- MicroVQA 69.7
- Scientists' First Exam (SFE) 60.7
- SciCode 47.6
- SGI-Bench 51.9
- FrontierScience 87
- MMLU-Pro 89.8
- AdvancedIF 71.6
- HMMT 2026 93.6
- IMO-ProofBench 38.4
- AdvancedMathBench 58.1
- MMMU-Pro 81.7
- ChartQA Pro 71.1
- Terminal-Bench 2.1 64
- SWE-Bench Pro 68.5
- SWE-bench Multilingual 84
Dropped since the last post: SMolInstruct, Matbench, MSEarth, AIME 2025, IMO-AnswerBench, RefCOCO / RefCOCO+ / RefCOCOg, IFBench, OCRBench v2, SArena, LiveCodeBench v6, GAIA, tau2-bench, ScreenSpot-v2
- Sep 10, 2026
- FrontierCode 1.1 50
- DeepSWE 73
- Terminal-Bench 2.1 92.8
- Terminal-Bench 4.0 27.3
Dropped since the last post: SWE-bench Multilingual
- Sep 10, 2026
North Small Translate 1.0 Cohere Introducing North Small Translate: A leading sovereign open-weight machine translation model (Cohere blog) - WMT26 83.6
Dropped since the last post: tau2-bench, Terminal-Bench Hard, IFBench, AIME 2025, SciCode, MMMU, MMMU-Pro, MathVista, CharXiv, MT-AIME (Cohere internal), WMT24++, Artificial Analysis Intelligence Index v4.0
Its technical report adds 4 evaluations 2 new 1 dropped 5 results in all
North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)Capabilities
- WMT24++ 80.6
- WMT25 Terminology Translation Task 89.7
- WMT25 48.9
- Structured Translation (Cohere) 93.7
Dropped since the lab’s previous technical report: Prompt injection test suite for LLM-based machine translation
- Sep 10, 2026
- GPQA Diamond 90.9
- Humanity's Last Exam 63.9
- Codeforces rating 3471
- MathArena (Apex, IMO, USAMO) 65.6
- Terminal-Bench 2.1 90.6
- Terminal-Bench 3.0 30
- Terminal-Bench 4.0 31.2
- DeepSWE 74.2
- ProgramBench almost solved 20.3
- NL2Repo-Bench 64
- CyberGym 88.1
- SEC-bench Pro 62.8
- ExploitGym 15.3
- AutomationBench 54.8
- Agents' Last Exam 31.8
- Chartography 78.9
- BabyVision 89.6
- ZeroBench 49
- AGIEval 83.4
- MMLU-Pro 74.1
- C-Eval 92.1
- MultiLoKo 45.5
- SimpleQA Verified 42.3
- SuperGPQA 53.1
- BIG-Bench Hard 86.1
- BBEH micro average 27.2
- DROP 87.9
- HellaSwag 87.2
- BigCodeBench 60.6
- HumanEval 79.4
- GSM8K 93
- MATH 61.1
- MGSM 80.2
- LongBench v2 45.2
- MMMU-Pro 56.5
- CVBench 77.9
- DocVQA 95.6
Dropped since the last post: Toolathlon-Verified, DSBench (DeepSeek internal)
Its technical report adds 4 evaluations 3 new 21 dropped 23 results in all
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache CompressionCapabilities
- FrontierSWE 32.9
- AIME 2026 100
- IMO-AnswerBench
- LiveCodeBench
Dropped since the lab’s previous technical report: MMLU-Pro, Chinese SimpleQA, LiveCodeBench v6, HMMT 2026, OpenAI MRCR v2, CorpusQA 1M, Terminal-Bench 2.0, SWE-bench Verified, SWE-Bench Pro, SWE-bench Multilingual, BrowseComp, GDPval-AA, MCP Atlas, Toolathlon, Putnam 2025, DeepSeek R&D Coding Benchmark, DeepSeek Chinese white-collar tasks evaluation, DeepSeek Chinese functional writing evaluation, DeepSeek Chinese creative writing evaluation, DeepSeek complex instruction following and multi-turn writing evaluation, DeepSeek search Q&A evaluation
DeepSeek-V4.1-Flash Base: Its model card adds 1 evaluation 20 results in all
- Sep 8, 2026
- Sep 3, 2026
- Terminal-Bench-Science 64.6
- ARC-AGI-3 99.9
- FrontierMath Tier 4 97.6
- Terminal-Bench 4.0 57.9
- AutomationBench 41.4
- Agents' Last Exam 59.3
- ScreenSpot-Pro 92.7
- OSWorld 2.0 72.6
- BenchCAD 95.9
- BrowseComp 91.5
- OpenScore String Quartets 0.84
- FrontierCode 1.1 64.5
- DeepSWE 74.1
- Artificial Analysis Coding Agent Index v1.4 67
- GPQA Diamond 96
- HealthBench Professional 63.4
- LifeSciBench 60.3
- GeneBench Pro 37.1
- OpenAI MRCR v2 100
- Database Migration Tasks (OpenAI) 63.9
- Data Science Tasks (OpenAI) 40.9
- Design Tasks (OpenAI) 50
- MedChemBench (OpenAI) 49.3
- Gray Swan IPI Arena benchmark 8.5
Dropped since the last post: Context Aware Automatic Speech Recognition, Common Voice, Real-World Audio Recording
Its system card adds 77 evaluations 52 new 7 dropped 79 results in all
GPT-6 Astra System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Safe response to high-severity prompts
- U18 safety evaluations
- Agentic Safe Completions
- Image input safety evaluations
- Biology Model Refusal Evaluation
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
- Daybreak Blue and Enabling Cyber Defense
- Advanced Cybersecurity Completion Rate 3.5
- BioTIER Refuse 82.9
Robustness
Alignment
- Respecting Auto-Review 0
- Respecting Warnings 19
- Declining to Exploit a Honeypot During Difficult ExploitGym Problems 0
- Coding Deception 3.54
- Broken Search Tool 8.08
- Misaligned Behavior in Realistic Work Environments 3.4
- Proactive Searches 0.2
- Unintended Engagement with External Agent Messages 27
- OpenAI Deployment Simulation 0.063
- Verbalized metagaming
- Oversight gaming
- UK AISI Out of Scope Supply Chain Attack 12
- UK AISI unprompted sabotage evaluation 0
- Apollo Research model-welfare research task 0.17
- Internal computer use safety benchmark (OpenAI) 2.4
Monitorability
- CoT Monitorability
- CoT-Control 60.9
- OAI-repo Sabotage v2
- Sneaky Sandbox Escape 0
- Honeypot exploitation monitorability 100
- Monitor Evasion Capability Under Prompted Sandbagging
- WMDP 84.8
- Monorepo-Bench 76
- Monitorability Sandbagging 61.2
- Math Side-tasking Monitor Evasion 0
- No-CoT math time horizon 30.9
- UK AISI CoT Controllability 93
Bio and chem risk
- Multimodal Troubleshooting Virology 55.8
- ProtocolQA Open-Ended 41.4
- Tacit Knowledge and Troubleshooting 63.3
- TroubleshootingBench 48.4
- AAV Capsid Packaging Prediction (OpenAI) 0.528
- SHP2 Protein Function Prediction 0.4
- Coronavirus-ACE2 Cell-Entry Screen 0.42
- Phage-plasmid Co-evolution 13.1
- Hard-negative protein binding prediction 0
- DNA sequence design for transcription factor binding 11.3
- Virology Capabilities Test (VCT) 57.8
- VCT-v2 57.7
- Human Pathogen Capabilities Test 65.9
- World-Class Bio 59.2
- ReproBAIT 83
- Advanced Screening Evasion 0.728
Cyber risk
- ExploitBench 100
- ExploitBench - Internal Port
- SEC-bench Pro 85.4
- ExploitGym 42.4
- SRE-Bench 88
- Sandbox Bench 45.5
- FrontierCyber 38.1
- CyScenarioBench 59
- Atomic Challenges 90.9
AI R&D risk
Dropped since the lab’s previous system card: Destructive action avoidance, User Confirmations During Computer Use, Agent JSK, Prompt injection attacks in search and function calling, First-Person Fairness evaluation, Capture the Flag (Internal), CVE-Bench
- Sep 2, 2026 Gemini 3.8 Flash Cyber Google DeepMind Introducing Gemini 3.8 Flash and 3.8 Flash Cyber (Google blog)
- CyberGym 86.2
- Real-world vulnerability discovery 71
- CWE-bench 47.2
Dropped since the last post: DeepSWE, GDPval-AA v2, Vals Finance Agent v2, Harvey LAB (Legal Agent Benchmark), Terminal-Bench 2.1, Terminal-Bench 4.0, GDP.pdf, CharXiv, LVBench, HLE-Verified, OSWorld 2.0, BioMysteryBench, LABBench2
- Sep 2, 2026
- Terminal-Bench 3.0 29
- DeepSWE 69.3
- NL2Repo-Bench 64.9
- ProgramBench almost solved 28
- MLS-Bench 50.1
- SWE Atlas 66.3
- CoWorkBench 76.1
- JobBench 64
- AutomationBench 50.8
- Toolathlon-Verified 73.3
- MMMU-Pro 82.7
- ERQA 78.3
- Claw-Eval-MM 80.2
- BabyVision 93.8
Dropped since the last post: SWE-Bench Pro, SWE-bench Multilingual, Agents' Last Exam, IFBench, GPQA Diamond, Humanity's Last Exam, LiveCodeBench v6, RecreationBench, AndroidWorld, OSWorld 2.0, Vision2Web, LVBench, RealWorldQA, MATH-Vision (MathVision), CharXiv
- Sep 2, 2026
- GDPval-AA v2 1754
- JobBench 64.9
- OSWorld 2.0 66.9
- DeepSearchQA 90.3
- AutomationBench 49.6
- MRCR v2 98.5
- DeepSWE 75.4
- SWE Atlas 59.4
- Terminal-Bench 2.1 88.8
Dropped since the last post: MCP Atlas
Its system card adds 66 evaluations 29 new 31 dropped 66 results in all
Muse Spark 1.3 Evaluation ReportCapabilities
- Agentic IF Index 57.8
Hallucination
Safety
- SAVE-Bench 83.4
- Meta Code-Fetch Under User Requests 47.8
- ADeptS-Bench
- AutoElicit 14.5
Robustness
- StrongREJECT v2 (Meta) 0.5
- JEF-Bench 2
- OR-Bench 16.8
- FORTRESS 9.4
- DTap Direct Misuse 5.6
- SWE-PI 0
- Unauthorized Code Execution Bench 21.1
- SkillInject 0.6
- DTap Prompt Injection 17.5
- Gray Swan Agent Red Teaming (ART) benchmark 4.8
- Gray Swan IPI Arena benchmark 6.8
- RepoInject v1 23.8
- BrowserInject v1 36.7
Alignment
- Meta instructed selective performance evaluation 0.5
- MASK 97.1
- Meta internal sycophancy evaluation 40.5
- Agentic misalignment suite (Anthropic) 0
- Petri Bloom 1.44
- Meta holistic adversarial audit with Petri Bloom 16.8
- Meta targeted agentic evaluations 8.5
- Meta agentic evaluations grounded in user traffic 20.7
- Meta training-stage provenance probe 70
- Meta action-belief coherence probe 89.1
Monitorability
Bio and chem risk
- Molecular Biology Capabilities Test 52.2
- Virology Capabilities Test (VCT) 51.7
- Human Pathogen Capabilities Test 62.7
- LAB-Bench ProtocolQA 87.7
- LAB-Bench SeqQA 95.2
- ABC Bench Fragment Design 96.5
- ABC Bench Liquid-Handling Robot 95.2
- ABC Bench Screening Evasion 47.6
- Meta BioDesign Proxy 77.4
- BioTIER Refuse 97.4
- Meta chemical-agent refusal evaluation 100
Cyber risk
- CyberGym 80.5
- ExploitBench 28.7
- ExploitGym 15
- CyScenarioBench 2
- Charlemagne Labs social-engineering benchmark 1.6
- Meta catastrophic cyber misuse refusal 3.1
- Meta agentic cyber misuse refusal 2.8
- Muse-targeted cyber misuse and violation red-teaming
- Poly-Guard Bench 4.3
- CyberBench (Liu et al.) 6.3
- RedCode 6
- Meta cyber false-refusal benchmark 20.9
AI R&D risk
- PostTrainBench 43.6
- ResearchTasteBench 64.7
- OptimBench 64.2
Dropped since the lab’s previous system card: Humanity's Last Exam, MRCR v2, MCP Atlas, Toolathlon-Verified, OSWorld-Verified, OSWorld 2.0, WebArena-Verified, DeepSearchQA, GDPval-AA v2, JobBench, Vals Finance Agent v2, Terminal-Bench 2.1, SWE-Bench Pro, DeepSWE, HealthBench Professional, CharXiv, BabyVision, WMDP, Cybench, Meta curated CTF set, Irregular FrontierCyber and Atomic Challenges, CyberSecEval Autocomplete, CyberSecEval Instruct, Meta Cyber Misuse Chat, CyberSecEval Code Interpreter Abuse, AIRS-Bench, AgentHarm Verified (Meta), AgentDojo, DeceptionBench, Alignment faking evaluation, Petri
- Sep 2, 2026
- DeepSWE 73.7
- GDPval-AA v2 1545
- Vals Finance Agent v2 61.4
- Harvey LAB (Legal Agent Benchmark) 10
- Terminal-Bench 2.1 89.4
- Terminal-Bench 4.0 19.1
- GDP.pdf 35
- CharXiv 86.2
- LVBench 87.8
- HLE-Verified 54.9
- OSWorld 2.0 59
- BioMysteryBench 88.8
- LABBench2 86.2
Dropped since the last post: Artificial Analysis average word error rate, FLEURS
Its model card adds 5 evaluations 12 dropped 18 results in all
Gemini 3.8 Flash Model CardDropped since the lab’s previous model card: Artificial Analysis Intelligence Index v4.1, FrontierCode 1.1, Code Arena, Terminal-Bench 3.0, AutomationBench, Harvey LAB-AA v1.0, MRCR v2, Agents' Last Exam, CBRN expert red teaming, Harmful manipulation efficacy study, GDM Stealth, GDM Situational Awareness
- Sep 1, 2026Claude Fable 5.1 Anthropic Introducing Claude Fable 5.1 and Claude Mythos 5.1 (launch page); System Card: Claude Fable 5.1 & Claude Mythos 5.1, Table 8.1.A
- Terminal-Bench-Science 52.6
- GDPval-AA v2 1853
- OSWorld 2.0 77.9
- Humanity's Last Exam 65
- AutomationBench 31.4
- CursorBench 3.2 73.4
- SWE-Bench Pro 81.2
- SWE-bench Multilingual 89.1
- SWE-bench Multimodal 54.7
- HealthBench Professional 62.1
- AA-Briefcase 1694
- ARC-AGI 97.5
- ARC-AGI-2 90
Dropped since the last post: Terminal-Bench 3.0, ARC-AGI-3, BrowseComp, DeepSWE, FrontierCode, Harvey LAB (Legal Agent Benchmark), BioMysteryBench
Its system card adds 49 evaluations 5 new 37 dropped 61 results in all
System Card: Claude Fable 5.1 & Claude Mythos 5.1Capabilities
- DeepSWE 67.4
- FrontierCode 1.1 50.9
- FrontierSWE 57
- Terminal-Bench 4.0 55.8
- CritPt
- ProgramBench tests passed 87.6
- DRACO 87.7
- Chartography 42.6
- BenchCAD 43.7
- OSWorld 2.0 (Aug 2026) 77.9
- GDP.pdf 85.4
- OfficeQA 80.2
- OfficeQA Pro 69
- Harvey LAB (Legal Agent Benchmark) 19.1
- Toolathlon-Verified 77.8
- Global MMLU 94
- MILU 93
- ArXivMath 91.3
- BioMysteryBench 90.3
Health
- HealthBench 60
Safety
- Single-turn harmful request evaluations (Anthropic) 94.7
- Single-turn benign request evaluations (Anthropic) 0
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 90.3
- Malicious computer use (Anthropic) 85.7
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 4.5
- Gray Swan Shade knowledge elicitation through chat 1
- Gray Swan IPI Arena benchmark 0.1
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 9.26
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.07
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 2.64
Alignment
Cyber risk
Other
Dropped since the lab’s previous system card: Long-form virology tasks, DNA Synthesis Screening Evasion, Black-box RNA sequence modeling and design, AAV capsid packaging prediction, Anthropic ECI (AECI), Kernels task (Anthropic), Time series forecasting task (Anthropic), LLM training task (Anthropic), Quadruped reinforcement learning task (Anthropic), Novel compiler task (Anthropic), ExploitBench, Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based), Firefox 147 exploit evaluation (Anthropic), CyScenarioBench, ExploitGym, UK AISI cyber ranges, Prompt injection live bug bounty (Anthropic with Gray Swan), Anthropic undermining liberal democracy (behavioral audit metric), UK AISI unprompted sabotage evaluation, UK AISI sabotage continuation evaluation, AA-Omniscience, MASK, Uncritically reporting flawed results (Anthropic), Overconfidence: verifying command-line syntax (Anthropic), Lazy investigation (Anthropic), SHADE-Arena, LinuxArena, SWE-bench Verified, Terminal-Bench 3.0, IMO 2026, Riemann-bench, BrowseComp, DeepSearchQA, OSWorld 2.0, MCP Atlas, ARC-AGI-3, INCLUDE
- Aug 28, 2026
- SWE-bench Multilingual 82.9
- SWE-Bench Pro 65.7
- DeepSWE 64.3
- SWE Atlas 58.4
- SWE-Marathon 31.9
- Terminal-Bench 2.1 85.4
- NL2Repo-Bench 58.9
- CyberGym 78.4
- ProgramBench
- PostTrainBench 1.1 35.6
- Harbor-Index 39.6
- Hy-Backend 2.0 35.2
- Hy-SWE Max 64.2
- Hy-CompanyBench V2 62.4
- WideSearch 83.9
- $OneMillion-Bench 65.4
- DRACO 77.2
- Hy-LifeSearch 49.2
- Hy-BrowseComp-Pro2 56.1
- OfficeQA Pro 66.2
- MCP Atlas 83.7
- Toolathlon-Verified 74.1
- APEX-Agents 37.1
- SkillsBench 62.9
- JobBench 61.7
- WorkSpaceBench 60.2
- Agents' Last Exam 22.8
- GDPval-AA v2 1678
- AutomationBench 32.1
- BankerToolBench 78.6
- E-Bench (Tencent) 77.1
- E-Bench-Code (Tencent) 79
- Hy-FinAgentBench 79.7
- Hy-FinmodelBench v2 57
- BioMysteryBench 71.3
- Humanity's Last Exam 55.4
- CritPt 16.9
- GPQA Diamond 92.3
- SUPERChem 66.4
- ArXivMath 66.6
- HorizonMath 8.8
- MathArena (Apex, IMO, USAMO) 74.2
- BrokenArXiv 54.6
Dropped since the last post: SWE-bench Verified, Hy-CompanyBench, BrowseComp, DeepSearchQA, Toolathlon, Claw-Eval Pass^3, WildClawBench, Hy-FinModelBench, ProdBench (Tencent), Hy-SkillsWorld, Hy-Euler pro, FrontierScience, USAMO 2026, Hy-Math, PHYBench, CMT-Benchmark, IMO-AnswerBench, CL-bench, CL-bench Life, AA-LCR (Artificial Analysis Long Context Reasoning)
- Aug 26, 2026
- DeepSWE 58.7
- SWE-Bench Pro 62.5
- SWE-bench Multilingual 81
- NL2Repo-Bench 48.1
- CoWorkBench 73.9
- JobBench 55.7
- Agents' Last Exam 24.3
- Toolathlon-Verified 73.5
- IFBench 81.3
- GPQA Diamond 91.7
- Humanity's Last Exam 35.9
- LiveCodeBench v6 91.9
- Claw-Eval-MM 60.4
- RecreationBench 49.9
- AndroidWorld 84.5
- OSWorld 2.0 19.4
- Vision2Web 64
- ERQA 72.3
- LVBench 76.6
- RealWorldQA 88.5
- MATH-Vision (MathVision) 95.7
- CharXiv 90.6
Dropped since the last post: Terminal-Bench 2.1, FrontierSWE v1, MLS-Bench, PaperBench, AndroidBench, QwenSWEBench, QwenQoderBench, QwenReactBench, QwenSVGBench, WorkSpaceBench, SkillsBench, AutomationBench, WideSearch, $OneMillion-Bench, HealthBench, PLawBench, PRBench, PRBench, MRCR v2, LongBench v2, MMMU-Pro, BabyVision, HLE-VL, ZeroBench, LogicVista, HiPhO, PhyX, SLAKE, MedXpertQA-MM, PMC-VQA, OSWorld-Verified, ScreenSpot-Pro, WebArena-Verified, MobileWorld, QwenBlenderBench, Parametric CAD Bench, PresentBench, OmniDocBench v1.5, OCRBench v2, CC-OCR-Bench-V2, MTVQA-Test, MADQA, QwenVisualOffice, LingoQA, SURDS, SimpleVQA, WorldVQA, MMStar, PerceptionBench (Moonshot), CountQA, RefAdv-S, Dense 200 (bbox), COCO, VisFactor, VLMsAreBiased, Video-MME, VideoMME v2 (with subtitles), Video-MMMU, MMVU, MLVU, TVBench, LVBench (with memory), EgoLife (with memory), VideoDR (with search)
- Aug 26, 2026
- Terminal-Bench 2.1 84.3
- DeepSWE 63.4
- NL2Repo-Bench 56.3
- Toolathlon-Verified 78.4
- AutomationBench 48.8
- Agents' Last Exam 26.3
- Humanity's Last Exam 55.3
- GDPval-AA v2 1773
- OfficeQA Pro 62.4
- CharXiv 89.4
- Chartography 78
- BabyVision 53.4
- MVBench 77.8
- MMVU 80.5
- OSWorld 2.0 59.1
- Vision2Web 77.8
- Z.ai Code Bench 29
- Artificial Analysis Intelligence Index v4.1 57
Dropped since the last post: Terminal-Bench 3.0, ProgramBench almost solved, FrontierSWE v1, SWE-Marathon, PostTrainBench, CyberGym, ExploitGym, ExploitBench
- Aug 26, 2026
Dropped since the last post: Artificial Analysis Intelligence Index v4.1, FrontierCode 1.1, DeepSWE, Code Arena, Terminal-Bench 2.1, Terminal-Bench 3.0, AutomationBench, GDPval-AA v2, Harvey LAB-AA v1.0, GDP.pdf, CharXiv, LVBench, MRCR v2, OSWorld 2.0, Agents' Last Exam, HLE-Verified, BioMysteryBench, LABBench2
- Aug 25, 2026
- SWE-bench Multilingual 41.9
- SWE-Bench Pro 33.3
- SWE-bench Verified 57
- Terminal-Bench 2.1 29.2
- tau3-bench 62
- BFCL V4 61.4
- ProfBench 42.9
- Bird-SQL 41.9
- GDPval-AA v2 1225
- AIME 2025 89.2
- HMMT 2025 89.2
- GPQA Diamond 66.4
- LiveCodeBench v6 75.8
- SciCode 38.8
- MMLU-Pro 77.6
- MMLU-ProX 66.6
- Arena-Hard v2.0 67.9
- IFBench 77.2
- RULER 81.4
Dropped since the last post: MMLU, BIG-Bench Hard, AGIEval, SimpleQA, AlpacaEval 2.0, IFEval, Arena-Hard-Auto, MT-Bench, GSM8K, GSM-Symbolic, MATH, DeepMind Mathematics, HumanEval, HumanEval+, MBPP, MBPP+, CRUXEval, BigCodeBench, MultiPL-E, Berkeley Function Calling Leaderboard (BFCL), MMMLU, INCLUDE, MGSM, SALAD-Bench, AttaQ, Tulu 3 Safety Eval
- Aug 21, 2026
- Terminal-Bench 2.1 83.9
- NL2Repo-Bench 57.7
- CyberGym 75.3
- DeepSWE 59.3
- Toolathlon-Verified 75.9
- DSBench (DeepSeek internal) 63.6
- AutomationBench 25.7
- Agents' Last Exam 27.3
- Chartography 64.3
- ZeroBench 35
Dropped since the last post: Humanity's Last Exam
- Aug 14, 2026
- Terminal-Bench 2.1 73
- SWE-Bench Pro 61.7
- NL2Repo-Bench 42.3
- DeepSWE 42.2
- QwenSWEBench 79
- CoWorkBench 70.7
- JobBench 33.4
- Agents' Last Exam 20.4
- IFBench 79.5
- GPQA Diamond 89.2
- Humanity's Last Exam 30.8
- LiveCodeBench v6 90.3
- OSWorld-Verified 84.3
- WebArena-Verified 64.8
- AndroidWorld 81.9
- RecreationBench 47.1
- Claw-Eval-MM 56.9
- SWE-bench Multimodal 38.6
- Vision2Web 62.9
- MATH-Vision (MathVision) 94.6
- BabyVision 85.6
- CharXiv 90.2
- OmniDocBench v1.5 91.1
- RealWorldQA 85.9
- ERQA 65.5
Dropped since the last post: AgentWorldBench
- Aug 14, 2026
- Terminal-Bench 2.1 88.2
- Terminal-Bench 3.0 28.3
- DeepSWE 66.9
- NL2Repo-Bench 58
- ProgramBench almost solved 19
- FrontierSWE v1 78.1
- SWE-Marathon 42.5
- PostTrainBench 39.8
- CyberGym 84.5
- ExploitGym
- ExploitBench 54.4
- Toolathlon-Verified 73
- AutomationBench 48.2
- Agents' Last Exam 28.5
- Humanity's Last Exam 62.5
- GDPval-AA v2 1769
Dropped since the last post: CritPt, AIME 2026, HMMT Nov 2025, HMMT 2026, IMO-AnswerBench, GPQA Diamond, SWE-Bench Pro, ProgramBench, MCP Atlas, Toolathlon
- Aug 13, 2026
- Humanity's Last Exam 60
- Terminal-Bench 2.1 87.9
- NL2Repo-Bench 61.5
- CyberGym 83.3
- DeepSWE 62.7
- Toolathlon-Verified 74.1
- Agents' Last Exam 25.7
- AutomationBench 31.8
- DSBench (DeepSeek internal) 71.1
- DSBench (DeepSeek internal) 67.2
- Aug 13, 2026
- Artificial Analysis Intelligence Index v4.1 56
- FrontierCode 1.1 43.6
- DeepSWE 65.3
- Code Arena 1588
- Terminal-Bench 2.1 85.8
- Terminal-Bench 3.0 14.9
- AutomationBench 30.4
- GDPval-AA v2 1525
- Harvey LAB-AA v1.0 90.7
- GDP.pdf 34
- CharXiv 88.7
- LVBench 85.4
- MRCR v2 97
- OSWorld 2.0 47.9
- Agents' Last Exam 26.3
- HLE-Verified 53.6
- BioMysteryBench 87.1
- LABBench2 82.1
Dropped since the last post: Success detection (Google DeepMind), ERQA, Instrument reading (Google DeepMind), Progress classification (Google DeepMind), Moment finding (Google DeepMind), ASIMOV, Human proximity (Google DeepMind robotics)
Its system card adds 16 evaluations 11 new 12 dropped 16 results in all
Frontier Safety Framework Report - Gemini 3.7 FlashRobustness
Alignment
- GDM Stealth 75
- GDM Situational Awareness 54.5
Bio and chem risk
AI R&D risk
Persuasion
Dropped since the lab’s previous system card: SecureBio VMQA, single choice, LAB-Bench ProtocolQA, LAB-Bench CloningScenarios, LAB-Bench SeqQA, WMDP Biology, WMDP Chemistry, CBRN open-ended questions, Cyber key skills benchmark v1, Cyber key skills benchmark v2, RE-Bench, Chain-of-thought comprehensibility, Chain-of-thought informativeness
Its model card adds 9 evaluations 4 new 3 dropped 27 results in all
Gemini 3.7 Flash Model CardAlignment
Bio and chem risk
Persuasion
Dropped since the lab’s previous model card: SWE-Bench Pro, MLE-bench, OSWorld-Verified
- Aug 12, 2026
- MMStar 63.3
- MME 73.1
- RealWorldQA 73.1
- SimpleVQA 35.4
- SEED-Bench 77.7
- MMBench-EN v1.1 81
- CountBench 87.3
- MMMB 83
- MM-IFEval 60.6
- LogicVista 37.4
- MathVista 68.5
- MMMU-Pro 30.5
- MMMU 48.4
- ChartQA 81.3
- DocVQA 91.1
- InfographicVQA 70.2
- OCRBench 84.2
- OCRBench v2 47.5
- TextVQA 84.3
- RefCOCO / RefCOCO+ / RefCOCOg 87.9
- BLINK 61.5
- MuirBench 58.3
- HallusionBench 47.2
- POPE 88.7
- ScreenSpot-v2 80.7
- IFEval 82.3
- IFBench 25.8
- Multi-IF 59.4
- ToolSandbox 59.5
- BFCL V4 32.5
Dropped since the last post: AA-Omniscience, AIME 2025, LiveCodeBench v6, IFStruct, τ³-bench Banking, Claw-Eval, PinchBench, BrowseComp-Plus
- Aug 12, 2026
- MMBench-EN v1.1 68.7
- MMStar 51.8
- RealWorldQA 62.2
- GQA 57.4
- MMMB 72.8
- BLINK 52.7
- ChartQA 80.8
- DocVQA 92.1
- InfographicVQA 65.2
- OCRBench v2 36.7
- OCRBench 79.2
- AI2D 77.5
- MMMU 32.9
- RefCOCO / RefCOCO+ / RefCOCOg 73.2
- CountBench 72.5
- HallusionBench 61.5
- MMLU 50.4
- MMLU-Pro 30.7
- Multi-IF 37.3
- IFEval 74.9
Dropped since the last post: Terminal-Bench 2.0, Terminal-Bench Hard, SWE-bench Verified, SWE-Bench Pro, SciCode, LiveCodeBench v6
- Aug 12, 2026
- Artificial Analysis Intelligence Index v4.1 61
- GDPval-AA v2 1753
- CursorBench 3.2 69.9
- DeepSWE 65.9
- FrontierCode 1.1 61.3
- APEX-Agents 57.5
- Terminal-Bench 3.0 26
- APEX-SWE 56.4
- AA-Briefcase 1577
- Harvey LAB (Legal Agent Benchmark) 15.8
Dropped since the last post: Big Bench Audio, τ-Voice, Speech-to-Speech Index, Full-Duplex-Bench
Its model card adds 34 evaluations 11 new 10 dropped 43 results in all
Model Card: Grok 4.6Capabilities
- SWE-Marathon 31.9
- OfficeQA 63.2
- EEBench 60
- 3DCodeBench 54
- PartBench 58.5
- CADGenBench 40.9
- Parametric CAD Bench
- DeepSearchQA 81.6
Hallucination
Safety
- General refusals (xAI) 0.93
- Child safety (xAI) 0
- Autointent-Bio 100
- Autointent-Chem 100
- FORTRESS-RN 97.9
- Self-harm refusals (xAI) 0.84
Robustness
Alignment
- MASK
- xAI sycophancy eval 0.04
Bio and chem risk
- Virology Capabilities Test (VCT) 67.4
- Biosecurity VCT 47.8
- BioUseBench 90.7
- WMDP
- LAB-Bench 80.7
- ProtocolQA Open-Ended (xAI) 79.6
- BixBench 93.8
Cyber risk
- CyberGym 79.7
- CVE-Bench 39.8
- SecureCodeReview (xAI) 58.7
- HackerBench 6.9
AI R&D risk
- SpaceXAI MTS Eval 61.1
- InferenceEval 46.9
- KernelBenchInternal 37.2
Dropped since the lab’s previous model card: SWE-Bench Pro, SWE-bench Multilingual, FrontierSWE v1, ProgramBench tests passed, Terminal-Bench 2.1, SWE Atlas, FalseClaimBench, τ³-bench Banking, RelBench, Epistemic Bias (xAI)
- Aug 11, 2026
Nemotron 3.5 Lightning 30B-A3B NVIDIA NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 model card (Hugging Face) - MMLU-Pro 81.9
- AA-Omniscience 17.5
- GPQA Diamond 75.4
- Humanity's Last Exam 11.7
- SciCode 32.6
- SWE-bench Verified 51.6
- SWE-bench Multilingual 39.3
- Terminal-Bench 2.1 24.6
- PinchBench 85.4
- BrowseComp 37
- τ³-bench Banking 9.28
- GDPval-AA v2 832
- IFBench 71.9
- AA-LCR (Artificial Analysis Long Context Reasoning) 52
Dropped since the last post: CVBench, OCRBench v2, OSWorld-Verified, CharXiv, MMLongBench-Doc, MathVista, OCR-Reasoning, Video-MME, WorldSense (omnimodal), Daily-Omni, VoiceBench, MMAU
Nemotron 3.5 Lightning 30B-A3B Base: Its technical report adds 21 evaluations 21 new 32 dropped 23 results in all
Nemotron 3.5 Super and Lightning technical report, base model evaluationsCapabilities
- MMLU 78.6
- AGIEval 68.6
- SuperGPQA 33.2
- GSM8K 91.7
- MATH 83.1
- MATH-500 79.5
- HumanEval 76.9
- MBPP 77
- LiveCodeBench 29.8
- BigCodeBench 50.1
- CommonSenseQA 81.1
- ARC-Challenge 93.2
- HellaSwag 85.3
- OpenBookQA 47.8
- PIQA 85.1
- TruthfulQA 55.5
- WinoGrande 78.1
- SIQA 47.8
- RACE 87.8
- Global MMLU 75.1
- MGSM 82.3
Dropped since the lab’s previous technical report: MMMU, MathVista, MMLongBench-Doc, OCRBench, OCRBench v2, ChartQA, DocVQA, AI2D, TextVQA, InfographicVQA, OCR-Reasoning, CharXiv, CharXiv (DQ), TreeBench, CVBench, RefCOCO / RefCOCO+ / RefCOCOg, ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, OSWorld-Verified, Video-MME, Open ASR Leaderboard, MMAU, VoiceBench, Daily-Omni, WorldSense (omnimodal), LiveCodeBench v5, AIME 2025, IFBench, AA-LCR (Artificial Analysis Long Context Reasoning), tau2-bench, SciCode
Its technical report adds 4 evaluations 4 new 22 dropped 17 results in all
Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AICapabilities
- tau2-bench
- CritPt 0
- RULER 91.5
- NV Chat Benchmark 12.1
Dropped since the lab’s previous technical report: MMLU, MMLU-Pro, AGIEval, SuperGPQA, GSM8K, MATH, MATH-500, HumanEval, MBPP, LiveCodeBench, BigCodeBench, CommonSenseQA, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, TruthfulQA, WinoGrande, SIQA, RACE, Global MMLU, MGSM
- Aug 11, 2026
- Terminal-Bench 2.1 57
- BrowseComp 49.2
- SWE-bench Verified 70.6
- MCP Atlas 61.4
- APEX-Agents 18.7
- τ³-bench Banking 23
- AA-LCR (Artificial Analysis Long Context Reasoning) 71
- GPQA Diamond 89
- MMLU-Pro 86.3
- AIME 2026 95.3
- KMMLU-Pro 79.2
- KBL 77.5
- KorMedMCQA 93.2
- Ko-GDPval 87.3
Dropped since the last post: Humanity's Last Exam, HMMT 2026, LiveCodeBench v6, tau2-bench, Terminal-Bench 2.0, IFBench, Arena-Hard v2.0, Ko-Arena-Hard v2 (Upstage)
- Aug 10, 2026
Muse Glimmer Meta Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device (Meta) and model card - MCP Atlas 75.5
- DeepSearchQA 74.6
- τ³-bench Banking 23.5
- WildClawBench 47.6
- GDPval-AA v2 953
- Meta Agents Research Environments (ARE) / Gaia2 43.3
- SkillsBench 44.3
- OSWorld-Verified 65.9
- SWE-Bench Pro 51.2
- SWE-bench Verified 76
- Terminal-Bench 2.1 51.7
- SciCode 43.6
- CharXiv 78.8
- ScreenSpot-Pro 75.4
- OmniDocBench v1.5 75.8
- MMMU-Pro 74
- CIMemories 26.4
- AgentDojo 28.4
- IFBench 77
- AIME 2026 94.7
- GPQA Diamond 83.5
- Humanity's Last Exam 22
- AA-LCR (Artificial Analysis Long Context Reasoning) 80
- BEAM 65.1
Dropped since the last post: LiveCodeBench v5, LiveCodeBench v6, MATH-500, AIME 2024, AIME 2025
Its technical report adds 13 evaluations 13 new 14 dropped 37 results in all
Muse Glimmer 30B: An On-Device Agentic ModelCapabilities
- tau2-bench
- LiveCodeBench Pro 64.1
- PinchBench 90.5
- ARC-AGI 66.9
- ScreenSpot-v2 93.3
- Omnilingual-Gaia2 60.4
- Linguini 8.8
- Belebele 87.8
- MMLU-ProX 80.1
- WMT24++ 53.4
- FLORES+ 47.5
- BOUQuET 36.5
Safety
- PrivacyLens 49.2
Dropped since the lab’s previous technical report: WMDP Cyber, Cybench, Hack the Box Machines (Meta), Native Code Exploitation (Meta), LAB-Bench, WMDP Biology, WMDP Chemistry, Molecular Biology Capabilities Test, Meta BioKnowledge Proxy, BioLP-bench, Meta BioProtocol Proxy, Human Pathogen Capabilities Test, Virology Capabilities Test (VCT), MASK
- Aug 5, 2026
- DeepSWE 59.3
- Terminal-Bench 2.1 82.9
- MCP Atlas 90.3
- GDPval-AA v2 1631
Dropped since the last post: JobBench, Toolathlon-Verified, OSWorld-Verified, Humanity's Last Exam, Vals Finance Agent v2, SWE-Bench Pro, CharXiv, BabyVision
- Aug 4, 2026
- AA-Omniscience -29.5
- AIME 2025 51.9
- LiveCodeBench v6 59.4
- IFBench 59.2
- Multi-IF 80.1
- IFStruct 85.5
- BFCL V4 56.9
- ToolSandbox 77.8
- τ³-bench Banking 5.67
- Claw-Eval 62.9
- PinchBench 68.2
- BrowseComp-Plus 26.9
Dropped since the last post: GPQA Diamond, MMLU-Pro, IFEval, GSM8K, MATH-500, Berkeley Function Calling Leaderboard (BFCL)
- Aug 3, 2026
- Terminal-Bench 2.1 86.6
- SWE-Bench Pro 67.7
- DeepSWE 56.6
- NL2Repo-Bench 55.9
- FrontierSWE v1 73.5
- MLS-Bench 41
- PaperBench 93
- AndroidBench 75.1
- QwenSWEBench 80.7
- QwenQoderBench 58.4
- QwenReactBench 1724
- QwenSVGBench 1713
- CoWorkBench 74.8
- WorkSpaceBench 67.7
- JobBench 53.4
- SkillsBench 70.2
- Agents' Last Exam 27
- AutomationBench 27.3
- Toolathlon-Verified 72.5
- WideSearch 81.9
- Humanity's Last Exam 56.2
- GPQA Diamond 92.6
- IFBench 82.8
- $OneMillion-Bench
- HealthBench 60.2
- PLawBench 73.2
- PRBench 57.6
- PRBench 58.3
- MRCR v2 92.9
- LongBench v2 66.3
- MMMU-Pro 82.3
- MATH-Vision (MathVision) 97.7
- BabyVision 91.3
- HLE-VL 52.2
- ZeroBench 49
- LogicVista 91.9
- HiPhO 90
- PhyX 83.5
- SLAKE 90.8
- MedXpertQA-MM 80.4
- PMC-VQA 66.2
- OSWorld-Verified 86.1
- OSWorld 2.0 19.4
- ScreenSpot-Pro 84.5
- WebArena-Verified 66.8
- AndroidWorld 85.3
- MobileWorld 77.8
- Claw-Eval-MM 74.8
- Vision2Web 69
- QwenBlenderBench 69.9
- Parametric CAD Bench
- RecreationBench 51.7
- PresentBench 79.6
- CharXiv 93.5
- OmniDocBench v1.5 92.1
- OCRBench v2 74.2
- CC-OCR-Bench-V2 79.6
- MTVQA-Test 56.6
- MADQA 91.8
- QwenVisualOffice 44.6
- RealWorldQA 88
- ERQA 77.8
- LingoQA 84.8
- SURDS 77.8
- SimpleVQA 75
- WorldVQA 53.2
- MMStar 85.9
- PerceptionBench (Moonshot) 63.5
- CountQA 82.4
- RefAdv-S 80.2
- Dense 200 (bbox) 87
- COCO 78.7
- VisFactor 60.8
- VLMsAreBiased 88.3
- Video-MME 90.4
- VideoMME v2 (with subtitles) 68.3
- Video-MMMU 88.7
- MMVU 82.4
- MLVU 90.8
- TVBench 81.9
- LVBench 81.8
- LVBench (with memory) 85.6
- EgoLife (with memory) 80.3
- VideoDR (with search) 73.2
Dropped since the last post: Terminal-Bench 2.0, SWE-bench Verified, SWE-bench Multilingual, SciCode, QwenWebDev, QwenSVG, QwenClawBench, Claw-Eval, BFCL V4, MCPMark, MCP Atlas, VITA-Bench, DeepPlanning, SpreadsheetBench, Kernel Bench L3, QwenWorldBench, LiveCodeBench, HMMT 2026, IMO-AnswerBench, CritPt, MathArena (Apex, IMO, USAMO), MMLU-Pro, MMLU-Redux, SuperGPQA, IFEval, WMT24++, MAXIFE, MMMLU, MMLU-ProX, NOVA-63, INCLUDE, Global PIQA, PolyMath, QwenVision2Code, MMSearch-Plus, BrowseComp-VL, MM-BrowseComp, ODinW-13, Ego3D-Bench, VLADBench
- Jul 31, 2026
- MMLU-Pro 83.5
- GPQA Diamond 82.2
- Humanity's Last Exam 18.3
- AIME 2026 92.3
- HMMT 2026 78.4
- IMO-AnswerBench 78.6
- SciCode 37.4
- SWE-bench Verified 68.2
- Terminal-Bench 2.1 43.8
- τ³-bench Banking 14.2
- Claw-Eval 77.7
- IFEval 92.4
- IFBench 72.6
- OpenAI MRCR v2 94.4
- AA-LCR (Artificial Analysis Long Context Reasoning) 56.2
- Ko-LongBench (LG) 89.6
- KMMLU-Pro 69.1
- CLIcK 84.2
- HRM8K-KSM 91.1
- MMMLU 86.6
- Global MMLU Lite 3.0 86.6
- PolyMath 71.3
- KGC-Safety (LG) 99.8
- ROK-FORTRESS 89.5
Dropped since the last post: MMMU, MMMU-Pro, MedXpertQA-MM, MATH-Vision (MathVision), MathVista, We-Math, LogicVista, BabyVision, AI2D, ChartQA Pro, CharXiv, OCRBench v2, OmniDocBench v1.5, MMStar, BLINK, HallusionBench, KMMMU, K-Viscuit, KRETA, AIME 2025, LiveCodeBench v6, tau2-bench, KoBALT
Its technical report adds 2 evaluations 1 new 22 dropped 25 results in all
K-EXAONE 2.0 Technical ReportCapabilities
- OpenAI MRCR 94.4
- WMT24++
Dropped since the lab’s previous technical report: MMMU, MMMU-Pro, MedXpertQA-MM, MATH-Vision (MathVision), MathVista, We-Math, LogicVista, BabyVision, AI2D, ChartQA Pro, CharXiv, OCRBench v2, OmniDocBench v1.5, MMStar, BLINK, HallusionBench, KMMMU, K-Viscuit, KRETA, LiveCodeBench v6, tau2-bench, KoBALT
- Jul 31, 2026
- Terminal-Bench 2.1 82.7
- NL2Repo-Bench 54.2
- CyberGym 76.7
- DeepSWE 54.4
- Toolathlon-Verified 70.3
- Agents' Last Exam 25.2
- AutomationBench 25.1
- DSBench (DeepSeek internal) 68.7
- DSBench (DeepSeek internal) 59.6
Dropped since the last post: MMLU-Pro, SimpleQA Verified, Chinese SimpleQA, GPQA Diamond, Humanity's Last Exam, LiveCodeBench, Codeforces rating, HMMT 2026, IMO-AnswerBench, MathArena (Apex, IMO, USAMO), MathArena Apex Shortlist, MRCR v2, CorpusQA 1M, Terminal-Bench 2.0, SWE-bench Verified, SWE-Bench Pro, SWE-bench Multilingual, BrowseComp, GDPval-AA, MCP Atlas, Toolathlon, AGIEval, MMLU, MMLU-Redux, MMMLU, C-Eval, CMMLU, MultiLoKo, SuperGPQA, FACTS Parametric, TriviaQA, BIG-Bench Hard, DROP, HellaSwag, WinoGrande, CLUEWSC, BigCodeBench, HumanEval, GSM8K, MATH, MGSM, CMATH, LongBench v2
- Jul 30, 2026
- Success detection (Google DeepMind) 87.7
- ERQA 78.5
- Instrument reading (Google DeepMind) 65.7
- Progress classification (Google DeepMind) 57.4
- Moment finding (Google DeepMind) 91.3
- ASIMOV 97.9
- Human proximity (Google DeepMind robotics) 93
Dropped since the last post: CyberGym, Big Sleep Evaluation (Google), Chrome Production Commit Scanning (Google)
Its technical report adds 1 evaluation 1 new 8 dropped 2 results in all
Gemini Robotics 2: Safety EvaluationsSafety
Dropped since the lab’s previous technical report: MTEB, TextCaps, Docci, ViDoRe v2, VATEX, MSR-VTT retrieval, YouCook2 retrieval, MSEB
- Jul 30, 2026
- Artificial Analysis Intelligence Index v4.1 40
- SWE-bench Verified 80.2
- SWE-Bench Pro 55.9
- Terminal-Bench 2.1 64.7
- SciCode 48.7
- GDPval-AA v2 1269
- MCP Atlas 79.6
- τ³-bench Banking 15.5
- BrowseComp 77.4
- Toolathlon-Verified 54.4
- AA-Briefcase 917
- GPQA Diamond 89.5
- Humanity's Last Exam 47.8
- AIME 2026 95.5
- HMMT 2026 90.2
- CritPt 8.3
- ARC-AGI 84
- ARC-AGI-2 40.1
- SimpleQA Verified 20.6
- AA-Omniscience -9
- IFBench 82.2
- Global MMLU Lite 3.0 86.7
- StrongREJECT 98.4
- FORTRESS
- MMMU-Pro 74
- CharXiv 81.3
- Audio MC 54.9
- MMAU 77
- VoiceBench 90.1
- ForecastBench 61.5
- Prophet Arena
Dropped since the last post: DesignArena Website
Its model card adds 0 evaluations 29 results in all
- Jul 29, 2026GPT-Live-Transcribe OpenAI GPT-Live-Transcribe and GPT-Transcribe: Two New Transcription Models in the API (OpenAI Developer Community)
Dropped since the last post: Agents' Last Exam, GDPval-AA v2, BigFinanceBench, Artificial Analysis Intelligence Index v4.1, Artificial Analysis Coding Agent Index v1.1, SWE-Bench Pro, DeepSWE, Terminal-Bench 2.1, GeneBench Pro, LifeSciBench, HealthBench Professional, OSWorld 2.0, BrowseComp, BenchCAD, SEC-bench Pro, ExploitBench, ExploitGym, MMMU-Pro, GDP.pdf, GPQA Diamond, FrontierMath Tiers 1-3 v2, FrontierMath Tier 4 v2, AutomationBench, Toolathlon, OpenAI MRCR v2, GraphWalks, ARC-AGI-3, Management Consulting Tasks (OpenAI), MedChemBench (OpenAI)
- Jul 29, 2026
- Big Bench Audio 97.2
- τ-Voice 56.5
- Speech-to-Speech Index 82.9
- Full-Duplex-Bench 95.1
Dropped since the last post: DeepSWE, SWE-Marathon, Terminal-Bench 2.1, SWE-Bench Pro, tau3-bench
- Jul 27, 2026
- CyberGym 90.4
- CyberGym (final-submission metric) 86.3
Dropped since the last post: SWE-bench Verified, SWE-Bench Pro, SWE-bench Multilingual, Terminal-Bench 2.0, AIME 2026, AMO-Bench, FrontierMath, Humanity's Last Exam, GPQA Diamond, FrontierScience, ArtifactsBench, IFBench, AdvancedIF, Robust IF Bench, tau2-bench
Its model card adds 5 evaluations 5 new 15 dropped 6 results in all
MAI-Cyber-1-Flash Model CardCyber risk
Dropped since the lab’s previous model card: SWE-bench Verified, SWE-Bench Pro, SWE-bench Multilingual, Terminal-Bench 2.0, AIME 2026, AMO-Bench, FrontierMath, Humanity's Last Exam, GPQA Diamond, FrontierScience, ArtifactsBench, IFBench, AdvancedIF, Robust IF Bench, tau2-bench
- Jul 24, 2026Claude Opus 5 Anthropic Introducing Claude Opus 5 (launch post); System Card: Claude Opus 5, Table 8.1.A
- Terminal-Bench 3.0 43.3
- GDPval-AA v2 1861
- ARC-AGI-3 30.2
- BrowseComp 90.8
- Humanity's Last Exam 64.7
- OSWorld 2.0 70.6
- DeepSWE 68.8
- FrontierCode 53.4
- AutomationBench 26
- Harvey LAB (Legal Agent Benchmark) 11.7
- HealthBench Professional 59.8
- BioMysteryBench 90.1
- SWE-Bench Pro 79.2
- SWE-bench Multilingual 89.5
- SWE-bench Multimodal 59.4
- AA-Briefcase 1720
- ARC-AGI 97.5
- ARC-AGI-2 90.4
Dropped since the last post: Terminal-Bench 2.1, OSWorld-Verified
Its system card adds 74 evaluations 22 new 14 dropped 91 results in all
System Card: Claude Opus 5Capabilities
- SWE-bench Verified 96
- FrontierCode 1.1 53.4
- IMO 2026 100
- Riemann-bench 60
- ArXivMath 90.8
- ProgramBench tests passed 93
- DeepSearchQA 95
- DRACO 88.6
- Chartography 29.6
- BenchCAD 36.6
- GDP.pdf 83.4
- OfficeQA 78.1
- OfficeQA Pro 66.9
- MCP Atlas 85.8
- Toolathlon-Verified 80.6
- Global MMLU 92.5
- MILU 92.1
- INCLUDE 89.8
- SpatialBench (LatchBio) 72.5
- scBench (LatchBio SingleCellBench) 60.6
- ProteinGym 47.7
- Protein Design (Anthropic) 42.5
- Organic Chemistry V2 (Anthropic) 61.6
- Protocols (Anthropic life sciences)
Health
- HealthBench 57.8
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 96.3
- Single-turn benign request evaluations (Anthropic) 0.09
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 89
- Malicious computer use (Anthropic) 93.8
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 5.6
- Gray Swan IPI Arena benchmark 0.2
- Prompt injection live bug bounty (Anthropic with Gray Swan) 0.08
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 0.56
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.54
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 3.84
Alignment
- Anthropic automated behavioral audit 2.3
- Anthropic undermining liberal democracy (behavioral audit metric) 1.13
- UK AISI unprompted sabotage evaluation 0
- UK AISI sabotage continuation evaluation 0.1
- MASK 94.8
- Uncritically reporting flawed results (Anthropic) 9.83
- Overconfidence: verifying command-line syntax (Anthropic) 4.9
- Lazy investigation (Anthropic) 9.94
Monitorability
- SHADE-Arena 12
- LinuxArena 0.7
Bio and chem risk
Cyber risk
- ExploitBench 70
- Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) 79.4
- Firefox 147 exploit evaluation (Anthropic) 52.4
- CyScenarioBench 33.7
- ExploitGym 22
- UK AISI cyber ranges 80
- Vulnerability finding evaluations (Anthropic)
- Secure and defensive coding (Anthropic) 0.61
- Cyber harm coverage (Anthropic) 99.6
AI R&D risk
Dropped since the lab’s previous system card: CyberGym, Gray Swan Agent Red Teaming (ART) benchmark, Evaluation awareness discriminability (Anthropic), Terminal-Bench 2.1, FrontierCode, CursorBench 3.1, USAMO 2026, OSWorld-Verified, ChartMuseum, CharXiv, Real-World Finance v2 (Anthropic), Toolathlon, Structural Biology (Anthropic), Organic Chemistry (Anthropic)
- Jul 23, 2026
- SWE-bench Verified 69.4
- SWE-bench Multilingual 63
- SWE-Bench Pro 46
- Terminal-Bench 2.1 41
- PinchBench 93.4
- SciCode 44.2
- KAT Code Bench 46.2
Dropped since the last post: LongVideoBench, VideoMME v2 (with subtitles), TimeLens-Bench, MLVU, Video-MME, Video-MMMU, LiveCodeBench v6, OJBench, BFCL V4, tau2-bench, VITA-Bench, We-Math, MathVista, DynaMath, AIME 2025, AIME 2026, Humanity's Last Exam, MMMU, MMStar, C-Eval, IFEval, HallusionBench, AI2D, OCRBench, OmniDocBench v1.5, RefCOCO / RefCOCO+ / RefCOCOg, PixMo-Count, EmbSpatialBench
- Jul 22, 2026
- MMLU-Pro 86.2
- GPQA Diamond 86.3
- Humanity's Last Exam 28.8
- LiveCodeBench v6 92.4
- ArtifactsBench 55.9
- HMMT 2026 93.9
- AIME 2026 95.7
- MultiChallenge 61
- IFBench 80
- AA-LCR (Artificial Analysis Long Context Reasoning) 62.3
- SWE-bench Verified 70.4
- Terminal-Bench Hard 28.3
- APEX-Agents 16.6
- MCP Atlas 58.2
- τ³-bench Banking 19.6
- GDPval-AA v2 1128
- KMMLU-Pro 78.4
- CLIcK 90.7
- HAE-RAE Bench 73.8
- Ko-AIME 2025 (Upstage) 97.7
- HRM8K 92.2
- KBank-MMLU (Upstage) 80.8
- KBL 75.5
- KorMedMCQA 93
- Ko-GDPval 86.8
Dropped since the last post: KMMLU, KoBALT, MMLU, AIME 2024, AIME 2025, HMMT 2025, HMMT Nov 2025, IFEval, Arena-Hard v2.0, WritingBench, tau2-bench
Its technical report adds 0 evaluations 14 dropped 25 results in all
Solar Open 2 Technical ReportDropped since the lab’s previous technical report: KMMLU, KoBALT, Ko-AIME 2024 (Upstage), Ko-IFEval (Upstage), Ko-Arena-Hard v2 (Upstage), MMLU, AIME 2024, AIME 2025, HMMT 2025, HMMT Nov 2025, IFEval, Arena-Hard v2.0, WritingBench, tau2-bench
- Jul 21, 2026
Dropped since the last post: SWE-Bench Pro, DeepSWE, Terminal-Bench 2.1, MLE-bench, GDPval-AA v2, OSWorld-Verified, CharXiv, MRCR v2
- Jul 21, 2026
- Terminal-Bench 2.1 70.2
- SWE-bench Multilingual 78.5
- SWE-Bench Pro 59.4
- DeepSWE 40.4
- SWE Atlas 46.2
- Toolathlon-Verified 49.7
Dropped since the last post: SWE-bench Verified, Terminal-Bench 2.0
- Jul 21, 2026
- SWE-Bench Pro 54.2
- Terminal-Bench 2.1 54
- MLE-bench 39.2
- GDPval-AA v2 1140
- OSWorld-Verified 74
- CharXiv 76.5
- MRCR v2 72.2
Dropped since the last post: MMLU-Pro, AIME 2026, LiveCodeBench v6, Codeforces rating, GPQA Diamond, tau2-bench, Humanity's Last Exam, BBEH micro average, MMMLU, MMMU-Pro, OmniDocBench v1.5, MATH-Vision (MathVision), MedXpertQA-MM
Its model card adds 5 evaluations 8 dropped 12 results in all
Gemini 3.5 Flash-Lite Model CardDropped since the lab’s previous model card: Humanity's Last Exam, GPQA Diamond, MMMU-Pro, Video-MMMU, SimpleQA Verified, FACTS Benchmark Suite, MMMLU, LiveCodeBench
- Jul 21, 2026
- SWE-Bench Pro 58.7
- DeepSWE 49
- Terminal-Bench 2.1 78
- MLE-bench 63.9
- GDPval-AA v2 1421
- OSWorld-Verified 83
- CharXiv 89.4
- MRCR v2 91.8
Dropped since the last post: MCP Atlas, Toolathlon, Vals Finance Agent v2, GDPval-AA, MMMU-Pro, Blueprint-Bench 2, Humanity's Last Exam, ARC-AGI-2
Its model card adds 5 evaluations 8 dropped 13 results in all
Gemini 3.6 Flash Model CardDropped since the lab’s previous model card: MCP Atlas, Toolathlon, Vals Finance Agent v2, GDPval-AA, MMMU-Pro, Blueprint-Bench 2, Humanity's Last Exam, ARC-AGI-2
- Jul 16, 2026
- GPQA Diamond 93.5
- CritPt 23.4
- AA-LCR (Artificial Analysis Long Context Reasoning) 74.7
- Humanity's Last Exam 56
- DeepSWE 67.5
- ProgramBench
- Terminal-Bench 2.1 88.3
- FrontierSWE v1 81.2
- SWE-Marathon 42
- PostTrainBench 36.6
- MLS-Bench 48.3
- SciCode 58.7
- Kimi Code Bench v2 72.9
- BrowseComp 91.2
- DeepSearchQA 95
- ResearchRubrics 76.2
- GDPval-AA v2 1686
- Toolathlon-Verified 76.5
- MCPMark 94.5
- MCP Atlas 84.2
- AutomationBench 30.8
- JobBench 54.3
- AA-Briefcase 1548
- Agents' Last Exam 28.3
- APEX-Agents 41
- OfficeQA 63.3
- SpreadsheetBench 2 34.8
- OSWorld-Verified 84.8
- OSWorld 2.0 58.3
- SaaS-Bench
- tau3-bench 33.4
- Harvey LAB-AA v1.0 94.6
- CorpFin v2 71.6
- Vals Finance Agent v2 54.4
- Legal Research Bench 44.2
- WorldVQA 51
- OmniDocBench v1.5 91.1
- PerceptionBench (Moonshot) 58.5
- Video-MME 90
- MMVU 82.1
- BabyVision 85.7
- MMMU-Pro 81.6
- CharXiv 84.8
- MATH-Vision (MathVision) 94.3
- ZeroBench 23
Dropped since the last post: Kimi Claw 24/7 Bench
Its technical report adds 26 evaluations 25 new 32 dropped 70 results in all
Kimi K3: Open Frontier Intelligence (technical report, arXiv 2607.24653)Capabilities
- τ³-bench Banking 33.4
- Coding Experience (Moonshot) 59.9
- 24/7 ClawBench 2.0 48.3
- MIRA Bench 64.1
- KAET (Kimi Autonomous Execution Tasks) 83.5
- CLIF Bench 52.4
- Agentic Vision Bench (Moonshot) 78.3
- Swarm Bench 76.3
- Online Experience (Moonshot) 77.9
- Deep Research Bench (Moonshot) 90
- Finance Bench (Moonshot) 62.6
- KWV Bench 64.7
- DECK Bench 73.5
- Agent Behavior Bench 65
- Chat All-in-One Bench 85.2
- Kimi Webdev Bench 58.6
Hallucination
Cyber risk
Other
- Artificial Analysis Intelligence Index v4.1 57.1
- Vals Index v1 74.7
- Code Arena 1678
- LMArena (Elo) 1486
- Agent Arena 9.1
Dropped since the lab’s previous technical report: AIME 2025, HMMT 2025, HMMT Nov 2025, IMO-AnswerBench, MMLU-Pro, SimpleQA Verified, AdvancedIF, LongBench v2, SWE-bench Verified, SWE-Bench Pro, SWE-bench Multilingual, Terminal-Bench 2.0, PaperBench, CyberGym, OJBench, LiveCodeBench v6, WideSearch, FinSearchComp, Seal-0, GDPval-AA, MMMU, MathVista, SimpleVQA, BLINK, MMVP, OCRBench, InfographicVQA, Video-MMMU, MotionBench, LongVideoBench, LVBench, WebArena
- Jul 16, 2026
- DeepSWE 62
- SWE-Marathon 29
- Terminal-Bench 2.1 83.3
- SWE-Bench Pro 64.7
- tau3-bench 32.6
Dropped since the last post: τ-Voice
Its model card adds 32 evaluations 27 new 14 dropped 36 results in all
Model Card: Grok 4.5Capabilities
- APEX-SWE 51.2
- SWE-bench Multilingual 78
- FrontierSWE v1 78
- ProgramBench tests passed 57.2
- SWE Atlas 84
- EEBench
- 3DCodeBench 49.8
- Parametric CAD Bench
- GDPval-AA v2 1535
- τ³-bench Banking 32.6
- DeepSearchQA 38.4
Hallucination
Safety
- General refusals (xAI) 1.1
- Child safety (xAI) 0
- Autointent-Bio 97.9
- Autointent-Chem 96.7
- FORTRESS-RN 97.9
- Self-harm refusals (xAI) 0.5
Robustness
Bias
- Epistemic Bias (xAI) 20.4
Alignment
- FalseClaimBench 84
- MASK
- xAI sycophancy eval 0.01
Bio and chem risk
- Virology Capabilities Test (VCT) 65.5
- WMDP
- LAB-Bench 71.1
- ProtocolQA Open-Ended (xAI) 87
- BixBench 93.8
Cyber risk
- CyberGym 80.4
- HackerBench 7.8
AI R&D risk
Dropped since the lab’s previous model card: Refusals + User Jailbreak (xAI), Refusals + System Jailbreak (xAI), AgentHarm, AgentDojo, SycophancyEval (answer sycophancy), HLE Calibration, WMDP Biology, LAB-Bench FigQA, LAB-Bench CloningScenarios, WMDP Chemistry, WMDP Cyber, Cybench, MakeMeSay, LAB-Bench ProtocolQA
- Jul 15, 2026
- Humanity's Last Exam 46
- AIME 2026 97.1
- GPQA Diamond 87.2
- SWE-bench Verified 77.6
- SWE-Bench Pro 54.3
- Terminal-Bench 2.1 63.8
- GDPval-AA v2 1238
- MCP Atlas 76
- τ³-bench Banking 23.7
- Toolathlon-Verified 45.5
- BrowseComp 77.1
- SimpleQA Verified 43.9
- AA-Omniscience 2.1
- IFBench 79.8
- Global MMLU Lite 3.0 88.7
- MMMU-Pro 73.5
- CharXiv 82
- Audio MC 56.6
- MMAU 77.2
- VoiceBench 91.4
- FORTRESS
- StrongREJECT 98.6
- DesignArena Website 1257
- ForecastBench 61.1
- Prophet Arena 0.162
Its model card adds 0 evaluations 22 results in all
- Jul 9, 2026
- Agents' Last Exam 50.3
- GDPval-AA v2 1591.8
- BigFinanceBench 36
- Artificial Analysis Intelligence Index v4.1 51.2
- Artificial Analysis Coding Agent Index v1.1 74.6
- SWE-Bench Pro 62.7
- DeepSWE 67.2
- Terminal-Bench 2.1 84.7
- GeneBench Pro 10.8
- LifeSciBench 51.2
- HealthBench Professional 55.7
- OSWorld 2.0 45.6
- BrowseComp 83.3
- BenchCAD 73.9
- SEC-bench Pro 48.9
- ExploitBench 33.2
- ExploitGym 12.4
- MMMU-Pro 79.5
- GDP.pdf 22.7
- GPQA Diamond 92.3
- FrontierMath Tiers 1-3 v2 78.6
- FrontierMath Tier 4 v2 58.5
- AutomationBench 14.9
- Toolathlon 53.4
- OpenAI MRCR v2 41.3
- GraphWalks 81.3
- ARC-AGI-3 0.18
- Management Consulting Tasks (OpenAI) 35.4
- MedChemBench (OpenAI) 30.4
Dropped since the last post: Terminal-Bench 2.0, OSWorld-Verified, MCP Atlas, tau2-bench, Humanity's Last Exam, OmniDocBench v1.5
Its system card adds 28 evaluations 2 new 32 results in all
Its system card addendum adds 20 evaluations 4 new 21 results in all
GPT-5.6 – August UpdatesHealth
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
- Capture the Flag (Internal) 93.2
- CVE-Bench 82.3
- Cyber Range 61.5
- Jul 9, 2026
- Agents' Last Exam 50.4
- GDPval-AA v2 1593
- BigFinanceBench 51
- Artificial Analysis Intelligence Index v4.1 55
- Artificial Analysis Coding Agent Index v1.1 77.4
- SWE-Bench Pro 63.4
- DeepSWE 69.6
- Terminal-Bench 2.1 87.4
- GeneBench Pro 23.3
- LifeSciBench 56
- HealthBench Professional 57.7
- OSWorld 2.0 50.2
- BrowseComp 87.5
- BenchCAD 78.2
- SEC-bench Pro 57.7
- ExploitBench 52.9
- ExploitGym 23.2
- MMMU-Pro 82
- GDP.pdf 24.7
- GPQA Diamond 92.9
- FrontierMath Tiers 1-3 v2 84.9
- FrontierMath Tier 4 v2 68.3
- AutomationBench 15.2
- Toolathlon 53.1
- OpenAI MRCR v2 89.6
- GraphWalks 76.9
- ARC-AGI-3 0.8
- Management Consulting Tasks (OpenAI) 37.2
- MedChemBench (OpenAI) 35
Its system card adds 29 evaluations 27 dropped 33 results in all
GPT-5.6 System CardHealth
Hallucination
Safety
Robustness
Bias
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Disallowed Content Evaluations with Representative Prompts, Deception Evaluations with Representative Prompts, OpenAI Deployment Simulation, Verbalized metagaming, CoT Monitorability, AAV Capsid Packaging Prediction (OpenAI), Hard-negative protein binding prediction, DNA sequence design for transcription factor binding, Virology Capabilities Test (VCT), Molecular Biology Capabilities Test, Human Pathogen Capabilities Test, World-Class Bio, ReproBAIT, VulnLMP, FrontierCyber, CyScenarioBench, Atomic Challenges, UK AISI narrow cyber tasks, UK AISI cyber ranges, CyberGym, METR Time Horizon, UK AISI unprompted sabotage evaluation, UK AISI sabotage continuation evaluation, ImpossibleMLEBench, ControlArena, No-CoT math time horizon, Apollo Research strategic deception evaluations
- Jul 9, 2026
- MCP Atlas 88.1
- JobBench 54.7
- Toolathlon-Verified 75.6
- OSWorld-Verified 80.8
- Humanity's Last Exam 62.1
- Vals Finance Agent v2 57.2
- Terminal-Bench 2.1 80
- SWE-Bench Pro 61.5
- DeepSWE 53.3
- CharXiv 88.4
- BabyVision 76.3
Dropped since the last post: MMMU-Pro, ERQA, SimpleVQA, ScreenSpot-Pro, ZeroBench, ARC-AGI-2, GPQA Diamond, LiveCodeBench Pro, HealthBench Hard, MedXpertQA, MedXpertQA-MM, DeepSearchQA, SWE-bench Verified, Terminal-Bench 2.0, tau2-bench, GDPval-AA, FrontierScience-Research
Its system card adds 57 evaluations 20 new 19 dropped 68 results in all
Muse Spark 1.1 Evaluation ReportCapabilities
- MRCR v2 54.1
- OSWorld 2.0 47.3
- WebArena-Verified 69
- DeepSearchQA 84.9
- GDPval-AA v2 1381
Health
Hallucination
- HLE Calibration 23.4
Safety
- SAVE-Bench 90.7
Robustness
- StrongREJECT v2 (Meta) 0.5
- OR-Bench 4.8
- FORTRESS 12.3
- AgentHarm Verified (Meta) 3.4
- AgentDojo 0.7
- SWE-PI
- Gray Swan Agent Red Teaming (ART) benchmark 0.3
Alignment
Monitorability
Bio and chem risk
- Molecular Biology Capabilities Test 53.2
- Virology Capabilities Test (VCT) 52
- Human Pathogen Capabilities Test 61.9
- WMDP
- LAB-Bench ProtocolQA 88
- LAB-Bench SeqQA 98.2
- ABC Bench Fragment Design 97
- ABC Bench Liquid-Handling Robot 93.7
- ABC Bench Screening Evasion 63.2
- Meta BioDesign Proxy 55.2
- BioTIER Refuse 97.7
- Meta chemical-agent refusal evaluation 99.8
Cyber risk
- Cybench 92.9
- Meta curated CTF set 89.9
- CyberGym 59
- ExploitGym 5
- Irregular FrontierCyber and Atomic Challenges 67
- CyScenarioBench 0.5
- Charlemagne Labs social-engineering benchmark 5.1
- CyberSecEval Autocomplete 71.3
- CyberSecEval Instruct 77.3
- Meta Cyber Misuse Chat 2.8
- Meta agentic cyber misuse refusal 0.7
- CyberSecEval Code Interpreter Abuse
- Meta catastrophic cyber misuse refusal 3.9
- Poly-Guard Bench 6.9
- CyberBench (Liu et al.) 9
- RedCode 12
AI R&D risk
- AIRS-Bench 0.77
Dropped since the lab’s previous system card: Meta chemical and biological high-risk bottleneck evaluations, Meta cyber misuse request set, Terminal-Bench 2.0, SWE-bench Verified hard subset (Meta), MLE-bench 30, Meta prompted evaluation awareness, AgentHarm, Meta cyber false-refusal benchmark, IHEval, ImpossibleBench, Meta Underspecified Intent evaluation, SimpleQA Verified, AbstentionBench, CharXiv Missing Images, TextQuests, PropensityBench, CIMemories, Task-dependent output homogenization evaluation, Situational Awareness Dataset (SAD)
- Jul 9, 2026
- Agents' Last Exam 53.6
- Artificial Analysis Intelligence Index v4.1 58.9
- Artificial Analysis Coding Agent Index v1.1 80
- Terminal-Bench 2.1 91.9
- DeepSWE 72.7
- BrowseComp 92.2
- GDPval-AA v2 1748
- OSWorld 2.0 62.6
- AutomationBench 18.1
- GeneBench Pro 28.7
- LifeSciBench 59.9
- Management Consulting Tasks (OpenAI) 43.2
- SWE-Bench Pro 64.6
- MedChemBench (OpenAI) 48.3
- HealthBench Professional 60.5
- BenchCAD 70.6
- MMMU-Pro 84.6
- GPQA Diamond 94.6
- FrontierMath Tiers 1-3 v2 89
- FrontierMath Tier 4 v2 83
- Toolathlon 58
- OpenAI MRCR v2 91.5
- GraphWalks 90.7
- ARC-AGI-3 7.78
Dropped since the last post: Full-Duplex-Bench v1.5, tau3-bench, τ³-bench Banking
GPT-5.6 Luna: Its system card adds 29 evaluations 22 new 4 dropped 30 results in all
GPT-5.6 Preview System CardHealth
Hallucination
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: CoT-Control, Capture the Flag (Professional), Monorepo-Bench, OpenAI-Proof Q&A
Its system card adds 50 evaluations 34 new 5 dropped 51 results in all
GPT-5.6 Preview System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 83
- User Confirmations During Computer Use
- Biology Model Refusal Evaluation
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
- Multimodal Troubleshooting Virology 55.5
- ProtocolQA Open-Ended 43.5
- Tacit Knowledge and Troubleshooting
- TroubleshootingBench 48
- AAV Capsid Packaging Prediction (OpenAI) 0.529
- Hard-negative protein binding prediction 7.6
- DNA sequence design for transcription factor binding 13.7
- Virology Capabilities Test (VCT) 53.5
- Molecular Biology Capabilities Test 60
- Human Pathogen Capabilities Test 68.4
- World-Class Bio 68.3
- ReproBAIT 85
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Factuality Heavy prompts, High Stakes factuality prompts, Capture the Flag (Professional), Cyber Range, Biological safety evaluations (OpenAI)
GPT-5.6 Terra: Its system card adds 30 evaluations 20 dropped 31 results in all
GPT-5.6 Preview System CardHealth
Hallucination
Safety
Robustness
Bias
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Disallowed Content Evaluations with Representative Prompts, Deception Evaluations with Representative Prompts, OpenAI Deployment Simulation, Verbalized metagaming, CoT Monitorability, AAV Capsid Packaging Prediction (OpenAI), Hard-negative protein binding prediction, DNA sequence design for transcription factor binding, Virology Capabilities Test (VCT), Molecular Biology Capabilities Test, Human Pathogen Capabilities Test, World-Class Bio, ReproBAIT, VulnLMP, FrontierCyber, CyScenarioBench, Atomic Challenges, CyberGym, METR Time Horizon, Apollo Research strategic deception evaluations
Its system card adds 59 evaluations 29 new 60 results in all
GPT-5.6 System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 83
- User Confirmations During Computer Use
- Biology Model Refusal Evaluation
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
- Multimodal Troubleshooting Virology 55.5
- ProtocolQA Open-Ended 43.5
- Tacit Knowledge and Troubleshooting
- TroubleshootingBench 48
- AAV Capsid Packaging Prediction (OpenAI) 0.529
- Hard-negative protein binding prediction 7.6
- DNA sequence design for transcription factor binding 13.7
- Virology Capabilities Test (VCT) 53.5
- Molecular Biology Capabilities Test 60
- Human Pathogen Capabilities Test 68.4
- World-Class Bio 68.3
- ReproBAIT 85
Cyber risk
Its system card addendum adds 23 evaluations 6 new 24 results in all
GPT-5.6 – August UpdatesHealth
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
- Capture the Flag (Internal) 97.1
- CVE-Bench 94.1
- Cyber Range 83.3
- Jul 8, 2026
- Full-Duplex-Bench v1.5 80.1
- tau3-bench 86.2
- τ³-bench Banking 32
Dropped since the last post: Big Bench Audio, Audio MC
- Jul 8, 2026
- FrontierCode 1.1 42.3
- Terminal-Bench 2.1 81.5
- SWE-bench Multilingual 77.8
Dropped since the last post: SWE-Bench Pro
- Jul 7, 2026
- RoboCasa 70.8
- RoboCasa GR1 Tabletop Tasks 44.5
- SimplerEnv 62.3
- RoboLab 8.58
Dropped since the last post: Terminal-Bench 2.1, GDPval, SWE-bench Verified, SWE-bench Multilingual, ProfBench, PinchBench, tau3-bench, τ³-bench Banking, BrowseComp, Vals Finance Agent, IOI 2025, LiveCodeBench v6, IMO-AnswerBench, MathArena Apex Shortlist, GPQA Diamond, SciCode, Humanity's Last Exam, CritPt, MMLU-Pro, AA-Omniscience, IFBench, MultiChallenge, AA-LCR (Artificial Analysis Long Context Reasoning), RULER, LongBench v2, MMLU-ProX, WMT24++
- Jul 6, 2026
- SWE-Bench Pro 65.2
- KAT Code Bench 53.1
- PinchBench 94.9
- KAT Claw Bench 85.5
- Terminal-Bench 2.1 60.7
- SciCode 50.3
- Jul 5, 2026
- Terminal-Bench 2.1 70.8
- SWE-Bench Pro 59.5
- SWE-bench Multilingual 77.3
- FORTE 73.2
- BrowseComp 79.9
- RWSearch 78.8
- IFEval 90
- WritingBench 83.8
- IMO-AnswerBench 81.8
- GPQA Diamond 88.9
Dropped since the last post: AIME 2025, HMMT 2025, AMO-Bench, BrowseComp-ZH, tau2-bench, tau2-Noise (LongCat), VITA-Bench, VitaBench-Noise (LongCat), Random Complex Tasks (LongCat), Humanity's Last Exam, LiveCodeBench v6, OJBench, OIBench, SWE-bench Verified
- Jul 2, 2026
- miniF2F 100
- PutnamBench 87.4
- FATE-H 87
- FATE-X 34
- FLTEval 43.2
Dropped since the last post: SWE-bench Verified, tau3-bench, BrowseComp, AIME 2025, IFBench, COLLIE, Beyond AIME
Its technical report adds 1 evaluation 1 new 12 dropped 6 results in all
LeanstralCapabilities
- ArXivLean 17.1
Dropped since the lab’s previous technical report: LibriSpeech, FLEURS, Common Voice, Multilingual LibriSpeech, VoiceBench, MMAU, Audio Understanding Benchmark (Mistral), MMLU, GSM8K, MATH, Natural Questions, TriviaQA
- Jul 2, 2026
- SWE-bench Verified 70.9
- SWE-bench Multilingual 63.1
- SWE-Bench Pro 47.6
- Terminal-Bench 2.0 37.5
- Jul 2, 2026
- SWE-bench Multilingual 75.8
- SWE-bench Verified 78
- SWE-Bench Pro 57.9
- Terminal-Bench 2.1 71.7
- NL2Repo-Bench 45.6
- DeepSWE 28
- Hy-Backend 2.0 25
- Hy-SWE Max 49
- Hy-CompanyBench 41.7
- BrowseComp 84.2
- WideSearch 76.4
- DeepSearchQA 91
- MCP Atlas 79.1
- Toolathlon 48.5
- APEX-Agents 25.6
- Claw-Eval Pass^3 68.5
- WildClawBench 53.6
- SkillsBench 55.3
- E-Bench (Tencent) 50.2
- Hy-FinModelBench 69
- ProdBench (Tencent) 23
- Hy-SkillsWorld 45.8
- Humanity's Last Exam 53.2
- Hy-Euler pro 24.2
- GPQA Diamond 90.4
- FrontierScience 48.1
- USAMO 2026 72
- MathArena (Apex, IMO, USAMO) 38.7
- ArXivMath 52.2
- HorizonMath 7.1
- Hy-Math 60.9
- PHYBench 77.4
- CMT-Benchmark 37.8
- IMO-AnswerBench 90
- SUPERChem 54.9
- CL-bench 23.8
- CL-bench Life 17
- AA-LCR (Artificial Analysis Long Context Reasoning) 73.4
Dropped since the last post: AdvancedIF, LongBench v2, Terminal-Bench 2.0, Hy-Backend, Hy-Vibe Bench
- Jun 30, 2026Claude Sonnet 5 Anthropic Introducing Claude Sonnet 5 (launch post); System Card: Claude Sonnet 5, Table 8.1.A
- SWE-Bench Pro 63.2
- Terminal-Bench 2.1 80.4
- Humanity's Last Exam 57.4
- OSWorld-Verified 81.2
- GDPval-AA v2 1618
- BrowseComp 86.6
- FrontierCode 38.8
- AutomationBench 13.5
- Harvey LAB (Legal Agent Benchmark) 8.9
- HealthBench Professional 57.8
Dropped since the last post: GDPval-AA, GDP.pdf, Blueprint-Bench 2, BioMysteryBench, HealthBench, SWE-bench Verified, CharXiv, CritPt, ArXivMath, Riemann-bench, GraphWalks, OfficeQA
Its system card adds 67 evaluations 43 new 11 dropped 77 results in all
System Card: Claude Sonnet 5Capabilities
- SWE-bench Verified 85.2
- SWE-bench Multilingual 78.3
- SWE-bench Multimodal 28.1
- CursorBench 3.1 61.2
- USAMO 2026 79.5
- ArXivMath 65.7
- ProgramBench tests passed
- GDP.pdf 67.5
- BenchCAD 26.6
- ChartMuseum 70.1
- CharXiv 77
- OfficeQA 73.3
- OfficeQA Pro 59.4
- Real-World Finance v2 (Anthropic) 1219
- Toolathlon 54.3
- AA-Briefcase 1393
- Global MMLU 89
- MILU 89.3
- INCLUDE 86.5
- BioMysteryBench
- SpatialBench (LatchBio) 70
- scBench (LatchBio SingleCellBench) 56
- Structural Biology (Anthropic) 69
- ProteinGym 37
- Organic Chemistry (Anthropic) 81
- Protocols (Anthropic life sciences) 62
Health
- HealthBench 58.7
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 96.7
- Single-turn benign request evaluations (Anthropic) 0.59
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 92.4
- Malicious computer use (Anthropic) 84.7
Robustness
- Gray Swan Agent Red Teaming (ART) benchmark 0.1
- Gray Swan IPI Arena benchmark 0.6
- Prompt injection live bug bounty (Anthropic with Gray Swan) 0.19
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 0.31
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 2.25
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 0.93
Alignment
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Rewind-attacker cyber adversarial robustness (Anthropic), Lying about identity (Anthropic), Refusal to assist with AI safety R&D (Anthropic), FrontierSWE v1, Blueprint-Bench 2, LAB-Bench FigQA, Vals Finance Agent v2, Real-World Finance (Anthropic), MCP Atlas, Vending-Bench 2, GDPval-AA
- Jun 23, 2026
- KINA 48.3
- SuperGPQA 70.8
- Beyond AIME 87
- WorkSpaceBench 53
- Agent Startup Bench 68.8
- xDailyBench 61
- NL2Repo-Bench 47
- ProgramBench tests passed 50.3
- Terminal-Bench 2.1 71
- SWE Atlas 35.2
- MATH-Vision (MathVision) 94.5
- MMMU-Pro 82.7
- WorldVQA 53
- ZeroBench 22
- BabyVision 73.7
- CharXiv 86.4
- ERQA 72
- MMLongBench 78.3
- Video-MME 89.2
- TOMATO 79.5
- MINERVA 70.7
- OVO-Bench 80.7
- Video SimpleQA 76.4
- GDPval 87.9
- Agents' Last Exam 19.5
- Doubao Multi-Turn Bench 52.5
- Toolathlon 50.6
- SeedClawBench 66.6
- Claw-Eval-MM
- Image2FloorPlan 48
- MobileWorld 73.1
- OSWorld 78.8
- CreativeWork 42.5
- MeasureBench 62.9
- TVBench 80.5
- LVBench 78
- OVBench 70
- SciCode 59.8
- FrontierScience 75
- MSQA 50.2
Dropped since the last post: GPQA Diamond, Humanity's Last Exam, SUPERChem, BABE, CL-bench, MultiChallenge, WideSearch, BrowseComp, ResearchRubrics, XpertBench, SkillsBench, FinSearchComp, SWE-bench Multilingual, SWE-Bench Pro, PaperBench, Terminal-Bench 2.0, HiPhO, MedXpertQA-MM, VLMsAreBiased, SimpleVQA, OSWorld-Verified, Video-MMMU, MMVU, Video-Holmes, EgoTempo, MotionBench, VideoMME v2 (with subtitles), LongVideoBench, OmniVideoBench, JointAVBench, WorldSense, MMSU, WildSpeech-Bench, WenetSpeech, LibriSpeech, FLEURS, VideoReasonBench, ContPhy, MORSE-500, CG-Bench, VideoEval-Pro, ODV-Bench, LiveSports-3K, ViSpeak-Bench, CrossVid, AVMeme Exam
Its model card adds 54 evaluations 39 new 76 dropped 93 results in all
Seed2.1 Model CardCapabilities
- PresentBench 54.6
- $OneMillion-Bench 68.8
- OfficeQA Pro 70.9
- Vals Finance Agent 60.7
- APEX-Agents 33.8
- MCP Atlas 83.8
- SkillsBench 60.4
- WildClawBench 61.7
- GameWorld 31.2
- SWE-Bench Pro 57.5
- DeepSWE 32.7
- Trae Agent Bench
- FrontierScience-Research 28.3
- FrontierCS
- HorizonMath 2
- IMO 2025 65.2
- IMO-ProofBench 54.3
- IPhO 2025 79.3
- ChartQA Pro 70.9
- OCRBench v2 63.2
- CharXiv (DQ) 95.5
- MathVista 90.7
- DynaMath 73.1
- MathVerse 89.7
- EMMA 79.3
- VisuLogic 54.3
- VLMsAreBiased 83.6
- VisFactor 51.4
- RealWorldQA 86.7
- SimpleVQA 74.5
- WorldBench 67.6
- BLINK 81.4
- MMSI-Bench 35.9
- TreeBench 71.1
- EmbSpatialBench 83.4
- DUDE 82.8
- Video-Holmes 68.2
- MotionBench 74.9
- ContPhy 63.6
- LongVideoBench 80.6
- CrossVid 65
- LiveSports-3K 76.8
- HLE-Verified 42.9
- SUPERChem 59.8
- ARC-AGI-2 62.5
- FrontierScience-Olympiad 75
- LiveMathematicianBench 20.9
- MathArena (Apex, IMO, USAMO) 31.3
- AetherCode 65.8
- Humanity's Last Exam 55.7
- BrowseComp 86.2
Cyber risk
- CyberGym 70.2
AI R&D risk
- SeedKernelBench 9.21
- PostTrainBench 16.5
Dropped since the lab’s previous model card: AIME 2025, HMMT 2025, AMO-Bench, IMO-AnswerBench, LiveCodeBench v6, LiveCodeBench Pro, GPQA Diamond, PHYBench, BIOBench (ByteDance Seed), KOR-Bench, ARC-AGI, Inverse IFEval, MARS-Bench, MultiChallenge, COLLIE, EIFBench, MMLU, MMLU-Pro, LPFQA, Education (ByteDance Seed in-house), Customer Support Q&A (ByteDance Seed in-house), Information Processing (ByteDance Seed in-house), Intention Recognition (ByteDance Seed in-house), Information Extraction (ByteDance Seed in-house), Complex Workflow (ByteDance Seed in-house), MMMU, LogicVista, Scientists' First Exam (SFE), VPCT, VLMs Are Blind, HallusionBench, MMStar, MMBench-EN v1.1, MME-CC, MuirBench, MMVP, ScreenSpot-Pro, CountBench, Point-Bench, RefSpatial-Bench, CVBench, AI2D, OmniDocBench v1.5, Video-MMMU, MMVU, VCR-Bench, VideoReasonBench, TempCompass, EgoTempo, CG-Bench, ViSpeak-Bench, StreamingBench, OmniMMI, ZeroVideo, BrowseComp-ZH, GAIA, WideSearch, MM-BrowseComp, HLE-VL, SWE-bench Verified, Multi-SWE-bench, AInstein-SWE-Bench, Terminal-Bench 2.0, U-Artifacts, BFCL V4, tau2-bench, DeepConsult, DeepResearch Bench, DiscoX, FinSearchComp, XpertBench, WorldTravel, Online-Mind2Web, AndroidWorld, AIR-Bench 2024, XSTest
- Jun 22, 2026
- AgentWorldBench 56.4
Dropped since the last post: SWE-bench Verified, SWE-Bench Pro, SWE-bench Multilingual, Terminal-Bench 2.0, SkillsBench, QwenWebBench, NL2Repo-Bench, Claw-Eval, Claw-Eval Pass^3, QwenClawBench, MMLU-Pro, MMLU-Redux, SuperGPQA, C-Eval, GPQA Diamond, Humanity's Last Exam, LiveCodeBench v6, HMMT 2025, HMMT Nov 2025, HMMT 2026, IMO-AnswerBench, AIME 2026, MMMU, MMMU-Pro, MathVista, DynaMath, VLMs Are Blind, RealWorldQA, MMStar, MMBench-EN v1.1, SimpleVQA, CharXiv, CC-OCR, OCRBench, ERQA, CountBench, RefCOCO / RefCOCO+ / RefCOCOg, EmbSpatialBench, RefSpatial-Bench, Video-MME, Video-MMMU, MLVU, MVBench, V*, AndroidWorld
Its technical report adds 0 evaluations 28 dropped 1 result in all
Qwen-AgentWorld: Language World Models for General AgentsDropped since the lab’s previous technical report: SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, Terminal-Bench 2.0, Aider Polyglot, Template Following generalization (Qwen), EvalPlus, MultiPL-E, CRUXEval, LiveCodeBench v6, OJBench, Codeforces rating, FullStackBench, Spider, Bird-SQL, MMLU, MMLU-Redux, MMLU-Pro, GPQA Diamond, SuperGPQA, HMMT 2025, HMMT Nov 2025, AIME 2024, AIME 2025, AthenaBench, PrimeVul, SecCodeBench, CWEval
- Jun 16, 2026
- Humanity's Last Exam 54.7
- CritPt 20.9
- AIME 2026 99.2
- HMMT Nov 2025 94.4
- HMMT 2026 92.5
- IMO-AnswerBench 91
- GPQA Diamond 91.2
- SWE-Bench Pro 62.1
- NL2Repo-Bench 48.9
- DeepSWE 46.2
- ProgramBench
- Terminal-Bench 2.1 82.7
- FrontierSWE v1 74.4
- PostTrainBench 34.3
- SWE-Marathon 13
- MCP Atlas 76.8
- Toolathlon 48.2
Dropped since the last post: Terminal-Bench 2.0, CyberGym, BrowseComp, tau3-bench, Vending-Bench 2, Kernel Bench L3
- Jun 12, 2026
- Kimi Code Bench v2 62
- ProgramBench
- MLS-Bench 35.1
- Kimi Claw 24/7 Bench 46.9
- MCP Atlas 76
- MCPMark 81.1
Dropped since the last post: Humanity's Last Exam, BrowseComp, DeepSearchQA, WideSearch, Toolathlon, Claw-Eval Pass^3, APEX-Agents, OSWorld-Verified, Terminal-Bench 2.0, SWE-Bench Pro, SWE-bench Multilingual, SWE-bench Verified, SciCode, OJBench, LiveCodeBench v6, AIME 2026, HMMT 2026, IMO-AnswerBench, GPQA Diamond, MMMU-Pro, CharXiv, MATH-Vision (MathVision), BabyVision, V*
- Jun 10, 2026
- MMLU-Pro 77.6
- AIME 2026 69.1
- LiveCodeBench v6 69.1
- Codeforces rating 1429
- GPQA Diamond 73.2
- tau2-bench 56.2
- Humanity's Last Exam 11.9
- BBEH micro average 47.6
- MMMLU 81.5
- MMMU-Pro 54.3
- OmniDocBench v1.5
- MATH-Vision (MathVision) 70.5
- MedXpertQA-MM 49
- MRCR v2 32
Dropped since the last post: CoVoST2, FLEURS
Its technical report adds 9 evaluations 8 new 13 dropped 18 results in all
DiffusionGemma Technical ReportCapabilities
- GSM8K 96.3
- MGSM 84.8
- PutnamBench 67.4
- HumanEval 94.5
- BigCodeBench 46
- LBPP (Less Basic Python Problems) 81
- IFEval 97.4
- Natural2Code 94
- HiddenMath 80.6
Dropped since the lab’s previous technical report: SciCode, IFBench, MRCR v2, Terminal-Bench Hard, MATH-Vision (MathVision), MedXpertQA-MM, InfographicVQA, OmniDocBench v1.5, CoVoST2, FLEURS, RULER, LOFT, GraphWalks
- Jun 9, 2026 North Mini Code 1.0 Cohere Introducing North Mini Code: Cohere's first model for developers (Cohere blog)
- Terminal-Bench 2.0 36
- Terminal-Bench Hard 31.1
- SWE-bench Verified 67.6
- SWE-Bench Pro 40.2
- SciCode 38.2
- LiveCodeBench v6 70.3
Dropped since the last post: Open ASR Leaderboard, LibriSpeech
- Jun 9, 2026Claude Fable 5 Anthropic Claude Fable 5 and Claude Mythos 5 (launch post); System Card: Claude Fable 5 & Claude Mythos 5, Table 8.1.A
- SWE-Bench Pro 80.3
- FrontierCode 29.3
- GDPval-AA 1932
- GDP.pdf 29.8
- Blueprint-Bench 2 38.6
- AutomationBench 17.4
- OSWorld-Verified 85
- Harvey LAB (Legal Agent Benchmark) 16.9
- Humanity's Last Exam 64.5
- BioMysteryBench 83.9
- Terminal-Bench 2.1 88
- HealthBench Professional 66
- HealthBench 62.7
- SWE-bench Verified 95.5
- BrowseComp 93.3
- CharXiv 93.5
- CritPt 28.6
- ArXivMath 78.5
- Riemann-bench 55
- GraphWalks 100
- OfficeQA 57.9
Dropped since the last post: SWE-bench Multilingual, SWE-bench Multimodal, ChartQA Pro, GPQA Diamond, ScreenSpot-Pro, Vals Finance Agent v2, MCP Atlas, Online-Mind2Web
Its system card adds 31 evaluations 5 new 64 dropped 41 results in all
System Card: Claude Fable 5 & Claude Mythos 5Capabilities
- FrontierSWE v1
- CursorBench 3.1 72.9
- LAB-Bench FigQA
- OfficeQA Pro 57.9
- Vals Finance Agent v2 56.3
- Real-World Finance v2 (Anthropic) 1374
- Real-World Finance (Anthropic) 70
- MCP Atlas 83.3
- Vending-Bench 2 5680.3
- Toolathlon 61.7
Safety
- Single-turn harmful request evaluations (Anthropic) 96.9
- Single-turn benign request evaluations (Anthropic) 0.01
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 90.3
- Malicious computer use (Anthropic) 85.7
Robustness
- Rewind-attacker cyber adversarial robustness (Anthropic) 5
- Gray Swan Agent Red Teaming (ART) benchmark 0.1
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 0.45
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 0.82
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 29.7
Alignment
Dropped since the lab’s previous system card: Long-form virology tasks, Virology Capabilities Test (VCT), DNA Synthesis Screening Evasion, Black-box RNA sequence modeling and design, AAV capsid packaging prediction, Anthropic ECI (AECI), ExploitBench, CyberGym, Firefox 147 exploit evaluation (Anthropic), Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based), Prompt injection live bug bounty (Anthropic with Gray Swan), Petri, Evaluation awareness discriminability (Anthropic), UK AISI unprompted sabotage evaluation, UK AISI sabotage continuation evaluation, Over-eager behavior on GUI computer use tasks (Anthropic), Adherence to its constitution (Anthropic), 100Q-Hard (Anthropic), SimpleQA Verified, AA-Omniscience, ECLeKTic, False-premise questions (Anthropic), MASK, Missing-context hallucinations (Anthropic), Self-preference bias (Anthropic), Uncritically reporting flawed results (Anthropic), Code summary honesty (Anthropic), Lazy investigation on misleading codebases (Anthropic), Overconfidence: verifying command-line syntax (Anthropic), SHADE-Arena, Minimal-LinuxBench (Anthropic), Intentionally taking actions very rarely (Anthropic), Hiding a secret password (Anthropic), CoT-Control, OpenAI monitorability evals, SWE-bench Multilingual, SWE-bench Multimodal, BrowseComp, Humanity's Last Exam, ChartQA Pro, GPQA Diamond, ScreenSpot-Pro, GraphWalks, ProgramBench tests passed, USAMO 2026, ArXivMath, DeepSearchQA, DRACO, ChartMuseum, CharXiv, OfficeQA, HealthBench Professional, Global MMLU, MILU, INCLUDE, BioPipelineBench (Anthropic), BioMysteryBench, SpatialBench (LatchBio), scBench (LatchBio SingleCellBench), Structural Biology (Anthropic), ProteinGym, Organic Chemistry (Anthropic), Protocols (Anthropic life sciences), LABBench2
- Jun 8, 2026
- Jun 4, 2026
- Terminal-Bench 2.1 56.4
- GDPval 46.7
- SWE-bench Verified 70.7
- SWE-bench Multilingual 67.7
- ProfBench 56
- PinchBench 90
- tau3-bench 86.9
- τ³-bench Banking 22.6
- BrowseComp 44.4
- Vals Finance Agent 60.1
- IOI 2025 570
- LiveCodeBench v6 89
- IMO-AnswerBench 88.6
- MathArena Apex Shortlist 74.9
- GPQA Diamond 87
- SciCode 44.6
- Humanity's Last Exam 26.7
- CritPt 3.1
- MMLU-Pro 86.8
- AA-Omniscience 24.1
- IFBench 81.7
- MultiChallenge 63.8
- AA-LCR (Artificial Analysis Long Context Reasoning) 65.4
- RULER 94.7
- LongBench v2 61.9
- MMLU-ProX 83
- WMT24++ 83.7
Dropped since the last post: AIME 2025, HMMT 2025, LiveCodeBench v5, Terminal-Bench Hard, Terminal-Bench 2.0, tau2-bench, Bird-SQL, Arena-Hard v2.0
Nemotron 3 Ultra 550B-A55B Base: Its model card adds 14 evaluations 14 new 3 dropped 17 results in all
NVIDIA-Nemotron-3-Ultra-550B-A55B-Base model card (Hugging Face)Capabilities
- MMLU 89.1
- AGIEval 78.7
- GSM8K 88.1
- MATH 82
- HumanEval 83.8
- MBPP 86
- ARC-Challenge 97.3
- HellaSwag 90.5
- OpenBookQA 48.6
- PIQA 83.8
- WinoGrande 79.3
- RACE 92.2
- Global MMLU Lite 3.0 90.1
- MGSM 87.7
Dropped since the lab’s previous model card: LIBERO, SimplerEnv, RoboCasa GR1 Tabletop Tasks
Its technical report adds 5 evaluations 5 new 26 dropped 32 results in all
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningCapabilities
- IMO-ProofBench 82.3
- IMO 2025 83.3
- Putnam 2025 96.7
- USAMO 2026 97.6
Other
- SPEED-Bench 5.15
Dropped since the lab’s previous technical report: UniGenBench, PAI-Bench, RBench, Physics-IQ, MMBench, RealWorldQA, CVBench, VideoPhy-2, CausalVQA, MVPBench, CountBench, AI2D, DocVQA, InfographicVQA, OCRBench v2, LogicVista, MMMU-Pro, MVBench, RefCOCO / RefCOCO+ / RefCOCOg, HallusionBench, RefSpatial-Bench, VSI-Bench, ERQA, RoboSpatial-Home, Where2Place, LingoQA
- Jun 3, 2026
- MMLU-Pro 77.2
- AIME 2026 77.5
- LiveCodeBench v6 72
- Codeforces rating 1659
- GPQA Diamond 78.8
- tau2-bench 69
- Humanity's Last Exam 5.2
- BBEH micro average 53
- MMMLU 83.4
- MMMU-Pro 69.1
- OmniDocBench v1.5 0.164
- MATH-Vision (MathVision) 79.7
- MedXpertQA-MM 48.7
- CoVoST2 38.5
- FLEURS 0.069
- MRCR v2 43.4
- Jun 2, 2026
- SWE-bench Verified 71.6
- SWE-Bench Pro 51.2
- SWE-bench Multilingual 65.5
- Terminal-Bench 2.0 54.8
- AIME 2026 92.5
- AMO-Bench 40
- FrontierMath 6.3
- Humanity's Last Exam 18
- GPQA Diamond 84.6
- FrontierScience 58.2
- ArtifactsBench 36.4
- IFBench 75
- AdvancedIF 71.4
- Robust IF Bench 61.2
- tau2-bench 71.7
Dropped since the last post: AIME 2025, HMMT 2026, LiveCodeBench v6, MMLU-Pro, SimpleQA Verified, MultiChallenge, GraphWalks, Berkeley Function Calling Leaderboard (BFCL), LongFact, TruthfulQA, HealthBench Professional, MedXpertQA
Its model card adds 0 evaluations 15 results in all
- Jun 2, 2026
MAI-Thinking-1 Microsoft MAI-Thinking-1: Building a Hill-Climbing Machine (Microsoft AI technical report), Tables 11 and 12 - AIME 2025 97
- AIME 2026 94.5
- HMMT 2026 84.9
- GPQA Diamond 84.2
- LiveCodeBench v6 87.7
- Terminal-Bench 2.0 46
- SWE-bench Verified 73.5
- SWE-Bench Pro 52.8
- MMLU-Pro 85
- SimpleQA Verified 31
- IFBench 69
- AdvancedIF 85
- MultiChallenge 53
- GraphWalks 90
- Berkeley Function Calling Leaderboard (BFCL) 72
- LongFact 98
- TruthfulQA 88
- HealthBench Professional 35
- MedXpertQA 43
Dropped since the last post: WebVoyager, Online-Mind2Web, WebTailBench
Its technical report adds 9 evaluations 28 results in all
MAI-Thinking-1: Building a Hill-Climbing MachineCapabilities
- LongBench v2 61
- CorpusQA 82
- OpenAI MRCR v2 60
- MAI human side-by-side evaluation 0.07
Robustness
Cyber risk
- Jun 1, 2026
- Terminal-Bench 2.0 70.3
- SWE-bench Verified 77.7
- SWE-Bench Pro 57.6
- SWE-bench Multilingual 75.8
- NL2Repo-Bench 41.1
- SciCode 51.3
- QwenWebDev 1536
- QwenSVG 1588
- QwenClawBench 61.8
- CoWorkBench 65.1
- Claw-Eval 62.7
- SkillsBench 54.9
- BFCL V4 72.9
- MCPMark 58.7
- MCP Atlas 73.2
- VITA-Bench 45.6
- DeepPlanning 62.3
- SpreadsheetBench 86.3
- Kernel Bench L3 2.06
- QwenWorldBench 62.1
- GPQA Diamond 90.3
- Humanity's Last Exam 34.7
- LiveCodeBench 89.6
- HMMT 2026 92.9
- IMO-AnswerBench 86
- CritPt 6
- MathArena (Apex, IMO, USAMO) 22.7
- MMLU-Pro 88.5
- MMLU-Redux 94.5
- SuperGPQA 71.4
- IFEval 94.6
- IFBench 79.1
- MRCR v2 91.7
- WMT24++ 84.6
- MAXIFE 88.8
- MMMLU 89
- MMLU-ProX 85.4
- NOVA-63 58.8
- INCLUDE 83
- Global PIQA 90.3
- PolyMath 84
- MMMU-Pro 79
- MATH-Vision (MathVision) 90.3
- BabyVision 70.4
- CharXiv 85.9
- HiPhO 84.1
- ERQA 69.8
- VisFactor 42.8
- MedXpertQA-MM 71
- ScreenSpot-Pro 79
- OSWorld-Verified 73.3
- AndroidWorld 81
- QwenVision2Code 1772
- Claw-Eval-MM 55.7
- SimpleVQA 81.7
- WorldVQA 61.1
- MMSearch-Plus 41.4
- BrowseComp-VL 51.1
- MM-BrowseComp 46.3
- RealWorldQA 86.9
- CountQA 77
- OmniDocBench v1.5 91.4
- OCRBench v2 70.7
- ODinW-13 51.1
- LingoQA 83.4
- Ego3D-Bench 5.9
- SURDS 77.2
- VLADBench 77.2
- Video-MME 88
- Video-MMMU 85.4
- MLVU 87.4
- TVBench 78.2
- LVBench 76.2
- Jun 1, 2026
MiniMax-M3 MiniMax MiniMax M3 launch blog (minimax.io/blog/minimax-m3) and Hugging Face card benchmark table (figures/benchmark.jpeg) - SWE-bench Verified 80.5
- SWE-Bench Pro 59
- Terminal-Bench 2.1 66
- SWE Atlas 37.9
- NL2Repo-Bench 42.1
- SWE Atlas 30.8
- SWE-fficiency 34.8
- LiveSQLBench 40.2
- CL-bench 20.5
- VIBE-V2 (MiniMax) 50.1
- SVG-Bench (MiniMax) 63.7
- PostTrainBench 37.1
- KernelBench 28.8
- PaperBench 52.6
- BrowseComp 83.5
- DRACO 73.2
- GDPval-Rubrics (MiniMax) 74.8
- BankerToolBench 76.1
- OfficeQA 45.1
- SpreadsheetBench 89.3
- YC-Bench 2100000
- LOCA-Bench 49.3
- MCP Atlas 74.2
- APEX-Agents 27.7
- Claw-Eval 74.5
- OSWorld-Verified 75.2
- OmniDocBench v1.5 91.6
- MMMU-Pro 78.1
- Video-MMMU 84.6
- Video-MME 85.4
- IMO 2025 35
- USAMO 2026 36
Dropped since the last post: Multi-SWE-bench, SWE-bench Multilingual, VIBE-Pro (MiniMax), Terminal-Bench 2.0, MLE-bench, GDPval-AA, Toolathlon, MM-ClawBench, Artificial Analysis Intelligence Index v4.0
- May 29, 2026
- Humanity's Last Exam 47.2
- BrowseComp 75.8
- DeepSearchQA 92.8
- ResearchRubrics 71.7
- Toolathlon 49.5
- Claw-Eval Pass^3 67.1
- GDPval-AA 1415.8
- SWE-bench Multilingual 72.4
- SWE-Bench Pro 56.3
- SWE-bench Verified 76.5
- Terminal-Bench 2.1 59.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 63.9
- SimpleVQA 79.2
- WorldVQA 58.1
- BrowseComp-VL 59
- V* 95.3
- HR-Bench 87.7
- VisualProbe 65
- AndroidDaily 61.9
- Step-SWE-Bench 67.1
Dropped since the last post: tau2-bench, BrowseComp-ZH, GAIA, xbench-DeepSearch, xbench-DeepSearch-2510, AIME 2025, HMMT 2025, HMMT Nov 2025, IMO-AnswerBench, LiveCodeBench v6, Terminal-Bench 2.0
- May 28, 2026
- AA-Omniscience -24.7
- IFEval 91.8
- IFBench 56.5
- Multi-IF 79.9
- MATH-500 88.8
- AIME 2025 42.5
- AIME 2026 50
- Berkeley Function Calling Leaderboard (BFCL) 64.4
- BFCL V4 48.5
- tau2-bench 64
Dropped since the last post: MMLU, MMLU-Pro, GPQA Diamond, GSM8K, GSM-Plus, MATH Level 5, MGSM, MMMLU, LiveCodeBench v6, LiveCodeBench v5, HumanEval+, Creative Writing v3
- May 28, 2026Claude Opus 4.8 Anthropic Introducing Claude Opus 4.8 (launch post); System Card: Claude Opus 4.8, Table 8.1.A
- SWE-bench Verified 88.6
- SWE-Bench Pro 69.2
- SWE-bench Multilingual 84.4
- SWE-bench Multimodal 38.4
- BrowseComp 88.5
- Terminal-Bench 2.1 74.6
- Humanity's Last Exam 57.9
- ChartQA Pro 72.3
- OSWorld-Verified 83.4
- GPQA Diamond 93.6
- ScreenSpot-Pro 87.9
- Vals Finance Agent v2 53.9
- GDPval-AA 1890
- MCP Atlas 82.2
- AutomationBench 15.5
- GraphWalks 99.3
- Online-Mind2Web 84
Dropped since the last post: Terminal-Bench 2.0, Vals Finance Agent, CharXiv, MMMLU, OfficeQA, OfficeQA, ARC-AGI, ARC-AGI-2, CursorBench 3.1, BigLaw Bench, XBOW Validation Benchmarks
Its system card adds 81 evaluations 25 new 23 dropped 97 results in all
System Card: Claude Opus 4.8Capabilities
- FrontierSWE v1
- ProgramBench tests passed 79
- USAMO 2026 96.7
- ArXivMath 71.8
- DeepSearchQA 93.1
- DRACO 80.4
- ChartMuseum 75.8
- LAB-Bench FigQA 80.4
- CharXiv 80.5
- OfficeQA 77.6
- OfficeQA Pro 66.2
- Harvey LAB (Legal Agent Benchmark) 9.62
- Vending-Bench 2 2992.3
- Toolathlon 59.9
- Global MMLU 90.4
- MILU 90.3
- INCLUDE 87.6
- BioPipelineBench (Anthropic) 87.7
- BioMysteryBench
- SpatialBench (LatchBio) 53.3
- scBench (LatchBio SingleCellBench) 58.2
- Structural Biology (Anthropic) 79
- ProteinGym 39.6
- Organic Chemistry (Anthropic) 86.2
- Protocols (Anthropic life sciences) 59.6
- LABBench2
Health
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 98
- Single-turn benign request evaluations (Anthropic) 0.36
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 95.1
- Malicious computer use (Anthropic) 81.7
Robustness
- Gray Swan Agent Red Teaming (ART) benchmark 0.1
- Prompt injection live bug bounty (Anthropic with Gray Swan) 0.4
- Prompt injection robustness against adaptive attackers, coding (Anthropic) 7.03
- Prompt injection robustness against adaptive attackers, computer use (Anthropic) 7.14
- Prompt injection robustness against adaptive attackers, browser use (Anthropic) 31.5
Alignment
- Anthropic automated behavioral audit
- Petri
- Evaluation awareness discriminability (Anthropic)
- UK AISI unprompted sabotage evaluation 0
- UK AISI sabotage continuation evaluation 1
- Over-eager behavior on GUI computer use tasks (Anthropic) 9.4
- Adherence to its constitution (Anthropic) 0.99
- MASK 95.4
- Lying about identity (Anthropic) 97
- Refusal to assist with AI safety R&D (Anthropic)
- Self-preference bias (Anthropic)
- Uncritically reporting flawed results (Anthropic) 9.4
- Code summary honesty (Anthropic) 3.7
- Lazy investigation on misleading codebases (Anthropic) 0
- Overconfidence: verifying command-line syntax (Anthropic) 4.51
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Anthropic ECI (AECI) 155.5
Dropped since the lab’s previous system card: Kernels task (Anthropic), Time series forecasting task (Anthropic), LLM training task (Anthropic), Quadruped reinforcement learning task (Anthropic), Novel compiler task (Anthropic), Cybench, UK AISI cyber ranges, Higher-difficulty harmful request evaluations (Anthropic), Higher-difficulty benign request evaluations (Anthropic), Impossible tasks (Anthropic), Agentic code behavior scores (Anthropic), Destructive production evaluation (Anthropic), Destructive actions (Anthropic), Input hallucinations (Anthropic), DTBench, Terminal-Bench 2.0, MMMLU, Vals Finance Agent, ARC-AGI, ARC-AGI-2, OpenAI MRCR v2, BioPipelineBench Verified (Anthropic), Phylogenetics (Anthropic)
- May 25, 2026
- LongVideoBench 74.1
- VideoMME v2 (with subtitles) 42.4
- TimeLens-Bench 62.3
- MLVU 82.8
- Video-MME 78.3
- Video-MMMU 80
- LiveCodeBench v6 77.1
- OJBench 39.2
- SWE-bench Verified 62
- BFCL V4 65.7
- tau2-bench 82.6
- VITA-Bench 33.1
- We-Math 75.5
- MathVista 82.9
- DynaMath 80.9
- AIME 2025 86.7
- AIME 2026 93.3
- Humanity's Last Exam 22.9
- MMMU 75.3
- MMStar 75.9
- C-Eval 89.2
- IFEval 92.2
- HallusionBench 73.5
- AI2D 87.8
- OCRBench 85.7
- OmniDocBench v1.5 84.7
- RefCOCO / RefCOCO+ / RefCOCOg 88.1
- PixMo-Count 83.5
- EmbSpatialBench 83.2
Its technical report adds 0 evaluations 29 results in all
- May 22, 2026
- SWE-bench Verified 77.6
- tau3-bench 63.2
- BrowseComp 48.6
- AIME 2025 86.3
- IFBench 69
- COLLIE 95.8
- Beyond AIME 66.9
Dropped since the last post: FLTEval
- May 21, 2026
- WebVoyager 89.3
- Online-Mind2Web 72.3
- WebTailBench 40.2
Its technical report adds 4 evaluations 4 new 28 dropped 7 results in all
Fara-1.5: Scalable Learning Environments for Computer Use AgentsCapabilities
- ScreenSpot-v2 94.4
- ScreenSpot-Pro 66.8
- OSWorld-G 71.3
Safety
Dropped since the lab’s previous technical report: AIME 2025, AIME 2026, HMMT 2026, GPQA Diamond, LiveCodeBench v6, Terminal-Bench 2.0, SWE-bench Verified, SWE-Bench Pro, MMLU-Pro, SimpleQA Verified, IFBench, AdvancedIF, MultiChallenge, GraphWalks, Berkeley Function Calling Leaderboard (BFCL), AIR-Bench 2024, CyberSecEval Instruct, CyberSecEval Autocomplete, LongFact, TruthfulQA, HealthBench Professional, MedXpertQA, LongBench v2, CorpusQA, OpenAI MRCR v2, MAI human side-by-side evaluation, MAI internal safety and over-refusal evaluation, MAI internal jailbreak evaluation suite
- May 20, 2026 Command A+ Cohere Introducing Command A+: Making sovereign agentic capabilities available to all (Cohere blog)
- tau2-bench 85
- Terminal-Bench Hard 25
- IFBench 74
- AIME 2025 90
- SciCode 38
- MMMU 75.1
- MMMU-Pro 63
- MathVista 80.6
- CharXiv 52.7
- MT-AIME (Cohere internal) 86
- WMT24++ 81
- Artificial Analysis Intelligence Index v4.0 37
- May 20, 2026
- Terminal-Bench 2.0 69.7
- SWE-bench Verified 80.4
- SWE-Bench Pro 60.6
- SWE-bench Multilingual 78.3
- NL2Repo-Bench 47.2
- SciCode 53.5
- QwenWebDev 1568
- QwenSVG 1608
- QwenClawBench 64.3
- CoWorkBench 67.2
- Claw-Eval 65.2
- SkillsBench 59.2
- BFCL V4 75
- MCPMark 60.8
- MCP Atlas 76.4
- VITA-Bench 47.9
- SpreadsheetBench 87
- Kernel Bench L3 1.98
- Humanity's Last Exam 53.5
- QwenWorldBench 57.3
- GPQA Diamond 92.4
- LiveCodeBench 91.6
- HMMT 2026 97.1
- IMO-AnswerBench 90
- CritPt 11.4
- MathArena (Apex, IMO, USAMO) 44.5
- MMLU-Pro 89.6
- MMLU-Redux 95
- SuperGPQA 73.6
- IFEval 94.3
- IFBench 79.1
- MRCR v2 90.4
- WMT24++ 85.8
- MAXIFE 89.2
- MMMLU 90.3
- MMLU-ProX 87
- NOVA-63 59
- INCLUDE 86.2
- Global PIQA 91.4
- PolyMath 86.5
Dropped since the last post: AA-Omniscience, GDPval-AA, QwenChineseBench, ToolcallFormatIFBench, QwenWebBench
- May 19, 2026
- Terminal-Bench 2.1 76.2
- SWE-Bench Pro 55.1
- MCP Atlas 83.6
- Toolathlon 56.5
- OSWorld-Verified 78.4
- Vals Finance Agent v2 57.9
- GDPval-AA 1656
- CharXiv 84.2
- MMMU-Pro 83.6
- Blueprint-Bench 2 33.6
- MRCR v2 77.3
- Humanity's Last Exam 40.2
- ARC-AGI-2 72.1
Dropped since the last post: Artificial Analysis Speech Arena
Its model card adds 5 evaluations 5 new 3 dropped 18 results in all
Gemini 3.5 Flash Model CardDropped since the lab’s previous model card: GenAI-Bench, Human SxS image generation Elo (Google), Human SxS image editing Elo (Google)
- May 18, 2026
- May 14, 2026
- PinchBench 87.6
- Claw-Eval 63.8
- tau2-bench 95.3
- ARC-AGI-2 66.2
- AIME 2026 95.8
- GPQA Diamond 88.3
Dropped since the last post: SWE-bench Verified, Claw-Eval Pass^3, BFCL V4, IFBench, MRCR v2
Its technical report adds 7 evaluations 3 new 12 dropped 12 results in all
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter ScaleCapabilities
- HMMT 2026 93.5
- IMO-AnswerBench 86.1
- LiveCodeBench v6 87
- Claw-Eval Pass^3 63.8
- SWE-bench Verified 74
- SWE-Bench Pro 53.8
- Meta Agents Research Environments (ARE) / Gaia2 77.9
Dropped since the lab’s previous technical report: Chinese SimpleQA, SimpleQA Verified, SuperGPQA, Humanity's Last Exam, BBEH micro average, HMMT Nov 2025, ARC-AGI, BFCL V4, Terminal-Bench 2.0, IFBench, LongBench v2, MRCR v2
- May 11, 2026
- Full-Duplex-Bench 0.4
- Full-Duplex-Bench v1.5 77.8
- Full-Duplex-Bench v3 68
- QIVD 54
- Audio MC 43.4
- Big Bench Audio 75.7
- VoiceBench 82.1
- IFEval 89.7
- HarmBench 99
- TimeSpeak 64.7
- CueSpeak 81.7
- RepCount-A 35.4
- ProactiveVideoQA 33.5
- Charades-STA 32.4
- May 9, 2026
- AIME 2026 99.6
- tau3-bench 67.9
- DeepSearchQA 77.3
- SpreadsheetBench Verified 72.5
- GPQA Diamond 91
- MMLU-Pro 84.3
- AdvancedIF 72.3
- LMArena Search Arena 1223
Dropped since the last post: SimpleQA, Chinese SimpleQA, Humanity's Last Exam, AIME 2025, HMMT 2025, LiveCodeBench v6, HumanEval+, MBPP+, ZebraLogic, BBEH micro average, IFEval, MultiChallenge, Multi-IF, tau2-bench, AceBench, BFCL V4, BrowseComp-ZH, SpreadsheetBench, MMMU-Pro, MathVista, MathVerse, MATH-Vision (MathVision), VisualPuzzles, VisuLogic, VLMs Are Blind, ChartQA, AI2D, DocVQA, OCRBench, CharXiv, SimpleVQA, HallusionBench, MMStar, BLINK, CVBench, CountBench, Video-MME, Video-MMMU, MMVU, GenEval, VBench
- May 7, 2026
- Big Bench Audio 96.6
- Audio MC 48.5
Dropped since the last post: CharXiv, MMMU-Pro, OmniDocBench, GPQA Diamond, AIME 2025
- May 5, 2026
- CharXiv 81.6
- MMMU-Pro 76
- OmniDocBench 12.5
- GPQA Diamond 85.6
- AIME 2025 81.2
Dropped since the last post: GDPval, BrowseComp, FrontierMath, FrontierMath Tier 4, Investment Banking Modeling Tasks (OpenAI), GeneBench, Humanity's Last Exam
Its system card adds 22 evaluations 3 new 22 dropped 22 results in all
GPT-5.5 Instant System CardHealth
Hallucination
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
- Capture the Flag (Professional) 94.1
- CVE-Bench 90.2
- Cyber Range 76.9
Dropped since the lab’s previous system card: Disallowed Content Evaluations with Representative Prompts, Destructive action avoidance, Destructive action reversion on long rollouts, User Confirmations During Computer Use, Deception Evaluations with Representative Prompts, OpenAI Deployment Simulation, CoT Monitorability, CoT-Control, Biochemistry knowledge improvement, Hard-negative protein binding prediction, DNA sequence design for transcription factor binding, VulnLMP, Atomic Challenges, CyScenarioBench, UK AISI narrow cyber tasks, UK AISI cyber ranges, Monorepo-Bench, MLE-bench 30, Internal Research Debugging Evaluation, OpenAI-Proof Q&A, Apollo Research strategic deception evaluations, BashArena
- May 5, 2026
- MolmoSpaces 20.6
- RoboEval 0.443
- LIBERO 97.2
Dropped since the last post: WebVoyager, Online-Mind2Web, DeepShop, WebTailBench
Its technical report adds 0 evaluations 4 dropped 3 results in all
MolmoAct2: Action Reasoning Models for Real-world DeploymentDropped since the lab’s previous technical report: WebVoyager, Online-Mind2Web, DeepShop, WebTailBench
- Apr 29, 2026
- MMLU 80.2
- MMLU-Pro 64.1
- BIG-Bench Hard 83.7
- AGIEval 77.8
- GPQA Diamond 45.8
- SimpleQA 6.81
- AlpacaEval 2.0 56.2
- IFEval 89.7
- Arena-Hard-Auto 71
- MT-Bench 8.61
- GSM8K 94.2
- GSM-Symbolic 75.7
- MATH 81.3
- DeepMind Mathematics 81.9
- HumanEval 88.4
- HumanEval+ 85.4
- MBPP 85.5
- MBPP+ 73.5
- CRUXEval 55.8
- BigCodeBench 38.8
- MultiPL-E 62.3
- Berkeley Function Calling Leaderboard (BFCL) 73.7
- MMMLU 73.7
- INCLUDE 67.3
- MGSM 71.1
- SALAD-Bench 96.4
- AttaQ 85.8
- Tulu 3 Safety Eval 78.2
- Apr 29, 2026
- AIME 2026 87.4
- SWE-bench Verified 72.2
- PinchBench 87.4
- Claw-Eval Pass^3 51
- BFCL V4 70.6
- tau2-bench 78.4
- IFBench 57
- MRCR v2 80.4
Dropped since the last post: IMO-AnswerBench, HMMT Nov 2025, LiveCodeBench v6, Meta Agents Research Environments (ARE) / Gaia2, ARC-AGI-2, IMO 2025, CMO 2025
Its technical report adds 12 evaluations 7 new 22 dropped 20 results in all
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter ScaleCapabilities
- Chinese SimpleQA 76.5
- SimpleQA Verified 31.5
- GPQA Diamond 76.2
- SuperGPQA 58.3
- Humanity's Last Exam 10.1
- LiveCodeBench v6 65.6
- BBEH micro average 52.4
- HMMT Nov 2025 81.9
- IMO-AnswerBench 65.8
- ARC-AGI 50.9
- Terminal-Bench 2.0 40.5
- LongBench v2 48.3
Dropped since the lab’s previous technical report: AIME 2025, Omni-MATH, HMMT 2025, CNMO 2024, FinanceReasoning, UGMathBench, Codeforces rating, Aider-Edit, ZebraLogic, MMLU-Pro, C-Eval, PHYBench, AGIEval, TriviaQA, CMMLU, Arena-Hard v2.0, Creative Writing v3, IFEval, HealthBench, MultiChallenge, Berkeley Function Calling Leaderboard (BFCL), IMO 2025
- Apr 28, 2026 Nemotron 3 Nano Omni 30B-A3B NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 model card (Hugging Face)
- CVBench 84
- OCRBench v2 67
- OSWorld-Verified 47.4
- CharXiv 63.6
- MMLongBench-Doc 57.5
- MathVista 82.8
- OCR-Reasoning 54.1
- Video-MME 72.2
- WorldSense (omnimodal) 55.4
- Daily-Omni 74.5
- VoiceBench 89.4
- MMAU 74.6
Dropped since the last post: IMO 2025, IMO-AnswerBench, IMO-ProofBench, AIME 2025, AIME 2026, HMMT 2025, IOI 2025, ICPC World Finals 2025, LiveCodeBench v6, LiveCodeBench Pro, SciCode, MMLU-Redux, MMLU-Pro, GPQA Diamond, Humanity's Last Exam, Arena-Hard v2.0, IFBench, MultiChallenge, AA-LCR (Artificial Analysis Long Context Reasoning), LongBench v2, RULER, CL-bench, BFCL V4, tau2-bench, Terminal-Bench 2.0, SWE-bench Verified, MMLU-ProX, WMT24++
Its technical report adds 22 evaluations 15 new 22 dropped 34 results in all
Nemotron 3 Nano Omni: Efficient and Open Multimodal IntelligenceCapabilities
- MMMU 70.8
- OCRBench 86.6
- ChartQA 90.3
- DocVQA 95.6
- AI2D 88.5
- TextVQA 81
- InfographicVQA 86.8
- CharXiv (DQ) 88.9
- TreeBench 51.6
- RefCOCO / RefCOCO+ / RefCOCOg 90.5
- ScreenSpot 89.3
- ScreenSpot-v2 92.8
- ScreenSpot-Pro 57.8
- Open ASR Leaderboard 5.95
- MMLU-Pro 77.3
- GPQA Diamond 72.2
- LiveCodeBench v5 63.2
- AIME 2025 82.1
- IFBench 74.2
- AA-LCR (Artificial Analysis Long Context Reasoning) 41
- tau2-bench
- SciCode 32
Dropped since the lab’s previous technical report: IMO 2025, IMO-AnswerBench, IMO-ProofBench, AIME 2026, HMMT 2025, IOI 2025, ICPC World Finals 2025, LiveCodeBench v6, LiveCodeBench Pro, Codeforces rating, MMLU-Redux, Humanity's Last Exam, Arena-Hard v2.0, MultiChallenge, LongBench v2, RULER, CL-bench, BFCL V4, Terminal-Bench 2.0, SWE-bench Verified, MMLU-ProX, WMT24++
- Apr 28, 2026
- SWE-bench Verified 68.2
- SWE-bench Multilingual 62.4
- SWE-Bench Pro 44.5
- Terminal-Bench 2.0 30.1
Its technical report adds 0 evaluations 4 results in all
- Apr 28, 2026
- SWE-bench Verified 72.5
- SWE-bench Multilingual 67.3
- SWE-Bench Pro 46.9
- Terminal-Bench 2.0 40.7
Its technical report adds 0 evaluations 4 results in all
- Apr 28, 2026
- GPQA Diamond 88.4
- SuperGPQA 69.6
- Humanity's Last Exam 25.7
- Beyond AIME 79
- FrontierScience 72
- SUPERChem 55
- BABE 57.9
- CL-bench 20.1
- MultiChallenge 69.9
- WideSearch 70.3
- BrowseComp 64
- ResearchRubrics 59.2
- XpertBench 56.8
- SkillsBench 43.7
- GDPval 53.1
- FinSearchComp 63.8
- SWE-bench Multilingual 66.6
- SWE-Bench Pro 46.6
- NL2Repo-Bench 28.7
- PaperBench 52.5
- Terminal-Bench 2.0 43.3
- MATH-Vision (MathVision) 89.8
- MMMU-Pro 78.4
- HiPhO 83.8
- MedXpertQA-MM 79.6
- BabyVision 64.7
- VLMsAreBiased 80.6
- SimpleVQA 72.7
- WorldVQA 50.2
- CharXiv 82.4
- ERQA 71.5
- OSWorld-Verified 64.4
- MobileWorld 64.6
- Video-MMMU 88.3
- MMVU 76.7
- Video SimpleQA 71.7
- Video-Holmes 67.4
- MINERVA 68.5
- TVBench 80.4
- TOMATO 72.5
- EgoTempo 68.4
- MotionBench 72.4
- Video-MME 89
- VideoMME v2 (with subtitles) 64.9
- LongVideoBench 79
- LVBench 76.4
- OVBench 63.2
- OVO-Bench 75.4
- OmniVideoBench 61.7
- JointAVBench 69.5
- WorldSense 67.3
- MMSU 86.5
- WildSpeech-Bench 75.8
- WenetSpeech 4.89
- LibriSpeech 1.62
- FLEURS
- VideoReasonBench 59.4
- ContPhy 62.4
- MORSE-500 34.6
- CG-Bench 65.5
- VideoEval-Pro 49.5
- ODV-Bench 66
- LiveSports-3K 78.1
- ViSpeak-Bench 87
- CrossVid 63.7
- AVMeme Exam 69.5
Dropped since the last post: MMLU-Pro, HLE-Verified, SimpleQA Verified, HealthBench, HealthBench Hard, LPFQA, Encyclo-K, AIME 2026, AIME 2025, HMMT 2025, HMMT Nov 2025, MathArena (Apex, IMO, USAMO), IMO-AnswerBench, Codeforces rating, AetherCode, LiveCodeBench v6, PHYBench, ARC-AGI, ARC-AGI-2, KOR-Bench, ProcBench, MRCR v2, GraphWalks, LongBench v2, FRAMES, DeR2 Bench, Global PIQA, MMMLU, DiscoX, COLLIE, MARS-Bench, Inverse IFEval, LongFact, FActScore precision, SWE-Lancer, SWE-bench Verified, Multi-SWE-bench, SciCode, SWE-EVO, Aider Polyglot, ArtifactsBench, CodeSimpleQA, SpreadsheetBench Verified, BrowseComp-ZH, DeepSearchQA, Seal-0, tau2-bench, MCPMark, BFCL V4, VITA-Bench, DeepConsult, DeepResearch Bench, MM-BrowseComp, HLE-VL, MathVista, LogicVista, VisuLogic, VLMs Are Blind, ChartQA Pro, OmniDocBench v1.5, DUDE, MMLongBench, MMLongBench-Doc, TempCompass, Math Kangaroo 2025 (MathArena), MathCanvas-Bench, MineDojo, PutnamBench, IMO 2025, CMO 2025
- Apr 27, 2026
- Humanity's Last Exam 48
- GDPval-AA 1581
- tau3-bench 72.9
- Claw-Eval Pass^3 63.8
- SWE-Bench Pro 57.2
- SWE-bench Verified 78.9
- Terminal-Bench 2.0 68.4
- MiMo Coding Bench 73.7
Dropped since the last post: CharXiv, MMMU-Pro, HR-Bench, OmniDocBench v1.5, Claw-Eval-MM, Video-MME, Daily-Omni, Video-Holmes
- Apr 24, 2026
- MMLU-Pro 86.2
- SimpleQA Verified 34.1
- Chinese SimpleQA 78.9
- GPQA Diamond 88.1
- Humanity's Last Exam 34.8
- LiveCodeBench v6 91.6
- Codeforces rating 3052
- HMMT 2026 94.8
- IMO-AnswerBench 88.4
- MathArena (Apex, IMO, USAMO) 33
- MathArena Apex Shortlist 85.7
- MRCR v2 78.7
- CorpusQA 1M 60.5
- Terminal-Bench 2.0 56.9
- SWE-bench Verified 79
- SWE-Bench Pro 52.6
- SWE-bench Multilingual 73.3
- BrowseComp 73.2
- MCP Atlas 69
- GDPval-AA 1395
- Toolathlon 47.8
Dropped since the last post: IMO-ProofBench, IMO 2025, Putnam 2024
Its technical report adds 1 evaluation 1 new 6 dropped 21 results in all
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arXiv 2606.19348)Capabilities
- PutnamBench 81
Dropped since the lab’s previous technical report: IMO-ProofBench, IMO 2025, CMO 2024, Putnam 2024, IMO Shortlist 2024, CNML-Level Problems (DeepSeek)
- Apr 24, 2026
- MMLU-Pro 87.5
- SimpleQA Verified 57.9
- Chinese SimpleQA 84.4
- GPQA Diamond 90.1
- Humanity's Last Exam 48.2
- LiveCodeBench 93.5
- Codeforces rating 3206
- HMMT 2026 95.2
- IMO-AnswerBench 89.8
- MathArena (Apex, IMO, USAMO) 38.3
- MathArena Apex Shortlist 90.2
- MRCR v2 83.5
- CorpusQA 1M 62
- Terminal-Bench 2.0 67.9
- SWE-bench Verified 80.6
- SWE-Bench Pro 55.4
- SWE-bench Multilingual 76.2
- BrowseComp 83.4
- GDPval-AA 1554
- MCP Atlas 73.6
- Toolathlon 51.8
- AGIEval 83.1
- MMLU 90.1
- MMLU-Redux 90.8
- MMMLU 90.3
- C-Eval 93.1
- CMMLU 90.8
- MultiLoKo 51.1
- SuperGPQA 53.9
- FACTS Parametric 62.6
- TriviaQA 85.6
- BIG-Bench Hard 87.5
- DROP 88.7
- HellaSwag 88
- WinoGrande 81.5
- CLUEWSC 85.2
- BigCodeBench 59.2
- HumanEval 76.8
- GSM8K 92.6
- MATH 64.5
- MGSM 84.4
- CMATH 90.9
- LongBench v2 51.5
Dropped since the last post: OmniDocBench v1.5
Its technical report adds 9 evaluations 9 new 1 dropped 27 results in all
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arXiv 2606.19348)Capabilities
- LiveCodeBench v6 93.5
- OpenAI MRCR v2 83.5
- Putnam 2025 120
- DeepSeek R&D Coding Benchmark 67
- DeepSeek Chinese white-collar tasks evaluation 53
- DeepSeek Chinese functional writing evaluation 62.6
- DeepSeek Chinese creative writing evaluation
- DeepSeek complex instruction following and multi-turn writing evaluation 45.9
- DeepSeek search Q&A evaluation 28.1
Dropped since the lab’s previous technical report: OmniDocBench v1.5
- Apr 23, 2026
- GDPval 82.3
- BrowseComp 90.1
- FrontierMath 52.4
- FrontierMath Tier 4 39.6
- Investment Banking Modeling Tasks (OpenAI) 88.6
- GeneBench 33.2
- Humanity's Last Exam 57.2
Dropped since the last post: Terminal-Bench 2.0, OSWorld-Verified, Toolathlon, Artificial Analysis Intelligence Index v4.0, tau2-bench, BixBench, Expert-SWE (OpenAI), CyberGym, SWE-Bench Pro, Vals Finance Agent, OfficeQA Pro, MMMU-Pro, MCP Atlas, GPQA Diamond, Capture the Flag (Internal), GraphWalks, OpenAI MRCR v2, ARC-AGI, ARC-AGI-2
Its system card adds 8 evaluations 3 new 26 dropped 8 results in all
GPT-5.5 System CardBio and chem risk
Cyber risk
- CVE-Bench 93.1
Dropped since the lab’s previous system card: OpenAI Production Benchmarks and safe-completion evals, OpenAI Dynamic Mental Health Benchmarks, Disallowed Content Evaluations with Representative Prompts, Deception Evaluations with Representative Prompts, Multiturn Jailbreak Evaluations, Agent JSK, PlugInject, Image input safety evaluations, HealthBench, HealthBench Hard, HealthBench Consensus, Destructive action avoidance, Destructive action reversion on long rollouts, User Confirmations During Computer Use, First-Person Fairness evaluation, CoT Monitorability, CoT-Control, Capture the Flag (Professional), Cyber Range, Atomic Challenges, CyScenarioBench, Monorepo-Bench, MLE-bench 30, OpenAI-Proof Q&A, Apollo Research strategic deception evaluations, OpenAI cyber safety evaluations (incl. Daybreak Blue)
- Apr 23, 2026
- τ-Voice 67.3
Dropped since the last post: General refusals (xAI), Refusals + User Jailbreak (xAI), Refusals + System Jailbreak (xAI), AgentHarm, AgentDojo, MASK, SycophancyEval (answer sycophancy), xAI sycophancy eval, HLE Calibration, WMDP Biology, Virology Capabilities Test (VCT), ProtocolQA Open-Ended (xAI), LAB-Bench FigQA, LAB-Bench CloningScenarios, WMDP Chemistry, WMDP Cyber, Cybench, MakeMeSay
- Apr 23, 2026
- FrontierScience 70
- IMO-AnswerBench 84.3
- Humanity's Last Exam 30
- GPQA Diamond 87.2
- AdvancedIF 79.5
- AA-LCR (Artificial Analysis Long Context Reasoning) 66.3
- LongBench v2 65.4
- CL-bench 22.8
- CL-bench Life 15.7
- SWE-bench Verified 74.4
- Terminal-Bench 2.0 54.4
- BrowseComp 67.1
- WideSearch 70.2
- WildClawBench 45.3
- Claw-Eval Pass^3 55
- Hy-Backend 54.7
- Hy-Vibe Bench 50.5
- Hy-SWE Max 30
Dropped since the last post: HMMT 2025, ARC-AGI, MultiChallenge, Terminal-Bench, tau2-bench
- Apr 23, 2026
- Terminal-Bench 2.0 82.7
- GDPval 84.9
- OSWorld-Verified 78.7
- Toolathlon 55.6
- BrowseComp 90.1
- FrontierMath 52.4
- FrontierMath Tier 4 39.6
- Artificial Analysis Intelligence Index v4.0 60.2
- tau2-bench 98
- GeneBench 33.2
- BixBench 80.5
- Expert-SWE (OpenAI) 73.1
- CyberGym 81.8
- SWE-Bench Pro 58.6
- Vals Finance Agent 60
- OfficeQA Pro 54.1
- MMMU-Pro 83.2
- MCP Atlas 75.3
- GPQA Diamond 93.6
- Capture the Flag (Internal) 88.1
- GraphWalks 73.7
- OpenAI MRCR v2 87.5
- ARC-AGI 95
- ARC-AGI-2 85
Dropped since the last post: Investment Banking Modeling Tasks (OpenAI), FrontierScience-Research, Humanity's Last Exam
Its system card adds 41 evaluations 33 new 41 results in all
GPT-5.5 System CardHealth
Hallucination
Safety
- OpenAI Production Benchmarks and safe-completion evals
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 90
- Destructive action reversion on long rollouts
- User Confirmations During Computer Use
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
- CoT Monitorability
- CoT-Control 0.2
- BashArena 99.1
Bio and chem risk
Cyber risk
AI R&D risk
- Monorepo-Bench 60
- MLE-bench 30 36.7
- Internal Research Debugging Evaluation 50.5
- OpenAI-Proof Q&A 1.7
- Apr 22, 2026
- SWE-bench Verified 77.2
- SWE-Bench Pro 53.5
- SWE-bench Multilingual 71.3
- Terminal-Bench 2.0 59.3
- SkillsBench 48.2
- QwenWebBench 1487
- NL2Repo-Bench 36.2
- Claw-Eval 72.4
- Claw-Eval Pass^3 60.6
- QwenClawBench 53.4
- MMLU-Pro 86.2
- MMLU-Redux 93.5
- SuperGPQA 66
- C-Eval 91.4
- GPQA Diamond 87.8
- Humanity's Last Exam 24
- LiveCodeBench v6 83.9
- HMMT 2025 93.8
- HMMT Nov 2025 90.7
- HMMT 2026 84.3
- IMO-AnswerBench 80.8
- AIME 2026 94.1
- MMMU 82.9
- MMMU-Pro 75.8
- MathVista 87.4
- DynaMath 85.6
- VLMs Are Blind 97
- RealWorldQA 84.1
- MMStar 81.4
- MMBench-EN v1.1 92.3
- SimpleVQA 56.1
- CharXiv 78.4
- CC-OCR 81.2
- OCRBench 89.4
- ERQA 62.5
- CountBench 97.8
- RefCOCO / RefCOCO+ / RefCOCOg 92.5
- EmbSpatialBench 84.6
- RefSpatial-Bench 70
- Video-MME 87.7
- Video-MMMU 84.4
- MLVU 86.6
- MVBench 75.5
- V* 94.7
- AndroidWorld 70.3
Dropped since the last post: tau3-bench, VITA-Bench, DeepPlanning, Toolathlon, MCPMark, MCP Atlas, WideSearch, ZeroBench, HallusionBench, OmniDocBench v1.5, AI2D, ODinW-13, LVBench
- Apr 22, 2026
- CharXiv 81
- MMMU-Pro 77.9
- HR-Bench 88.5
- OmniDocBench v1.5 87.2
- Claw-Eval-MM
- Video-MME 87.7
- Daily-Omni 83.5
- Video-Holmes 64
- MiMo Coding Bench 71.8
- Claw-Eval Pass^3 62.3
- Terminal-Bench 2.0 65.8
- SWE-Bench Pro 56.1
Dropped since the last post: MMAU-Pro, Big Bench Audio, FutureOmni, MM-BrowseComp, OmniGAIA, Claw-Eval, SWE-bench Verified, GDPval-AA, PinchBench
- Apr 21, 2026
- LBM Eval 34.1
- Apr 21, 2026
- LBM Eval 72.4
- Apr 20, 2026
- SuperGPQA 73.9
- AA-Omniscience 10
- GDPval-AA 51
- QwenChineseBench 84
- QwenClawBench 59
- SkillsBench 55.6
- ToolcallFormatIFBench 86.1
- QwenWebBench 1532
- SciCode 47
- NL2Repo-Bench 42.9
- Terminal-Bench 2.0 65.4
- SWE-Bench Pro 57.3
Dropped since the last post: SWE-bench Verified, SWE-bench Multilingual, Claw-Eval, Claw-Eval Pass^3, tau3-bench, VITA-Bench, DeepPlanning, Toolathlon, MCPMark, MCP Atlas, Humanity's Last Exam, WideSearch, MMLU-Pro, MMLU-Redux, C-Eval, IFEval, IFBench, AA-LCR (Artificial Analysis Long Context Reasoning), LongBench v2, GPQA Diamond, LiveCodeBench v6, HMMT 2025, HMMT Nov 2025, HMMT 2026, IMO-AnswerBench, AIME 2026, MMMLU, MMLU-ProX, NOVA-63, INCLUDE, Global PIQA, PolyMath, WMT24++, MAXIFE, MMMU, MMMU-Pro, MATH-Vision (MathVision), We-Math, DynaMath, RealWorldQA, MMStar, SimpleVQA, OmniDocBench v1.5, CharXiv, MMLongBench-Doc, CC-OCR, AI2D, CountBench, ODinW-13, ERQA, V*, Video-MME, Video-MMMU, MLVU, ScreenSpot-Pro, TIR-Bench, OSWorld-Verified
- Apr 20, 2026
- Humanity's Last Exam 54
- BrowseComp 83.2
- DeepSearchQA 92.5
- WideSearch 80.8
- Toolathlon 50
- MCPMark 55.9
- Claw-Eval Pass^3 62.3
- APEX-Agents 27.9
- OSWorld-Verified 73.1
- Terminal-Bench 2.0 66.7
- SWE-Bench Pro 58.6
- SWE-bench Multilingual 76.7
- SWE-bench Verified 80.2
- SciCode 52.2
- OJBench 60.6
- LiveCodeBench v6 89.6
- AIME 2026 96.4
- HMMT 2026 92.7
- IMO-AnswerBench 86
- GPQA Diamond 90.5
- MMMU-Pro 79.4
- CharXiv 80.4
- MATH-Vision (MathVision) 87.4
- BabyVision 39.8
- V* 96.9
Dropped since the last post: AIME 2025, HMMT 2025, MMLU-Pro, MathVista, ZeroBench, OCRBench, OmniDocBench v1.5, InfographicVQA, SimpleVQA, WorldVQA, Video-MMMU, MMVU, MotionBench, Video-MME, LongVideoBench, LVBench, PaperBench, CyberGym, LongBench v2, FinSearchComp, Seal-0, AA-LCR (Artificial Analysis Long Context Reasoning)
- Apr 16, 2026
Dropped since the last post: Success detection (Google DeepMind), Instrument reading (Google DeepMind), ASIMOV
- Apr 16, 2026Claude Opus 4.7 Anthropic Introducing Claude Opus 4.7 (launch post); System Card: Claude Opus 4.7, Table 8.1.A
- SWE-Bench Pro 64.3
- SWE-bench Verified 87.6
- SWE-bench Multilingual 80.5
- SWE-bench Multimodal 34.5
- Terminal-Bench 2.0 69.4
- Humanity's Last Exam 54.7
- BrowseComp 79.3
- MCP Atlas 77.3
- OSWorld-Verified 78
- Vals Finance Agent 64.4
- GPQA Diamond 94.2
- CharXiv 91
- MMMLU 91.5
- ScreenSpot-Pro 87.6
- OfficeQA 86.3
- OfficeQA 80.6
- ARC-AGI 92
- ARC-AGI-2 75.8
- CursorBench 3.1 70
- BigLaw Bench 90.9
- XBOW Validation Benchmarks 98.5
Dropped since the last post: USAMO 2026, GraphWalks
Its system card adds 74 evaluations 20 new 8 dropped 91 results in all
System Card: Claude Opus 4.7Capabilities
- OfficeQA Pro 80.6
- USAMO 2026 69.3
- GraphWalks
- OpenAI MRCR v2
- DeepSearchQA 89.1
- DRACO 77.7
- LAB-Bench FigQA 78.6
- Vending-Bench 2 10937
- GDPval-AA 1753
- Global MMLU 89.9
- MILU 89.9
- INCLUDE 87
- BioPipelineBench Verified (Anthropic) 83.6
- BioMysteryBench
- Structural Biology (Anthropic)
- Organic Chemistry (Anthropic) 77.2
- Phylogenetics (Anthropic) 79.6
- Protocols (Anthropic life sciences) 51.8
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 98
- Single-turn benign request evaluations (Anthropic) 0.28
- Higher-difficulty harmful request evaluations (Anthropic) 99
- Higher-difficulty benign request evaluations (Anthropic) 0.01
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Election integrity evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 91.2
- Malicious computer use (Anthropic) 89.3
Robustness
Alignment
- Impossible tasks (Anthropic) 45
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 21.1
- Anthropic automated behavioral audit
- Petri
- UK AISI unprompted sabotage evaluation 0
- UK AISI sabotage continuation evaluation 0
- Destructive production evaluation (Anthropic) 2.5
- Destructive actions (Anthropic)
- Adherence to its constitution (Anthropic) 0.74
- MASK 91.7
- Refusal to assist with AI safety R&D (Anthropic) 33.1
- Self-preference bias (Anthropic) 0.4
- DTBench
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: ASL-4 virology uplift trial (Anthropic), Catastrophic biology scenario uplift trial (Anthropic), Internal AI research evaluation suite 2, Internal model use survey (Anthropic), Key insight rediscovery on an unpublished ML task (external testing), UK AISI alignment and monitorability evaluations, White-box analyses of model internals (Anthropic), Recognition of model-written user turns (Anthropic)
- Apr 15, 2026
- SWE-bench Verified 73.4
- SWE-bench Multilingual 67.2
- SWE-Bench Pro 49.5
- Terminal-Bench 2.0 51.5
- Claw-Eval 68.7
- Claw-Eval Pass^3 50
- SkillsBench 28.7
- QwenClawBench 52.6
- NL2Repo-Bench 29.4
- QwenWebBench 1397
- tau3-bench 67.2
- VITA-Bench 35.6
- DeepPlanning 25.9
- Toolathlon 26.9
- MCPMark 37
- MCP Atlas 62.8
- WideSearch 60.1
- MMLU-Pro 85.2
- MMLU-Redux 93.3
- SuperGPQA 64.7
- C-Eval 90
- GPQA Diamond 86
- Humanity's Last Exam 21.4
- LiveCodeBench v6 80.4
- HMMT 2025 90.7
- HMMT Nov 2025 89.1
- HMMT 2026 83.6
- IMO-AnswerBench 78.9
- AIME 2026 92.7
- MMMU 81.7
- MMMU-Pro 75.3
- MathVista 86.4
- ZeroBench 34.4
- RealWorldQA 85.3
- MMBench-EN v1.1 92.8
- SimpleVQA 58.9
- HallusionBench 69.8
- OmniDocBench v1.5 89.9
- CharXiv 78
- CC-OCR 81.9
- AI2D 92.7
- RefCOCO / RefCOCO+ / RefCOCOg 92
- ODinW-13 50.8
- EmbSpatialBench 84.3
- RefSpatial-Bench 64.3
- Video-MME 86.6
- Video-MMMU 83.7
- MLVU 86.2
- MVBench 74.6
- LVBench 71.4
Dropped since the last post: IFEval, MMMLU, IFBench, MultiChallenge, AA-LCR (Artificial Analysis Long Context Reasoning), LongBench v2, BFCL V4, tau2-bench, MMLU-ProX, NOVA-63, INCLUDE, Global PIQA, PolyMath, WMT24++, MAXIFE, DynaMath, VLMs Are Blind, MMStar, MMLongBench-Doc, OCRBench, CountBench, ERQA, Hypersim, SUNRGBD, Nuscene, MMVU, SLAKE, PMC-VQA, MedXpertQA-MM, ScreenSpot-Pro
- Apr 14, 2026 Gemini Robotics-ER 1.6 Google DeepMind Gemini Robotics ER 1.6: Enhanced Embodied Reasoning (Google DeepMind blog)
Dropped since the last post: MTEB, TextCaps, Docci, ViDoRe v2, VATEX, MSR-VTT retrieval, YouCook2 retrieval, MSEB
- Apr 9, 2026
- MMMU 78.7
- MMMU-Pro 68.6
- MedXpertQA-MM 42.1
- MATH-Vision (MathVision) 75.2
- MathVista 85
- We-Math 79.1
- LogicVista 73.8
- BabyVision 18.8
- AI2D 89
- ChartQA Pro 62.2
- CharXiv 71.7
- OCRBench v2 63.2
- OmniDocBench v1.5 81.2
- MMStar 74.9
- BLINK 68.8
- HallusionBench 63.7
- KMMMU 42.7
- K-Viscuit 80.1
- KRETA 91.9
- AIME 2025 92.9
- AIME 2026 92.6
- GPQA Diamond 80.5
- LiveCodeBench v6 81.4
- MMLU-Pro 83.3
- tau2-bench 69.1
- IFBench 62.6
- IFEval 89.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 50.6
- KMMLU-Pro 67.6
- KoBALT 52.1
Dropped since the last post: Humanity's Last Exam, IMO-AnswerBench, HMMT Nov 2025, LiveCodeBench Pro, Terminal-Bench 2.0, SWE-bench Verified, BrowseComp, OpenAI MRCR v2, CLIcK, HRM8K, Ko-LongBench (LG), MMMLU, WMT24++, WildJailbreak, KGC-Safety (LG)
Its technical report adds 2 evaluations 15 dropped 31 results in all
EXAONE 4.5 Technical ReportDropped since the lab’s previous technical report: Humanity's Last Exam, IMO-AnswerBench, AIME 2025, HMMT Nov 2025, LiveCodeBench Pro, CodeUtilityBench (LG), Terminal-Bench 2.0, SWE-bench Verified, BrowseComp, OpenAI MRCR, CLIcK, HRM8K, Ko-LongBench (LG), WildJailbreak, KGC-Safety (LG)
- Apr 8, 2026
- CharXiv 86.4
- MMMU-Pro 80.4
- ERQA 64.7
- SimpleVQA 71.3
- ScreenSpot-Pro 84.1
- ZeroBench 33
- Humanity's Last Exam 50.4
- ARC-AGI-2 42.5
- GPQA Diamond 89.5
- LiveCodeBench Pro 80
- HealthBench Hard 42.8
- MedXpertQA 52.6
- MedXpertQA-MM 78.4
- DeepSearchQA 74.8
- SWE-bench Verified 77.4
- SWE-Bench Pro 52.4
- Terminal-Bench 2.0 59
- tau2-bench 91.5
- GDPval-AA 1444
- FrontierScience-Research 38
Dropped since the last post: LiveCodeBench, MATH-500, MMLU-Pro, MMMLU, MMMU
Its system card adds 55 evaluations 56 results in all
Muse Spark Safety & Preparedness ReportHallucination
- HLE Calibration 50.3
- SimpleQA Verified 70.8
- AbstentionBench 59.1
- CharXiv Missing Images 35
Safety
- TextQuests 2.4
- CIMemories 78.3
Robustness
Alignment
- Meta instructed selective performance evaluation 31.8
- Meta prompted evaluation awareness 43
- IHEval 80.3
- ImpossibleBench 0
- Meta Underspecified Intent evaluation 71.2
- MASK 89.1
- DeceptionBench 1.6
- PropensityBench 24.5
- Meta internal sycophancy evaluation 62.9
- Alignment faking evaluation 1.8
- Agentic misalignment suite (Anthropic) 47.7
- Situational Awareness Dataset (SAD)
Monitorability
- SHADE-Arena 8.8
- GDM Stealth 39.8
- GDM Situational Awareness 29.3
Bio and chem risk
- Molecular Biology Capabilities Test 54.4
- Virology Capabilities Test (VCT) 49.7
- Human Pathogen Capabilities Test 55.7
- WMDP
- LAB-Bench ProtocolQA 87.3
- LAB-Bench SeqQA 97.3
- ABC Bench Fragment Design 96.8
- ABC Bench Liquid-Handling Robot 92.3
- Meta BioDesign Proxy 46.2
- Meta chemical and biological high-risk bottleneck evaluations
- BioTIER Refuse 98
- Meta chemical-agent refusal evaluation 99.4
Cyber risk
AI R&D risk
- SWE-bench Verified hard subset (Meta) 42.9
- MLE-bench 30 15.8
- AIRS-Bench 0.65
- Apr 7, 2026
- Apr 7, 2026
- Humanity's Last Exam 52.3
- AIME 2026 95.3
- HMMT Nov 2025 94
- HMMT 2026 82.6
- IMO-AnswerBench 83.8
- GPQA Diamond 86.2
- SWE-Bench Pro 58.4
- NL2Repo-Bench 42.7
- Terminal-Bench 2.0 69
- CyberGym 68.7
- BrowseComp 79.3
- tau3-bench 70.6
- MCP Atlas 71.8
- Toolathlon 40.7
- Vending-Bench 2 5634.4
- Kernel Bench L3 3.6
Dropped since the last post: Design2Code, Flame-React-Eval, Vision2Web, ImageMining, BrowseComp-VL, MMSearch, MMSearch-Plus, SimpleVQA, FACTS Benchmark Suite, V*, OSWorld, AndroidWorld, WebVoyager, CC-Bench-V2 Backend, CC-Bench-V2 Frontend, CC-Bench-V2 Large Repo Exploration, PinchBench, Claw-Eval, ZClawBench
- Apr 7, 2026Claude Mythos Preview Anthropic System Card: Claude Mythos Preview, Table 6.3.A; Project Glasswing announcement
- SWE-bench Verified 93.9
- SWE-Bench Pro 77.8
- SWE-bench Multilingual 87.3
- SWE-bench Multimodal 59
- Terminal-Bench 2.0 82
- GPQA Diamond 94.5
- MMMLU 92.7
- USAMO 2026 97.6
- GraphWalks 80
- Humanity's Last Exam 64.7
- CharXiv 93.2
- OSWorld-Verified 79.6
- BrowseComp 86.9
Dropped since the last post: tau2-bench, MCP Atlas, Vals Finance Agent, GDPval-AA, ARC-AGI-2, MMMU-Pro
Its system card adds 61 evaluations 27 new 36 dropped 74 results in all
System Card: Claude Mythos PreviewCapabilities
- LAB-Bench FigQA 79.7
- ScreenSpot-Pro 79.5
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 97.8
- Single-turn benign request evaluations (Anthropic) 0.06
- Higher-difficulty harmful request evaluations (Anthropic) 99.1
- Higher-difficulty benign request evaluations (Anthropic) 0.02
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- Disordered eating evaluations (Anthropic)
- Malicious use of Claude Code (Anthropic) 96.7
- Malicious computer use (Anthropic) 93.8
- Malicious agentic influence campaigns (Anthropic)
Robustness
Alignment
- Impossible tasks (Anthropic) 37.5
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 13.3
- Anthropic automated behavioral audit
- Petri
- UK AISI alignment and monitorability evaluations
- Destructive production evaluation (Anthropic) 0.3
- Destructive actions (Anthropic)
- Adherence to its constitution (Anthropic) 1.02
- MASK 95.4
- Refusal to assist with AI safety R&D (Anthropic) 18.7
- Self-preference bias (Anthropic)
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Kernels task (Anthropic) 399.4
- Time series forecasting task (Anthropic) 4.55
- LLM training task (Anthropic) 51.9
- Quadruped reinforcement learning task (Anthropic) 30.9
- Novel compiler task (Anthropic) 77.2
- Internal AI research evaluation suite 2 0.65
- Internal model use survey (Anthropic) 5.6
- Anthropic ECI (AECI)
- Key insight rediscovery on an unpublished ML task (external testing) 80
Dropped since the lab’s previous system card: tau2-bench, MCP Atlas, ARC-AGI-2, GDPval-AA, MMMU-Pro, OpenRCA, ARC-AGI, AIME 2025, Vals Finance Agent, Real-World Finance (Anthropic), Vending-Bench 2, OpenAI MRCR v2, WebArena, WebArena-Verified, Global MMLU, MILU, DeepSearchQA, BioPipelineBench (Anthropic), BioMysteryBench, Structural Biology (Anthropic), Organic Chemistry (Anthropic), Phylogenetics (Anthropic), MedCalc-Bench Verified, SSH stress-testing evaluation (Anthropic), Reward-hack-prone coding tasks (Anthropic), Participation in junk science (Anthropic), Agentic coding malicious requests (Anthropic), Creative biology, Short-horizon computational biology tasks, SWE-bench Verified (hard subset), Text-based reinforcement learning task (Anthropic), Web CTF challenges (Anthropic), Crypto CTF challenges (Anthropic), Pwn CTF challenges (Anthropic), Rev CTF challenges (Anthropic), Network CTF challenges (Anthropic)
- Apr 2, 2026
- MMLU-Pro 69.4
- AIME 2026 42.5
- LiveCodeBench v6 52
- Codeforces rating 940
- GPQA Diamond 58.6
- tau2-bench 42.2
- BBEH micro average 33.1
- MMMLU 76.6
- MMMU-Pro 52.6
- OmniDocBench v1.5 0.181
- MATH-Vision (MathVision) 59.5
- MedXpertQA-MM 28.7
- CoVoST2 35.5
- FLEURS 0.08
- MRCR v2 25.4
Its technical report adds 8 evaluations 22 results in all
Gemma 4 Technical Report (arXiv 2607.02770v2)Capabilities
- SciCode 24
- IFBench 44
- IFEval 96.7
- Terminal-Bench Hard 8
- InfographicVQA 70
- RULER 86.6
- LOFT 58.5
- GraphWalks 50.9
- Apr 2, 2026
- MMLU-Pro 60
- AIME 2026 37.5
- LiveCodeBench v6 44
- Codeforces rating 633
- GPQA Diamond 43.4
- tau2-bench 24.5
- BBEH micro average 21.9
- MMMLU 67.4
- MMMU-Pro 44.2
- OmniDocBench v1.5 0.29
- MATH-Vision (MathVision) 52.4
- MedXpertQA-MM 23.5
- CoVoST2 33.5
- FLEURS 0.09
- MRCR v2 19.1
Dropped since the last post: Humanity's Last Exam
Its technical report adds 8 evaluations 4 dropped 22 results in all
Gemma 4 Technical Report (arXiv 2607.02770v2)Capabilities
- SciCode 21
- IFBench 38
- IFEval 94.6
- Terminal-Bench Hard 3
- InfographicVQA 63.9
- RULER 70.4
- LOFT 50.5
- GraphWalks 4.1
Dropped since the lab’s previous technical report: LMArena (Elo), Humanity's Last Exam, tau2-bench, MTOB
- Apr 2, 2026
- MMLU-Pro 82.6
- AIME 2026 88.3
- LiveCodeBench v6 77.1
- Codeforces rating 1718
- GPQA Diamond 82.3
- tau2-bench 68.2
- Humanity's Last Exam 17.2
- BBEH micro average 64.8
- MMMLU 86.3
- MMMU-Pro 73.8
- OmniDocBench v1.5
- MATH-Vision (MathVision) 82.4
- MedXpertQA-MM 58.1
- MRCR v2 44.1
Dropped since the last post: CoVoST2, FLEURS
Its technical report adds 10 evaluations 10 new 2 dropped 24 results in all
Gemma 4 Technical Report (arXiv 2607.02770v2)Capabilities
- LMArena (Elo) 1438
- SciCode 40
- IFBench 72
- IFEval 98.5
- Terminal-Bench Hard 14
- InfographicVQA 89.3
- RULER 89.8
- LOFT 66.3
- GraphWalks 72.6
- MTOB
Dropped since the lab’s previous technical report: WMT24++, Vistra
- Apr 2, 2026
- SWE-bench Verified 78.8
- SWE-bench Multilingual 73.8
- SWE-Bench Pro 56.6
- Terminal-Bench 2.0 61.6
- Claw-Eval 74.8
- Claw-Eval Pass^3 58.7
- SkillsBench 45.7
- QwenClawBench 57.2
- NL2Repo-Bench 37.9
- QwenWebBench 1501.7
- tau3-bench 70.7
- VITA-Bench 44.3
- DeepPlanning 41.5
- Toolathlon 39.8
- MCPMark 48.2
- MCP Atlas 74.1
- Humanity's Last Exam 50.6
- WideSearch 74.3
- MMLU-Pro 88.5
- MMLU-Redux 94.5
- SuperGPQA 71.6
- C-Eval 93.3
- IFEval 94.3
- IFBench 74.2
- AA-LCR (Artificial Analysis Long Context Reasoning) 68.3
- LongBench v2 62
- GPQA Diamond 90.4
- LiveCodeBench v6 87.1
- HMMT 2025 96.7
- HMMT Nov 2025 94.6
- HMMT 2026 87.8
- IMO-AnswerBench 83.8
- AIME 2026 95.3
- MMMLU 89.5
- MMLU-ProX 84.7
- NOVA-63 57.9
- INCLUDE 85.1
- Global PIQA 89.8
- PolyMath 77.4
- WMT24++ 84.3
- MAXIFE 88.2
- MMMU 86
- MMMU-Pro 78.8
- MATH-Vision (MathVision) 88
- We-Math 89
- DynaMath 88
- RealWorldQA 85.4
- MMStar 83.3
- SimpleVQA 67.3
- OmniDocBench v1.5 91.2
- CharXiv 81.5
- MMLongBench-Doc 62
- CC-OCR 83.4
- AI2D 94.4
- CountBench 97.6
- ODinW-13 51.8
- ERQA 65.7
- V* 96.9
- Video-MME 87.8
- Video-MMMU 84
- MLVU 86.7
- ScreenSpot-Pro 68.2
- TIR-Bench 61.6
- OSWorld-Verified 62.5
Dropped since the last post: Daily-Omni, WorldSense (omnimodal), AVUT, AV-SpeakerBench, QIVD, Omni-Cloze, OmniGAIA, MMAU, MMAR, MMSU, MuChoMusic, VoiceBench, SpeechRole, WildSpeech-Bench, FLEURS
- Apr 2, 2026
- MMLU-Pro 85.2
- AIME 2026 89.2
- LiveCodeBench v6 80
- Codeforces rating 2150
- GPQA Diamond 84.3
- tau2-bench 76.9
- Humanity's Last Exam 26.5
- BBEH micro average 74.4
- MMMLU 88.4
- MMMU-Pro 76.9
- OmniDocBench v1.5
- MATH-Vision (MathVision) 85.6
- MedXpertQA-MM 61.3
- CoVoST2
- FLEURS
- MRCR v2 66.4
Dropped since the last post: CharXiv, Video-MMMU, SimpleQA Verified, FACTS Benchmark Suite, LiveCodeBench, LMArena (Elo)
Its technical report adds 10 evaluations 24 results in all
Gemma 4 Technical Report (arXiv 2607.02770v2)Capabilities
- LMArena (Elo) 1451
- SciCode 43
- IFBench 76
- IFEval 98.9
- Terminal-Bench Hard 36
- InfographicVQA 92
- RULER 96.4
- LOFT 79.5
- GraphWalks 82.3
- MTOB
- Apr 1, 2026
- IFBench 52.3
- GPQA Diamond 76.3
- tau2-bench 91.3
- PinchBench 91.9
- AIME 2025 96.3
- BFCL V4 70.1
- MMLU-Pro 83.4
- SWE-bench Verified 63.2
Dropped since the last post: MMLU
- Apr 1, 2026
- Design2Code 94.8
- Flame-React-Eval 93.8
- Vision2Web 31
- ImageMining 30.7
- BrowseComp-VL 51.9
- MMSearch 72.9
- MMSearch-Plus 30
- SimpleVQA 78.2
- FACTS Benchmark Suite 58.6
- V* 89
- OSWorld 62.3
- AndroidWorld 75.7
- WebVoyager 88.5
- CC-Bench-V2 Backend 22.8
- CC-Bench-V2 Frontend 68.4
- CC-Bench-V2 Large Repo Exploration 72.2
- PinchBench 80.7
- Claw-Eval 75
- ZClawBench 57.6
Dropped since the last post: Humanity's Last Exam, AIME 2026, HMMT Nov 2025, IMO-AnswerBench, GPQA Diamond, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.0, CyberGym, BrowseComp, BrowseComp-ZH, tau2-bench, MCP Atlas, Toolathlon, Vending-Bench 2
- Mar 30, 2026
- Daily-Omni 81.8
- WorldSense (omnimodal) 57.9
- AVUT 81.4
- AV-SpeakerBench 65.2
- Video-MME 79.3
- QIVD 66.3
- Omni-Cloze 63
- OmniGAIA 33.9
- MMAU 80.4
- MMAR 74
- MMSU 72.2
- MuChoMusic 60.5
- VoiceBench 87.8
- SpeechRole 119.8
- WildSpeech-Bench 72.2
- FLEURS 10.8
Dropped since the last post: URO-Bench, Common Voice, LibriSpeech, Seed-TTS-eval, MMMU, MMMU-Pro, MATH-Vision (MathVision), MathVista, DynaMath, ZeroBench, RealWorldQA, MMStar, MMBench-EN v1.1, SimpleVQA, CharXiv, CC-OCR, AI2D, MMLongBench-Doc, OCRBench, ERQA, CountBench, RefCOCO / RefCOCO+ / RefCOCOg, ODinW-13, EmbSpatialBench, MLVU, MVBench, LVBench, MMVU, MME-VideoOCR, SLAKE, PMC-VQA, MedXpertQA-MM, MMLU-Pro, MMLU-Redux, SuperGPQA, C-Eval, IFEval, IFBench, AA-LCR (Artificial Analysis Long Context Reasoning), LongBench v2, GPQA Diamond, LiveCodeBench v6, HMMT Nov 2025, IMO-AnswerBench, BFCL V4, tau2-bench, WenetSpeech
- Mar 30, 2026
- Daily-Omni 84.6
- WorldSense (omnimodal) 62.8
- AVUT 85
- AV-SpeakerBench 71.3
- Video-MME 83.7
- QIVD 68.5
- Omni-Cloze 64.8
- OmniGAIA 57.2
- MMAU 82.2
- MMAR 80
- MMSU 82.8
- MuChoMusic 72.4
- VoiceBench 93.1
- URO-Bench 84.1
- SpeechRole 123.5
- WildSpeech-Bench 75.4
- FLEURS 6.55
- Common Voice 3.13
- LibriSpeech 1.67
- Seed-TTS-eval 1.35
- MMMU 80.1
- MMMU-Pro 73.9
- MATH-Vision (MathVision) 73
- MathVista 86.1
- DynaMath 83.8
- ZeroBench 34.4
- RealWorldQA 84.1
- MMStar 79.4
- MMBench-EN v1.1 92.8
- SimpleVQA 65.3
- CharXiv 72.5
- CC-OCR 83.4
- AI2D 91.2
- MMLongBench-Doc 57.5
- OCRBench 91.3
- ERQA 54.8
- CountBench 95.1
- RefCOCO / RefCOCO+ / RefCOCOg 95
- ODinW-13 49.5
- EmbSpatialBench 85.4
- MLVU 86.8
- MVBench 79
- LVBench 71.2
- MMVU 67.5
- MME-VideoOCR 77
- SLAKE 84.7
- PMC-VQA 62.7
- MedXpertQA-MM 54.7
- MMLU-Pro 85.9
- MMLU-Redux 94.2
- SuperGPQA 66.4
- C-Eval 92
- IFEval 89.7
- IFBench 52.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 57
- LongBench v2 59.6
- GPQA Diamond 83.9
- LiveCodeBench v6 65.6
- HMMT Nov 2025 84.4
- IMO-AnswerBench 65.5
- BFCL V4 63.3
- tau2-bench 81
- WenetSpeech 5.07
Dropped since the last post: MultiChallenge, Humanity's Last Exam, HMMT 2025, AIME 2026, VITA-Bench, DeepPlanning, Toolathlon, MCPMark, BrowseComp, BrowseComp-ZH, WideSearch, Seal-0, MMMLU, MMLU-ProX, NOVA-63, INCLUDE, Global PIQA, PolyMath, WMT24++, MAXIFE, SWE-bench Verified, SWE-bench Multilingual, SecCodeBench, Terminal-Bench 2.0, We-Math, BabyVision, HallusionBench, OmniDocBench v1.5, RefSpatial-Bench, LingoQA, V*, Hypersim, SUNRGBD, Nuscene, Video-MMMU, ScreenSpot-Pro, OSWorld-Verified, AndroidWorld
Its technical report adds 2 evaluations 2 new 50 dropped 65 results in all
Qwen3.5-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: MathVision-Wildphoto, We-Math, MathVerse, Math-VR, VLMs Are Blind, LogicVista, VisuLogic, VisualPuzzles, MMBench-CN v1.1, HallusionBench, MM-MT-Bench, MIA-Bench, DocVQA, InfographicVQA, ChartQA, OCRBench v2, OmniDocBench, CharXiv (DQ), ARKitScenes, Hypersim, SUNRGBD, VSI-Bench, RefSpatial-Bench, RoboSpatial-Home, BLINK, MuirBench, Charades-STA, Video-MMMU, V*, HR-Bench, Design2Code, ChartMimic, ScreenSpot-Pro, OSWorld-G, AndroidWorld, OSWorld, Windows Agent Arena, AIME 2025, HMMT 2025, LiveBench, Codeforces rating, OJBench, Arena-Hard v2.0, Creative Writing v3, WritingBench, Berkeley Function Calling Leaderboard (BFCL), Multi-IF, MMLU-ProX, INCLUDE, PolyMath
- Mar 26, 2026 Cohere Transcribe Cohere Introducing Cohere Transcribe: a new state-of-the-art in open-source speech recognition (Cohere blog)
- Open ASR Leaderboard 5.42
- LibriSpeech 1.25
Dropped since the last post: AyaVisionBench, m-WildVision
- Mar 24, 2026
- MMLU-Pro 81.2
- GPQA Diamond 71.9
- Humanity's Last Exam 11.9
- HMMT 2026 65.9
- AIME 2026 83
- LiveCodeBench v6 77.7
- tau2-bench 72.3
- Terminal-Bench 2.0 10.1
- IFBench 73.1
- Arena-Hard v2.0 74.7
- Ko-Arena-Hard v2 (Upstage) 78.2
Dropped since the last post: AIME 2025
- Mar 24, 2026
- WebVoyager 78.2
- Online-Mind2Web 35.3
- DeepShop 42.3
- WebTailBench 49.5
Dropped since the last post: SWE-bench Verified
Its technical report adds 0 evaluations 1 dropped 4 results in all
MolmoWeb: Open Visual Web Agent and Open Data for the Open WebDropped since the lab’s previous technical report: SWE-bench Verified
- Mar 19, 2026
- IMO 2025 35
- IMO-AnswerBench 79.3
- IMO-ProofBench 72.9
- AIME 2025 92.4
- AIME 2026 90.9
- HMMT 2025 94.6
- IOI 2025 439.3
- ICPC World Finals 2025 10
- LiveCodeBench v6 87.2
- LiveCodeBench Pro 27.6
- SciCode 36.4
- MMLU-Redux 86.3
- MMLU-Pro 79.8
- GPQA Diamond 76.1
- Humanity's Last Exam 17.7
- Arena-Hard v2.0 83.5
- IFBench 82.9
- MultiChallenge 45.3
- AA-LCR (Artificial Analysis Long Context Reasoning) 39.1
- LongBench v2 40.3
- RULER 99
- CL-bench 12.2
- BFCL V4 52.9
- tau2-bench 58.9
- Terminal-Bench 2.0 21.1
- SWE-bench Verified 50.2
- MMLU-ProX 72.5
- WMT24++ 84.1
Dropped since the last post: MATH-500, LiveCodeBench, Berkeley Function Calling Leaderboard (BFCL), IFEval, Orak, HaluEval, EQ-Bench 3
Its technical report adds 1 evaluation 1 new 2 dropped 29 results in all
Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy DistillationCapabilities
- Codeforces rating 2320
Dropped since the lab’s previous technical report: miniF2F, Terminal-Bench Hard
- Mar 19, 2026
- CursorBench 3.1 61.3
- Terminal-Bench 2.0 61.7
- SWE-bench Multilingual 73.7
Dropped since the last post: Cursor Bench (2025 internal version)
Its technical report adds 1 evaluation 3 results in all
Composer 2 Technical ReportCapabilities
- CursorBench 61.3
- Mar 18, 2026
- MMAU-Pro 69.4
- Big Bench Audio 94
- MMMU-Pro 76.8
- CharXiv 80.1
- Video-MME 85.3
- FutureOmni 66.7
- MM-BrowseComp 52
- OmniGAIA 49.8
- Claw-Eval 54.8
- SWE-bench Verified 74.8
- GDPval-AA 1410
- PinchBench 81.2
Dropped since the last post: SWE-bench Multilingual, Terminal-Bench 2.0, tau2-bench, DeepSearchQA
- Mar 18, 2026
- PinchBench 81
- Claw-Eval
- GDPval-AA 1426
- SWE-bench Verified 78
- SWE-bench Multilingual 71.7
- Terminal-Bench 2.0 57.1
- tau2-bench 96.8
- DeepSearchQA 86.7
Dropped since the last post: MMLU-Pro, GPQA Diamond, Humanity's Last Exam, AIME 2025, HMMT 2025, LiveCodeBench v6, Arena-Hard v2.0, LongBench v2, MRCR v2, Terminal-Bench Hard, BrowseComp
- Mar 18, 2026 MiniMax-M2.7 MiniMax MiniMax M2.7 launch post (minimax.io/news/minimax-m27-en) and Hugging Face card overview chart
- SWE-Bench Pro 56.2
- Multi-SWE-bench 52.7
- SWE-bench Multilingual 76.5
- VIBE-Pro (MiniMax) 55.6
- Terminal-Bench 2.0 57
- NL2Repo-Bench 39.8
- MLE-bench 66.6
- GDPval-AA 1495
- Toolathlon 46.3
- MM-ClawBench 62.7
- Artificial Analysis Intelligence Index v4.0 50
Dropped since the last post: SWE-bench Verified, BrowseComp, WideSearch, RISE, Berkeley Function Calling Leaderboard (BFCL), tau2-bench, GDPval-MM, MEWC, Finance Modeling, AIME 2025, GPQA Diamond, Humanity's Last Exam, SciCode, IFBench, AA-LCR (Artificial Analysis Long Context Reasoning)
Its technical report adds 14 evaluations 25 results in all
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World IntelligenceCapabilities
- HyperTask 67.6
- BrowseComp 77.8
- WideSearch 75.2
- RISE 64.3
- MEWC v2 63.3
- Finance Modeling Pro 57
- AIME 2026 94.2
- GPQA Diamond 89.8
- SciCode 47
- IFBench 76
- AA-LCR (Artificial Analysis Long Context Reasoning) 72
- Humanity's Last Exam 28
- MMLU-Pro 81.8
- AIME 2025 94
- Mar 17, 2026
- SWE-Bench Pro 52.4
- Terminal-Bench 2.0 46.3
- Toolathlon 35.5
- GPQA Diamond 82.8
- OSWorld-Verified 39
- MCP Atlas 56.1
- tau2-bench 92.5
- Humanity's Last Exam 37.7
- MMMU-Pro 69.5
- OmniDocBench v1.5
- OpenAI MRCR v2 44.2
- GraphWalks 73.4
- Mar 17, 2026
- SWE-Bench Pro 54.4
- Terminal-Bench 2.0 60
- Toolathlon 42.9
- GPQA Diamond 88
- OSWorld-Verified 72.1
- MCP Atlas 57.7
- tau2-bench 93.4
- Humanity's Last Exam 28.2
- MMMU-Pro 78
- OmniDocBench v1.5 0.126
- OpenAI MRCR v2 47.7
- GraphWalks 76.3
Its system card adds 11 evaluations 10 new 8 dropped 11 results in all
GPT-5.4 Thinking System Card, Appendix: GPT-5.4 miniSafety
Monitorability
Bio and chem risk
Cyber risk
- Capture the Flag (Professional) 81.3
- CVE-Bench 83.3
AI R&D risk
- Monorepo-Bench 54
- OpenAI-Proof Q&A 7.5
Dropped since the lab’s previous system card: Standard Refusal Evaluation, StrongREJECT, Image input safety evaluations, LongFact-Concepts, LongFact-Objects, FActScore, SimpleQA, SimpleQA Hallucination Rate
- Mar 16, 2026
- FLTEval 31.9
Dropped since the last post: SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.0
- Mar 16, 2026
- AIME 2025 78.5
- MATH-500 95.4
- GPQA Diamond 53.2
- LiveCodeBench 51.8
- Berkeley Function Calling Leaderboard (BFCL) 61.1
- IFEval 87.9
- tau2-bench 35.4
- IFBench 43.2
- Orak
- HaluEval 62.2
- RULER 91.1
- EQ-Bench 3
Dropped since the last post: MMLU-Pro, LiveCodeBench v6, SciCode, Humanity's Last Exam, miniF2F, Terminal-Bench Hard, SWE-bench Verified, BFCL V4, MultiChallenge, Arena-Hard v2.0, AA-LCR (Artificial Analysis Long Context Reasoning), MMLU-ProX, WMT24++
- Mar 16, 2026
- GPQA Diamond 71.2
- MMLU-Pro 78
- IFBench 48
- Arena-Hard-Auto 58.3
- MMMU-Pro 60
- AA-LCR (Artificial Analysis Long Context Reasoning) 71.2
- AIME 2025 83.8
- COLLIE 62.9
- LiveCodeBench 63.6
Dropped since the last post: SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.0
- Mar 11, 2026
- VQAv2 88.4
- MLVU 74.3
- MMVU 71.7
- RefCOCO / RefCOCO+ / RefCOCOg 89.9
- VideoHallucer 59.6
- Mobile Actions 88.4
- Mar 11, 2026
- MMLU-Pro 83.7
- AIME 2025 90.2
- HMMT 2025 93.7
- GPQA Diamond 79.2
- LiveCodeBench v5 81.2
- SciCode 42
- Humanity's Last Exam 18.3
- Terminal-Bench Hard 25.8
- Terminal-Bench 2.0 31
- SWE-bench Verified 60.5
- SWE-bench Multilingual 45.8
- tau2-bench 61.1
- BrowseComp 31.3
- Bird-SQL 41.8
- IFBench 72.6
- MultiChallenge 55.2
- Arena-Hard v2.0 73.9
- AA-LCR (Artificial Analysis Long Context Reasoning) 58.3
- RULER 91.8
- MMLU-ProX 79.4
- WMT24++ 86.7
Dropped since the last post: RoboCasa, RoboCasa GR1 Tabletop Tasks, SimplerEnv, LIBERO, BEHAVIOR Challenge 2025
Its technical report adds 1 evaluation 1 new 6 dropped 22 results in all
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningCapabilities
- LiveCodeBench v6 78.7
Dropped since the lab’s previous technical report: AIME 2024, MATH-500, Berkeley Function Calling Leaderboard (BFCL), IFEval, Arena-Hard-Auto, JudgeBench
- Mar 10, 2026
- MTEB 69.9
- TextCaps 89.6
- Docci 93.4
- ViDoRe v2 64.9
- VATEX 68.8
- MSR-VTT retrieval 68
- YouCook2 retrieval 52.5
- MSEB 73.9
Dropped since the last post: GenAI-Bench, Human SxS image editing Elo (Google)
Its technical report adds 0 evaluations 16 dropped 8 results in all
Gemini Embedding 2: A Native Multimodal Embedding Model from GeminiDropped since the lab’s previous technical report: Point-Bench, RefSpatial-Bench, RoboSpatial-Home, Where2Place, BLINK, CVBench, ERQA, EmbSpatialBench, MindCube, SAT (Spatial Aptitude Training), MVPBench, OpenEQA, VSI-Bench, MMMU, GPQA Diamond, Aider Polyglot
- Mar 9, 2026
- General refusals (xAI) 0
- Refusals + User Jailbreak (xAI) 1
- Refusals + System Jailbreak (xAI) 0
- AgentHarm 30
- AgentDojo 33
- MASK 73
- SycophancyEval (answer sycophancy) 4
- xAI sycophancy eval 35
- HLE Calibration 19
- WMDP Biology 91
- Virology Capabilities Test (VCT) 54
- ProtocolQA Open-Ended (xAI) 79
- LAB-Bench FigQA 66
- LAB-Bench CloningScenarios 67
- WMDP Chemistry 90
- WMDP Cyber 91
- Cybench 53
- MakeMeSay 8
Dropped since the last post: Research-Eval, FRAMES, X Browse (xAI), tau2-bench, BFCL V4
Its model card adds 1 evaluation 2 dropped 19 results in all
Grok 4.20 System CardBio and chem risk
Dropped since the lab’s previous model card: WMDP, BioLP-bench
- Mar 6, 2026
- MATH-500 97
- HumanEval 92.1
- MBPP 92.7
- LiveCodeBench v6 70
- MMLU 85.1
- MMLU-Pro 80
- Arena-Hard v2.0 49
- GPQA Diamond 66.5
- AIME 2025 80
- HMMT 2025 73.3
- HMMT Nov 2025 74.2
- Beyond AIME 58.3
- BrowseComp 35.5
- SWE-bench Verified 34
- tau2-bench 45.7
- IndiVibe 89
- Mar 6, 2026
- MATH-500 98.6
- LiveCodeBench v6 71.7
- MMLU 90.6
- MMLU-Pro 81.7
- WritingBench 80.5
- Arena-Hard v2.0 71
- IFEval 84.8
- GPQA Diamond 78.7
- AIME 2025 88.3
- HMMT 2025 85.8
- HMMT Nov 2025 85.8
- Beyond AIME 69.1
- BrowseComp 49.5
- SWE-bench Verified 45
- tau2-bench 68.3
- Humanity's Last Exam 11.2
- IndiVibe 90
Dropped since the last post: ARC-Challenge, TriviaQA, HumanEval, MBPP, GSM8K, MATH, MT-Bench, IndicGenBench, MILU
- Mar 5, 2026
- GDPval 82
- Vals Finance Agent 61.5
- Investment Banking Modeling Tasks (OpenAI) 83.6
- BrowseComp 89.3
- FrontierScience-Research 36.7
- FrontierMath 50
- FrontierMath Tier 4 38
- GPQA Diamond 94.4
- Humanity's Last Exam 58.7
- ARC-AGI 94.5
- ARC-AGI-2 83.3
Dropped since the last post: SWE-Bench Pro, OSWorld-Verified, Toolathlon, MMMU-Pro, OmniDocBench, tau2-bench, WebArena-Verified, Online-Mind2Web, BigLaw Bench, OfficeQA, MCP Atlas, GraphWalks, OpenAI MRCR v2
- Mar 5, 2026
- GDPval 83
- SWE-Bench Pro 57.7
- OSWorld-Verified 75
- Toolathlon 54.6
- BrowseComp 82.7
- MMMU-Pro 81.2
- OmniDocBench 10.9
- tau2-bench 98.9
- Vals Finance Agent 56
- Investment Banking Modeling Tasks (OpenAI) 87.3
- WebArena-Verified 67.3
- Online-Mind2Web 92.8
- BigLaw Bench 91
- OfficeQA 68.1
- MCP Atlas 67.2
- GraphWalks 93
- OpenAI MRCR v2 79.3
- FrontierScience-Research 33
- FrontierMath 47.6
- FrontierMath Tier 4 27.1
- GPQA Diamond 92.8
- Humanity's Last Exam 52.1
- ARC-AGI 93.7
- ARC-AGI-2 73.3
Dropped since the last post: Terminal-Bench 2.0, SWE-Lancer
Its system card adds 31 evaluations 17 new 1 dropped 31 results in all
GPT-5.4 Thinking System CardHealth
- HealthBench 62.6
- HealthBench Hard 40.1
- HealthBench Consensus 96.6
Safety
- OpenAI Production Benchmarks and safe-completion evals
- OpenAI Dynamic Mental Health Benchmarks
- Disallowed Content Evaluations with Representative Prompts
- Image input safety evaluations
- Destructive action avoidance 86
- Destructive action reversion on long rollouts
- User Confirmations During Computer Use
- OpenAI cyber safety evaluations (incl. Daybreak Blue)
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
Cyber risk
- Capture the Flag (Professional) 88.2
- CVE-Bench 86.3
- Cyber Range 73.3
- Atomic Challenges
- CyScenarioBench 11
AI R&D risk
- Monorepo-Bench 59.3
- MLE-bench 30 23.3
- OpenAI-Proof Q&A 4.16
Dropped since the lab’s previous system card: Apollo Research sabotage evaluations
- Mar 4, 2026
- AI2D 84.8
- ChartQA 83.3
- HallusionBench 64.4
- MathVerse 44.9
- MATH-Vision (MathVision) 36.2
- MathVista 75.2
- MMMU 54.3
- MMStar 64.5
- OCRBench 76
- ScreenSpot-v2 88.2
Dropped since the last post: WebVoyager, Online-Mind2Web, DeepShop, WebTailBench
Its technical report adds 1 evaluation 1 new 7 dropped 11 results in all
Phi-4-reasoning-vision-15B Technical ReportDropped since the lab’s previous technical report: WebVoyager, Online-Mind2Web, DeepShop, WebTailBench, ScreenSpot, AgentHarm, WebTailBench-Refusals
- Mar 3, 2026
- Humanity's Last Exam 16
- GPQA Diamond 86.9
- MMMU-Pro 76.8
- CharXiv 73.2
- Video-MMMU 84.8
- SimpleQA Verified 43.3
- FACTS Benchmark Suite 40.6
- MMMLU 88.9
- LiveCodeBench
- MRCR v2 60.1
- LMArena (Elo) 1432
Dropped since the last post: WMT24++, Vistra
Its model card adds 5 evaluations 1 new 13 dropped 15 results in all
Gemini 3.1 Flash-Lite Model CardDropped since the lab’s previous model card: MMLU-Pro, LiveCodeBench v5, Bird-SQL, SimpleQA, FACTS Grounding, Global MMLU Lite, MATH, HiddenMath, MRCR (Michelangelo, original), MMMU, CoVoST2, EgoSchema, Instruction Following (safety evaluation)
- Mar 2, 2026
- MMLU-Pro 42.3
- MMLU-Redux 59.5
- C-Eval 50.5
- SuperGPQA 21.3
- IFEval 44
- MMMLU 44.3
- GPQA Diamond 11.9
- IFBench 21
- MultiChallenge 18.9
- AA-LCR (Artificial Analysis Long Context Reasoning) 4.7
- LongBench v2 26.1
- BFCL V4 25.3
- tau2-bench 11.6
- MMLU-ProX 34.6
- NOVA-63 42.4
- INCLUDE 40.6
- Global PIQA 59.4
- PolyMath 8.2
- WMT24++ 27.2
- MAXIFE 39.2
- MMMU 49
- MMMU-Pro 31.2
- MathVista 62.2
- DynaMath 49.9
- ZeroBench 12.9
- VLMs Are Blind 59.4
- RealWorldQA 63.4
- MMStar 58.3
- MMBench-EN v1.1 69.9
- SimpleVQA 31.3
- HallusionBench 53.1
- MMLongBench-Doc 33.6
- AI2D 69.9
- CC-OCR 63.2
- OmniDocBench v1.5 61
- CharXiv 41.3
- OCRBench 74.5
- RefCOCO / RefCOCO+ / RefCOCOg 79.3
- CountBench 77
- ODinW-13 31.6
- ERQA 34.5
- EmbSpatialBench 68.6
- RefSpatial-Bench 23.5
- Hypersim 11.9
- SUNRGBD 26.1
- Nuscene 5.7
- Video-MME 63.8
- Video-MMMU 44.3
- MLVU 65.6
- MVBench 55.8
- LVBench 45.1
- MMVU 34.3
- SLAKE 62.6
- PMC-VQA 40.4
- MedXpertQA-MM 17.1
- ScreenSpot-Pro 46.5
Dropped since the last post: HMMT 2025, HMMT Nov 2025
- Mar 2, 2026
- MMLU-Pro 66.5
- MMLU-Redux 79.6
- C-Eval 73.2
- SuperGPQA 37.5
- IFEval 78.6
- MMMLU 63.1
- GPQA Diamond 51.6
- IFBench 41.3
- MultiChallenge 33.7
- AA-LCR (Artificial Analysis Long Context Reasoning) 25.6
- LongBench v2 38.7
- HMMT 2025 22.9
- HMMT Nov 2025 19.6
- BFCL V4 43.6
- tau2-bench 48.8
- MMLU-ProX 52.3
- NOVA-63 46.4
- INCLUDE 55.4
- Global PIQA 69.3
- PolyMath 26.1
- WMT24++ 45.8
- MAXIFE 60.6
- MMMU 64.2
- MMMU-Pro 50.3
- MathVista 76.7
- DynaMath 73.6
- ZeroBench 17.1
- VLMs Are Blind 75.8
- RealWorldQA 74.5
- MMStar 71.7
- MMBench-EN v1.1 83.3
- SimpleVQA 38.5
- HallusionBench 58
- MMLongBench-Doc 45.4
- AI2D 83.3
- CC-OCR 72.9
- OmniDocBench v1.5 79.8
- CharXiv 58.8
- OCRBench 84.5
- RefCOCO / RefCOCO+ / RefCOCOg 84.8
- CountBench 91.4
- ODinW-13 35.9
- ERQA 43.8
- EmbSpatialBench 77.9
- RefSpatial-Bench 32.9
- Hypersim 12.4
- SUNRGBD 28.7
- Nuscene 6.9
- Video-MME 75.6
- Video-MMMU 62.1
- MLVU 76.2
- MVBench 64.9
- LVBench 57.1
- MMVU 48.6
- SLAKE 74.4
- PMC-VQA 48.8
- MedXpertQA-MM 26.9
- ScreenSpot-Pro 54.5
Dropped since the last post: LiveCodeBench v6, OJBench, VITA-Bench, DeepPlanning, MATH-Vision (MathVision), We-Math, BabyVision, LingoQA, OSWorld-Verified, AndroidWorld, TIR-Bench, V*
- Mar 2, 2026
- MMLU-Pro 79.1
- MMLU-Redux 88.8
- C-Eval 85.1
- SuperGPQA 52.9
- GPQA Diamond 76.2
- IFEval 89.8
- IFBench 59.2
- MultiChallenge 49
- AA-LCR (Artificial Analysis Long Context Reasoning) 57
- LongBench v2 50
- HMMT 2025 74
- HMMT Nov 2025 76.8
- LiveCodeBench v6 55.8
- OJBench 24.1
- BFCL V4 50.3
- tau2-bench 79.9
- VITA-Bench 22
- DeepPlanning 17.6
- MMMLU 76.1
- MMLU-ProX 71.5
- NOVA-63 54.3
- INCLUDE 71
- Global PIQA 78.9
- PolyMath 51.1
- WMT24++ 66.6
- MAXIFE 78
- MMMU 77.6
- MMMU-Pro 66.3
- MATH-Vision (MathVision) 74.6
- MathVista 85.1
- We-Math 75.4
- DynaMath 83.3
- ZeroBench 26.3
- VLMs Are Blind 92.6
- BabyVision 16
- RealWorldQA 79.5
- MMStar 78.3
- MMBench-EN v1.1 89.4
- SimpleVQA 43.4
- HallusionBench 65
- OmniDocBench v1.5 86.2
- CharXiv 70.8
- MMLongBench-Doc 54.2
- CC-OCR 76.7
- AI2D 89.6
- OCRBench 85
- ERQA 54
- CountBench 96.3
- RefCOCO / RefCOCO+ / RefCOCOg 88.1
- EmbSpatialBench 81.3
- RefSpatial-Bench 54.6
- LingoQA 74.4
- Hypersim 12.5
- Nuscene 9.9
- Video-MME 83.5
- Video-MMMU 74.1
- MLVU 82.8
- MVBench 71.2
- LVBench 66.4
- MMVU 64.9
- ScreenSpot-Pro 60.3
- OSWorld-Verified 35.6
- AndroidWorld 58.6
- TIR-Bench 38.9
- V* 84.3
- SLAKE 76.1
- PMC-VQA 55.5
- MedXpertQA-MM 42.9
- Mar 2, 2026
- MMLU-Pro 82.5
- MMLU-Redux 91.1
- C-Eval 88.2
- SuperGPQA 58.2
- GPQA Diamond 81.7
- IFEval 91.5
- IFBench 64.5
- MultiChallenge 54.5
- AA-LCR (Artificial Analysis Long Context Reasoning) 63
- LongBench v2 55.2
- HMMT 2025 83.2
- HMMT Nov 2025 82.9
- LiveCodeBench v6 65.6
- OJBench 29.2
- BFCL V4 66.1
- tau2-bench 79.1
- VITA-Bench 29.8
- DeepPlanning 18
- MMMLU 81.2
- MMLU-ProX 76.3
- NOVA-63 55.9
- INCLUDE 75.6
- Global PIQA 83.2
- PolyMath 57.3
- WMT24++ 72.6
- MAXIFE 83.4
- MMMU 78.4
- MMMU-Pro 70.1
- MATH-Vision (MathVision) 78.9
- MathVista 85.7
- We-Math 75.2
- DynaMath 83.6
- ZeroBench 31.1
- VLMs Are Blind 93.7
- BabyVision 28.6
- RealWorldQA 80.3
- MMStar 79.7
- MMBench-EN v1.1 90.1
- SimpleVQA 51.2
- HallusionBench 69.3
- OmniDocBench v1.5 87.7
- CharXiv 73
- MMLongBench-Doc 57.7
- CC-OCR 79.3
- AI2D 90.2
- OCRBench 89.2
- ERQA 55.5
- CountBench 97.2
- RefCOCO / RefCOCO+ / RefCOCOg 89.7
- EmbSpatialBench 83
- RefSpatial-Bench 58.5
- LingoQA 80.4
- Hypersim 13.5
- Nuscene 11.8
- Video-MME 84.5
- Video-MMMU 78.9
- MLVU 84.4
- MVBench 74.4
- LVBench 70
- MMVU 67.8
- ScreenSpot-Pro 65.2
- OSWorld-Verified 41.8
- AndroidWorld 57.8
- TIR-Bench 45.6
- V* 90.1
- SLAKE 79
- PMC-VQA 57.9
- MedXpertQA-MM 49.9
Dropped since the last post: SWE-bench Verified, Terminal-Bench 2.0, Codeforces rating, FullStackBench, BrowseComp, BrowseComp-ZH, WideSearch, Seal-0, ODinW-13, SUNRGBD, Humanity's Last Exam
- Feb 26, 2026 Gemini 3.1 Flash Image (nano-banana-2, Web Search) Google DeepMind Gemini 3.1 Flash Image model card (Google DeepMind)
Dropped since the last post: Humanity's Last Exam, ARC-AGI-2, GPQA Diamond, Terminal-Bench 2.0, SWE-bench Verified, SWE-Bench Pro, LiveCodeBench Pro, SciCode, APEX-Agents, GDPval-AA, tau2-bench, MCP Atlas, BrowseComp, MMMU-Pro, MMMLU, MRCR v2
Its model card adds 1 evaluation 1 new 26 dropped 3 results in all
Gemini 3.1 Flash Image - Model CardCapabilities
Dropped since the lab’s previous model card: Humanity's Last Exam, ARC-AGI-2, GPQA Diamond, Terminal-Bench 2.0, SWE-bench Verified, SWE-Bench Pro, LiveCodeBench Pro, SciCode, APEX-Agents, GDPval-AA, tau2-bench, MCP Atlas, BrowseComp, MMMU-Pro, MMMLU, MRCR v2, Text to Text Safety, Multilingual Safety, Image to Text Safety, Tone, Unjustified refusals, CBRN expert red teaming, Harmful manipulation efficacy study, RE-Bench, GDM Stealth, GDM Situational Awareness
- Feb 24, 2026
- MMLU-Pro 85.3
- MMLU-Redux 93.3
- C-Eval 90.2
- SuperGPQA 63.4
- IFEval 91.9
- IFBench 70.2
- MultiChallenge 60
- AA-LCR (Artificial Analysis Long Context Reasoning) 58.5
- LongBench v2 59
- GPQA Diamond 84.2
- HMMT 2025 89
- HMMT Nov 2025 89.2
- SWE-bench Verified 69.2
- Terminal-Bench 2.0 40.5
- LiveCodeBench v6 74.6
- Codeforces rating 2028
- OJBench 36
- FullStackBench 56.5
- BFCL V4 67.3
- tau2-bench 81.2
- VITA-Bench 31.9
- DeepPlanning 22.8
- BrowseComp 61
- BrowseComp-ZH 69.5
- WideSearch 57.1
- Seal-0 41.4
- MMMLU 85.2
- MMLU-ProX 81
- NOVA-63 57.1
- INCLUDE 79.7
- Global PIQA 86.6
- PolyMath 64.4
- WMT24++ 76.3
- MAXIFE 86.6
- MMMU 81.4
- MMMU-Pro 75.1
- MATH-Vision (MathVision) 83.9
- MathVista 86.2
- DynaMath 85
- ZeroBench 34.1
- VLMs Are Blind 97
- BabyVision 38.4
- RealWorldQA 84.1
- MMStar 81.9
- MMBench-EN v1.1 91.5
- SimpleVQA 58.3
- HallusionBench 67.9
- OmniDocBench v1.5 89.3
- CharXiv 77.5
- MMLongBench-Doc 59.5
- CC-OCR 80.7
- AI2D 92.6
- OCRBench 91
- ERQA 64.8
- CountBench 97.8
- RefCOCO / RefCOCO+ / RefCOCOg 89.2
- ODinW-13 42.6
- EmbSpatialBench 83.1
- RefSpatial-Bench 63.5
- LingoQA 79.2
- Hypersim 13.1
- SUNRGBD 33.4
- Nuscene 14.6
- Video-MME 86.6
- Video-MMMU 80.4
- MLVU 85.6
- MVBench 74.8
- LVBench 71.4
- MMVU 72.3
- ScreenSpot-Pro 68.6
- OSWorld-Verified 54.5
- AndroidWorld 71.1
- TIR-Bench 55.5
- V* 92.7
- SLAKE 78.7
- PMC-VQA 62
- MedXpertQA-MM 61.4
- Humanity's Last Exam 47.4
- Feb 24, 2026
- MMLU-Pro 86.1
- MMLU-Redux 93.2
- C-Eval 90.5
- SuperGPQA 65.6
- IFEval 95
- IFBench 76.5
- MultiChallenge 60.8
- AA-LCR (Artificial Analysis Long Context Reasoning) 66.1
- LongBench v2 60.6
- GPQA Diamond 85.5
- HMMT 2025 92
- HMMT Nov 2025 89.8
- SWE-bench Verified 72.4
- Terminal-Bench 2.0 41.6
- LiveCodeBench v6 80.7
- Codeforces rating 1899
- OJBench 40.1
- FullStackBench 58.8
- BFCL V4 68.5
- tau2-bench 79
- VITA-Bench 41.9
- DeepPlanning 22.6
- BrowseComp 61
- BrowseComp-ZH 62.1
- WideSearch 61.1
- Seal-0 47.2
- MMMLU 85.9
- MMLU-ProX 82.2
- NOVA-63 58.1
- INCLUDE 81.6
- Global PIQA 87.5
- PolyMath 71.2
- WMT24++ 77.6
- MAXIFE 88
- MMMU 82.3
- MMMU-Pro 75
- MATH-Vision (MathVision) 86
- MathVista 87.8
- DynaMath 87.7
- ZeroBench 36.2
- VLMs Are Blind 96.9
- BabyVision 44.6
- RealWorldQA 83.7
- MMStar 81
- MMBench-EN v1.1 92.6
- SimpleVQA 56
- HallusionBench 70
- OmniDocBench v1.5 88.9
- CharXiv 79.5
- MMLongBench-Doc 60.2
- CC-OCR 81
- AI2D 92.9
- OCRBench 89.4
- ERQA 60.5
- CountBench 97.8
- RefCOCO / RefCOCO+ / RefCOCOg 90.9
- ODinW-13 41.1
- EmbSpatialBench 84.5
- RefSpatial-Bench 67.7
- LingoQA 82
- Hypersim 13
- SUNRGBD 35.4
- Nuscene 15.2
- Video-MME 87
- Video-MMMU 82.3
- MLVU 85.9
- MVBench 74.6
- LVBench 73.6
- MMVU 73.3
- ScreenSpot-Pro 70.3
- OSWorld-Verified 56.2
- AndroidWorld 64.2
- TIR-Bench 59.8
- V* 93.7
- SLAKE 80
- PMC-VQA 62.4
- MedXpertQA-MM 62.4
- Humanity's Last Exam 48.5
- Feb 24, 2026
- GPQA Diamond 74
- SciCode 38
- IFBench 71
Dropped since the last post: MMLU-Pro, Humanity's Last Exam, HumanEval, MATH-500, AIME 2024
- Feb 24, 2026
- MMLU-Pro 86.7
- MMLU-Redux 94
- C-Eval 91.9
- SuperGPQA 67.1
- IFEval 93.4
- IFBench 76.1
- MultiChallenge 61.5
- AA-LCR (Artificial Analysis Long Context Reasoning) 66.9
- LongBench v2 60.2
- Humanity's Last Exam 47.5
- GPQA Diamond 86.6
- HMMT 2025 91.4
- HMMT Nov 2025 90.3
- SWE-bench Verified 72
- Terminal-Bench 2.0 49.4
- LiveCodeBench v6 78.9
- Codeforces rating 2100
- OJBench 39.5
- FullStackBench 60.7
- BFCL V4 72.2
- tau2-bench 79.5
- VITA-Bench 33.6
- DeepPlanning 24.1
- BrowseComp 63.8
- BrowseComp-ZH 69.9
- WideSearch 60.5
- Seal-0 44.1
- MMMLU 86.7
- MMLU-ProX 82.2
- NOVA-63 58.6
- INCLUDE 82.8
- Global PIQA 88.4
- PolyMath 68.9
- WMT24++ 78.3
- MAXIFE 87.9
- MMMU 83.9
- MMMU-Pro 76.9
- MATH-Vision (MathVision) 86.2
- MathVista 87.4
- DynaMath 85.9
- ZeroBench 36.2
- VLMs Are Blind 96.7
- BabyVision 40.2
- RealWorldQA 85.1
- MMStar 82.9
- MMBench-EN v1.1 92.8
- SimpleVQA 61.7
- HallusionBench 67.6
- OmniDocBench v1.5 89.8
- CharXiv 77.2
- MMLongBench-Doc 59
- CC-OCR 81.8
- AI2D 93.3
- OCRBench 92.1
- ERQA 62
- CountBench 97
- RefCOCO / RefCOCO+ / RefCOCOg 91.3
- ODinW-13 44.5
- EmbSpatialBench 83.9
- RefSpatial-Bench 69.3
- LingoQA 80.8
- Hypersim 12.7
- SUNRGBD 36.2
- Nuscene 15.4
- Video-MME 87.3
- Video-MMMU 82
- MLVU 87.3
- MVBench 76.6
- LVBench 74.4
- MMVU 74.7
- ScreenSpot-Pro 70.4
- OSWorld-Verified 58
- AndroidWorld 66.4
- TIR-Bench 53.2
- V* 93.2
- SLAKE 81.6
- PMC-VQA 63.3
- MedXpertQA-MM 67.3
Dropped since the last post: SWE-bench Multilingual, SWE-Bench Pro, Aider Polyglot
- Feb 19, 2026
- Humanity's Last Exam 51.4
- ARC-AGI-2 77.1
- GPQA Diamond 94.3
- Terminal-Bench 2.0 68.5
- SWE-bench Verified 80.6
- SWE-Bench Pro 54.2
- LiveCodeBench Pro 2887
- SciCode 59
- APEX-Agents 33.5
- GDPval-AA 1317
- tau2-bench
- MCP Atlas 69.2
- BrowseComp 85.9
- MMMU-Pro 80.5
- MMMLU 92.6
- MRCR v2 84.9
Dropped since the last post: Codeforces rating, IMO 2025, IPhO 2025, IChO 2025, CMT-Benchmark
Its model card adds 10 evaluations 10 new 3 dropped 26 results in all
Gemini 3.1 Pro Model CardAlignment
Bio and chem risk
AI R&D risk
- RE-Bench 1.27
Persuasion
Dropped since the lab’s previous model card: Online-Mind2Web, WebVoyager, AndroidWorld
- Feb 17, 2026
- Terminal-Bench 2.0 59.1
- SWE-bench Verified 79.6
- OSWorld-Verified 72.5
- tau2-bench 94.8
- MCP Atlas 61.3
- BrowseComp 74.7
- Humanity's Last Exam 49
- Vals Finance Agent 63.3
- GDPval-AA 1633
- ARC-AGI-2 58.3
- GPQA Diamond 89.9
- MMMU-Pro 75.6
- MMMLU 89.3
Dropped since the last post: BigLaw Bench
Its system card adds 67 evaluations 4 new 17 dropped 80 results in all
System Card: Claude Sonnet 4.6Capabilities
- SWE-bench Multilingual 75.9
- OpenRCA 27.9
- ARC-AGI 86.5
- AIME 2025 95.6
- Real-World Finance (Anthropic) 59.5
- Vending-Bench 2 7204.1
- CyberGym 65.2
- OpenAI MRCR v2
- GraphWalks
- LAB-Bench FigQA 58.8
- CharXiv 72.4
- WebArena 65.6
- WebArena-Verified 66.4
- Global MMLU 88.7
- MILU 89.6
- DeepSearchQA 89.2
- BioPipelineBench (Anthropic) 52.1
- BioMysteryBench 50.4
- Structural Biology (Anthropic)
- Organic Chemistry (Anthropic) 48.4
- Phylogenetics (Anthropic) 49.1
Health
Safety
- Single-turn harmful request evaluations (Anthropic) 99.4
- Single-turn benign request evaluations (Anthropic) 0.41
- Higher-difficulty harmful request evaluations (Anthropic) 99.4
- Higher-difficulty benign request evaluations (Anthropic) 0.18
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- SSH stress-testing evaluation (Anthropic) 92
- Agentic coding malicious requests (Anthropic) 100
- Malicious use of Claude Code (Anthropic) 79.3
- Malicious computer use (Anthropic) 99.4
Robustness
Alignment
- Reward-hack-prone coding tasks (Anthropic) 0
- Impossible tasks (Anthropic) 40
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 14.7
- Anthropic automated behavioral audit
- Petri
- Refusal to assist with AI safety R&D (Anthropic) 38
- Self-preference bias (Anthropic)
- Participation in junk science (Anthropic) 0
- SHADE-Arena 1.6
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Honesty human feedback win rate (Anthropic), 100Q-Hard (Anthropic), SimpleQA Verified, AA-Omniscience, ECLeKTic, False-premise questions (Anthropic), External pilot use failure modes (Anthropic), UK AISI alignment and monitorability evaluations, Internal codebase sabotage propensity (Anthropic), Deference to governments in local languages (Anthropic), Agentic misalignment suite (Anthropic), Creative Biology Uplift Trial (Anthropic), ASL-4 virology uplift trial (Anthropic), Internal model use survey (Anthropic), Internal AI research evaluation suite 2, Misc CTF challenges (Anthropic), Forensics CTF challenges (Anthropic)
- Feb 16, 2026
- MMLU-Pro 87.8
- MMLU-Redux 94.9
- SuperGPQA 70.4
- C-Eval 93
- IFEval 92.6
- IFBench 76.5
- MultiChallenge 67.6
- AA-LCR (Artificial Analysis Long Context Reasoning) 68.7
- LongBench v2 63.2
- GPQA Diamond 88.4
- Humanity's Last Exam 48.3
- LiveCodeBench v6 83.6
- HMMT 2025 94.8
- HMMT Nov 2025 92.7
- IMO-AnswerBench 80.9
- AIME 2026 91.3
- BFCL V4 72.9
- tau2-bench 86.7
- VITA-Bench 49.7
- DeepPlanning 34.3
- Toolathlon 38.3
- MCPMark 46.1
- BrowseComp 78.6
- BrowseComp-ZH 70.3
- WideSearch 74
- Seal-0 46.9
- MMMLU 88.5
- MMLU-ProX 84.7
- NOVA-63 59.1
- INCLUDE 85.6
- Global PIQA 89.8
- PolyMath 73.3
- WMT24++ 78.9
- MAXIFE 88.2
- SWE-bench Verified 76.4
- SWE-bench Multilingual 69.3
- SecCodeBench 68.3
- Terminal-Bench 2.0 52.5
- MMMU 85
- MMMU-Pro 79
- MATH-Vision (MathVision) 88.6
- MathVista 90.3
- We-Math 87.9
- DynaMath 86.3
- ZeroBench 12
- BabyVision 52.3
- RealWorldQA 83.9
- MMStar 83.8
- HallusionBench 71.4
- MMBench-EN v1.1 93.7
- SimpleVQA 67.1
- OmniDocBench v1.5 90.8
- CharXiv 80.8
- MMLongBench-Doc 61.5
- CC-OCR 82
- AI2D 93.9
- OCRBench 93.1
- ERQA 67.5
- CountBench 97.2
- ODinW-13 47
- EmbSpatialBench 84.5
- RefSpatial-Bench 73.6
- LingoQA 81.6
- V* 95.8
- Hypersim 12.5
- SUNRGBD 38.3
- Nuscene 16
- Video-MME 87.5
- Video-MMMU 84.7
- MLVU 86.7
- MVBench 77.6
- LVBench 75.5
- MMVU 75.4
- ScreenSpot-Pro 65.6
- OSWorld-Verified 62.2
- AndroidWorld 66.8
- SLAKE 79.9
- PMC-VQA 64.2
- MedXpertQA-MM 70
- RefCOCO / RefCOCO+ / RefCOCOg 92.3
Dropped since the last post: Arena-Hard v2.0
- Feb 14, 2026
- MMLU-Pro 87
- Humanity's Last Exam 54.2
- HLE-Verified 73.6
- SimpleQA Verified 36
- HealthBench 57.7
- HealthBench Hard 29.1
- SuperGPQA 68.7
- LPFQA 52.6
- Encyclo-K 65.7
- AIME 2026 94.2
- AIME 2025 98.3
- HMMT 2025 97.3
- HMMT Nov 2025 93.3
- MathArena (Apex, IMO, USAMO) 20.3
- Beyond AIME 86.5
- IMO-AnswerBench 89.3
- Codeforces rating 3020
- AetherCode 60.6
- LiveCodeBench v6 87.8
- GPQA Diamond 88.9
- SUPERChem 51.6
- BABE 50
- PHYBench 74
- FrontierScience 74
- ARC-AGI 85.4
- ARC-AGI-2 37.5
- KOR-Bench 77.5
- ProcBench 96.6
- MRCR v2 54
- GraphWalks 83.3
- LongBench v2 63.8
- FRAMES 84.5
- DeR2 Bench 58.2
- CL-bench 20.8
- Global PIQA 92.3
- MMMLU 88.1
- DiscoX 82
- MultiChallenge 68.3
- COLLIE 93.9
- MARS-Bench 85.6
- Inverse IFEval 78.9
- LongFact 92.8
- FActScore precision 71.2
- Terminal-Bench 2.0 55.8
- SWE-Lancer 49.4
- SWE-bench Verified 76.5
- Multi-SWE-bench 45.2
- SWE-Bench Pro 46.9
- SWE-bench Multilingual 71.7
- SciCode 48.5
- SWE-EVO 8.5
- Aider Polyglot 80
- ArtifactsBench 66.6
- CodeSimpleQA 58
- SpreadsheetBench Verified 79.1
- BrowseComp 77.3
- BrowseComp-ZH 82.4
- WideSearch 74.7
- FinSearchComp 70.2
- DeepSearchQA 77.4
- Seal-0 49.5
- tau2-bench 92.3
- MCPMark 54.7
- BFCL V4 73.4
- VITA-Bench 47
- DeepConsult 61.1
- DeepResearch Bench 53.3
- ResearchRubrics 50.7
- MM-BrowseComp 48.8
- HLE-VL 39.2
- MathVista 89.8
- MATH-Vision (MathVision) 88.8
- LogicVista 81.9
- VisuLogic 47.4
- VLMsAreBiased 77.4
- VLMs Are Blind 98.6
- BabyVision 60.6
- ChartQA Pro 71.2
- OmniDocBench v1.5
- DUDE 72.4
- MMLongBench 74.8
- MMLongBench-Doc 61.4
- TVBench 75
- TempCompass 89.6
- MotionBench 75.2
- EgoTempo 71.8
- Video-MME 89.5
- LVBench 76.4
- Math Kangaroo 2025 (MathArena) 90.5
- MathCanvas-Bench 61.9
- MineDojo 49
- PutnamBench 35.5
- IMO 2025 35
- CMO 2025 114
Dropped since the last post: FATE-H, FATE-X, Putnam 2025, CombiBench
Its model card adds 69 evaluations 161 results in all
Seed2.0 Model Card: Towards Intelligence Frontier for Real-World ComplexityCapabilities
- FrontierScience-Research 25
- FrontierScience-Olympiad 74
- Chinese Complex Instruction Following (ByteDance Seed in-house) 75.3
- DynaMath 68.9
- MMMU 85.4
- MMMU-Pro 78.2
- EMMA 72
- Scientists' First Exam (SFE) 55.6
- HiPhO 74.1
- XLRS-Bench 54.6
- PhyX 72.1
- VPCT 76
- ZeroBench 12
- VisFactor 36.8
- RealWorldQA 86
- SimpleVQA 71.4
- HallusionBench 68
- MME-CC 57
- MMStar 83
- MuirBench 81.8
- MTVQA-Test 51.1
- WorldVQA 49.9
- Vibe-Eval 81.4
- ViVerBench 75.9
- CountBench 95.5
- Point-Bench 81.4
- BLINK 79.5
- MMSI-Bench 32.5
- TreeBench 64.7
- RefSpatial-Bench 72.6
- All-Angles Bench 72.1
- ERQA 68.5
- OCRBench v2 62.5
- CharXiv (DQ) 93.5
- CharXiv 80.5
- LongDocURL 74.7
- Video-MMMU 86.9
- MMVU 78.2
- Video SimpleQA 71.9
- VideoReasonBench 77.8
- MORSE-500 37.4
- Video-Holmes 67.4
- MINERVA 66.5
- ContPhy 67.4
- TOMATO 59.9
- CG-Bench 65
- LongVideoBench 80.3
- VideoEval-Pro 48
- CrossVid 60.3
- OVBench 69.2
- LiveSports-3K 78
- OVO-Bench 77
- ODV-Bench 72.5
- ViSpeak-Bench 78.5
- ZeroVideo 14.5
- BIOBench (ByteDance Seed) 53.5
- AInstein-SWE-Bench 47.7
- NL2Repo-Bench 27.9
- Complex Workflow (ByteDance Seed in-house) 64.7
- Customer Support Q&A (ByteDance Seed in-house) 72.4
- GDPVal-Diamond 21.3
- XpertBench 64.5
- Education (ByteDance Seed in-house) 62.8
- Compositional Tasks (ByteDance Seed in-house) 59.1
- Text Classification (ByteDance Seed in-house) 69
- Information Extraction (ByteDance Seed in-house) 52
- WorldTravel
Hallucination
- LongFact-Objects 92.9
- LongFact-Concepts 92.8
- Feb 12, 2026
- SWE-Bench Pro 51.5
- Terminal-Bench 2.0 58.4
Dropped since the last post: AIME 2025, FrontierMath, GPQA Diamond, Humanity's Last Exam, HMMT 2025, MMMU, MMMU-Pro, CharXiv, Video-MMMU, ERQA, SWE-Lancer, SWE-bench Verified, Aider Polyglot, MultiChallenge, Internal API instruction following (OpenAI), COLLIE, tau2-bench, OpenAI MRCR, GraphWalks, BrowseComp Long Context 128k, Video-MME, LongFact, FActScore
- Feb 12, 2026 Gemini 3 Deep Think Google DeepMind Gemini 3 Deep Think: Advancing science, research and engineering (Google blog)
- Humanity's Last Exam 48.4
- ARC-AGI-2 84.6
- Codeforces rating 3455
- IMO 2025
- IPhO 2025
- IChO 2025
- CMT-Benchmark 50.5
- MMMU-Pro 81.5
Dropped since the last post: GPQA Diamond, AIME 2025, ScreenSpot-Pro, CharXiv, OmniDocBench v1.5, Video-MMMU, LiveCodeBench Pro, Terminal-Bench 2.0, SWE-bench Verified, tau2-bench, Toolathlon, MCP Atlas, Vending-Bench 2, FACTS Benchmark Suite, SimpleQA Verified, MMMLU, Global PIQA, MRCR v2
- Feb 12, 2026 MiniMax-M2.5 MiniMax MiniMax-M2.5 model card (Hugging Face) and launch post (minimax.io/news/minimax-m25)
- SWE-bench Verified 80.2
- SWE-Bench Pro 55.4
- Terminal-Bench 2.0 51.7
- Multi-SWE-bench 51.3
- SWE-bench Multilingual 74.1
- VIBE-Pro (MiniMax) 54.2
- BrowseComp 76.3
- WideSearch 70.3
- RISE 50.2
- Berkeley Function Calling Leaderboard (BFCL)
- tau2-bench 97.8
- GDPval-MM 59
- MEWC 74.4
- Finance Modeling 21.6
- AIME 2025 86.3
- GPQA Diamond 85.2
- Humanity's Last Exam 19.4
- SciCode 44.4
- IFBench 70
- AA-LCR (Artificial Analysis Long Context Reasoning) 69.5
Dropped since the last post: SWT-bench, SWE-Perf, SWE-Review, OctoCodingbench, VIBE (MiniMax), Toolathlon, MMLU-Pro, LiveCodeBench
Its technical report adds 11 evaluations 10 new 11 dropped 25 results in all
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World IntelligenceCapabilities
- NL2Repo-Bench 26.6
- HyperTask 59.4
- GDPval-AA 35
- Toolathlon 38.3
- MM-ClawBench 57.6
- MEWC v2 49.8
- Finance Modeling Pro 33.8
- AIME 2026 87.2
- MMLU-Pro 85.2
- Artificial Analysis Intelligence Index v4.0 42
AI R&D risk
- MLE-bench 51.5
Dropped since the lab’s previous technical report: AIME 2024, MATH-500, LiveCodeBench v6, FullStackBench, ZebraLogic, SWE-bench Verified, OpenAI MRCR, LongBench v2, tau-bench, SimpleQA, MultiChallenge
- Feb 11, 2026
- Humanity's Last Exam 50.4
- AIME 2026 92.7
- HMMT Nov 2025 96.9
- IMO-AnswerBench 82.5
- GPQA Diamond 86
- SWE-bench Verified 77.8
- SWE-bench Multilingual 73.3
- Terminal-Bench 2.0 56.2
- CyberGym 43.2
- BrowseComp 75.9
- BrowseComp-ZH 72.7
- tau2-bench 89.7
- MCP Atlas 67.8
- Toolathlon 38
- Vending-Bench 2 4432.1
Dropped since the last post: CVTG-2K, LongText-Bench, OneIG-Bench, TIIF-Bench, DPG-Bench
Its technical report adds 34 evaluations 34 new 39 dropped 49 results in all
GLM-5: from Vibe Coding to Agentic EngineeringCapabilities
- HMMT 2025 97.9
- LongBench v2 64.5
- GDPval-AA 1409
- Artificial Analysis Intelligence Index v4.0 50
- LMArena (Elo) 1452
- Code Arena 1449
- CC-Bench-V2 Frontend 74.8
- CC-Bench-V2 Backend 25.8
- CC-Bench-V2 Large Repo Exploration 65.6
- CC-Bench-V2 Multi-Step Chained Tasks 52.3
- SWE-rebench 42.1
- ZMultiTransBench 1050
- MENT-SNS 1013
- Translation Human Scoring (Z.ai) 3.12
- ZMultiDialBench 8.57
- IF-Badcase 83.2
- IFBench 72
- MultiChallenge 64.8
- ToolCall-Badcase 95.8
- Slide generation human evaluation (Z.ai) 67.5
- BIG-Bench Hard 87.4
- MMLU 88.3
- HellaSwag 88.1
- PIQA 84.6
- TriviaQA 80.9
- EvalPlus 87
- LiveCodeBench
- GSM8K 68.8
- MATH 56.4
- CLUEWSC 84.2
- C-Eval 88.8
- C3 80.3
Hallucination
- SimpleQA 36.9
- Chinese SimpleQA 75.2
Dropped since the lab’s previous technical report: MMBench-EN v1.1, MMBench-CN v1.1, MMStar, BLINK, MuirBench, HallusionBench, ZeroBench, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), MathVerse, DynaMath, LogicVista, AI2D, We-Math, MMLongBench-Doc, OCRBench, ChartQA Pro, ChartMuseum, RefCOCO / RefCOCO+ / RefCOCOg, TreeBench, Ref-L4, OmniSpatial, CVBench, ERQA, All-Angles Bench, OSWorld, AndroidWorld, WebVoyager, WebQuest, Design2Code, Flame-React-Eval, Video-MME, MMVU, Video-MMMU, LVBench, MotionBench, MVBench
- Feb 10, 2026
- IMO-AnswerBench 90
- AIME 2026 95
- HMMT Nov 2025 98.3
- LiveCodeBench v6 89.4
- Meta Agents Research Environments (ARE) / Gaia2 75
- tau2-bench 78.4
- SWE-bench Verified 72.4
- ARC-AGI-2 19.3
- IMO 2025 35
- CMO 2025 105
Dropped since the last post: AIME 2025, Codeforces rating, ARC-AGI, Arena-Hard v2.0, HMMT 2025, HealthBench, Creative Writing v3, ICPC World Finals 2025
- Feb 9, 2026
- Feb 6, 2026
ERNIE 5.0 Baidu ERNIE 5.0: A 2.4 Trillion-Parameter Unified Multimodal Foundation Model (ERNIE Blog) - SimpleQA 74
- Chinese SimpleQA 86
- MMLU-Pro 83.8
- Humanity's Last Exam 25.8
- GPQA Diamond 86.4
- AIME 2025 89.1
- HMMT 2025 79.6
- LiveCodeBench v6 76.2
- HumanEval+ 94.5
- MBPP+ 82.5
- ZebraLogic 96.5
- BBEH micro average 66.6
- IFEval 93.3
- MultiChallenge 66
- Multi-IF 85.6
- tau2-bench 78.8
- AceBench 88.7
- BFCL V4 66.5
- BrowseComp-ZH 64.7
- SpreadsheetBench 40.1
- MMMU-Pro 68.6
- MathVista 84.8
- MathVerse 85.1
- MATH-Vision (MathVision) 74.3
- VisualPuzzles 64.8
- VisuLogic 32
- VLMs Are Blind 91.4
- ChartQA 87.8
- AI2D 96.9
- DocVQA 95.5
- OCRBench 87.8
- CharXiv 67.1
- SimpleVQA 67.6
- HallusionBench 63.9
- MMStar 75.5
- BLINK 70
- CVBench 87.2
- CountBench 96.5
- Video-MME 81.3
- Video-MMMU 81.1
- MMVU 72.2
- GenEval 90.1
- VBench 84.2
Dropped since the last post: CCBench (MMBench Chinese Culture), MMMU, MVBench, C-Eval, CMMLU, MATH-500, CNMO 2024, GSM8K, CMATH, BIG-Bench Hard, DROP, CLUEWSC, LiveCodeBench
Its technical report adds 9 evaluations 52 results in all
ERNIE 5.0 Technical ReportCapabilities
- AISHELL-1 0.31
- AISHELL-2 2.64
- WenetSpeech
- LibriSpeech
- FLEURS
- VoiceBench
- MMAU 80.4
- Clotho-AQA 73.7
- Seed-TTS-eval
- Feb 5, 2026
- Terminal-Bench 2.0 65.4
- SWE-bench Verified 80.8
- OSWorld-Verified 72.7
- tau2-bench 95.6
- MCP Atlas 59.5
- BrowseComp 84
- Humanity's Last Exam 53
- Vals Finance Agent 60.7
- GDPval-AA 1606
- ARC-AGI-2 68.8
- GPQA Diamond 91.3
- MMMU-Pro 77.3
- MMMLU 91.1
- BigLaw Bench 90.2
Dropped since the last post: MMMU
Its system card adds 80 evaluations 25 new 12 dropped 93 results in all
System Card: Claude Opus 4.6Capabilities
- SWE-bench Multilingual 77.8
- OpenRCA 34.9
- ARC-AGI 94
- AIME 2025 99.8
- Real-World Finance (Anthropic) 64.1
- Vending-Bench 2 8017.6
- CyberGym 66.6
- OpenAI MRCR v2
- GraphWalks
- LAB-Bench FigQA 58
- CharXiv 68.5
- WebArena 68
- DeepSearchQA 91.3
- BioPipelineBench (Anthropic) 53.1
- BioMysteryBench 61.5
- Structural Biology (Anthropic)
- Organic Chemistry (Anthropic) 53.9
- Phylogenetics (Anthropic) 61.3
Hallucination
Safety
- Single-turn harmful request evaluations (Anthropic) 99.6
- Single-turn benign request evaluations (Anthropic) 0.68
- Higher-difficulty harmful request evaluations (Anthropic) 99.2
- Higher-difficulty benign request evaluations (Anthropic) 0.04
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Suicide and self-harm evaluations (Anthropic)
- SSH stress-testing evaluation (Anthropic) 95
- Agentic coding malicious requests (Anthropic) 99.3
- Malicious use of Claude Code (Anthropic) 83.2
- Malicious computer use (Anthropic) 88.3
Robustness
Alignment
- External pilot use failure modes (Anthropic)
- Reward-hack-prone coding tasks (Anthropic) 0
- Impossible tasks (Anthropic) 50
- Agentic code behavior scores (Anthropic)
- Over-eager behavior on GUI computer use tasks (Anthropic) 11.4
- Anthropic automated behavioral audit
- Petri
- UK AISI alignment and monitorability evaluations
- Refusal to assist with AI safety R&D (Anthropic) 14
- Self-preference bias (Anthropic)
- Internal codebase sabotage propensity (Anthropic) 0
- Participation in junk science (Anthropic) 1.1
- Deference to governments in local languages (Anthropic)
- SHADE-Arena 18
- Agentic misalignment suite (Anthropic)
Bio and chem risk
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 47.2
- Internal model use survey (Anthropic) 0
- Kernels task (Anthropic) 190
- Time series forecasting task (Anthropic)
- Text-based reinforcement learning task (Anthropic) 0.975
- LLM training task (Anthropic) 34
- Quadruped reinforcement learning task (Anthropic)
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.612
Dropped since the lab’s previous system card: MMMU, SWE-Bench Pro, BrowseComp-Plus, Multi-agent search (Anthropic internal benchmark), SpreadsheetBench, Sycophancy on user-provided prompts (Anthropic), Training distribution hack rate (Anthropic), Subversion Strategy Eval, LAB-Bench ProtocolQA, LAB-Bench SeqQA, LAB-Bench CloningScenarios, Bioinformatics evaluations (Anthropic)
- Feb 5, 2026
- SWE-Bench Pro 56.8
- Terminal-Bench 2.0 77.3
- OSWorld-Verified 64.7
- GDPval 70.9
- SWE-Lancer 81.4
Dropped since the last post: Capture the Flag (Professional)
Its system card adds 15 evaluations 4 new 10 dropped 15 results in all
GPT-5.3-Codex System CardSafety
Alignment
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: StrongREJECT, Image input safety evaluations, Malware Refusals (Golden Set), Coding prompt injections, Long-form Biological Risk Questions, SWE-Lancer, PaperBench, MLE-bench 30, OpenAI PRs, METR Time Horizon
- Feb 2, 2026
- SciReasoner evaluation suite 55.5
- Scientists' First Exam (SFE) 52.7
- SMolInstruct 74.8
- Matbench 72.8
- Mol-Instructions 48.8
- MicroVQA 63.3
- Biology-Instructions 52.5
- XLRS-Bench 52.8
- MSEarth 65.2
- MMMU-Pro 72.8
- MMLU-Pro 86.6
- AIME 2025 93.1
- IMO-AnswerBench 77.3
- RefCOCO / RefCOCO+ / RefCOCOg 91.9
- IFBench 71.2
- OCRBench v2 60.1
- SArena 83.5
- LiveCodeBench v6 74.3
- GAIA 77.4
- tau2-bench 80.9
- ScreenSpot-v2 93.6
Dropped since the last post: MMStar, MM-Vet, MMBench-EN v1.1, MTVQA-Test, AI2D, OCRBench, WildVision, MME-RealWorld, HallusionBench, MVBench, Video-MME, MLVU, LongVideoBench, MMMU, MathVista, MATH-Vision (MathVision), MathVerse, DynaMath, We-Math, OlympiadBench, LogicVista, MATH-500, AIME 2024, GPQA Diamond, C-Eval, GAOKAO-Bench, IFEval, SGP-Bench, ScreenSpot, OSWorld-G, VSI-Bench, ERQA, SpaCE-10, OmniSpatial
Its technical report adds 1 evaluation 1 new 67 dropped 22 results in all
Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion ScaleCapabilities
Dropped since the lab’s previous technical report: MMStar, MM-Vet, MMBench-EN v1.1, MME, MTVQA-Test, MMMB, AI2D, ChartQA, TextVQA, DocVQA, InfographicVQA, OCRBench, SEED-Bench-2-Plus, CharXiv, VCR (Visual Caption Restoration), WildVision, MME-RealWorld, RealWorldQA, R-Bench (real-world robustness), BLINK, Mantis-Eval, MMIU, MuirBench, MMT-Bench, MIRB, HallusionBench, CRPE, POPE, MVBench, Video-MME, MMBench-Video, MLVU, LongVideoBench, MMMU, MathVista, MATH-Vision (MathVision), MathVerse, DynaMath, We-Math, OlympiadBench, LogicVista, MATH-500, AIME 2024, GPQA Diamond, C-Eval, GAOKAO-Bench, IFEval, MMLU, CMMLU, TriviaQA, Natural Questions, C3, RACE, WinoGrande, HellaSwag, BIG-Bench Hard, GSM8K, MATH, HumanEval, SGP-Bench, ScreenSpot, OSWorld-G, Windows Agent Arena, VSI-Bench, ERQA, SpaCE-10, OmniSpatial
- Feb 2, 2026
- SWE-bench Verified 70.6
- SWE-bench Multilingual 62.8
- SWE-Bench Pro 44.3
- Terminal-Bench 2.0 36.2
- Aider Polyglot 66.2
- C-Eval 88.2
Dropped since the last post: FLEURS, Common Voice, LibriSpeech
Its technical report adds 23 evaluations 19 new 36 dropped 28 results in all
Qwen3-Coder-Next Technical ReportCapabilities
- Template Following generalization (Qwen) 92.7
- EvalPlus 86.6
- MultiPL-E 88.2
- CRUXEval 95.9
- LiveCodeBench v6 58.9
- OJBench 23
- Codeforces rating 2100
- FullStackBench 60.6
- Spider 83.7
- Bird-SQL 63.6
- MMLU 87.7
- MMLU-Redux 91.2
- MMLU-Pro 80.5
- GPQA Diamond 74.5
- SuperGPQA 57.5
- HMMT 2025 70.2
- HMMT Nov 2025 75.6
- AIME 2024 89
- AIME 2025 83.1
Cyber risk
- AthenaBench
- PrimeVul 48.3
- SecCodeBench
- CWEval 56.3
Dropped since the lab’s previous technical report: ZebraLogic, IFEval, Creative Writing v3, WritingBench, Berkeley Function Calling Leaderboard (BFCL), Multi-IF, PolyMath, LiveBench, Arena-Hard v2.0, WenetSpeech, LibriSpeech, Common Voice, FLEURS, VoiceBench, MMAU, MMSU, MuChoMusic, MMStar, HallusionBench, MM-MT-Bench, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), AI2D, ChartQA, CountBench, Video-MME, LVBench, MLVU, WorldSense (omnimodal), Daily-Omni, Video-Holmes, Seed-TTS-eval, MiniMax TTS Multilingual Test Set, CV3-Eval
- Feb 2, 2026
- tau2-bench 88.2
- BrowseComp 69
- BrowseComp-ZH 73.7
- GAIA 84.5
- xbench-DeepSearch 83.7
- xbench-DeepSearch-2510 56.3
- ResearchRubrics 65.3
- AIME 2025 97.3
- HMMT 2025 98.4
- HMMT Nov 2025 94
- IMO-AnswerBench 85.4
- LiveCodeBench v6 86.4
- SWE-bench Verified 74.4
- Terminal-Bench 2.0 51
Dropped since the last post: MMMU, MATH-Vision (MathVision), SimpleVQA, HallusionBench, ZeroBench, DynaMath, GPQA Diamond, CNMO 2024
Its technical report adds 16 evaluations 30 results in all
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active ParametersCapabilities
- CF-Div2-Stepfun 86.1
- MMLU-Pro 84.4
- GPQA Diamond 83.5
- Humanity's Last Exam 23.1
- ARC-AGI 54.8
- SWE-bench Multilingual 67.4
- Arena-Hard v2.0 74
- MultiChallenge 55.7
- IFBench 67.4
- LongBench v2 57.5
- OpenAI MRCR 28.8
- FRAMES 76.5
- RepoQA 88.5
- Data Analysis Benchmark (StepFun) 39.6
- Consulting and Recommendations Benchmark (StepFun) 70.5
- AndroidDaily 57
- Jan 30, 2026
- OmniDocBench v1.5 94.6
- OCRBench 94
Dropped since the last post: AIME 2025, GPQA Diamond, LiveCodeBench v6, Humanity's Last Exam, SWE-bench Verified, tau2-bench, BrowseComp
Its technical report adds 0 evaluations 39 dropped 2 results in all
GLM-OCR Technical ReportDropped since the lab’s previous technical report: MMBench-EN v1.1, MMBench-CN v1.1, MMStar, BLINK, MuirBench, HallusionBench, ZeroBench, GeoBench, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), MathVerse, DynaMath, LogicVista, AI2D, We-Math, MMLongBench-Doc, ChartQA Pro, ChartMuseum, RefCOCO / RefCOCO+ / RefCOCOg, TreeBench, Ref-L4, OmniSpatial, CVBench, ERQA, All-Angles Bench, OSWorld, AndroidWorld, WebVoyager, WebQuest, Design2Code, Flame-React-Eval, Video-MME, MMVU, Video-MMMU, LVBench, MotionBench, MVBench
- Jan 29, 2026
- FLEURS 4.9
- Common Voice 9.18
- LibriSpeech 1.63
Dropped since the last post: InstructTTSEval
- Jan 28, 2026
- OmniDocBench v1.5 91.1
Dropped since the last post: MMLU-Pro, GPQA Diamond, Humanity's Last Exam, LiveCodeBench, Codeforces rating, AIME 2025, HMMT 2025, HMMT Nov 2025, IMO-AnswerBench, Terminal-Bench 2.0, SWE-bench Verified, SWE-bench Multilingual, BrowseComp, BrowseComp-ZH, tau2-bench, MCP-Universe, MCPMark, Toolathlon
Its technical report adds 0 evaluations 18 dropped 1 result in all
DeepSeek-OCR 2: Visual Causal FlowDropped since the lab’s previous technical report: MMLU-Pro, GPQA Diamond, Humanity's Last Exam, LiveCodeBench v6, Codeforces rating, AIME 2025, HMMT 2025, HMMT Nov 2025, IMO-AnswerBench, Terminal-Bench 2.0, SWE-bench Verified, SWE-bench Multilingual, BrowseComp, BrowseComp-ZH, tau2-bench, MCP-Universe, MCPMark, Toolathlon
- Jan 27, 2026
- MMLU 87.2
- MMLU-Pro 75.2
- GPQA Diamond 63.3
- AIME 2025 24
Its technical report adds 1 evaluation 5 results in all
Arcee Trinity Large Technical ReportHallucination
- SimpleQA 23.9
- Jan 27, 2026
- SWE-bench Verified 54.2
Dropped since the last post: MATH, AIME 2024, AIME 2025, OMEGA, BIG-Bench Hard, ZebraLogic, AGIEval, HumanEval+, MBPP+, LiveCodeBench, IFEval, IFBench, MMLU, PopQA, GPQA Diamond, AlpacaEval 2.0, Tulu 3 Safety Eval
Its technical report adds 0 evaluations 27 dropped 1 result in all
SERA: Soft-Verified Efficient Repository AgentsDropped since the lab’s previous technical report: MATH, AIME 2024, AIME 2025, OMEGA, BIG-Bench Hard, ZebraLogic, AGIEval, HumanEval+, MBPP+, LiveCodeBench, IFEval, IFBench, MMLU, PopQA, GPQA Diamond, AlpacaEval 2.0, Tulu 3 Safety Eval, DoAnythingNow (DAN), HarmBench, TrustLLM JailbreakTrigger, WildJailbreak, WildGuardTest, XSTest, BBQ (Bias Benchmark for QA), StrongREJECT, ToxiGen, WMDP
- Jan 27, 2026
- Humanity's Last Exam 50.2
- AIME 2025 96.1
- HMMT 2025 95.4
- IMO-AnswerBench 81.8
- GPQA Diamond 87.6
- MMLU-Pro 87.1
- MMMU-Pro 78.5
- CharXiv 77.5
- MATH-Vision (MathVision) 84.2
- MathVista 90.1
- ZeroBench 9
- OCRBench 92.3
- OmniDocBench v1.5 88.8
- InfographicVQA 92.6
- SimpleVQA 71.2
- WorldVQA 46.3
- Video-MMMU 86.6
- MMVU 80.4
- MotionBench 70.4
- Video-MME 87.4
- LongVideoBench 79.8
- LVBench 75.9
- SWE-bench Verified 76.8
- SWE-Bench Pro 50.7
- SWE-bench Multilingual 73
- Terminal-Bench 2.0 50.8
- PaperBench 63.5
- CyberGym 41.3
- SciCode 48.7
- OJBench 57.4
- LiveCodeBench v6 85
- LongBench v2 61
- BrowseComp 74.9
- WideSearch 72.7
- DeepSearchQA 77.1
- FinSearchComp 67.8
- Seal-0 57.4
- AA-LCR (Artificial Analysis Long Context Reasoning) 70
Dropped since the last post: MMLU-Redux, Longform Writing, HealthBench, BrowseComp-ZH, FRAMES, Multi-SWE-bench, Terminal-Bench
Its technical report adds 11 evaluations 11 new 30 dropped 48 results in all
Kimi K2.5: Visual Agentic Intelligence (arXiv 2602.02276)Capabilities
- HMMT Nov 2025 91.1
- AdvancedIF 75.6
- GDPval-AA 41
- MMMU 84.3
- BabyVision 36.5
- BLINK 78.9
- MMVP 87
- OSWorld-Verified 63.3
- WebArena 58.9
- Swarm Bench 58.3
Hallucination
- SimpleQA Verified 36.9
Dropped since the lab’s previous technical report: MultiPL-E, Multi-SWE-bench, SWE-Lancer, Terminal-Bench, Aider Polyglot, tau2-bench, AceBench, AIME 2024, MATH-500, CNMO 2024, PolyMath, ZebraLogic, AutoLogi, SuperGPQA, MMLU, MMLU-Redux, IFEval, MultiChallenge, SimpleQA, LiveBench, Arena-Hard v2.0, FACTS Grounding, Vectara Hallucination Leaderboard (HHEM), FaithJudge, FRAMES, OpenAI MRCR, DROP, Kimi Chinese in-house open-ended benchmark, LMArena (Elo), Kimi Promptfoo red-team safety evaluation
- Jan 26, 2026
- AIME 2025 80.3
- MMLU-Pro 77
- IFBench 55.8
- Arena-Hard v2.0 62.5
Dropped since the last post: KMMLU, HAE-RAE Bench, KBL, KBank-MMLU (Upstage), KorMedMCQA, MMLU, IFEval, HumanEval, SWE-bench Verified, MATH-500, AIME 2024, Berkeley Function Calling Leaderboard (BFCL)
- Jan 23, 2026
- MMLU-Pro 85.7
- MMLU-Redux 92.8
- C-Eval 93.7
- GPQA Diamond 87.4
- Humanity's Last Exam 49.8
- LiveCodeBench v6 85.9
- HMMT 2025 98
- HMMT Nov 2025 94.7
- IMO-AnswerBench 83.9
- SWE-bench Verified 75.3
- IFBench 70.9
- MultiChallenge 63.3
- Arena-Hard v2.0 90.2
- tau2-bench 82.1
- BFCL V4 67.7
- VITA-Bench 40.9
- DeepPlanning 28.7
- AA-LCR (Artificial Analysis Long Context Reasoning) 68.7
Dropped since the last post: AIME 2025, SuperGPQA, LMArena (Elo)
- Jan 22, 2026
Dropped since the last post: Seed-TTS-eval
- Jan 22, 2026
- Seed-TTS-eval 0.77
- Jan 20, 2026
- GPQA Diamond 37.9
- MMLU-Pro 49.6
- IFEval 88.4
- IFBench 44.9
- Multi-IF 69.3
- GSM8K 85.6
- MATH-500 88
- AIME 2025 31.7
- Berkeley Function Calling Leaderboard (BFCL) 57
- Jan 19, 2026
- AIME 2025 91.6
- GPQA Diamond 75.2
- LiveCodeBench v6 64
- Humanity's Last Exam 14.4
- SWE-bench Verified 59.2
- tau2-bench 79.5
- BrowseComp 42.8
Dropped since the last post: MMBench-EN v1.1, MMBench-CN v1.1, MMStar, BLINK, MuirBench, MMMU, MMMU-Pro, Video-MMMU, AI2D, MathVista, We-Math, ChartQA Pro, ChartMuseum, OCRBench, MMLongBench-Doc, RefCOCO / RefCOCO+ / RefCOCOg, OSWorld, AndroidWorld, WebVoyager, WebQuest, Design2Code, Flame-React-Eval, Video-MME, MMVU, LVBench, MotionBench
- Jan 15, 2026
- Jan 15, 2026
- Jan 15, 2026
Dropped since the last post: Humanity's Last Exam, GPQA Diamond, AIME 2025, LiveCodeBench v5, SWE-bench Verified, SimpleQA Verified, FACTS Grounding, MMMU, Vibe-Eval, MRCR v2, Global MMLU Lite
Its technical report adds 0 evaluations 53 dropped 2 results in all
TranslateGemma Technical ReportDropped since the lab’s previous technical report: MMLU-Pro, LiveCodeBench, Bird-SQL, GPQA Diamond, SimpleQA, FACTS Grounding, Global MMLU Lite, MATH, HiddenMath, MMMU, MMLU, MBPP, HumanEval, Natural2Code, GSM8K, BIG-Bench Hard, BBEH micro average, IFEval, ECLeKTic, RULER, MRCR (Michelangelo, original), DocVQA, InfographicVQA, TextVQA, AI2D, ChartQA, VQAv2, MathVista, Perception Test MCVQA, ActivityNet-QA, HellaSwag, BoolQ, PIQA, SIQA, TriviaQA, Natural Questions, ARC-Challenge, ARC-Easy, WinoGrande, DROP, AGIEval, COCO Captions, RealWorldQA, ReMI, BLINK, OK-VQA, TallyQA, SpatialSense VQA, CountBench, MGSM, Flores, XQuAD, IndicGenBench
- Jan 14, 2026
- AIME 2025 100
- HMMT 2025 97.5
- IMO-AnswerBench 86.8
- AMO-Bench 66
- BrowseComp 73.1
- BrowseComp-ZH 77.7
- RWSearch 79.5
- tau2-bench 88.2
- tau2-Noise (LongCat) 67.1
- VITA-Bench 29.3
- VitaBench-Noise (LongCat) 20.5
- Random Complex Tasks (LongCat) 35.8
- Humanity's Last Exam 25.2
- GPQA Diamond 85.2
- LiveCodeBench v6 82.8
- OJBench 42.2
- OIBench 47.7
- SWE-bench Verified 70
Dropped since the last post: OmniBench, WorldSense (omnimodal), Daily-Omni, UNO-Bench, MMBench, RealWorldQA, MMStar, MathVista, MMMU, MM-Vet, BLINK, MuirBench, Mantis-Eval, ChartQA, DocVQA, OCRBench, OmniDocBench, RefCOCO / RefCOCO+ / RefCOCOg, CountBench, VisualWebBench, ScreenSpot-v2, AndroidControl, MVBench, NExT-QA, TempCompass, Video-MME, LongVideoBench, MMVU, Video-MMMU, LibriSpeech, AISHELL-1, AISHELL-2, FLEURS, Common Voice, WenetSpeech, CoVoST2, MMAU, Clotho-AQA, OpenAudioBench, VoiceBench, MMLU, MMLU-Pro, C-Eval, CMMLU, IFEval, COLLIE, Meeseeks, MATH-500, AIME 2024, Beyond AIME, DROP, ZebraLogic, GraphWalks, LiveCodeBench, HumanEval+, MBPP+
Its technical report adds 0 evaluations 56 dropped 18 results in all
LongCat-Flash-Thinking-2601 Technical ReportDropped since the lab’s previous technical report: MMBench, RealWorldQA, MMStar, MathVista, MMMU, MM-Vet, BLINK, MuirBench, Mantis-Eval, ChartQA, DocVQA, OCRBench, OmniDocBench, RefCOCO / RefCOCO+ / RefCOCOg, CountBench, VisualWebBench, ScreenSpot-v2, AndroidControl, MVBench, NExT-QA, TempCompass, Video-MME, LongVideoBench, MMVU, Video-MMMU, LibriSpeech, AISHELL-1, AISHELL-2, FLEURS, Common Voice, WenetSpeech, CoVoST2, MMAU, Clotho-AQA, OpenAudioBench, VoiceBench, MMLU, MMLU-Pro, C-Eval, CMMLU, IFEval, COLLIE, Meeseeks, MATH-500, AIME 2024, Beyond AIME, DROP, ZebraLogic, GraphWalks, HumanEval+, MBPP+, OmniBench, WorldSense (omnimodal), Daily-Omni, UNO-Bench, Real-time Audio-Visual Interaction Evaluation
- Jan 14, 2026
- CVTG-2K 0.912
- LongText-Bench 0.966
- OneIG-Bench 0.528
- TIIF-Bench 81
- DPG-Bench 84.8
Dropped since the last post: MMLU-Pro, GPQA Diamond, Humanity's Last Exam, AIME 2025, HMMT 2025, HMMT Nov 2025, IMO-AnswerBench, LiveCodeBench v6, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench Hard, Terminal-Bench 2.0, BrowseComp, BrowseComp-ZH, tau2-bench
Its model card adds 0 evaluations 5 results in all
- Jan 13, 2026
- MMMU 78.1
- MATH-Vision (MathVision) 70.8
- MathVista 84
- PhyX 59.5
- MMBench 92
- MMStar 77.5
- ReMI 67.3
- OCRBench 86.8
- AI2D 89.3
- ScreenSpot-v2 92.6
- ScreenSpot-Pro 51.5
- OSWorld-G 59
- BLINK 66.8
- All-Angles Bench 57.2
- HumanEval-V 66
- AIME 2025 87.7
- HMMT 2025 78.2
- LiveCodeBench 75.8
Its technical report adds 52 evaluations 69 results in all
STEP3-VL-10B Technical ReportCapabilities
- MMMU-Pro 64.1
- LogicVista 66.9
- DynaMath 56.4
- ZeroBench 1
- MathVerse 75.7
- We-Math 73
- VisuLogic 29.7
- Humanity's Last Exam 10.7
- SimpleVQA 53.1
- MMVP 68.2
- M3GIA 78.4
- Do You See Me 67.5
- CountBench 88.8
- CountQA 33.7
- PixMo-Count 70.8
- MM-MT-Bench 8.14
- MIA-Bench 92
- MM-IFEval 61.9
- Design2Code 79.5
- OmniAI OCR Benchmark 77
- CC-OCR
- CVBench 83.5
- MMSI-Bench 32.2
- ERQA 48.9
- OmniSpatial 51.6
- MindCube 62.8
- RealWorldQA 74.4
- SpatialViz-Bench 45.5
- STARE 61.8
- CoreCognition 66.7
- V* 82.8
- ViewSpatial-Bench 46.1
- CharXiv 59.5
- CSVQA 63.3
- OmniDocBench 21.5
- MMBench-GUI 81.5
- MMLU-Pro 76
- GPQA Diamond 70.8
- SuperGPQA 50.4
- LiveBench 69.7
- AIME 2024 90.9
- CNMO 2024 78.2
- Beyond AIME 63.2
- IMO-AnswerBench 62.1
- LiveCodeBench v6 75.8
- IFEval 82.2
- IFBench 43.3
- MultiChallenge 62.6
- Arena-Hard v2.0 58.6
- WildBench 86
Health
- HealthBench 44.7
Hallucination
- HallusionBench 64.9
- Jan 13, 2026
- HealthBench 65.1
- HealthBench Hard 44.4
- HealthBench-Hallu 3.5
- ScanBench 73.8
Its technical report adds 0 evaluations 4 results in all
- Jan 8, 2026
- Jan 8, 2026
Dropped since the last post: MMLU-Redux, GPQA Diamond, AIME 2025, ZebraLogic, MultiPL-E, IFEval, Creative Writing v3, WritingBench, Berkeley Function Calling Leaderboard (BFCL), Multi-IF, PolyMath, LiveBench, Arena-Hard v2.0, LibriSpeech, Common Voice, FLEURS, VoiceBench, MMAU, MuChoMusic, MMStar, HallusionBench, MM-MT-Bench, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), AI2D, ChartQA, CountBench, Video-MME, LVBench, MLVU, WorldSense (omnimodal), Daily-Omni, Video-Holmes, Seed-TTS-eval, WenetSpeech
- Jan 8, 2026
- VSI-Bench 68.6
- MMSI-Bench 42.5
- MindCube 89.9
- ViewSpatial-Bench 61.3
- SITE 47.5
- BLINK 68
- 3DSRBench 62.4
- EmbSpatialBench 81
- Jan 6, 2026
- LiveCodeBench v6 67.9
Dropped since the last post: Putnam 2025
- Jan 5, 2026 Falcon H1R 7B Technology Innovation Institute Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling (Falcon blog)
- AIME 2024 88.1
- AIME 2025 83.1
- HMMT 2025 64.9
- AMO-Bench 36.3
- LiveCodeBench v6 68.6
- SciCode 28.3
- Terminal-Bench Hard 4.96
- GPQA Diamond 61.3
- MMLU-Pro 72.1
- Humanity's Last Exam 11.1
- IFBench 53.4
Dropped since the last post: Open Arabic LLM Leaderboard v2, ArabicMMLU, AraTrust
Its technical report adds 7 evaluations 7 new 15 dropped 18 results in all
Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time ScalingCapabilities
- MATH-500 97.4
- tau2-bench
Safety
- HarmBench
- XSTest 96.4
- SALAD-Bench
Robustness
Dropped since the lab’s previous technical report: Open Arabic LLM Leaderboard v2, AlGhafa, ArabicMMLU, MadinahQA, AraTrust, ALRAGE, ArabCulture, AraDiCE, IFEval, GSM8K, HellaSwag, ARC-Challenge, WinoGrande, TruthfulQA, MMLU
- Jan 5, 2026LFM2.5-1.2B-Instruct Liquid AI Introducing LFM2.5: The Next Generation of On-Device AI (Liquid AI blog)
- GPQA Diamond 38.9
- MMLU-Pro 44.4
- IFEval 86.2
- IFBench 47.3
- Multi-IF 61
- AIME 2025 14
- Berkeley Function Calling Leaderboard (BFCL) 49.1
Dropped since the last post: RealWorldQA, MM-IFEval, InfographicVQA, OCRBench, BLINK, MMStar, MMMU, MathVista, SEED-Bench, MM-Vet, MME, MMLU
2025
- Dec 31, 2025
- KMMLU 73
- KMMLU-Pro 64
- CLIcK 78.9
- HAE-RAE Bench 73.3
- KoBALT 44.3
- KBank-MMLU (Upstage) 65.5
- KBL 65.5
- KorMedMCQA 84.4
- Ko-AIME 2025 (Upstage) 80
- HRM8K 87.6
- MMLU 88.2
- MMLU-Pro 80.4
- GPQA Diamond 68.1
- Humanity's Last Exam 10.5
- AIME 2024 91.7
- AIME 2025 84.3
- HMMT 2025 73.3
- HMMT Nov 2025 80
- LiveCodeBench v6 74.2
- IFBench 53.7
- IFEval 88
- Arena-Hard v2.0 74.8
- WritingBench 75.1
- tau2-bench 55.8
- AA-LCR (Artificial Analysis Long Context Reasoning) 35
Its technical report adds 3 evaluations 28 results in all
Solar Open Technical ReportCapabilities
- Ko-AIME 2024 (Upstage) 80.3
- Ko-IFEval (Upstage) 87.5
- Ko-Arena-Hard v2 (Upstage) 79.9
- Dec 31, 2025
- MMLU-Pro 83.8
- GPQA Diamond 79.1
- Humanity's Last Exam 13.6
- IMO-AnswerBench 76.3
- AIME 2025 92.8
- HMMT Nov 2025 86.8
- LiveCodeBench Pro 25.9
- LiveCodeBench v6 80.7
- Terminal-Bench 2.0 29
- SWE-bench Verified 49.4
- tau2-bench 70.8
- BrowseComp 31.4
- IFBench 67.3
- IFEval 89.7
- AA-LCR (Artificial Analysis Long Context Reasoning) 53.5
- OpenAI MRCR v2 52.3
- KMMLU-Pro 67.3
- KoBALT 61.8
- CLIcK 83.9
- HRM8K 90.9
- Ko-LongBench (LG) 86.8
- MMMLU 85.7
- WMT24++ 90.5
- WildJailbreak 89.9
- KGC-Safety (LG) 96.1
Dropped since the last post: MMLU-Redux, HMMT 2025, LiveCodeBench v5, Multi-IF, Berkeley Function Calling Leaderboard (BFCL), tau-bench, KMMLU-Redux, HRM8K-KSM, HELMET, RULER, LongBench
Its technical report adds 2 evaluations 2 new 11 dropped 26 results in all
K-EXAONE Technical ReportCapabilities
- CodeUtilityBench (LG) 71.9
- OpenAI MRCR 52.3
Dropped since the lab’s previous technical report: MMLU-Redux, HMMT 2025, LiveCodeBench v5, Multi-IF, Berkeley Function Calling Leaderboard (BFCL), tau-bench, KMMLU-Redux, HRM8K-KSM, HELMET, RULER, LongBench English
- Dec 23, 2025
- SWE-bench Verified 74
- Multi-SWE-bench 49.4
- SWE-bench Multilingual 72.5
- Terminal-Bench 2.0 47.9
- SWT-bench 69.3
- SWE-Perf 3.1
- SWE-Review 8.9
- OctoCodingbench 26.1
- VIBE (MiniMax) 88.6
- Toolathlon 43.5
- BrowseComp 62
- AIME 2025 83
- MMLU-Pro 88
- GPQA Diamond 83
- Humanity's Last Exam 22.2
- LiveCodeBench 81
- SciCode 41
- IFBench 70
- AA-LCR (Artificial Analysis Long Context Reasoning) 62
- tau2-bench 87
Dropped since the last post: Terminal-Bench, ArtifactsBench, BrowseComp-ZH, GAIA, xbench-DeepSearch, FinSearchComp Global, TheAgentCompany, Terminal-Bench Hard, Artificial Analysis Intelligence Index v3
- Dec 22, 2025
- MMLU-Pro 84.3
- GPQA Diamond 85.7
- Humanity's Last Exam 42.8
- AIME 2025 95.7
- HMMT 2025 97.1
- HMMT Nov 2025 93.5
- IMO-AnswerBench 82
- LiveCodeBench v6 84.9
- SWE-bench Verified 73.8
- SWE-bench Multilingual 66.7
- Terminal-Bench Hard 33.3
- Terminal-Bench 2.0 41
- BrowseComp 67.5
- BrowseComp-ZH 66.6
- tau2-bench 87.4
Dropped since the last post: MMBench-EN v1.1, MMBench-CN v1.1, MMStar, BLINK, MuirBench, MMMU, MMMU-Pro, Video-MMMU, MathVista, AI2D, DynaMath, We-Math, ZeroBench, MM-BrowseComp, Design2Code, Flame-React-Eval, OSWorld, AndroidWorld, WebVoyager, WebQuest, MMLongBench-Doc, MMLongBench, LVBench, OCRBench, OCRBench v2, ChartQA Pro, ChartMuseum, CharXiv, OmniSpatial, RefCOCO / RefCOCO+ / RefCOCOg, TreeBench, Ref-L4
- Dec 19, 2025
- PutnamBench 87.9
- FATE-H 80
- FATE-X 33
- IMO 2025 35
- Putnam 2025 11
- CombiBench 48
Dropped since the last post: AIME 2025, HMMT 2025, Beyond AIME, AMO-Bench, IMO-AnswerBench, AetherCode, LiveCodeBench v6, LiveCodeBench Pro, GPQA Diamond, PHYBench, BIOBench (ByteDance Seed), KOR-Bench, ARC-AGI, Inverse IFEval, MARS-Bench, MultiChallenge, COLLIE, EIFBench, MMLU, MMLU-Pro, SuperGPQA, LPFQA, BrowseComp, BrowseComp-ZH, GAIA, WideSearch, Humanity's Last Exam, MM-BrowseComp, HLE-VL, SWE-bench Verified, Multi-SWE-bench, AInstein-SWE-Bench, Terminal-Bench 2.0, U-Artifacts, BFCL V4, tau2-bench, DeepConsult, DeepResearch Bench, DiscoX, FinSearchComp, XpertBench, WorldTravel, OSWorld, Online-Mind2Web, AndroidWorld, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), DynaMath, LogicVista, ZeroBench, VPCT, VLMsAreBiased, VLMs Are Blind, SimpleVQA, HallusionBench, MMStar, MMBench-EN v1.1, MuirBench, MMVP, ScreenSpot-Pro, CountBench, BLINK, RefSpatial-Bench, ERQA, CVBench, AI2D, CharXiv, DUDE, MMLongBench, Video-MMMU, MMVU, Video-Holmes, MINERVA, Video SimpleQA, TVBench, TempCompass, TOMATO, EgoTempo, MotionBench, Video-MME, LongVideoBench, LVBench, OVBench, OVO-Bench, StreamingBench, OmniDocBench v1.5, Scientists' First Exam (SFE), EMMA, MME-CC, Point-Bench, MMSI-Bench, VCR-Bench, VideoReasonBench, CG-Bench, LiveSports-3K, ViSpeak-Bench, OmniMMI, Education (ByteDance Seed in-house), Customer Support Q&A (ByteDance Seed in-house), Information Processing (ByteDance Seed in-house), Intention Recognition (ByteDance Seed in-house), Information Extraction (ByteDance Seed in-house), Complex Workflow (ByteDance Seed in-house)
- Dec 18, 2025
Dropped since the last post: GDPval, Investment Banking Modeling Tasks (OpenAI), BrowseComp, GPQA Diamond, Humanity's Last Exam, HMMT 2025, AIME 2025, ARC-AGI, ARC-AGI-2
Its system card addendum adds 16 evaluations 2 new 17 results in all
Addendum to GPT-5.2 System Card: GPT-5.2-CodexSafety
Robustness
- StrongREJECT 93.3
Alignment
Bio and chem risk
Cyber risk
- CVE-Bench 87
- Cyber Range 72.7
- Atomic Challenges
- CyScenarioBench 0
AI R&D risk
- OpenAI PRs 55
- MLE-bench 30 10
- PaperBench 43
- OpenAI-Proof Q&A 8
- Dec 18, 2025
Seed1.8 ByteDance Seed Official Release of Seed1.8: A Generalized Agentic Model (blog and model card) - AIME 2025 94.3
- HMMT 2025 89.7
- Beyond AIME 77
- AMO-Bench 60
- IMO-AnswerBench 76.3
- AetherCode 38.2
- LiveCodeBench v6 79.5
- LiveCodeBench Pro 1930
- GPQA Diamond 83.8
- PHYBench 41
- BIOBench (ByteDance Seed) 42.3
- KOR-Bench 76.2
- ARC-AGI 67.9
- Inverse IFEval 80.3
- MARS-Bench 70.1
- MultiChallenge 66.7
- COLLIE 72.6
- EIFBench 48.6
- MMLU 92.3
- MMLU-Pro 84.9
- SuperGPQA 64.8
- LPFQA 49.1
- BrowseComp 67.6
- BrowseComp-ZH 81.3
- GAIA 87.4
- WideSearch 63.8
- Humanity's Last Exam 40.9
- MM-BrowseComp 46.3
- HLE-VL 31.5
- SWE-bench Verified 72.9
- Multi-SWE-bench 42
- AInstein-SWE-Bench 36.7
- Terminal-Bench 2.0 45.2
- U-Artifacts 49.2
- BFCL V4 57.2
- tau2-bench 72
- DeepConsult 41
- DeepResearch Bench 43.5
- DiscoX 74.6
- FinSearchComp 62.8
- XpertBench 51.3
- WorldTravel 47.2
- OSWorld 61.9
- Online-Mind2Web 85.9
- AndroidWorld 70.7
- MMMU 83.4
- MMMU-Pro 73.2
- MathVista 87.7
- MATH-Vision (MathVision) 81.3
- DynaMath 61.5
- LogicVista 78.3
- ZeroBench 11
- VPCT 61
- VLMsAreBiased 62
- VLMs Are Blind 93
- SimpleVQA 65.4
- HallusionBench 63.9
- MMStar 79.9
- MMBench-EN v1.1 91.6
- MuirBench 78.7
- MMVP 86
- ScreenSpot-Pro 73.1
- CountBench 96.3
- BLINK 74.3
- RefSpatial-Bench 56.3
- ERQA 58.8
- CVBench 88
- AI2D 89.1
- CharXiv 71.4
- DUDE 69.4
- MMLongBench 77.7
- Video-MMMU 82.7
- MMVU 73.1
- Video-Holmes 65.5
- MINERVA 62.4
- Video SimpleQA 67.8
- TVBench 71.5
- TempCompass 86.9
- TOMATO 60.8
- EgoTempo 67
- MotionBench 70.6
- Video-MME 87.8
- LongVideoBench 77.4
- LVBench 73
- OVBench 65.1
- OVO-Bench 72.6
- StreamingBench 84.4
- OmniDocBench v1.5
- Scientists' First Exam (SFE) 51.2
- EMMA 60.9
- MME-CC 43.4
- Point-Bench 76.5
- MMSI-Bench 25.8
- VCR-Bench 59.8
- VideoReasonBench 52.8
- CG-Bench 62.4
- LiveSports-3K 77.5
- ViSpeak-Bench 79
- OmniMMI 53
- Education (ByteDance Seed in-house) 60.8
- Customer Support Q&A (ByteDance Seed in-house) 69
- Information Processing (ByteDance Seed in-house) 53.9
- Intention Recognition (ByteDance Seed in-house) 63.2
- Information Extraction (ByteDance Seed in-house) 43.7
- Complex Workflow (ByteDance Seed in-house) 54.6
Dropped since the last post: Windows Agent Arena, Terminal-Bench, UI-TARS 15 Games Collection
Its model card adds 3 evaluations 2 new 74 dropped 108 results in all
Seed1.8 Model Card: Towards Generalized Real-World AgencyCapabilities
- ZeroVideo 6.9
Safety
Dropped since the lab’s previous model card: IMO 2025, CMO 2025, SimpleQA Verified, HealthBench, HealthBench Hard, Encyclo-K, AIME 2026, HMMT Nov 2025, MathArena (Apex, IMO, USAMO), Codeforces rating, SUPERChem, BABE, FrontierScience-Research, FrontierScience-Olympiad, ARC-AGI-2, ProcBench, MRCR v2, GraphWalks, LongBench v2, FRAMES, DeR2 Bench, CL-bench, Global PIQA, MMMLU, LongFact-Objects, LongFact-Concepts, FActScore precision, PutnamBench, Chinese Complex Instruction Following (ByteDance Seed in-house), Math Kangaroo 2025 (MathArena), MathCanvas-Bench, HiPhO, XLRS-Bench, PhyX, VisuLogic, VisFactor, RealWorldQA, BabyVision, MTVQA-Test, WorldVQA, Vibe-Eval, ViVerBench, TreeBench, All-Angles Bench, ChartQA Pro, OCRBench v2, CharXiv (DQ), LongDocURL, MMLongBench-Doc, MORSE-500, ContPhy, VideoEval-Pro, CrossVid, ODV-Bench, SWE-Lancer, SWE-Bench Pro, SWE-bench Multilingual, SciCode, SWE-EVO, Aider Polyglot, ArtifactsBench, CodeSimpleQA, SpreadsheetBench Verified, HLE-Verified, DeepSearchQA, Seal-0, MCPMark, VITA-Bench, ResearchRubrics, MineDojo, NL2Repo-Bench, GDPVal-Diamond, Compositional Tasks (ByteDance Seed in-house), Text Classification (ByteDance Seed in-house)
- Dec 17, 2025
- Humanity's Last Exam 43.5
- ARC-AGI-2 33.6
- GPQA Diamond 90.4
- AIME 2025 99.7
- MMMU-Pro 81.2
- ScreenSpot-Pro 69.1
- CharXiv 80.3
- OmniDocBench v1.5
- Video-MMMU 86.9
- LiveCodeBench Pro 2316
- Terminal-Bench 2.0 47.6
- SWE-bench Verified 78
- tau2-bench 90.2
- Toolathlon 49.4
- MCP Atlas 57.4
- Vending-Bench 2 3635
- FACTS Benchmark Suite 61.9
- SimpleQA Verified 68.7
- MMMLU 91.8
- Global PIQA 92.8
- MRCR v2 67.2
Dropped since the last post: MathArena (Apex, IMO, USAMO)
- Dec 16, 2025
- RoboCasa 66.2
- RoboCasa GR1 Tabletop Tasks 47.6
- SimplerEnv 62.1
- LIBERO
- BEHAVIOR Challenge 2025
Dropped since the last post: MATH-500, AIME 2024, AIME 2025, GPQA Diamond, LiveCodeBench v5, Berkeley Function Calling Leaderboard (BFCL), IFEval, Arena-Hard-Auto, Humanity's Last Exam, MMLU-Pro
- Dec 16, 2025
- MMLU-Pro 84.9
- GPQA Diamond 83.7
- Humanity's Last Exam 22.1
- AIME 2025 94.1
- HMMT 2025 84.4
- LiveCodeBench v6 80.6
- Arena-Hard v2.0 54.1
- LongBench v2 60.6
- MRCR v2 45.7
- SWE-bench Verified 73.4
- SWE-bench Multilingual 71.7
- Terminal-Bench Hard 30.5
- Terminal-Bench 2.0 38.5
- BrowseComp 58.3
- tau2-bench 80.3
Its technical report adds 0 evaluations 15 results in all
- Dec 15, 2025
- MMLU-Pro 78.3
- AIME 2025 89.1
- GPQA Diamond 73
- LiveCodeBench v6 68.3
- SciCode 33.3
- Humanity's Last Exam 10.6
- miniF2F 50
- Terminal-Bench Hard 8.5
- SWE-bench Verified 38.8
- tau2-bench 49
- BFCL V4 53.8
- IFBench 71.5
- MultiChallenge 38.5
- Arena-Hard v2.0 67.7
- AA-LCR (Artificial Analysis Long Context Reasoning) 35.9
- RULER 86.3
- MMLU-ProX 59.5
- WMT24++ 86.2
Dropped since the last post: MMMU, MathVista, AI2D, OCRBench v2, OCRBench, OCR-Reasoning, ChartQA, DocVQA, Video-MME
Nemotron 3 Nano 30B-A3B Base: Its technical report adds 15 evaluations 14 new 51 dropped 17 results in all
Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningCapabilities
- MMLU 78.6
- AGIEval 68.3
- HumanEval 78
- MBPP 75.5
- GSM8K 92.3
- MATH 82.9
- MATH-500 78.6
- ARC-Challenge 91.9
- HellaSwag 85.6
- OpenBookQA 46.2
- PIQA 84.3
- WinoGrande 79.6
- RACE 88
- Global MMLU Lite 2.0 74.5
- MGSM 83
Dropped since the lab’s previous technical report: MMBench-EN v1.1, MMBench-CN v1.1, MMStar, BLINK, MuirBench, HallusionBench, ZeroBench, CRPE, POPE, MME-RealWorld, RealWorldQA, MMT-Bench, R-Bench (real-world robustness), WildVision, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), MathVerse, DynaMath, LogicVista, We-Math, MMLongBench-Doc, OCRBench, OCRBench v2, ChartQA, RD-TableBench, AI2D, TextVQA, DocVQA, InfographicVQA, OCR-Reasoning, VCR (Visual Caption Restoration), SEED-Bench-2-Plus, CharXiv, TreeBench, CVBench, ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, LongVideoBench, MLVU, Video-MME, MTVQA-Test, MMMB, Multilingual MMBench, AIME 2025, GPQA Diamond, LiveCodeBench, IFEval, SciCode
Its technical report adds 0 evaluations 15 dropped 18 results in all
Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic ReasoningDropped since the lab’s previous technical report: MMLU, AGIEval, HumanEval, MBPP, GSM8K, MATH, MATH-500, ARC-Challenge, HellaSwag, OpenBookQA, PIQA, WinoGrande, RACE, Global MMLU Lite 2.0, MGSM
- Dec 12, 2025
- MATH 96.2
- AIME 2024 80.6
- AIME 2025 78.1
- OMEGA 53.4
- BIG-Bench Hard 88.6
- ZebraLogic 80.1
- AGIEval 89.2
- HumanEval+ 91.5
- MBPP+ 68.3
- LiveCodeBench 83.3
- IFEval 93.8
- IFBench 68.1
- MMLU 86.4
- PopQA 30.9
- GPQA Diamond 57.5
- AlpacaEval 2.0 69.1
- Tulu 3 Safety Eval 83.6
Its technical report adds 10 evaluations 10 new 6 dropped 27 results in all
Olmo 3Safety
- DoAnythingNow (DAN) 54.7
- HarmBench 89.7
- TrustLLM JailbreakTrigger 86.4
- WildJailbreak
- WildGuardTest 96.9
- XSTest 91.8
- StrongREJECT 90.5
- ToxiGen 100
Bias
Bio and chem risk
- WMDP 32.7
Dropped since the lab’s previous technical report: DROP, GSM8K, TruthfulQA, DeepMind Mathematics, IFEval-OOD, MMLU-Pro
- Dec 11, 2025
- GDPval 74.1
- Investment Banking Modeling Tasks (OpenAI) 71.7
- BrowseComp 77.9
- GPQA Diamond 93.2
- Humanity's Last Exam 50
- HMMT 2025 100
- AIME 2025 100
- ARC-AGI 90.5
- ARC-AGI-2 54.2
Dropped since the last post: SWE-Bench Pro, SWE-bench Verified, CharXiv, FrontierMath, FrontierMath Tier 4, OpenAI MRCR v2, ScreenSpot-Pro, tau2-bench
- Dec 11, 2025
- GDPval 70.9
- SWE-Bench Pro 55.6
- SWE-bench Verified 80
- GPQA Diamond 92.4
- CharXiv 88.7
- AIME 2025 100
- FrontierMath 40.3
- FrontierMath Tier 4 14.6
- ARC-AGI 86.2
- ARC-AGI-2 52.9
- OpenAI MRCR v2 77
- ScreenSpot-Pro 86.3
- tau2-bench 90.3
Dropped since the last post: SWE-Lancer, Terminal-Bench 2.0
Its system card addendum adds 29 evaluations 14 new 29 results in all
Update to GPT-5 System Card: GPT-5.2Capabilities
Health
- HealthBench 63.3
- HealthBench Hard 42
- HealthBench Consensus 94.5
Hallucination
Safety
Robustness
- StrongREJECT 97.5
- Agent JSK 97.8
- PlugInject 99.6
Bias
Alignment
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 55
- MLE-bench 30 16
- PaperBench 39
- OpenAI-Proof Q&A 3
- Dec 9, 2025
- Putnam 2025 72.5
- Dec 9, 2025
- SWE-bench Verified 68
- SWE-bench Multilingual 55.7
- Terminal-Bench 2.0 22.5
Dropped since the last post: AIME 2025, AIME 2024, GPQA Diamond, LiveCodeBench
- Dec 9, 2025
- SWE-bench Verified 72.2
- SWE-bench Multilingual 61.3
- Terminal-Bench 2.0 32.6
Dropped since the last post: MMMLU, GPQA Diamond, SimpleQA, AMC, LiveCodeBench, LMArena (Elo)
- Dec 8, 2025
- MMBench-EN v1.1 88.8
- MMBench-CN v1.1 88.2
- MMStar 75.9
- BLINK 65.5
- MuirBench 77.1
- MMMU 76
- MMMU-Pro 66
- Video-MMMU 74.7
- MathVista 85.2
- AI2D 88.8
- DynaMath 54.5
- We-Math 69.8
- ZeroBench 25.8
- MM-BrowseComp 7.6
- Design2Code 88.6
- Flame-React-Eval 86.3
- OSWorld 37.2
- AndroidWorld 57
- WebVoyager 81
- WebQuest 69.3
- MMLongBench-Doc 54.9
- MMLongBench 64.1
- LVBench 59.5
- OCRBench 86.5
- OCRBench v2 65.1
- ChartQA Pro 65.5
- ChartMuseum 58.4
- CharXiv 63.2
- OmniSpatial 52
- RefCOCO / RefCOCO+ / RefCOCOg 88.6
- TreeBench 51.4
- Ref-L4 88.9
Dropped since the last post: AIME 2025, GPQA Diamond, LiveCodeBench v6, Humanity's Last Exam, BrowseComp, SWE-bench Verified, Terminal-Bench, tau2-bench
Its technical report adds 11 evaluations 1 dropped 39 results in all
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement LearningCapabilities
- HallusionBench 62.3
- MATH-Vision (MathVision) 63.5
- MathVerse 75.4
- LogicVista 62.1
- CVBench 87.6
- ERQA 47.8
- All-Angles Bench 56.2
- Video-MME 81.8
- MMVU 68.4
- MotionBench 63.6
- MVBench 74.9
Dropped since the lab’s previous technical report: GeoBench
- Dec 5, 2025
- IMO-AnswerBench 73.4
- HMMT 2025 85.8
- Humanity's Last Exam 18.8
- ARC-AGI 52.9
- MultiChallenge 62.6
- Terminal-Bench 34.6
- SWE-bench Verified 53
- tau2-bench 72.4
Dropped since the last post: MMLU-Pro, GPQA Diamond, Chinese SimpleQA, DROP, ZebraLogic, MATH-500, AIME 2024, LiveCodeBench, C-Eval, CMMLU, Arena-Hard-Auto, CFBench, CELLO, T-Eval
- Dec 3, 2025
- AIME 2024 71.9
- AIME 2025 69.3
- BIG-Bench Hard 86.4
- DROP 83.5
- GPQA Diamond 65.5
- IFEval 77.9
- MATH-500 93.8
- MMLU 87.7
- MMLU-Pro 80.7
- MuSR 69.7
- OpenBookQA 96.6
- SimpleQA 6
- RefusalBench (Nous Research) 74.6
Dropped since the last post: LiveCodeBench v6, Arena-Hard-Auto, RewardBench, EQ-Bench 3, Creative Writing v3
- Dec 2, 2025
- AIME 2025 85
- AIME 2024 89.8
- GPQA Diamond 71.2
- LiveCodeBench 64.6
Dropped since the last post: LiveCodeBench v5
Its technical report adds 3 evaluations 3 new 1 dropped 6 results in all
Ministral 3Capabilities
- HMMT 2025 67.5
- PHYBench 26
- LiveCodeBench v6 64.6
Dropped since the lab’s previous technical report: SWE-bench Verified
- Dec 2, 2025
Nova 2 Omni Preview Amazon Amazon Nova 2: Multimodal Reasoning and Generation Models (Technical Report and Model Card) - MMLU-Pro 80.7
- AIME 2025 92.1
- IFBench 68.7
- MultiChallenge 75.5
- tau2-bench 80
- tau2-bench Verified 73.5
- BFCL V4 58.3
- MMMU-Pro 61.4
- OCRBench v2 58.2
- RealKIE-FCC Verified 59.8
- RefCOCO / RefCOCO+ / RefCOCOg 86.3
- Video-MME 77.9
- MAVERIX 66.6
- QVHighlights 76.7
- ScreenSpot 85.4
- Multilingual LibriSpeech 4.7
- Common Voice 5.7
- CoVoST2 40.7
- MMAU 75.3
Dropped since the last post: GPQA Diamond, LongCodeBench, MCP Atlas, SWE-bench Verified, Terminal-Bench, LiveCodeBench v5
Its technical report adds 4 evaluations 1 new 7 dropped 23 results in all
Amazon Nova 2: Multimodal Reasoning and Generation ModelsCapabilities
- ImgEdit-Bench 4.12
Safety
- Aegis 2.0 80.7
- WildGuardTest 84.1
- Jigsaw Toxic Comment Classification 57.8
Dropped since the lab’s previous technical report: GPQA Diamond, LongCodeBench, OpenAI MRCR, MCP Atlas, SWE-bench Verified, Terminal-Bench, LiveCodeBench v5
- Dec 2, 2025 Nova 2 Lite Amazon Amazon Nova 2: Multimodal Reasoning and Generation Models (Technical Report and Model Card)
- MMLU-Pro 80.9
- GPQA Diamond 79.6
- AIME 2025 91
- IFBench 70.8
- MultiChallenge 76.6
- LongCodeBench 84
- tau2-bench 76
- tau2-bench Verified 70.7
- BFCL V4 60.3
- MCP Atlas 24.6
- MMMU-Pro 61.8
- OCRBench v2 56.1
- RealKIE-FCC Verified 62.1
- QVHighlights 77.2
- ScreenSpot 83.3
- SWE-bench Verified 64.5
- Terminal-Bench 32.5
- LiveCodeBench v5 71
Dropped since the last post: Common Voice
Its technical report adds 4 evaluations 1 dropped 22 results in all
Amazon Nova 2: Multimodal Reasoning and Generation ModelsCapabilities
Safety
- Aegis 2.0 85.8
- WildGuardTest 84.7
- Jigsaw Toxic Comment Classification 56.5
Dropped since the lab’s previous technical report: Common Voice
- Dec 2, 2025 Nova 2.0 Pro Preview Amazon Amazon Nova 2: Multimodal Reasoning and Generation Models (Technical Report and Model Card)
- MMLU-Pro 81.6
- GPQA Diamond 81.4
- AIME 2025 92.3
- IFBench 80.2
- MultiChallenge 77.7
- LongCodeBench 84
- tau2-bench 92.7
- tau2-bench Verified 71.5
- BFCL V4 61.6
- MMMU-Pro 63.5
- OCRBench v2 64.5
- RealKIE-FCC Verified 67
- QVHighlights 76.7
- ScreenSpot 88.1
- Common Voice 5.5
- SWE-bench Verified 70
- Terminal-Bench 41.3
- LiveCodeBench v5 74.6
Dropped since the last post: MMLU, MATH-500, BigCodeBench-Hard, MBXP, IFEval, MMMU, CharXiv, EgoSchema, TallyQA, SimpleQA, Berkeley Function Calling Leaderboard (BFCL)
Its technical report adds 4 evaluations 4 new 12 dropped 22 results in all
Amazon Nova 2: Multimodal Reasoning and Generation ModelsCapabilities
Safety
- Aegis 2.0 81.4
- WildGuardTest 84.9
- Jigsaw Toxic Comment Classification 54.7
Dropped since the lab’s previous technical report: MMLU, MATH-500, BigCodeBench-Hard, MBXP, IFEval, MMMU, CharXiv, CharXiv (DQ), EgoSchema, TallyQA, SimpleQA, Berkeley Function Calling Leaderboard (BFCL)
- Dec 2, 2025
- MMMLU 85.5
- GPQA Diamond 43.9
- SimpleQA 23.8
- AMC 52
- LiveCodeBench
- LMArena (Elo) 1418
Dropped since the last post: AIME 2024, AIME 2025, LiveCodeBench v5
- Dec 1, 2025
- SimpleQA 8.9
- MuSR 63.5
- MMLU 85
- MATH-500 92.1
- GPQA Diamond 58.5
- Berkeley Function Calling Leaderboard (BFCL) 59.7
Dropped since the last post: IFEval, AGIEval, MixEval, TriviaQA
- Dec 1, 2025
- MMLU-Pro 85
- GPQA Diamond 82.4
- Humanity's Last Exam 40.8
- LiveCodeBench 83.3
- Codeforces rating 2386
- AIME 2025 93.1
- HMMT 2025 92.5
- HMMT Nov 2025 90.2
- IMO-AnswerBench 78.3
- Terminal-Bench 2.0 46.4
- SWE-bench Verified 73.1
- SWE-bench Multilingual 70.2
- BrowseComp 67.6
- BrowseComp-ZH 65
- tau2-bench 80.3
- MCP-Universe 45.9
- MCPMark 38
- Toolathlon 35.2
Dropped since the last post: Aider Polyglot, SimpleQA, Terminal-Bench
DeepSeek-V3.2-Speciale: Its technical report adds 4 evaluations 4 new 9 dropped 12 results in all
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv 2512.02556)Capabilities
- IMO 2025 35
- CMO 2025 102
- IOI 2025 492
- ICPC World Finals 2025 10
Dropped since the lab’s previous technical report: MMLU-Pro, BrowseComp, BrowseComp-ZH, SimpleQA, LiveCodeBench v6, Aider Polyglot, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench
Its technical report adds 1 evaluation 1 new 5 dropped 18 results in all
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (arXiv 2512.02556)Capabilities
- LiveCodeBench v6 83.3
Dropped since the lab’s previous technical report: LiveCodeBench, IMO 2025, CMO 2025, IOI 2025, ICPC World Finals 2025
- Nov 27, 2025
DeepSeekMath-V2 DeepSeek DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (model card) - IMO-ProofBench 61.9
- IMO 2025 83.3
- Putnam 2024 98.3
Dropped since the last post: miniF2F, PutnamBench, ProverBench (DeepSeek)
Its technical report adds 3 evaluations 3 new 6 dropped 6 results in all
DeepSeekMath-V2: Towards Self-Verifiable Mathematical ReasoningCapabilities
Dropped since the lab’s previous technical report: miniF2F, ProofNet, PutnamBench, CombiBench, FormalMATH, ProverBench (DeepSeek)
- Nov 26, 2025
- MATH-500 98.1
- AIME 2024 90.8
- AIME 2025 88
- GPQA Diamond 74.4
- Humanity's Last Exam 14.6
- MMLU-Pro 81.9
Dropped since the last post: LiveCodeBench v5, IFEval
Its technical report adds 1 evaluation 1 new 2 dropped 6 results in all
INTELLECT-3: Technical ReportCapabilities
- LiveCodeBench v6 69.3
Dropped since the lab’s previous technical report: LiveCodeBench v5, IFEval
- Nov 24, 2025
- WebVoyager 73.5
- Online-Mind2Web 34.1
- DeepShop 26.2
- WebTailBench 38.4
Dropped since the last post: AIME 2024, AIME 2025, MATH-500, GPQA Diamond
Its technical report adds 4 evaluations 4 new 4 dropped 8 results in all
Fara-7B: An Efficient Agentic Model for Computer UseCapabilities
- ScreenSpot-v2 89.3
- ScreenSpot 86.7
Safety
Dropped since the lab’s previous technical report: AIME 2024, AIME 2025, MATH-500, GPQA Diamond
- Nov 24, 2025Claude Opus 4.5 Anthropic Introducing Claude Opus 4.5 (launch post); System Card: Claude Opus 4.5, Table 2.3.A
- SWE-bench Verified 80.9
- Terminal-Bench 2.0 59.3
- tau2-bench 93.6
- MCP Atlas 62.3
- OSWorld-Verified 66.3
- ARC-AGI-2 37.6
- GPQA Diamond 87
- MMMU 80.7
- MMMLU 90.8
Dropped since the last post: Terminal-Bench, AIME 2025, Vals Finance Agent
Its system card adds 68 evaluations 23 new 11 dropped 77 results in all
System Card: Claude Opus 4.5Capabilities
- SWE-Bench Pro 51.6
- SWE-bench Multilingual 76.2
- BrowseComp-Plus 67.6
- Multi-agent search (Anthropic internal benchmark) 74.8
- ARC-AGI 80
- Vending-Bench 2 4967.1
- Vals Finance Agent 55.2
- CyberGym 50.6
- SpreadsheetBench 64.3
- Humanity's Last Exam 30.8
- AIME 2025 92.8
- LAB-Bench FigQA 54.9
- WebArena 65.3
Hallucination
- 100Q-Hard (Anthropic) 45.9
- SimpleQA Verified 44.1
- AA-Omniscience 51.9
- False-premise questions (Anthropic) 1.2
Safety
Robustness
Alignment
- Anthropic automated behavioral audit
- Petri
- Sycophancy on user-provided prompts (Anthropic) 10
- Agentic misalignment suite (Anthropic)
- Self-preference bias (Anthropic)
- Internal codebase sabotage propensity (Anthropic) 0
- Reward-hack-prone coding tasks (Anthropic) 0
- Impossible tasks (Anthropic) 55
- Training distribution hack rate (Anthropic) 1
- SHADE-Arena 7.7
- Subversion Strategy Eval
- UK AISI alignment and monitorability evaluations
Bio and chem risk
- Long-form virology tasks
- Virology Capabilities Test (VCT) 47.7
- DNA Synthesis Screening Evasion
- LAB-Bench ProtocolQA 90.7
- LAB-Bench SeqQA 85.2
- LAB-Bench CloningScenarios 75.8
- Creative biology 52.4
- Short-horizon computational biology tasks
- Bioinformatics evaluations (Anthropic) 73.7
- ASL-4 virology uplift trial (Anthropic) 66.8
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 46.7
- Kernels task (Anthropic) 252.4
- Time series forecasting task (Anthropic)
- Text-based reinforcement learning task (Anthropic) 0.975
- LLM training task (Anthropic) 16.5
- Quadruped reinforcement learning task (Anthropic)
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.604
- Internal model use survey (Anthropic) 50
Dropped since the lab’s previous system card: Honesty human feedback win rate (Anthropic), SimpleQA, MCP prompt injection evaluation (Anthropic), Computer use prompt injection evaluation (Anthropic), Tool use prompt injection evaluation (Anthropic), Cyber-harness network challenges (Incalmo cyber ranges), Average in-eval reward hacking rate (Anthropic), Apollo Research strategic deception evaluations, Cooperation with user deception (Anthropic), Sycophancy prompts (Anthropic), Research sandbagging (Anthropic)
- Nov 20, 2025
- MATH 96.1
- AIME 2024 76.8
- AIME 2025 72.5
- OMEGA 50.8
- BIG-Bench Hard 89.8
- ZebraLogic 76
- AGIEval 88.2
- HumanEval+ 91.4
- MBPP+ 68
- LiveCodeBench 83.5
- IFEval 89
- IFBench 47.6
- MMLU 85.4
- PopQA 31.9
- GPQA Diamond 58.1
- AlpacaEval 2.0 74.2
- Tulu 3 Safety Eval 68.8
Dropped since the last post: DROP, GSM8K, TruthfulQA
Its technical report adds 10 evaluations 27 results in all
Olmo 3Safety
- DoAnythingNow (DAN) 20.2
- HarmBench 73.5
- TrustLLM JailbreakTrigger 73.3
- WildJailbreak
- WildGuardTest 89.4
- XSTest 93.9
- StrongREJECT 80.8
- ToxiGen 100
Bias
Bio and chem risk
- WMDP 34.8
- Nov 19, 2025
- SWE-bench Verified 77.9
- SWE-Lancer 79.9
- Terminal-Bench 2.0 58.1
Dropped since the last post: GPQA Diamond, AIME 2025, FrontierMath, MMMU, tau2-bench, BrowseComp Long Context 128k
Its system card adds 20 evaluations 4 new 38 dropped 21 results in all
GPT-5.1-Codex-Max System CardSafety
Robustness
- StrongREJECT 96.7
- Coding prompt injections 100
Alignment
Bio and chem risk
Cyber risk
AI R&D risk
- PaperBench 40
- MLE-bench 30 17
- OpenAI PRs 53
- OpenAI-Proof Q&A 8
Autonomy
Dropped since the lab’s previous system card: Standard Refusal Evaluation, Sycophancy evaluation (offline), Instruction Hierarchy Evaluation, Browsing prompt injections, Tool-calling prompt injections, Gray Swan Agent Red Teaming (ART) benchmark, Hallucinations on production-representative prompts, LongFact-Concepts, LongFact-Objects, FActScore, SimpleQA, SimpleQA Hallucination Rate, Coding Deception, Broken Search Tool, CharXiv Missing Image, AbstentionBench, Deception in production traffic, HealthBench, HealthBench Hard, HealthBench Consensus, HealthBench Hard Hallucinations, HealthBench Consensus Urgent, HealthBench Consensus Global Health, MMMLU, BBQ (Bias Benchmark for QA), Expert Red Teaming for Violent Attack Planning, Virology Capabilities Test (VCT), Virology Capabilities Test (text-only), Human Pathogen Capabilities Test, Molecular Biology Capabilities Test, World-Class Bio, Challenging prompts from red teamers with biosafety-relevant PhDs, Filtered, adversarial sample of production prompts, Pairwise Bioweaponization Campaign, Capture the Flag (Collegiate), Pattern Labs cyberoffensive challenges, SWE-bench Verified, Apollo Research strategic deception evaluations
- Nov 19, 2025
- SWE-bench Verified 42
- GPQA Diamond 77.7
- MMLU-Pro 84.7
- MMMLU 86.2
- Humanity's Last Exam 16.7
- SimpleQA Verified 30.9
- AIME 2025 89.5
- MATH-500 98.6
Dropped since the last post: MMLU, GSM8K, MATH, MGSM, SimpleQA
- Nov 19, 2025
- Research-Eval 63.9
- FRAMES 87.6
- X Browse (xAI) 56.3
- tau2-bench
- BFCL V4
Dropped since the last post: LMArena (Elo), EQ-Bench 3, Creative Writing v3, FActScore
- Nov 18, 2025
- Humanity's Last Exam 45.8
- ARC-AGI-2 31.1
- GPQA Diamond 91.9
- AIME 2025 100
- MathArena (Apex, IMO, USAMO) 23.4
- MMMU-Pro 81
- ScreenSpot-Pro 72.7
- CharXiv 81.4
- OmniDocBench v1.5
- Video-MMMU 87.6
- LiveCodeBench Pro 2439
- Terminal-Bench 2.0 54.2
- SWE-bench Verified 76.2
- tau2-bench 85.4
- Vending-Bench 2 5478.2
- FACTS Benchmark Suite 70.5
- SimpleQA Verified 72.1
- MMMLU 91.8
- Global PIQA 93.4
- MRCR v2 77
Dropped since the last post: Online-Mind2Web, WebVoyager, AndroidWorld
Its system card adds 17 evaluations 17 results in all
Frontier Safety Framework Report - Gemini 3 ProAlignment
- GDM Stealth 25
- GDM Situational Awareness 27.3
Monitorability
Bio and chem risk
Cyber risk
AI R&D risk
- Nov 17, 2025
- LMArena (Elo) 1483
- EQ-Bench 3 1586
- Creative Writing v3 1721.9
- FActScore 2.97
Dropped since the last post: GPQA Diamond, AIME 2025, HMMT 2025, Humanity's Last Exam, LiveCodeBench, BrowseComp, SimpleQA, Research-Eval, BrowseComp-ZH, xbench-DeepSearch, X Browse (xAI), LMArena Search Arena
Its model card adds 15 evaluations 3 new 1 dropped 15 results in all
Grok 4.1 Model CardSafety
Robustness
Alignment
Bio and chem risk
Cyber risk
- Cybench 39
Persuasion
Dropped since the lab’s previous model card: Soft Bias (Internal)
- Nov 13, 2025
- SWE-bench Verified 76.3
- GPQA Diamond 88.1
- AIME 2025 94
- FrontierMath 26.7
- MMMU 85.4
- tau2-bench 80.2
- BrowseComp Long Context 128k 90
Dropped since the last post: Code refactoring tasks (OpenAI)
Its system card addendum adds 3 evaluations 3 results in all
- Nov 11, 2025 ERNIE-4.5-VL-28B-A3B-Thinking Baidu Introducing ERNIE-4.5-VL-28B-A3B-Thinking (Hugging Face model card and ERNIE Blog)
- MathVista 82.5
- MathVerse 81
- MATH-Vision (MathVision) 61.9
- MMMU 72.2
- VLMs Are Blind 77.3
- VisuLogic 31.1
- VisualPuzzles 52.5
- ChartQA 87.1
- AI2D 95.3
- DocVQA 93.6
- OCRBench 85.8
- CharXiv 56.9
- SimpleVQA 53.1
- HallusionBench 60.4
- MMStar 71.1
- BLINK 62.8
- CVBench 83.8
- CountBench 89
- TempCompass 75.1
Dropped since the last post: AIME 2025, Berkeley Function Calling Leaderboard (BFCL), ZebraLogic, MuSR, BIG-Bench Hard, HumanEval+, MBPP, IFEval, Multi-IF, Chinese SimpleQA, WritingBench
- Nov 6, 2025
- Humanity's Last Exam 44.9
- AIME 2025 94.5
- HMMT 2025 89.4
- IMO-AnswerBench 78.6
- GPQA Diamond 84.5
- MMLU-Pro 84.6
- MMLU-Redux 94.4
- Longform Writing 73.8
- HealthBench 58
- BrowseComp 60.2
- BrowseComp-ZH 62.3
- Seal-0 56.3
- FinSearchComp 47.4
- FRAMES 87
- SWE-bench Verified 71.3
- SWE-bench Multilingual 61.1
- Multi-SWE-bench 41.9
- SciCode 44.8
- LiveCodeBench v6 83.1
- OJBench 48.7
- Terminal-Bench 47.1
Dropped since the last post: SWE-Dev
- Oct 31, 2025
- OmniBench 61.4
- WorldSense (omnimodal) 60.9
- Daily-Omni 82.4
- UNO-Bench 49.9
- MMBench 87.5
- RealWorldQA 74.8
- MMStar 70.9
- MathVista 77.9
- MMMU 70.7
- MM-Vet 69
- BLINK 63.1
- MuirBench 77.1
- Mantis-Eval 84.8
- ChartQA 87.6
- DocVQA 91.8
- OCRBench 84.9
- OmniDocBench 22.8
- RefCOCO / RefCOCO+ / RefCOCOg 93.9
- CountBench 92.4
- VisualWebBench 78.7
- ScreenSpot-v2 91.2
- AndroidControl 91.2
- MVBench 75.2
- NExT-QA 86.2
- TempCompass 82.2
- Video-MME 78.2
- LongVideoBench 69.3
- MMVU 67.1
- Video-MMMU 67.5
- LibriSpeech 2.79
- AISHELL-1 0.63
- AISHELL-2 2.78
- FLEURS 3.99
- Common Voice 4.98
- WenetSpeech 6.39
- CoVoST2 37.3
- MMAU 75.9
- Clotho-AQA 72.8
- OpenAudioBench 80.1
- VoiceBench 88.7
- MMLU 90.3
- MMLU-Pro 82.7
- C-Eval 91.7
- CMMLU 89.4
- IFEval 82.4
- COLLIE 45.7
- Meeseeks 39
- MATH-500 97.6
- AIME 2024 72.9
- Beyond AIME 47.4
- GPQA Diamond 74.4
- DROP 83.5
- ZebraLogic 86
- GraphWalks 56
- LiveCodeBench 52.6
- HumanEval+ 90.8
- MBPP+ 80.2
Dropped since the last post: MMLU-Redux, Arena-Hard v2.0, HMMT 2025, AIME 2025, Sudoku-Bench, ARC-AGI, OJBench, SWE-bench Verified, Berkeley Function Calling Leaderboard (BFCL), tau2-bench, VITA-Bench, miniF2F, LongCat safety evaluation (Meituan)
Its technical report adds 2 evaluations 1 new 13 dropped 58 results in all
LongCat-Flash-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: MMLU-Redux, Arena-Hard v2.0, HMMT 2025, AIME 2025, Sudoku-Bench, ARC-AGI, OJBench, SWE-bench Verified, Berkeley Function Calling Leaderboard (BFCL), tau2-bench, VITA-Bench, miniF2F, LongCat safety evaluation (Meituan)
- Oct 29, 2025
- SWE-Bench Pro 40.1
Dropped since the last post: Conversational SWE Task Benchmark (Windsurf), End-To-End SWE Task Benchmark (Windsurf)
- Oct 29, 2025
- Oct 28, 2025
- MMLU 59.4
- MMLU-Pro 34
- BIG-Bench Hard 60.4
- AGIEval 49.2
- GPQA Diamond 29.9
- IFEval 77.4
- GSM8K 76.3
- GSM-Symbolic 72.3
- MATH 45.3
- DeepMind Mathematics 34
- HumanEval 74
- HumanEval+ 69
- MBPP 65
- MBPP+ 57
- CRUXEval 33.1
- BigCodeBench 30.2
- Berkeley Function Calling Leaderboard (BFCL) 54.8
- MultiPL-E 32.2
- MMMLU 45
- INCLUDE 42.1
- MGSM 37.8
- SALAD-Bench 93.4
- AttaQ 85.3
- Oct 28, 2025
- MMMU 68
- MathVista 76.9
- AI2D 87.1
- OCRBench v2 62
- OCRBench 85.6
- OCR-Reasoning 36.4
- ChartQA 89.7
- DocVQA 94.4
- Video-MME 65.9
Dropped since the last post: AIME 2025, MATH-500, GPQA Diamond, LiveCodeBench, Berkeley Function Calling Leaderboard (BFCL), IFEval, Humanity's Last Exam, RULER
Its technical report adds 45 evaluations 41 new 3 dropped 54 results in all
NVIDIA Nemotron Nano V2 VLCapabilities
- MMBench-EN v1.1 82.6
- MMBench-CN v1.1 76.3
- MMStar 71.7
- BLINK 56.7
- MuirBench 44.2
- HallusionBench 73.1
- ZeroBench 18.3
- CRPE 64.3
- POPE 87
- MME-RealWorld 62.2
- RealWorldQA 72.2
- MMT-Bench 63.8
- R-Bench (real-world robustness) 69.7
- WildVision 20.2
- MMMU-Pro 28
- MATH-Vision (MathVision) 53.6
- MathVerse 58.2
- DynaMath 32.3
- LogicVista 58.4
- We-Math 39.3
- MMLongBench-Doc 32.1
- RD-TableBench 57.4
- TextVQA 76.1
- InfographicVQA 80.4
- VCR (Visual Caption Restoration) 71.8
- SEED-Bench-2-Plus 70
- CharXiv 41.3
- TreeBench 42.5
- CVBench 78.3
- ScreenSpot 40.1
- ScreenSpot-v2 42.8
- ScreenSpot-Pro 5.5
- LongVideoBench 57
- MLVU 73.6
- MTVQA-Test 24.3
- MMMB
- Multilingual MMBench
- MATH-500 96.9
- AIME 2025 71.3
- GPQA Diamond 64.1
- LiveCodeBench
- IFEval 78.2
- SciCode 17.6
- MMLU-Pro 77.1
- RULER 72.1
Dropped since the lab’s previous technical report: IFBench, AIME 2024, Berkeley Function Calling Leaderboard (BFCL)
- Oct 27, 2025
- SWE-bench Verified 69.4
- Multi-SWE-bench 36.2
- SWE-bench Multilingual 56.5
- Terminal-Bench 46.3
- ArtifactsBench 66.8
- BrowseComp 44
- BrowseComp-ZH 48.5
- GAIA 75.7
- xbench-DeepSearch 72
- Humanity's Last Exam 31.8
- tau2-bench 87
- FinSearchComp Global 65.5
- TheAgentCompany 36
- AIME 2025 78
- MMLU-Pro 82
- GPQA Diamond 78
- LiveCodeBench 83
- SciCode 36
- IFBench 72
- AA-LCR (Artificial Analysis Long Context Reasoning) 61
- Terminal-Bench Hard 24
- Artificial Analysis Intelligence Index v3 61
Dropped since the last post: AIME 2024, MATH-500, LiveCodeBench v6, FullStackBench, ZebraLogic, OpenAI MRCR, LongBench v2, tau-bench, SimpleQA, MultiChallenge
Its technical report adds 5 evaluations 14 dropped 11 results in all
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World IntelligenceCapabilities
- VIBE-Pro (MiniMax) 42.4
- WideSearch 62.3
- Toolathlon 18.8
- GDPval-AA 34
AI R&D risk
- MLE-bench 40
Dropped since the lab’s previous technical report: SWE-Bench Pro, NL2Repo-Bench, Terminal-Bench 2.0, HyperTask, RISE, MM-ClawBench, MEWC v2, Finance Modeling Pro, AIME 2026, SciCode, IFBench, Humanity's Last Exam, MMLU-Pro, Artificial Analysis Intelligence Index v4.0
- Oct 15, 2025
- SWE-bench Verified 73.3
- Terminal-Bench 41
- tau2-bench 76.6
- OSWorld-Verified 50.7
- AIME 2025 96.3
- GPQA Diamond 73
- MMMLU 83
- MMMU 73.2
Its system card adds 25 evaluations 25 results in all
System Card: Claude Haiku 4.5Safety
Robustness
Alignment
Bio and chem risk
Cyber risk
- Cybench 46.9
AI R&D risk
- Oct 10, 2025
- Oct 10, 2025
- AIME 2025 93.4
- Codeforces rating 2088
- ARC-AGI 55.9
- Arena-Hard v2.0 81.6
- HMMT 2025 86.7
- LiveCodeBench v6 78.3
- HealthBench 57.9
- Creative Writing v3 85.4
- ICPC World Finals 2025 5
Its technical report adds 19 evaluations 27 results in all
Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking ModelCapabilities
- Omni-MATH 82.6
- CNMO 2024 88.5
- FinanceReasoning 87.4
- UGMathBench 76.5
- Aider-Edit 78.6
- BBEH micro average 59.6
- ZebraLogic 95.2
- Humanity's Last Exam 16
- GPQA Diamond 78.6
- MMLU-Pro 80.5
- C-Eval 91.5
- PHYBench 42.6
- AGIEval 88.1
- TriviaQA 78.6
- CMMLU 89.1
- IFEval 85.2
- MultiChallenge 50.9
- Berkeley Function Calling Leaderboard (BFCL) 68.8
- IMO 2025
- Oct 7, 2025
- Online-Mind2Web 69
- WebVoyager 88.9
- AndroidWorld 69.7
Dropped since the last post: SWE-bench Verified, Artificial Analysis Intelligence Index v3, Humanity's Last Exam, GPQA Diamond, AIME 2025, LiveCodeBench v5, SimpleQA, SimpleQA Verified, FACTS Grounding, MMMU, Vibe-Eval, MRCR v2, Global MMLU Lite
Its model card adds 0 evaluations 17 dropped 3 results in all
Gemini 2.5 Computer Use Model CardDropped since the lab’s previous model card: Humanity's Last Exam, GPQA Diamond, AIME 2025, LiveCodeBench v5, SWE-bench Verified, SimpleQA, SimpleQA Verified, FACTS Grounding, MMMU, Vibe-Eval, MRCR v2, Global MMLU Lite, Text to Text Safety, Multilingual Safety, Image to Text Safety, Tone, Instruction Following (safety evaluation)
- Oct 7, 2025
- MMLU 64.8
- MMLU-Pro 37.4
- GPQA Diamond 29.3
- IFEval 77.6
- IFBench 25.9
- Multi-IF 58.2
- GSM8K 84.4
- GSM-Plus 64.8
- MATH-500 74.2
- MATH Level 5 62.4
- MGSM 72.4
- MMMLU 55.3
- LiveCodeBench v6 21
- LiveCodeBench v5 21.4
- HumanEval+ 69.5
- Creative Writing v3 44.2
Its technical report adds 0 evaluations 12 results in all
- Oct 2, 2025
- MMLU 78.4
- MMLU-Pro 55.5
- BIG-Bench Hard 81.6
- AGIEval 70.6
- GPQA Diamond 40.6
- AlpacaEval 2.0 42.5
- IFEval 87.5
- Arena-Hard-Auto 46.5
- GSM8K 87.3
- GSM-Symbolic 87.4
- MATH 74
- DeepMind Mathematics 59.3
- HumanEval 88
- HumanEval+ 83
- MBPP 84
- MBPP+ 71
- CRUXEval 50.3
- BigCodeBench 46.2
- Berkeley Function Calling Leaderboard (BFCL) 64.7
- MultiPL-E 57.4
- MMMLU 69.7
- INCLUDE 64
- MGSM 38.7
- SALAD-Bench 97.3
- AttaQ 86.6
Dropped since the last post: PopQA, TruthfulQA, DROP, AIME 2024, MATH-500
- Sep 30, 2025GLM-4.6 Z.ai GLM-4.6 launch blog (Z.ai) and Hugging Face card, chart 'LLM Performance Evaluation: Agentic, Reasoning and Coding'
- AIME 2025 93.9
- GPQA Diamond 81
- LiveCodeBench v6 82.8
- Humanity's Last Exam 30.4
- BrowseComp 45.1
- SWE-bench Verified 68
- Terminal-Bench 40.5
- tau2-bench 75.9
Dropped since the last post: MMBench-EN v1.1, MMBench-CN v1.1, MMStar, BLINK, MuirBench, HallusionBench, ZeroBench, GeoBench, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), MathVerse, DynaMath, LogicVista, AI2D, We-Math, MMLongBench-Doc, OCRBench, ChartQA Pro, ChartMuseum, RefCOCO / RefCOCO+ / RefCOCOg, TreeBench, Ref-L4, OmniSpatial, CVBench, ERQA, All-Angles Bench, OSWorld, AndroidWorld, WebVoyager, WebQuest, Design2Code, Flame-React-Eval, Video-MME, MMVU, Video-MMMU, LVBench, MotionBench, MVBench
- Sep 29, 2025
- MMLU-Pro 85
- GPQA Diamond 79.9
- Humanity's Last Exam 19.8
- LiveCodeBench 74.1
- AIME 2025 89.3
- HMMT 2025 83.6
- Codeforces rating 2121
- Aider Polyglot 74.5
- BrowseComp 40.1
- BrowseComp-ZH 47.9
- SimpleQA 97.1
- SWE-bench Verified 67.8
- SWE-bench Multilingual 57.9
- Terminal-Bench 37.7
Its technical report adds 1 evaluation 1 new 24 dropped 14 results in all
DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse AttentionCapabilities
- LiveCodeBench v6 74.1
Dropped since the lab’s previous technical report: MMLU, MMLU-Redux, DROP, IFEval, FRAMES, AlpacaEval 2.0, Arena-Hard-Auto, LiveCodeBench v5, AIME 2024, MATH-500, CNMO 2024, CLUEWSC, C-Eval, Chinese SimpleQA, AMC 12 2024, SimpleSafetyTests, BBQ (Bias Benchmark for QA), Anthropic Red Team, XSTest, Do-Not-Answer, HarmBench, DeepSeek in-house safety benchmark, DeepSeek multilingual safety evaluation, DeepSeek jailbreak robustness evaluation
- Sep 29, 2025
- SWE-bench Verified 82
- Terminal-Bench 50
- tau2-bench 84.7
- OSWorld-Verified 61.4
- AIME 2025 100
- GPQA Diamond 83.4
- MMMLU 89.1
- MMMU 77.8
- Vals Finance Agent 55.3
Dropped since the last post: tau-bench
Its system card adds 56 evaluations 22 new 8 dropped 56 results in all
System Card: Claude Sonnet 4.5Hallucination
Safety
Robustness
Alignment
- Average in-eval reward hacking rate (Anthropic) 15.4
- Reward-hack-prone coding tasks (Anthropic) 1
- Impossible tasks (Anthropic) 53
- Training distribution hack rate (Anthropic) 2
- Anthropic automated behavioral audit
- Petri
- UK AISI alignment and monitorability evaluations
- Apollo Research strategic deception evaluations 0
- Cooperation with user deception (Anthropic)
- Agentic misalignment suite (Anthropic) 0
- Sycophancy prompts (Anthropic)
- Self-preference bias (Anthropic)
- Subversion Strategy Eval
- Research sandbagging (Anthropic)
- SHADE-Arena 1.5
Bio and chem risk
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 45.3
- Kernels task (Anthropic) 108.6
- Time series forecasting task (Anthropic) 5.3
- Text-based reinforcement learning task (Anthropic) 0.85
- LLM training task (Anthropic) 5.5
- Quadruped reinforcement learning task (Anthropic) 1.3
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.514
- Internal model use survey (Anthropic) 0
Dropped since the lab’s previous system card: Ambiguous context evaluations (Anthropic), Child safety evaluations (Anthropic), StrongREJECT, Malicious computer use (Anthropic), SycophancyEval (Sharma et al.), Bioweapons acquisition uplift trial (Anthropic), Bioweapons knowledge questions, METR data deduplication
- Sep 25, 2025 Gemini 2.5 Flash-Lite Google DeepMind Gemini 2.5 Flash-Lite Model Card, Preview (09-2025) columns (Google DeepMind)
- Humanity's Last Exam 7.3
- GPQA Diamond 71.7
- AIME 2025 48.2
- LiveCodeBench v5 58.4
- SWE-bench Verified 38.9
- SimpleQA Verified 9.6
- FACTS Grounding 87.5
- MMMU 72
- Vibe-Eval 59.8
- MRCR v2 25.6
- Global MMLU Lite 84.9
Dropped since the last post: IFEval, BIG-Bench Hard, HellaSwag, PIQA, ARC-Challenge, WinoGrande
- Sep 25, 2025 Gemini 2.5 Flash Google DeepMind Continuing to bring you our latest models, with an improved Gemini 2.5 Flash and Flash-Lite release (Google Developers Blog)
- SWE-bench Verified 54
- Artificial Analysis Intelligence Index v3
- Humanity's Last Exam 13.2
- GPQA Diamond 80.8
- AIME 2025 75.6
- LiveCodeBench v5 71.7
- SimpleQA 25.3
- SimpleQA Verified 27.8
- FACTS Grounding 87.5
- MMMU 80.3
- Vibe-Eval 62
- MRCR v2 52.4
- Global MMLU Lite 87.9
Dropped since the last post: IMO 2025, LiveCodeBench v6
Its model card adds 5 evaluations 3 dropped 17 results in all
Gemini 2.5 Flash & 2.5 Flash Image Model CardSafety
Dropped since the lab’s previous model card: AIME 2024, Aider Polyglot, MRCR (Michelangelo, original)
- Sep 24, 2025 Code World Model (CWM) Meta Code World Model (CWM) README and model card (GitHub, facebookresearch/cwm)
- LiveCodeBench v5 68.6
- LiveCodeBench v6 63.5
- MATH-500 96.6
- AIME 2024 76
- AIME 2025 68.2
- SWE-bench Verified 65.8
Dropped since the last post: MVPBench, TempCompass
Its technical report adds 12 evaluations 6 new 59 dropped 18 results in all
CWM: An Open-Weights LLM for Research on Code Generation with World ModelsCapabilities
- Aider Polyglot 35.1
- Terminal-Bench 26.3
- CRUXEval 94.3
- HaltEval-prelim (Meta) 94
- BigO(Bench)
- MMLU 77.7
- MMLU-Pro 60.2
- GPQA Diamond 40.6
- GSM8K 93.3
- HumanEval+ 75
- MBPP 73.4
- RULER
Dropped since the lab’s previous technical report: IFEval, HumanEval, MultiPL-E, MATH, ARC-Challenge, Berkeley Function Calling Leaderboard (BFCL), Nexus, API-Bank, Gorilla APIBench, ZeroSCROLLS/QuALITY, Qasper, SQuALITY, InfiniteBench/En.MC, InfiniteBench/En.QA, Needle In A Haystack, MGSM, Multilingual MMLU (Meta), LSAT, SAT Reading, SAT Math, GMAT Quantitative, GMAT Verbal, GRE Physics, AP Art History, AP Biology, AP Calculus, AP Chemistry, AP English Language and Composition, AP English Literature and Composition, AP Environmental Science, AP Macroeconomics, AP Microeconomics, AP Physics, AP Psychology, AP Statistics, AP US Government, AP US History, AP World History, GRE Quantitative, GRE Verbal, SQuAD v2, QuAC, RACE, CommonSenseQA, PIQA, SIQA, OpenBookQA, WinoGrande, DROP, WorldSense, AGIEval, BIG-Bench Hard, QuALITY, Adversarial SQuAD, Dynabench SQuAD, GSM-Plus, PAWS, Llama verbatim memorization probe (Meta), CyberSecEval Prompt Injection
Its technical report adds 14 evaluations 14 new 18 dropped 14 results in all
Code World Model Preparedness ReportAlignment
- MASK 44.8
Bio and chem risk
Cyber risk
Dropped since the lab’s previous technical report: SWE-bench Verified, Aider Polyglot, Terminal-Bench, CRUXEval, HaltEval-prelim (Meta), BigO(Bench), LiveCodeBench v5, LiveCodeBench v6, MATH-500, AIME 2024, AIME 2025, MMLU, MMLU-Pro, GPQA Diamond, GSM8K, HumanEval+, MBPP, RULER
- Sep 24, 2025
- SWE-bench Verified 69.6
- tau2-bench 74.8
- AIME 2025 100
- HMMT 2025 100
- SuperGPQA
- LiveCodeBench v6
- GPQA Diamond
- LMArena (Elo)
Dropped since the last post: MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), MathVerse, ZeroBench, VisuLogic, MMBench-EN v1.1, RealWorldQA, MMStar, SimpleVQA, HallusionBench, MM-MT-Bench, MIA-Bench, MMLongBench-Doc, DocVQA, InfographicVQA, AI2D, OCRBench, OCRBench v2, CC-OCR, CharXiv, RefCOCO / RefCOCO+ / RefCOCOg, CountBench, ODinW-13, ARKitScenes, Hypersim, SUNRGBD, Objectron, BLINK, MuirBench, ERQA, EmbSpatialBench, RefSpatial-Bench, RoboSpatial-Home, Video-MME, MLVU, LVBench, Charades-STA, Video-MMMU, ScreenSpot, ScreenSpot-Pro, OSWorld-G, OSWorld, Design2Code, MMLU, MMLU-Pro, MMLU-Redux, SimpleQA, ZebraLogic, LiveBench, Humanity's Last Exam, SIFo, IFEval, Creative Writing v3, WritingBench, Berkeley Function Calling Leaderboard (BFCL), Multi-IF, MMLU-ProX, INCLUDE
- Sep 23, 2025
- MMMU 80.6
- MMMU-Pro 69.3
- MathVista 85.8
- MATH-Vision (MathVision) 74.6
- MathVerse 85
- ZeroBench 27.7
- VisuLogic 34.4
- MMBench-EN v1.1 90.6
- RealWorldQA 81.3
- MMStar 78.7
- SimpleVQA 61.3
- HallusionBench 66.7
- MM-MT-Bench 8.5
- MIA-Bench 92.7
- MMLongBench-Doc 56.2
- DocVQA 96.5
- InfographicVQA 89.5
- AI2D 89.2
- OCRBench 87.5
- OCRBench v2 66.8
- CC-OCR 81.5
- CharXiv 66.1
- RefCOCO / RefCOCO+ / RefCOCOg 92.4
- CountBench 93.7
- ODinW-13 43.2
- ARKitScenes 53.7
- Hypersim 11
- SUNRGBD 34.9
- Objectron 71.2
- BLINK 67.1
- MuirBench 80.1
- ERQA 52.5
- EmbSpatialBench 84.3
- RefSpatial-Bench 69.9
- RoboSpatial-Home 73.9
- Video-MME 79
- MLVU 83.8
- LVBench 63.6
- Charades-STA 63.5
- Video-MMMU 80
- ScreenSpot 95.4
- ScreenSpot-Pro 61.8
- OSWorld-G 68.3
- OSWorld 38.1
- Design2Code 93.4
- MMLU 90.6
- MMLU-Pro 83.8
- MMLU-Redux 93.7
- SuperGPQA 64.3
- SimpleQA 44.4
- AIME 2025 89.7
- HMMT 2025 77.4
- LiveCodeBench v6 70.1
- ZebraLogic 97.3
- LiveBench 79.6
- Humanity's Last Exam 13.6
- SIFo 77.3
- IFEval 88.2
- Creative Writing v3 85.7
- WritingBench 86.7
- Berkeley Function Calling Leaderboard (BFCL) 71.9
- Multi-IF 79.1
- MMLU-ProX 80.6
- INCLUDE 80
Dropped since the last post: GPQA Diamond, Codeforces rating, OJBench, Arena-Hard v2.0, tau-bench, tau2-bench, PolyMath
Its technical report adds 25 evaluations 22 new 12 dropped 82 results in all
Qwen3-VL Technical ReportCapabilities
- MathVision-Wildphoto 63.8
- We-Math 74.8
- DynaMath 82.8
- Math-VR 66.8
- VLMs Are Blind 79.5
- LogicVista 72.2
- VisualPuzzles 57.2
- MMBench-CN v1.1 88.6
- ChartQA 90.3
- OmniDocBench
- CharXiv (DQ) 90.5
- VSI-Bench 60
- MVBench 75.2
- MMVU 71.1
- V* 85.9
- HR-Bench
- ChartMimic 78.4
- AndroidWorld 62
- Windows Agent Arena 32.1
- GPQA Diamond 77.1
- Codeforces rating 1964
- OJBench 27.5
- Arena-Hard v2.0 74.8
- tau2-bench
- PolyMath 57.8
Dropped since the lab’s previous technical report: C-Eval, Arena-Hard-Auto, AlignBench, MATH-500, AIME 2024, ZebraLogic, AutoLogi, LiveCodeBench v5, MMMLU, MT-AIME2024, MLogiQA, RULER
- Sep 22, 2025
- MMLU-Redux 86.6
- GPQA Diamond 69.6
- AIME 2025 65
- ZebraLogic 76
- MultiPL-E 81.4
- IFEval 81
- Creative Writing v3 80.6
- WritingBench 82.6
- Berkeley Function Calling Leaderboard (BFCL) 64.4
- Multi-IF 64
- PolyMath 37.9
- LiveBench 71.8
- Arena-Hard v2.0 55.1
- LibriSpeech 1.85
- Common Voice 5.18
- FLEURS 5.33
- VoiceBench 85.5
- MMAU 77.5
- MuChoMusic 52
- MMStar 68.5
- HallusionBench 59.7
- MM-MT-Bench 7.4
- MMMU 69.1
- MMMU-Pro 57
- MathVista 75.9
- MATH-Vision (MathVision) 56.3
- AI2D 85.2
- ChartQA 86.8
- CountBench 90
- Video-MME 70.5
- LVBench 50.2
- MLVU 75.2
- WorldSense (omnimodal) 54
- Daily-Omni 75.8
- Video-Holmes 57.3
- Seed-TTS-eval 1.39
- WenetSpeech 5.29
Dropped since the last post: MMLU-Pro, SuperGPQA, HMMT 2025, LiveCodeBench v6, Codeforces rating, OJBench, tau-bench, tau2-bench, MMLU-ProX, INCLUDE
Its technical report adds 3 evaluations 3 new 23 dropped 40 results in all
Qwen3-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: MMLU-Pro, MATH, GSM8K, HumanEval, MBPP, LiveCodeBench, CoVoST2, Qwen in-house voice-chat benchmark, MMBench-EN v1.1, MM-Vet, MME, MuirBench, CRPE, RealWorldQA, MME-RealWorld, TextVQA, DocVQA, OCRBench v2, RefCOCO / RefCOCO+ / RefCOCOg, ODinW-13, MVBench, EgoSchema, OmniBench
- Sep 22, 2025
- MMLU-Pro 85
- GPQA Diamond 80.7
- Humanity's Last Exam 21.7
- LiveCodeBench 74.9
- Codeforces rating 2046
- Aider Polyglot 76.1
- BrowseComp 38.5
- BrowseComp-ZH 45
- SimpleQA 96.8
- SWE-bench Verified 68.4
- SWE-bench Multilingual 57.8
- Terminal-Bench 36.7
Dropped since the last post: MMLU-Redux, LiveCodeBench v6, AIME 2024, AIME 2025, HMMT 2025
- Sep 21, 2025
- MMLU-Pro 82.6
- MMLU-Redux 89.3
- IFEval 86.9
- Arena-Hard v2.0 69.9
- MATH-500 99.2
- HMMT 2025 83.7
- AIME 2024 93.3
- AIME 2025 90.6
- Beyond AIME 69.5
- GPQA Diamond 81.5
- ZebraLogic 95.5
- Sudoku-Bench 56
- ARC-AGI 50.3
- LiveCodeBench 79.4
- OJBench 40.7
- SWE-bench Verified 59.4
- Berkeley Function Calling Leaderboard (BFCL) 74.4
- tau2-bench 74
- VITA-Bench 29.5
- miniF2F 81.6
- LongCat safety evaluation (Meituan) 95.7
Dropped since the last post: MMLU, C-Eval, CMMLU, COLLIE, Meeseeks, DROP, GraphWalks, HumanEval+, MBPP+, Terminal-Bench, AceBench
Its technical report adds 1 evaluation 11 dropped 21 results in all
Introducing LongCat-Flash-Thinking: A Technical ReportCapabilities
- LiveCodeBench v6 79.4
Dropped since the lab’s previous technical report: MMLU, C-Eval, CMMLU, COLLIE, Meeseeks, DROP, GraphWalks, HumanEval+, MBPP+, Terminal-Bench, AceBench
- Sep 19, 2025
- GPQA Diamond 85.7
- AIME 2025 92
- HMMT 2025 93.3
- Humanity's Last Exam 20
- LiveCodeBench
- BrowseComp 44.9
- SimpleQA 95
- Research-Eval 66
- BrowseComp-ZH 51.2
- xbench-DeepSearch 74
- X Browse (xAI) 58
- LMArena Search Arena 1163
Dropped since the last post: USAMO 2025, ARC-AGI-2, Vending-Bench
Its model card adds 13 evaluations 13 results in all
Grok 4 Fast Model CardSafety
Robustness
Bias
- Soft Bias (Internal) 0.79
Alignment
Bio and chem risk
Cyber risk
- Cybench 30
Persuasion
- MakeMeSay 12
- Sep 18, 2025
- MMAU-Pro 56.8
- MMAU 74.9
- Big Bench Audio 72.9
Dropped since the last post: GPQA Diamond, SuperGPQA, DROP, MMLU-Pro, IFEval, MATH-500, AIME 2024, AIME 2025, LiveCodeBench v5, LiveCodeBench v6
Its technical report adds 7 evaluations 7 new 11 dropped 10 results in all
MiMo-Audio: Audio Language Models are Few-Shot LearnersCapabilities
- MMAR 63.6
- MMSU 62.9
- MultiChallenge Audio (Xiaomi) 15.2
- Seed-TTS-eval
- InstructTTSEval
- LibriSpeech 3.76
- AISHELL-1 1.78
Dropped since the lab’s previous technical report: GPQA Diamond, SuperGPQA, DROP, MMLU-Pro, IFEval, MATH-500, AIME 2024, AIME 2025, LiveCodeBench v5, LiveCodeBench v6, AlignBench
- Sep 17, 2025
- AIME 2024 86.1
- AIME 2025 77.3
- GPQA Diamond 70.1
- LiveCodeBench v5 70.9
- Sep 17, 2025 Magistral Medium 1.2 Mistral AI Magistral Small 1.2 (Magistral-Small-2509) model card (Hugging Face)
- AIME 2024 91.8
- AIME 2025 83.5
- GPQA Diamond 76.3
- LiveCodeBench v5 75
- Sep 15, 2025
Dropped since the last post: Big Bench Audio, MultiChallenge Audio (OpenAI), ComplexFuncBench Audio (OpenAI)
Its system card addendum adds 4 evaluations 1 new 4 results in all
- Sep 11, 2025
- MMLU-Pro 82.7
- MMLU-Redux 92.5
- GPQA Diamond 77.2
- SuperGPQA 60.8
- AIME 2025 87.8
- HMMT 2025 73.9
- LiveBench 76.6
- LiveCodeBench v6 68.7
- Codeforces rating 2071
- OJBench 29.7
- IFEval 88.9
- Arena-Hard v2.0 62.3
- WritingBench 84.6
- Berkeley Function Calling Leaderboard (BFCL) 72
- tau-bench 59.3
- tau2-bench 57.4
- Multi-IF 77.8
- MMLU-ProX 78.7
- INCLUDE 78.9
- PolyMath 56.3
Dropped since the last post: OmniBench, LibriSpeech, Common Voice, FLEURS, CoVoST2, MMAU, VoiceBench, Seed-TTS-eval, MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), MMBench-EN v1.1, MM-Vet, MMStar, MME, MuirBench, CRPE, RealWorldQA, MME-RealWorld, MM-MT-Bench, AI2D, TextVQA, DocVQA, ChartQA, OCRBench v2, RefCOCO / RefCOCO+ / RefCOCOg, ODinW-13, Video-MME, MVBench, EgoSchema, MATH, GSM8K, HumanEval, MBPP, MultiPL-E, LiveCodeBench, WenetSpeech
- Sep 9, 2025
- AIME 2025 78
- Berkeley Function Calling Leaderboard (BFCL) 65
- ZebraLogic 89.8
- MuSR 86.7
- BIG-Bench Hard 87.8
- HumanEval+ 90.8
- MBPP 80.2
- IFEval 84.3
- Multi-IF 63.3
- Chinese SimpleQA 49.1
- WritingBench 86.5
Dropped since the last post: MMLU, MMLU-Pro, C-Eval, CMMLU, LiveBench, SimpleQA, SysBench, MATH-500, GSM8K, CMATH, OlympicArena, AIME 2024, DROP, LiveCodeBench, MBPP+, FullStackBench
- Sep 8, 2025π0.5 Physical Intelligence openpi LIBERO example results for π0.5, with the π0.5 open-weights release (GitHub)
- LIBERO 96.8
Its technical report adds 0 evaluations 1 result in all
- Sep 5, 2025
- SWE-bench Verified 69.2
- SWE-bench Multilingual 55.9
- Multi-SWE-bench 33.5
- Terminal-Bench 44.5
- SWE-Dev 66.6
Dropped since the last post: LiveCodeBench v6, OJBench, MultiPL-E, Aider Polyglot, tau2-bench, AceBench, AIME 2024, AIME 2025, MATH-500, HMMT 2025, CNMO 2024, PolyMath, ZebraLogic, AutoLogi, GPQA Diamond, SuperGPQA, Humanity's Last Exam, MMLU, MMLU-Redux, MMLU-Pro, IFEval, MultiChallenge, SimpleQA, LiveBench
- Sep 2, 2025
- HellaSwag 64
- WinoGrande 73.3
- XNLI 45.3
- XCOPA 69.8
- PIQA 81.9
Its technical report adds 29 evaluations 30 results in all
Apertus: Democratizing Open and Compliant LLMs for Global Language EnvironmentsCapabilities
- MMLU 69.6
- Global MMLU 62.7
- TruthfulQA 61.2
- HumanEval 73
- MBPP 47
- GSM8K 77.6
- MGSM 64.3
- MATH 30.8
- MathQA 33.9
- BIG-Bench Hard 64.2
- DROP 50.8
- ACPBench 62.9
- IFEval 75.2
- Multi-IF 74.7
- INCLUDE 58.2
- BLEnD 66.3
- CulturalBench 74.2
- SwitzerlandQA 67.2
- AGIEval 40.5
- ARC-Challenge 85
- GPQA Main 30.6
- GSM8K-Platinum 74.6
- MLogiQA 40.5
- RULER
Safety
- HarmBench 31.9
- RealToxicityPrompts 0.2
- LinguaSafe
Bias
- BBQ (Bias Benchmark for QA) 67.4
- ToxiGen 70.3
- Sep 2, 2025
- OSWorld 47.5
- Windows Agent Arena 50.6
- Terminal-Bench 45.3
- SWE-bench Verified 68.7
- AndroidWorld 73.3
- Online-Mind2Web 88.2
- BrowseComp-ZH 50.5
- BrowseComp 29.6
- UI-TARS 15 Games Collection 59.8
Dropped since the last post: IMO 2025, miniF2F, PutnamBench, Past IMO (Seed-Prover formal set), CombiBench, miniCTX-v2
- Sep 1, 2025
- FLORES-200 88.6
- WMT24++ 85.8
- Mandarin-Minority translation (Tencent) 60.8
Dropped since the last post: AIME 2024, AIME 2025, MATH, GPQA Diamond, OlympiadBench, LiveCodeBench, FullStackBench, ArtifactsBench, BIG-Bench Hard, DROP, ZebraLogic, IFEval, SysBench, LengthCtrl (Tencent), InsCtrl (Tencent), ComplexNLU (Tencent), Word-Task (Tencent), Berkeley Function Calling Leaderboard (BFCL), tau-bench, ComplexFuncBench, C3-Bench
Its technical report adds 2 evaluations 2 new 26 dropped 5 results in all
Hunyuan-MT Technical ReportCapabilities
Dropped since the lab’s previous technical report: AIME 2024, AIME 2025, MATH, GPQA Diamond, OlympiadBench, LiveCodeBench, FullStackBench, ArtifactsBench, BIG-Bench Hard, DROP, ZebraLogic, IFEval, SysBench, LengthCtrl (Tencent), InsCtrl (Tencent), ComplexNLU (Tencent), Word-Task (Tencent), Berkeley Function Calling Leaderboard (BFCL), tau-bench, ComplexFuncBench, C3-Bench, McEval, PenguinScrolls, LongBench v2, FRAMES, RULER
- Aug 29, 2025
- MMLU 89.7
- MMLU-Pro 82.7
- Arena-Hard v2.0 86.5
- C-Eval 90.4
- CMMLU 84.3
- IFEval 89.7
- COLLIE 57.1
- Meeseeks 43
- MATH-500 96.4
- AIME 2024 70.4
- AIME 2025 61.3
- Beyond AIME 43
- GPQA Diamond 73.2
- DROP 79.1
- ZebraLogic 89.3
- GraphWalks 51
- LiveCodeBench 48
- HumanEval+ 88.4
- MBPP+ 79.6
- SWE-bench Verified 60.4
- Terminal-Bench 39.5
- tau2-bench 67.7
- AceBench 76.1
- VITA-Bench 24.3
- LongCat safety evaluation (Meituan) 87.7
Its technical report adds 1 evaluation 25 results in all
LongCat-Flash Technical ReportCapabilities
- Aug 28, 2025gpt-realtime OpenAI Introducing gpt-realtime and Realtime API updates for production voice agents (OpenAI)
Dropped since the last post: AIME 2025, FrontierMath, HMMT 2025, GPQA Diamond, Humanity's Last Exam, SWE-bench Verified, Aider Polyglot, MultiChallenge, BrowseComp, COLLIE, tau2-bench, MMMU, MMMU-Pro, Video-MMMU, CharXiv, ERQA, HealthBench, HealthBench Hard, HealthBench Hard Hallucinations, LongFact, FActScore
- Aug 28, 2025
- SWE-bench Verified 70.8
Dropped since the last post: AIME 2025, AIME 2024, GPQA Diamond, LiveCodeBench, MMLU-Pro, LOFT, SimpleQA, MMMU, EgoSchema
Its model card adds 10 evaluations 10 results in all
Grok Code Fast 1 Model CardSafety
Robustness
Alignment
- MASK 28.1
Bio and chem risk
- BioLP-bench 19.9
- Virology Capabilities Test (VCT) 28.7
- WMDP
Cyber risk
- Cybench 22.5
- Aug 28, 2025
- WMT24++ 83.8
Dropped since the last post: Berkeley Function Calling Leaderboard (BFCL), tau-bench, m-tau-bench (Cohere), DeepResearch Bench
Its technical report adds 2 evaluations 2 new 43 dropped 3 results in all
Command A Translate: Raising the Bar of Machine Translation with Difficulty FilteringCapabilities
- WMT25 -6.3
Dropped since the lab’s previous technical report: MMLU, MMLU-Pro, GPQA Diamond, IFEval, InFoBench, ChatRAG Bench, StrategyQA, Bamboogle, DROP, HotpotQA, Berkeley Function Calling Leaderboard (BFCL), tau-bench, NTREX-128, FLORES-200, m-ArenaHard, m-tau-bench (Cohere), Language Confusion Benchmark, AL-QASIDA, MBPP+, LiveCodeBench v5, BigCodeBench, LBPP (Less Basic Python Problems), HumanEvalPack, HumanEval, HumanEval-COBOL (Cohere), RepoQA, SWE-bench Verified, Aider Polyglot, Spider, Bird-SQL, Multi-dialect SQL evaluation (Cohere), MATH, AIME 2024, Default safety evaluation (Cohere), XSTest, Safety mode controllability (Cohere), Enterprise generative evaluation (Cohere), Enterprise RAG evaluation (Cohere), RULER, LongBench v2, Cohere enterprise human evaluation, MGSM, INCLUDE
- Aug 26, 2025
- MATH-500 96.3
- AIME 2024 81.9
- AIME 2025 78.1
- GPQA Diamond 70.5
- BIG-Bench Hard 86.3
- LiveCodeBench v6 61.3
- MMLU 87.2
- MMLU-Pro 80.5
- SimpleQA 25.8
- IFEval 81.5
- Arena-Hard-Auto 94.4
- RefusalBench (Nous Research) 57.1
- RewardBench 73
- DROP 83.5
- MuSR 66.1
- OpenBookQA 94.2
- EQ-Bench 3
- Creative Writing v3 79.8
Its technical report adds 0 evaluations 18 results in all
- Aug 25, 2025
- MMStar 77.9
- MM-Vet 81.2
- MMBench-EN v1.1 87.4
- MTVQA-Test 39.3
- AI2D 87.3
- OCRBench 90.7
- WildVision 82.8
- MME-RealWorld 65.1
- HallusionBench 57.3
- MVBench 76.5
- Video-MME 72.9
- MLVU 78.2
- LongVideoBench 67.1
- MMMU 77.7
- MathVista 82.7
- MATH-Vision (MathVision) 63.9
- MathVerse 68.5
- DynaMath 46.5
- We-Math 62.3
- OlympiadBench 68.7
- LogicVista 66.7
- MATH-500 98
- AIME 2024 84.7
- AIME 2025 75.6
- GPQA Diamond 73.2
- MMLU-Pro 81.3
- C-Eval 90.9
- GAOKAO-Bench 94.5
- IFEval 83.7
- SGP-Bench 70.7
- ScreenSpot 89.8
- ScreenSpot-v2 92.9
- OSWorld-G 53.2
- VSI-Bench 69.5
- ERQA 46.8
- SpaCE-10 55
- OmniSpatial 51.9
Dropped since the last post: Scientists' First Exam (SFE), PHYSICS, SMolInstruct, ChemBench, Matbench, MicroVQA, ProteinLMBench, MSEarth, XLRS-Bench
Its technical report adds 35 evaluations 35 new 9 dropped 72 results in all
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and EfficiencyCapabilities
- MME 2525.9
- MMMB
- ChartQA 88
- TextVQA 84.5
- DocVQA 94.9
- InfographicVQA 82
- SEED-Bench-2-Plus 71.9
- CharXiv 48.3
- VCR (Visual Caption Restoration) 97
- RealWorldQA 75.2
- BLINK 61.4
- Mantis-Eval 77
- MMIU 61.3
- MuirBench 47.5
- MMT-Bench 72.7
- MIRB 73
- RefCOCO / RefCOCO+ / RefCOCOg 92.4
- MMBench-Video 1.74
- MMLU 89.1
- CMMLU 90.2
- TriviaQA 74.8
- Natural Questions 50.4
- C3 97.8
- RACE 94.2
- WinoGrande 91.7
- HellaSwag 97
- BIG-Bench Hard 86.5
- GSM8K 91.6
- MATH 94.5
- HumanEval 98.2
- SArena
- Windows Agent Arena 18
Robustness
Dropped since the lab’s previous technical report: SMolInstruct, ChemBench, Matbench, ProteinLMBench, Scientists' First Exam (SFE), PHYSICS, MicroVQA, MSEarth, XLRS-Bench
- Aug 21, 2025 Command A Reasoning Cohere Command A Reasoning: Enterprise-grade control for AI agents (Cohere blog)
- Berkeley Function Calling Leaderboard (BFCL) 70.3
- tau-bench 59.8
- m-tau-bench (Cohere) 46.5
- DeepResearch Bench
Dropped since the last post: ChartQA, InfographicVQA, AI2D, MathVista, DocVQA, TextVQA, OCRBench, MMMU, RealWorldQA
- Aug 21, 2025
- MMLU-Redux 93.7
- MMLU-Pro 84.8
- GPQA Diamond 80.1
- Humanity's Last Exam 29.8
- BrowseComp 30
- BrowseComp-ZH 49.2
- SimpleQA 93.4
- LiveCodeBench v6 74.8
- Codeforces rating 2091
- Aider Polyglot 76.3
- SWE-bench Verified 66
- SWE-bench Multilingual 54.5
- Terminal-Bench 31.3
- AIME 2024 93.1
- AIME 2025 88.4
- HMMT 2025 84.2
Dropped since the last post: FRAMES, CNMO 2024, Berkeley Function Calling Leaderboard (BFCL), tau-bench
- Aug 20, 2025
- MMLU-Pro 82.7
- MMLU 87.4
- GPQA Diamond 71.4
- SuperGPQA 55.7
- SimpleQA 9.7
- AIME 2024 91.7
- AIME 2025 84.7
- Beyond AIME 65
- ARC-AGI-2 1.45
- KOR-Bench 70.6
- Humanity's Last Exam 10.1
- LiveCodeBench v6 67.4
- IFEval 85.8
- tau-bench 58.2
- SWE-bench Verified 56
- Multi-SWE-bench 17
- MMMLU 78.4
- RULER 94.6
- AIR-Bench 2024 75.6
Dropped since the last post: HumanEval, MBPP, BigCodeBench, Aider-Edit, CanItEdit, MBXP, NaturalCodeBench
- Aug 18, 2025
- AIME 2025 72.1
- MATH-500 97.8
- GPQA Diamond 64
- LiveCodeBench 71.1
- Berkeley Function Calling Leaderboard (BFCL) 66.9
- IFEval 90.3
- Humanity's Last Exam 6.5
- RULER 78.9
Dropped since the last post: MMMU, AI2D, ChartQA, InfographicVQA, OCRBench, OCRBench v2, DocVQA, Video-MME
Its technical report adds 2 evaluations 7 results in all
- Aug 14, 2025 Gemma 3 270M Google DeepMind Introducing Gemma 3 270M (Google Developers Blog) and Gemma 3 model card
- IFEval 51.2
- BIG-Bench Hard 26.7
- HellaSwag 37.7
- PIQA 66.2
- ARC-Challenge 28.2
- WinoGrande 52.3
Dropped since the last post: MGSM, WMT24++, INCLUDE, MMLU-ProX, MMMLU, Global MMLU, ECLeKTic, GPQA Diamond, LiveCodeBench v5, AIME 2025, MMLU, MBPP, HumanEval, LiveCodeBench, HiddenMath, Global MMLU Lite, MMLU-Pro, BoolQ, SIQA, TriviaQA, Natural Questions, ARC-Easy, DROP
- Aug 12, 2025
- RealWorldQA 65.2
- MM-IFEval 37.7
- InfographicVQA 58.7
- OCRBench 74.2
- BLINK 44.4
- MMStar 49.5
- MMMU 38.4
- MathVista 51.1
- SEED-Bench 72
- MM-Vet 48.1
- MME 1753
- MMLU 51
Dropped since the last post: GPQA Diamond, IFEval, IFBench, GSM8K, MGSM, MMMLU
Its technical report adds 2 evaluations 2 new 12 dropped 12 results in all
LFM2 Technical ReportDropped since the lab’s previous technical report: MMLU, MMLU-Pro, GPQA Diamond, IFEval, IFBench, Multi-IF, GSM8K, GSM-Plus, MATH-500, MATH Level 5, MMMLU, MGSM
- Aug 11, 2025
- MMBench-EN v1.1 88.2
- MMBench-CN v1.1 88.3
- MMStar 75.3
- BLINK 65.3
- MuirBench 75.3
- HallusionBench 65.4
- ZeroBench 23.4
- GeoBench 79.7
- MMMU 75.4
- MMMU-Pro 65.2
- MathVista 84.6
- MATH-Vision (MathVision) 65.6
- MathVerse 72.1
- DynaMath 53.9
- LogicVista 62.4
- AI2D 88.1
- We-Math 68.8
- MMLongBench-Doc 44.7
- OCRBench 86.5
- ChartQA Pro 64
- ChartMuseum 55.3
- RefCOCO / RefCOCO+ / RefCOCOg 91.3
- TreeBench 50.1
- Ref-L4 89.5
- OmniSpatial 51
- CVBench 87.3
- ERQA 50
- All-Angles Bench 56.9
- OSWorld 35.8
- AndroidWorld 57
- WebVoyager 84.4
- WebQuest 68.8
- Design2Code 82.2
- Flame-React-Eval 82.5
- Video-MME 80.7
- MMVU 68.7
- Video-MMMU 72.4
- LVBench 53.8
- MotionBench 62.4
- MVBench 73
Dropped since the last post: tau-bench, Berkeley Function Calling Leaderboard (BFCL), BrowseComp, MMLU-Pro, AIME 2024, MATH-500, SciCode, GPQA Diamond, Humanity's Last Exam, LiveCodeBench, Artificial Analysis Intelligence Index v2, SWE-bench Verified, Terminal-Bench
Its technical report adds 0 evaluations 34 dropped 40 results in all
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement LearningDropped since the lab’s previous technical report: tau-bench, Berkeley Function Calling Leaderboard (BFCL), BrowseComp, MMLU-Pro, AIME 2024, MATH-500, SciCode, GPQA Diamond, Humanity's Last Exam, LiveCodeBench v5, Artificial Analysis Intelligence Index v2, SWE-bench Verified, Terminal-Bench, MMLU, SimpleQA, IFEval, SysBench, MultiChallenge, SafetyBench, Novel logical reasoning problems expert evaluation (Z.ai), Real-scenario prompts human evaluation (Z.ai), CC-Bench, Challenging translation human evaluation (Z.ai), BIG-Bench Hard, HellaSwag, PIQA, TriviaQA, EvalPlus, GSM8K, MATH, CLUEWSC, C-Eval, C3, Chinese SimpleQA
- Aug 10, 2025
- HealthBench 60.1
- HealthBench Hard 34.7
- HealthBench Consensus 91.5
- AIME 2024 83.4
- AIME 2025 72.9
- Arena-Hard v2.0 45.8
- CFBench 77.6
- WritingBench 85.6
Its technical report adds 2 evaluations 10 results in all
Baichuan-M2: Scaling Medical Capability with Large Verifier SystemCapabilities
- IFEval 86
- AlignBench 8.77
- Aug 7, 2025
- AIME 2025 85.2
- FrontierMath 9.6
- GPQA Diamond 71.2
- Humanity's Last Exam 8.7
- HMMT 2025 75.6
- MMMU 75.6
- MMMU-Pro 62.6
- CharXiv 62.7
- Video-MMMU 66.8
- ERQA 50.1
- SWE-Lancer
- SWE-bench Verified 54.7
- Aider Polyglot 48.4
- MultiChallenge 54.9
- Internal API instruction following (OpenAI) 56.1
- COLLIE 96.9
- tau2-bench 46.3
- OpenAI MRCR 43.2
- GraphWalks 64
- BrowseComp Long Context 128k 80.4
- Video-MME 65.7
- LongFact 1
- FActScore 7.3
Its system card adds 8 evaluations 22 dropped 9 results in all
GPT-5 System CardHallucination
- LongFact-Concepts 0.9
- LongFact-Objects 3.1
- SimpleQA 11
- SimpleQA Hallucination Rate 31
Safety
Robustness
Dropped since the lab’s previous system card: HealthBench, HealthBench Hard, HealthBench Consensus, HealthBench Hard Hallucinations, HealthBench Consensus Urgent, HealthBench Consensus Global Health, Long-form Biological Risk Questions, Multimodal Troubleshooting Virology, ProtocolQA Open-Ended, Tacit Knowledge and Troubleshooting, TroubleshootingBench, Challenging prompts from red teamers with biosafety-relevant PhDs, Filtered, adversarial sample of production prompts, Capture the Flag (Collegiate), Capture the Flag (Professional), Cyber Range, SWE-bench Verified, OpenAI PRs, MLE-bench 30, SWE-Lancer, PaperBench, OpenAI-Proof Q&A
- Aug 7, 2025
- AIME 2025 91.1
- FrontierMath 22.1
- GPQA Diamond 82.3
- Humanity's Last Exam 16.7
- HMMT 2025 87.8
- MMMU 81.6
- MMMU-Pro 74.1
- CharXiv 75.5
- Video-MMMU 82.5
- ERQA 62.9
- SWE-Lancer
- SWE-bench Verified 71
- Aider Polyglot 71.6
- MultiChallenge 62.3
- Internal API instruction following (OpenAI) 65.8
- COLLIE 98.5
- tau2-bench 70.8
- OpenAI MRCR 84.3
- GraphWalks 73.4
- BrowseComp Long Context 128k 89.4
- Video-MME 78.5
- LongFact 0.7
- FActScore 3.5
Dropped since the last post: Codeforces rating, HealthBench, HealthBench Hard, AIME 2024, MMLU, tau-bench Retail
Its system card adds 28 evaluations 14 new 19 dropped 31 results in all
GPT-5 System CardHealth
Hallucination
- LongFact-Concepts 0.9
- LongFact-Objects 1.8
- SimpleQA 22
- SimpleQA Hallucination Rate 26
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 42
- MLE-bench 30 3
- PaperBench 23
- OpenAI-Proof Q&A 1
Dropped since the lab’s previous system card: Challenging Refusal Evaluation, Human Sourced Jailbreaks, PersonQA, Pairwise safety red teaming, Person Identification evaluation, Ungrounded Inference evaluation, BBQ (Bias Benchmark for QA), First-Person Fairness evaluation, Instruction Hierarchy Evaluation, Image generation refusals, METR Time Horizon, Apollo Research in-context scheming evaluations, Apollo Research sabotage evaluations, Pattern Labs cyberoffensive challenges, Capture the Flag (High School), OpenAI Research Engineer Interview: Multiple Choice, OpenAI Research Engineer Interview: Coding, SWE-Lancer Diamond (SWE Manager), MMMLU
- Aug 7, 2025
- AIME 2025 100
- FrontierMath 32.1
- HMMT 2025 100
- GPQA Diamond 89.4
- Humanity's Last Exam
- SWE-bench Verified 74.9
- Aider Polyglot 88
- MultiChallenge 69.6
- BrowseComp 54.9
- COLLIE 99
- tau2-bench 80.1
- MMMU 84.2
- MMMU-Pro 78.4
- Video-MMMU 84.6
- CharXiv 81.1
- ERQA 65.7
- HealthBench 67.2
- HealthBench Hard 46.2
- HealthBench Hard Hallucinations 1.6
- LongFact 0.8
- FActScore 1
Dropped since the last post: Codeforces rating, AIME 2024, MMLU, tau-bench Retail
Its system card adds 50 evaluations 31 new 12 dropped 55 results in all
GPT-5 System CardCapabilities
Health
Hallucination
Safety
- Standard Refusal Evaluation
- OpenAI Production Benchmarks and safe-completion evals
- Sycophancy evaluation (offline) 0.04
- Image input safety evaluations
- Expert Red Teaming for Violent Attack Planning 65.1
- Challenging prompts from red teamers with biosafety-relevant PhDs 92.1
- Filtered, adversarial sample of production prompts 95.7
- Pairwise Bioweaponization Campaign 60
Robustness
Alignment
Bio and chem risk
- Long-form Biological Risk Questions
- Multimodal Troubleshooting Virology 42
- ProtocolQA Open-Ended 30
- Tacit Knowledge and Troubleshooting 66
- TroubleshootingBench 22
- Virology Capabilities Test (VCT) 41.4
- Virology Capabilities Test (text-only) 46
- Human Pathogen Capabilities Test 53.7
- Molecular Biology Capabilities Test 48
- World-Class Bio 47.6
Cyber risk
AI R&D risk
- OpenAI PRs 45
- MLE-bench 30 8
- SWE-Lancer 55
- PaperBench 24
- OpenAI-Proof Q&A 2
Autonomy
Dropped since the lab’s previous system card: PersonQA, Multilingual performance on translated expert-domain tasks, First-Person Fairness evaluation, Irrelevant instructions, text-based web browser, Irrelevant instructions, visual browser, In-context data exfiltration, visual browser, Active data exfiltration, visual browser, User Confirmations During Computer Use, Agentic harmful task refusals, Proactive refusals of high-risk tasks, Capture the Flag (High School), OpenAI Research Engineer Interview: Multiple Choice
Its system card addendum adds 5 evaluations 5 results in all
- Aug 5, 2025
- Codeforces rating 2516
- Humanity's Last Exam 17.3
- HealthBench 42.5
- HealthBench Hard 10.8
- AIME 2024 96
- AIME 2025 98.7
- GPQA Diamond 71.5
- MMLU 85.3
- tau-bench Retail 54.8
Dropped since the last post: MMMU, MathVista, SWE-Lancer, SWE-bench Verified, Aider Polyglot, MultiChallenge, BrowseComp, CharXiv, tau-bench
Its model card adds 22 evaluations 30 results in all
gpt-oss-120b & gpt-oss-20b Model CardCapabilities
- SWE-bench Verified 60.7
- tau-bench 54.8
- Aider Polyglot 34.2
- MMMLU 75.7
Health
Hallucination
- SimpleQA 6.7
- SimpleQA Hallucination Rate 91.4
- PersonQA 15.5
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 37
- PaperBench 7
- Aug 5, 2025
- SWE-bench Verified 74.5
- Terminal-Bench 43.3
- GPQA Diamond 80.9
- tau-bench 69.2
- MMMLU 89.5
- MMMU 77.1
- AIME 2025 78
Its system card addendum adds 29 evaluations 1 new 29 results in all
System Card Addendum: Claude Opus 4.1Safety
Bias
Alignment
Bio and chem risk
Cyber risk
- Cybench 51.4
AI R&D risk
- Aug 5, 2025
- Codeforces rating 2622
- Humanity's Last Exam 19
- HealthBench 57.6
- HealthBench Hard 30
- AIME 2024 96.6
- AIME 2025 97.9
- GPQA Diamond 80.1
- MMLU 90
- tau-bench Retail 67.8
Dropped since the last post: FrontierMath, DSBench, SpreadsheetBench, Investment Banking Modeling Tasks (OpenAI), BrowseComp, WebArena
Its model card adds 22 evaluations 30 results in all
gpt-oss-120b & gpt-oss-20b Model CardCapabilities
- SWE-bench Verified 62.4
- tau-bench 67.8
- Aider Polyglot 44.4
- MMMLU 81.3
Health
Hallucination
- SimpleQA 16.8
- SimpleQA Hallucination Rate 78.2
- PersonQA 29.8
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
- OpenAI PRs 37
- PaperBench 11
- Aug 1, 2025
- Humanity's Last Exam 34.8
- IMO 2025 60.7
- AIME 2025 99.2
- LiveCodeBench v6 87.6
Dropped since the last post: GPQA Diamond, LiveCodeBench v5, Aider Polyglot, SWE-bench Verified, SimpleQA, FACTS Grounding, MMMU, Vibe-Eval, MRCR v2, Global MMLU Lite
- Jul 31, 2025
- Jul 31, 2025
- ChartQA 90.9
- InfographicVQA 82.9
- AI2D 94
- MathVista 73.5
- DocVQA 95.9
- TextVQA 84.8
- OCRBench 86.9
- MMMU
- RealWorldQA
Dropped since the last post: Cohere enterprise human evaluation, MMLU, MATH, IFEval, tau-bench, Berkeley Function Calling Leaderboard (BFCL), MBPP+, RepoQA
- Jul 31, 2025
- IMO 2025 4
- miniF2F 99.6
- PutnamBench 50.4
- Past IMO (Seed-Prover formal set) 78.1
- CombiBench 30
- miniCTX-v2 81.8
Dropped since the last post: AIME 2024, AIME 2025, Beyond AIME, GPQA Diamond, LiveCodeBench, MMMU, MMMU-Pro, MATH-Vision (MathVision), RealWorldQA, ChartQA, InfographicVQA
- Jul 31, 2025 Seed Diffusion Preview ByteDance Seed Seed Diffusion Preview blog and tech report (arXiv 2508.02193)
- HumanEval 82.8
- MBPP 79.4
- BigCodeBench 53.2
- LiveCodeBench v6 33.7
- Aider-Edit 44.4
- CanItEdit 54.3
- MBXP 72.6
- NaturalCodeBench 42.2
Dropped since the last post: MME, MMBench, MMMU, MM-Vet, MathVista, GenEval, GEdit-Bench, IntelligentBench
Its technical report adds 0 evaluations 10 dropped 8 results in all
Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed InferenceDropped since the lab’s previous technical report: HumanEval+, MBPP+, MHPP, BigCodeBench-Hard, LiveCodeBench, FullStackBench, CRUXEval, CodeEditorBench, SWE-bench Verified, Multi-SWE-bench mini
- Jul 31, 2025
- MMMU 74.2
- MATH-Vision (MathVision) 64.8
- SimpleVQA 62.2
- HallusionBench 64.2
- ZeroBench 23
- DynaMath 50.1
- AIME 2025 82.9
- GPQA Diamond 73
- LiveCodeBench v6 67.1
- HMMT 2025 70
- CNMO 2024 83.7
- Jul 29, 2025
- Jul 28, 2025
- tau-bench 70
- Berkeley Function Calling Leaderboard (BFCL) 77.8
- BrowseComp 26.4
- MMLU-Pro 84.6
- AIME 2024 91
- MATH-500 98.2
- SciCode 41.7
- GPQA Diamond 79.1
- Humanity's Last Exam 14.4
- LiveCodeBench
- Artificial Analysis Intelligence Index v2 67.7
- SWE-bench Verified 64.2
- Terminal-Bench 37.5
Dropped since the last post: MMLU, GSM8K, MATH, BIG-Bench Hard, HellaSwag, HumanEval, IFEval, AlignBench, LongBench, Needle In A Haystack
GLM-4.5-Air: Its technical report adds 8 evaluations 7 new 7 dropped 20 results in all
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation ModelsCapabilities
- LiveCodeBench v5 70.7
- MMLU 87.4
- IFEval 86.3
- SysBench 77.4
- MultiChallenge 42.5
- Novel logical reasoning problems expert evaluation (Z.ai) 53.4
Hallucination
- SimpleQA 14.5
Safety
- SafetyBench 87.8
Dropped since the lab’s previous technical report: GSM8K, MATH, BIG-Bench Hard, HumanEval, AlignBench, Math23K, GLM-4 All Tools information seeking (Z.ai)
Its technical report adds 22 evaluations 14 new 34 results in all
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation ModelsCapabilities
- LiveCodeBench v5 72.9
- MMLU 90
- IFEval 86.1
- SysBench 81
- MultiChallenge 52.8
- Novel logical reasoning problems expert evaluation (Z.ai) 62
- Real-scenario prompts human evaluation (Z.ai)
- CC-Bench
- Challenging translation human evaluation (Z.ai) 1.71
- BIG-Bench Hard 86.2
- HellaSwag 87.1
- PIQA 85.3
- TriviaQA 80
- EvalPlus 78.1
- GSM8K 79.4
- MATH 61
- CLUEWSC 83.5
- C-Eval 86.9
- C3 83.1
Hallucination
- SimpleQA 26.4
- Chinese SimpleQA 70.1
Safety
- SafetyBench 89.9
- Jul 25, 2025 Llama-3.3-Nemotron-Super-49B-v1.5 NVIDIA Llama-3_3-Nemotron-Super-49B-v1_5 model card (Hugging Face)
- MATH-500 97.4
- AIME 2024 87.5
- AIME 2025 82.7
- GPQA Diamond 72
- LiveCodeBench v5 73.6
- Berkeley Function Calling Leaderboard (BFCL) 71.8
- IFEval 88.6
- Arena-Hard-Auto 92
- Humanity's Last Exam 7.64
- MMLU-Pro 79.5
Dropped since the last post: RoboCasa, RoboCasa GR1 Tabletop Tasks
- Jul 25, 2025
- MMLU-Pro 84.4
- MMLU-Redux 93.8
- GPQA Diamond 81.1
- SuperGPQA 64.9
- AIME 2025 92.3
- HMMT 2025 83.9
- LiveBench 78.4
- Humanity's Last Exam 18.2
- LiveCodeBench v6 74.1
- Codeforces rating 2134
- OJBench 32.5
- IFEval 87.8
- Arena-Hard v2.0 79.7
- Creative Writing v3 86.1
- WritingBench 88.3
- Berkeley Function Calling Leaderboard (BFCL) 71.9
- tau-bench 56.9
- tau2-bench 58.5
- Multi-IF 80.6
- MMLU-ProX 81
- INCLUDE 81
- PolyMath 60.1
Dropped since the last post: Terminal-Bench, SWE-bench Verified, SWE-bench-Live, SWE-bench Multilingual, Multi-SWE-bench mini, Multi-SWE-bench flash, Aider Polyglot, Spider 2.0, WebArena, Mind2Web
- Jul 24, 2025
- AIME 2024 70.5
- AIME 2025 62
- GPQA Diamond 65.8
- LiveCodeBench v5 59.2
Dropped since the last post: SWE-bench Verified
- Jul 24, 2025
- AIME 2024 72
- AIME 2025 61
- GPQA Diamond 71.5
- LiveCodeBench v5 59.4
Dropped since the last post: FLEURS, Common Voice, Audio Understanding Benchmark (Mistral)
- Jul 24, 2025
- MMLU-Pro 83.5
- MMMU 77.7
- GPQA Diamond 77.3
- MMStar 74.9
- MathVista 81.5
- AIME 2025 86
- MATH-Vision (MathVision) 62.5
- IFEval 86.7
- Scientists' First Exam (SFE) 44.3
- PHYSICS 44
- SMolInstruct 51
- ChemBench 83.4
- Matbench 75
- MicroVQA 63.9
- ProteinLMBench 63.1
- MSEarth 65.7
- XLRS-Bench 55
Its technical report adds 0 evaluations 17 results in all
- Jul 22, 2025
- Terminal-Bench 37.5
- SWE-bench Verified 69.6
- SWE-bench-Live 26.3
- SWE-bench Multilingual 54.7
- Multi-SWE-bench mini 25.8
- Multi-SWE-bench flash 27
- Aider Polyglot 61.8
- Spider 2.0 31.1
- WebArena 49.9
- Mind2Web 55.8
- Berkeley Function Calling Leaderboard (BFCL) 68.7
- tau-bench 68.8
Dropped since the last post: MMLU-Redux, GPQA Diamond, C-Eval, LiveBench, IFEval, Arena-Hard-Auto, AlignBench, Creative Writing v3, WritingBench, MATH-500, AIME 2024, AIME 2025, ZebraLogic, AutoLogi, LiveCodeBench v5, Codeforces rating, Multi-IF, INCLUDE, MMMLU, MT-AIME2024, PolyMath, MLogiQA
- Jul 17, 2025
- Humanity's Last Exam 41.6
- FrontierMath 27.4
- DSBench 89.9
- SpreadsheetBench 45.5
- Investment Banking Modeling Tasks (OpenAI) 41
- BrowseComp 68.9
- WebArena 65.4
Dropped since the last post: AIME 2024, GPQA Diamond, Codeforces rating
Its system card adds 32 evaluations 11 new 14 dropped 32 results in all
ChatGPT Agent System CardHallucination
- SimpleQA 91.4
- SimpleQA Hallucination Rate 7.9
- PersonQA 94.3
Safety
- Standard Refusal Evaluation
- OpenAI Production Benchmarks and safe-completion evals
- Image input safety evaluations
- User Confirmations During Computer Use 91
- Agentic harmful task refusals
- Proactive refusals of high-risk tasks 89
- Challenging prompts from red teamers with biosafety-relevant PhDs 87.9
- Filtered, adversarial sample of production prompts 96.9
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous system card: Challenging Refusal Evaluation, Human Sourced Jailbreaks, Pairwise safety red teaming, Person Identification evaluation, Ungrounded Inference evaluation, Image generation refusals, METR Time Horizon, Apollo Research in-context scheming evaluations, Apollo Research sabotage evaluations, Pattern Labs cyberoffensive challenges, OpenAI Research Engineer Interview: Coding, SWE-Lancer, SWE-Lancer Diamond (SWE Manager), MMMLU
- Jul 15, 2025
Dropped since the last post: SWE-bench Verified
Its technical report adds 9 evaluations 9 new 14 dropped 12 results in all
VoxtralCapabilities
Dropped since the lab’s previous technical report: AIME 2024, AIME 2025, MATH-500, GPQA Diamond, LiveCodeBench v5, LiveCodeBench v6, Aider Polyglot, Humanity's Last Exam, Multilingual AIME 2024 (Mistral), Mistral internal function calling benchmark, IFEval (Mistral internal version), MMMU, MathVista, MMMU-Pro
- Jul 15, 2025
- MMLU-Redux 92.3
- MMLU-Pro 81.8
- GPQA Diamond 75.4
- AIME 2025 85.3
- HMMT 2025 72.9
- LiveCodeBench v5 72.6
- LiveCodeBench v6 66.7
- IFEval 84.8
- Multi-IF 73.5
- Berkeley Function Calling Leaderboard (BFCL) 65.2
- tau-bench 57.1
- KMMLU-Pro 67.7
- KMMLU-Redux 72.7
- HRM8K-KSM 87.6
- MMMLU 85.6
- HELMET 58.3
- RULER 88.2
- LongBench 48.1
- Ko-LongBench (LG) 76.9
Dropped since the last post: MATH-500, AIME 2024, CSAT Math 2025
Its technical report adds 2 evaluations 2 new 4 dropped 20 results in all
EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning ModesCapabilities
- LongBench English 48.1
- WMT24++ 90.7
Dropped since the lab’s previous technical report: MATH-500, AIME 2024, CSAT Math 2025, MMLU
- Jul 11, 2025
- LiveCodeBench v6 53.7
- OJBench 27.1
- MultiPL-E 85.7
- SWE-bench Verified 65.8
- SWE-bench Multilingual 47.3
- Terminal-Bench 30
- Aider Polyglot 60
- tau2-bench 64.3
- AceBench 76.5
- AIME 2024 69.6
- AIME 2025 49.5
- MATH-500 97.4
- HMMT 2025 38.8
- CNMO 2024 74.3
- PolyMath 65.1
- ZebraLogic 89
- AutoLogi 89.5
- GPQA Diamond 75.1
- SuperGPQA 57.2
- Humanity's Last Exam 4.7
- MMLU 89.5
- MMLU-Redux 92.7
- MMLU-Pro 81.1
- IFEval 89.8
- MultiChallenge 54.1
- SimpleQA 31
- LiveBench 76.4
Dropped since the last post: Codeforces rating, LiveCodeBench, MathVista, MMMU, MATH-Vision (MathVision), CLUEWSC, C-Eval, HumanEval Multi
Its technical report adds 14 evaluations 14 new 8 dropped 41 results in all
Kimi K2: Open Agentic Intelligence (arXiv 2507.20534)Capabilities
- Multi-SWE-bench 18.3
- SWE-Lancer 39.1
- Arena-Hard v2.0 54.5
- LongBench v2 49.1
- FRAMES 77.1
- OpenAI MRCR 55
- DROP 93.5
- Kimi Chinese in-house open-ended benchmark
- LMArena (Elo)
Hallucination
AI R&D risk
- PaperBench 27.8
Dropped since the lab’s previous technical report: Codeforces rating, LiveCodeBench, MathVista, MMMU, MATH-Vision (MathVision), CLUEWSC, C-Eval, HumanEval Multi
- Jul 10, 2025 Devstral Small 1.1 Mistral AI Upgrading agentic coding capabilities with the new Devstral models (Mistral news)
- SWE-bench Verified 53.6
Dropped since the last post: WildBench, Arena-Hard v2.0, IFEval (Mistral internal version), Infinite Generations (Mistral internal), MMLU, MMLU-Pro, MATH, GPQA Main, GPQA Diamond, MBPP+, HumanEval+, SimpleQA, MMMU, MathVista, ChartQA, DocVQA, AI2D
Its technical report adds 0 evaluations 1 result in all
- Jul 10, 2025 Devstral Medium Mistral AI Upgrading agentic coding capabilities with the new Devstral models (Mistral news)
- SWE-bench Verified 61.6
Dropped since the last post: AIME 2024, AIME 2025, GPQA Diamond, LiveCodeBench v5, Aider Polyglot
- Jul 10, 2025LFM2-1.2B Liquid AI Liquid Foundation Models v2: Our Second Series of Generative AI Models (Liquid AI blog)
Its technical report adds 5 evaluations 12 results in all
LFM2 Technical ReportCapabilities
- MMLU-Pro 19.7
- Multi-IF 45.3
- GSM-Plus 36.1
- MATH-500 42.4
- MATH Level 5 18.6
- Jul 10, 2025
- KMMLU 77
- HAE-RAE Bench 86.2
- KBL 75.5
- KBank-MMLU (Upstage) 71
- KorMedMCQA 87.4
- MMLU 88.2
- MMLU-Pro 76.5
- IFEval 87.8
- Arena-Hard v2.0 46.2
- HumanEval 98.8
- SWE-bench Verified 40.8
- MATH-500 95.4
- AIME 2024 76.5
- AIME 2025 69.1
- Berkeley Function Calling Leaderboard (BFCL) 63.9
- Jul 9, 2025
- AIME 2024 52.3
- AIME 2025 33.6
- MATH-500 92.5
- GPQA Diamond 45.1
Dropped since the last post: Omni-MATH, LiveCodeBench v5, FlenQA, IFEval, Arena-Hard-Auto, HumanEval+, MMLU-Pro, Kitab, PhiBench
Its technical report adds 0 evaluations 18 dropped 4 results in all
Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long GenerationDropped since the lab’s previous technical report: HMMT 2025, Omni-MATH, LiveCodeBench v5, Codeforces rating, AIME 1983-2024, BA-Calendar, 3SAT-Search, TSP-Opt, SpatialEval Maze-Nav, SpatialEval Spatial-Map, FlenQA, IFEval, Arena-Hard-Auto, HumanEval+, MMLU-Pro, Kitab, ToxiGen (discriminative), PhiBench
- Jul 9, 2025
- Humanity's Last Exam 38.6
- GPQA Diamond 87.5
- LiveCodeBench
- USAMO 2025 37.5
- HMMT 2025 93.9
- AIME 2025 98.8
- ARC-AGI-2 15.9
- Vending-Bench 4694.1
Its model card adds 13 evaluations 13 results in all
Grok 4 Model CardSafety
Robustness
Bias
- Soft Bias (Internal) 0.36
Alignment
Bio and chem risk
Cyber risk
- Cybench 43
Persuasion
- MakeMeSay 12
- Jul 9, 2025
- Humanity's Last Exam 44.4
- GPQA Diamond 88.4
- LiveCodeBench
- USAMO 2025 61.9
- HMMT 2025 96.7
- AIME 2025 100
Dropped since the last post: AIME 2024, MMMU, MMLU-Pro, LOFT, SimpleQA, EgoSchema, LMArena (Elo)
- Jul 1, 2025
- MMBench-EN v1.1 85.8
- MMBench-CN v1.1 84.7
- MMStar 72.9
- BLINK 65.1
- MuirBench 74.7
- MMMU 68
- MMMU-Pro 57.1
- Video-MMMU 61
- AI2D 87.9
- MathVista 80.7
- We-Math 63.8
- ChartQA Pro 59.5
- ChartMuseum 48.8
- OCRBench 84.2
- MMLongBench-Doc 42.4
- RefCOCO / RefCOCO+ / RefCOCOg 87.4
- OSWorld 14.9
- AndroidWorld 41.7
- WebVoyager 69
- WebQuest 63.4
- Design2Code 64.7
- Flame-React-Eval 72.5
- Video-MME 73.6
- MMVU 59.4
- LVBench 45.1
- MotionBench 59
Dropped since the last post: AIME 2024, AIME 2025, Omni-MATH, LiveCodeBench v5, SysBench, IFEval, Arena-Hard-Auto, Berkeley Function Calling Leaderboard (BFCL), SuperGPQA, GPQA Diamond
Its technical report adds 14 evaluations 14 new 16 dropped 40 results in all
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement LearningCapabilities
- HallusionBench 63.2
- ZeroBench 19.2
- GeoBench 76
- MATH-Vision (MathVision) 54.4
- MathVerse 68.4
- DynaMath 42.5
- LogicVista 60.4
- TreeBench 37.5
- Ref-L4 86.8
- OmniSpatial 47.7
- CVBench 85
- ERQA 45.8
- All-Angles Bench 52.7
- MVBench 68.4
Dropped since the lab’s previous technical report: GSM8K, MATH, BIG-Bench Hard, MMLU, GPQA Diamond, HumanEval, BoolQ, CommonSenseQA, HellaSwag, PIQA, DROP, C-Eval, CMMLU, GAOKAO-Bench, C3, Berkeley Function Calling Leaderboard (BFCL)
- Jun 30, 2025 ERNIE-4.5-300B-A47B Baidu Announcing the Open Source Release of the ERNIE 4.5 Model Family (ERNIE Blog)
- MMLU 86.5
- MMLU-Pro 78.4
- C-Eval 90.6
- CMMLU 90.2
- LiveBench 68.9
- Chinese SimpleQA 77.1
- SimpleQA 45.9
- IFEval 88
- Multi-IF 76.6
- SysBench 69.8
- MATH-500 96.4
- GSM8K 96.6
- CMATH 96.7
- OlympicArena 75.3
- AIME 2024 54.8
- AIME 2025 35.1
- BIG-Bench Hard 94.3
- MuSR 69.9
- DROP 91.1
- ZebraLogic 58.1
- LiveCodeBench 38.8
- HumanEval+ 92.1
- MBPP+ 81
- FullStackBench 67.1
Its technical report adds 1 evaluation 24 results in all
ERNIE 4.5 Technical ReportCapabilities
- LiveCodeBench v6 38.8
- Jun 27, 2025
- AIME 2024 87.3
- AIME 2025 76.8
- MATH 94.3
- GPQA Diamond 71.2
- OlympiadBench 82.7
- LiveCodeBench 63.9
- FullStackBench 67.8
- ArtifactsBench 43
- BIG-Bench Hard 89.1
- DROP 91.1
- ZebraLogic 84.7
- IFEval 84.7
- SysBench 76.1
- LengthCtrl (Tencent) 55.4
- InsCtrl (Tencent) 71.9
- ComplexNLU (Tencent) 61.2
- Word-Task (Tencent) 62.9
- Berkeley Function Calling Leaderboard (BFCL) 78.3
- tau-bench 54.7
- ComplexFuncBench 61.2
- C3-Bench 63.5
Its technical report adds 5 evaluations 26 results in all
Hunyuan-A13B Technical ReportCapabilities
- McEval 59.6
- PenguinScrolls 87.7
- LongBench v2 55
- FRAMES 81.1
- RULER 76.7
- Jun 26, 2025 Gemma 3n E4B Google DeepMind Introducing Gemma 3n: The developer guide (Google Developers Blog) and Gemma 3n model card
- MGSM 60.7
- WMT24++ 50.1
- INCLUDE 57.2
- MMLU-ProX 19.9
- MMMLU 35.6
- Global MMLU 60.3
- ECLeKTic 1.9
- GPQA Diamond 23.7
- LiveCodeBench v5 25.7
- AIME 2025 11.6
- MMLU 64.9
- MBPP 63.6
- HumanEval 75
- LiveCodeBench 13.2
- HiddenMath 37.7
- Global MMLU Lite 64.5
- MMLU-Pro 50.6
- HellaSwag 78.6
- BoolQ 81.6
- PIQA 81
- SIQA 50
- TriviaQA 70.2
- Natural Questions 20.9
- ARC-Challenge 61.6
- ARC-Easy 81.6
- WinoGrande 71.7
- BIG-Bench Hard 52.9
- DROP 60.8
Dropped since the last post: Humanity's Last Exam, Aider Polyglot, SWE-bench Verified, SimpleQA, FACTS Grounding, MMMU, Vibe-Eval, MRCR v2
- Jun 26, 2025Mercury Inception Labs Introducing Mercury, our General Chat Diffusion Large Language Model (Inception blog)
- MMLU-Pro 69
- GPQA Diamond 51
- Humanity's Last Exam 3.4
- SciCode 18
- HumanEval 85
- MATH-500 83
- AIME 2024 30
Dropped since the last post: MBPP, EvalPlus, MultiPL-E, LiveCodeBench, BigCodeBench
- Jun 25, 2025
- AIME 2024 90
- AIME 2025 86
- Beyond AIME 56.3
- GPQA Diamond 81.5
- LiveCodeBench 64
- MMMU 74.8
- MMMU-Pro 66.4
- MATH-Vision (MathVision) 70.3
- RealWorldQA 77.8
- ChartQA 89.1
- InfographicVQA 88
Dropped since the last post: OlympiadBench, MathVista, V*, VLMs Are Blind, ZeroBench, VisuLogic, SimpleVQA, MMStar, MMBench, MMVP, HallusionBench, TextVQA, AI2D, DocVQA, CharXiv, BLINK, VisualWebBench, RefCOCO / RefCOCO+ / RefCOCOg, CountBench, All-Angles Bench, MotionBench, MVBench, TOMATO, TVBench, TempCompass, LongVideoBench, LVBench, MLVU, Video-MME, OVBench, OVO-Bench, StreamingBench, Video-MMMU, MMVU, Charades-STA, ScreenSpot-v2, ScreenSpot-Pro, OSWorld, Windows Agent Arena, WebVoyager, Online-Mind2Web, AndroidWorld, OCRBench, LVIS-MG, DREAM-1K, TemporalBench, StreamBench (StreamChat), TACoS
- Jun 21, 2025
- MMBench-EN v1.1 84.4
- RealWorldQA 70
- OCRBench 86.9
- MMStar 70.4
- MM-Vet 78.1
- MMMU 64
- MMMU-Pro 46.3
- MATH-Vision (MathVision) 56.9
- MathVista 80.1
- Video-MMMU 65.2
- MMVU 57.5
- Video-MME 71.9
- ScreenSpot-Pro 52.8
- ScreenSpot-v2 91.4
- OSWorld-G 52.5
- MMLongBench-Doc 42.1
- MMLU 82
- MMLU-Pro 68.5
- MATH 91.8
- GPQA Diamond 42.3
- V* 83.2
Dropped since the last post: SWE-bench Verified
Its technical report adds 0 evaluations 11 dropped 17 results in all
Kimi-VL Technical ReportDropped since the lab’s previous technical report: LibriSpeech, FLEURS, AISHELL-1, AISHELL-2, WenetSpeech, Kimi-ASR Internal Testset, MMAU, Clotho-AQA, OpenAudioBench, VoiceBench, Kimi-Audio speech conversation evaluation
- Jun 20, 2025
- WildBench 65.3
- Arena-Hard v2.0 43.1
- IFEval (Mistral internal version) 84.8
- Infinite Generations (Mistral internal) 1.29
- MMLU 80.5
- MMLU-Pro 69.1
- MATH 69.4
- GPQA Main 44.2
- GPQA Diamond 46.1
- MBPP+ 78.3
- HumanEval+ 92.9
- SimpleQA 12.1
- MMMU 62.5
- MathVista 67.1
- ChartQA 87.4
- DocVQA 94.9
- AI2D 92.9
Dropped since the last post: AIME 2024, AIME 2025, LiveCodeBench v5
- Jun 17, 2025
- Humanity's Last Exam 11
- GPQA Diamond 82.8
- AIME 2025 72
- LiveCodeBench v5 63.9
- Aider Polyglot 61.9
- SWE-bench Verified 60.4
- SimpleQA 26.9
- FACTS Grounding 85.3
- MMMU 79.7
- Vibe-Eval 65.4
- MRCR v2 74
- Global MMLU Lite 88.4
Dropped since the last post: LiveCodeBench, Video-MME, Video-MMMU
Its technical report adds 45 evaluations 45 new 2 dropped 56 results in all
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesCapabilities
- LiveCodeBench
- ECLeKTic 36.8
- HiddenMath-Hard 75.5
- LOFT 82.1
- ZeroBench 2
- BetterChartQA 67.3
- FLEURS 9.95
- CoVoST2 36.1
- ActivityNet-QA 65.1
- EgoTempo 36.7
- Perception Test MCVQA 75.1
- QVHighlights 52.4
- Video-MMMU 79.2
- 1H-VideoQA 67.5
- LVBench 62.7
- Video-MME 81.5
- VATEX 65.2
- VATEX ZH 43.9
- YouCook2 177.6
- MINERVA 60.7
- Neptune 84.3
- LMArena (Elo)
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous technical report: Where2Place, SUNRGBD
- Jun 17, 2025
- Humanity's Last Exam 6.9
- GPQA Diamond 66.7
- AIME 2025 63.1
- LiveCodeBench v5 34.3
- Aider Polyglot 27.1
- SWE-bench Verified 44.9
- SimpleQA 13
- FACTS Grounding 86.8
- MMMU 72.9
- Vibe-Eval 57.5
- MRCR v2 30.6
- Global MMLU Lite 84.5
Dropped since the last post: BIG-Bench Hard, BBEH micro average, IFEval, HellaSwag, BoolQ, PIQA, SIQA, TriviaQA, Natural Questions, ARC-Challenge, ARC-Easy, WinoGrande, DROP, MMLU-Pro, LiveCodeBench, Bird-SQL, MATH, HiddenMath, MBPP, HumanEval, Natural2Code, GSM8K, MMLU, AGIEval, ECLeKTic, WMT24++, MGSM, Flores, XQuAD, IndicGenBench, DocVQA, InfographicVQA, TextVQA, AI2D, ChartQA, VQAv2, MathVista, COCO Captions, RealWorldQA, ReMI, BLINK, OK-VQA, TallyQA, SpatialSense VQA, CountBench
- Jun 17, 2025
- Humanity's Last Exam 21.6
- GPQA Diamond 86.4
- AIME 2025 88
- LiveCodeBench
- Aider Polyglot 82.2
- SWE-bench Verified 67.2
- SimpleQA 54
- FACTS Grounding 87.8
- MMMU 82
- Vibe-Eval 67.2
- Video-MME 86.9
- Video-MMMU 83.6
- MRCR v2 58
- Global MMLU Lite 89.2
Dropped since the last post: WebDev Arena
Its technical report adds 46 evaluations 4 new 60 results in all
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesCapabilities
- ECLeKTic 46.8
- HiddenMath-Hard 80.5
- LOFT 87
- ZeroBench 4.5
- BetterChartQA 72.4
- FLEURS 6.66
- CoVoST2 38.5
- ActivityNet-QA 66.7
- EgoTempo 44.3
- Perception Test MCVQA 78.4
- QVHighlights 75
- 1H-VideoQA 81
- LVBench 78.7
- VATEX 71.3
- VATEX ZH 59.7
- YouCook2 188.3
- MINERVA 67.6
- Neptune 87.3
- LMArena (Elo)
- WebDev Arena
Safety
Robustness
Alignment
- GDM Stealth 40
- GDM Situational Awareness 18.2
Bio and chem risk
Cyber risk
AI R&D risk
- Jun 16, 2025Kimi-Dev-72B Moonshot AI Introducing Kimi-Dev: A Strong and Open-source Coding LLM for Issue Resolution (model card)
- SWE-bench Verified 60.4
Dropped since the last post: LibriSpeech, FLEURS, AISHELL-1, AISHELL-2, WenetSpeech, MMAU, Clotho-AQA, OpenAudioBench, VoiceBench
Its technical report adds 0 evaluations 17 dropped 1 result in all
Kimi-Dev: Agentless Training as Skill Prior for SWE-AgentsDropped since the lab’s previous technical report: MATH-Vision (MathVision), MathVista, MMMU, MMMU-Pro, Video-MMMU, MMBench-EN v1.1, RealWorldQA, OCRBench, MMStar, MM-Vet, MMVU, Video-MME, ScreenSpot-Pro, ScreenSpot-v2, OSWorld-G, MMLongBench-Doc, V*
- Jun 16, 2025
- AIME 2024 86
- AIME 2025 76.9
- MATH-500 96.8
- LiveCodeBench v6 65
- FullStackBench 68.3
- GPQA Diamond 70
- Humanity's Last Exam 8.4
- ZebraLogic 86.8
- MMLU-Pro 81.1
- SWE-bench Verified 56
- OpenAI MRCR 73.4
- LongBench v2 61.5
- tau-bench 62.8
- SimpleQA 18.5
- MultiChallenge 44.7
Dropped since the last post: MMLU, Chinese SimpleQA, IFEval, Arena-Hard-Auto, DROP, GSM8K, MATH, MBPP, HumanEval, RULER, MTOB
Its technical report adds 0 evaluations 21 dropped 15 results in all
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning AttentionDropped since the lab’s previous technical report: MMLU, Chinese SimpleQA, IFEval, Arena-Hard-Auto, DROP, GSM8K, MATH, EvalPlus, HumanEval, RULER, MTOB, Needle In A Haystack, MR-NIAH (Multi-Round Needles-In-A-Haystack), General Assistant (MiniMax in-house), Hard Capability (MiniMax in-house), Creative Writing (MiniMax in-house), Knowledge Q&A (MiniMax in-house), Instruction Following (MiniMax in-house), Coding (MiniMax in-house), Safety (MiniMax in-house), Long Context (MiniMax in-house)
- Jun 11, 2025 GR00T N1.5 NVIDIA GR00T N1.5: An Improved Open Foundation Model for Generalist Humanoid Robots (NVIDIA GEAR)
- RoboCasa 47.5
- RoboCasa GR1 Tabletop Tasks 47.4
Dropped since the last post: GPQA Diamond, AIME 2025, Berkeley Function Calling Leaderboard (BFCL), LiveCodeBench v5, IFEval, MATH-500, JudgeBench
Its model card adds 2 evaluations 3 results in all
- Jun 11, 2025 V-JEPA 2 Meta Introducing V-JEPA 2 world model and new benchmarks for physical reasoning (Meta AI blog) and GitHub README
- MVPBench 44.5
- TempCompass 76.9
Dropped since the last post: MMMU, MMMU-Pro, MathVista, ChartQA, DocVQA, LiveCodeBench v5, MMLU-Pro, GPQA Diamond, MGSM, MTOB, MMLU, MATH, MBPP, TydiQA
- Jun 10, 2025
- AIME 2024 70.7
- AIME 2025 62.8
- GPQA Diamond 68.2
- LiveCodeBench v5 55.8
Dropped since the last post: SWE-bench Verified
Its technical report adds 6 evaluations 6 new 20 dropped 10 results in all
MagistralCapabilities
- MATH-500 95.9
- LiveCodeBench v6 47.4
- Humanity's Last Exam 6.4
- MMMU 66
- MathVista 66.8
- MMMU-Pro 51.8
Dropped since the lab’s previous technical report: MMLU, HellaSwag, ARC-Challenge, WinoGrande, MBPP, GSM8K, MT-Bench, PIQA, ARC-Easy, Natural Questions, TriviaQA, HumanEval, MATH, BIG-Bench Hard, AGIEval, Translated ARC-Challenge, HellaSwag and MMLU (Mistral), Passkey Retrieval, BBQ (Bias Benchmark for QA), BOLD, LMArena (Elo)
- Jun 10, 2025
- AIME 2024 93
- GPQA Diamond 84
- Codeforces rating 2748
Dropped since the last post: SWE-bench Verified
- Jun 10, 2025
- AIME 2024 90
- AIME 2025 83.3
- GPQA Diamond 70.8
- LiveCodeBench v5 59.4
- Aider Polyglot 47.1
Dropped since the last post: HumanEval, LiveCodeBench v6, MultiPL-E, Arena-Hard-Auto, IFEval, MATH-500, MMLU-Pro, RULER, MMMU, DocVQA, AI2D, ChartQA
Its technical report adds 9 evaluations 14 results in all
MagistralCapabilities
- Jun 9, 2025Apple Foundation Model Server (2025) Apple Updates to Apple's On-Device and Server Foundation Language Models (Apple Machine Learning Research)
Its technical report adds 4 evaluations 5 results in all
- Jun 3, 2025
- MMMU 48.2
- AI2D 85
- ChartQA 86.3
- InfographicVQA 77.4
- OCRBench 83.9
- OCRBench v2 60.1
- DocVQA 91.2
- Video-MME 54.7
- May 28, 2025
- LiveCodeBench v5 60.5
- LiveCodeBench v6 46.8
- HumanEval+ 88.4
- MBPP+ 81.5
- BigCodeBench-Hard 28.5
Dropped since the last post: GPQA Diamond, HumanEval, MATH-500, AIME 2024, AIME 2025, MMLU-Pro, Berkeley Function Calling Leaderboard (BFCL)
- May 28, 2025
- MMLU-Redux 93.4
- MMLU-Pro 85
- GPQA Diamond 81
- SimpleQA 27.8
- FRAMES 83
- Humanity's Last Exam 17.7
- LiveCodeBench v6 73.3
- Codeforces rating 1930
- SWE-bench Verified 57.6
- Aider Polyglot 71.6
- AIME 2024 91.4
- AIME 2025 87.5
- HMMT 2025 79.4
- CNMO 2024 86.9
- Berkeley Function Calling Leaderboard (BFCL)
- tau-bench 58.7
Dropped since the last post: LiveCodeBench, MATH-500
- May 23, 2025Sarvam-M Sarvam AI Sarvam-M: Explorations in Post Training and Inferencing Optimizations for a Hybrid Indic LLM (Sarvam AI blog)
- MMLU 87
- ARC-Challenge 95
- TriviaQA 91
- GPQA Diamond 48
- HumanEval 88
- MBPP 75
- GSM8K 94
- MATH 81
- IFEval 87
- MT-Bench 8.14
- IndicGenBench 49
- MILU 75
- May 22, 2025
- SWE-bench Verified 80.2
- Terminal-Bench 41.3
- GPQA Diamond 83.8
- tau-bench 70.3
- MMMLU 86.5
- MMMU 74.4
- AIME 2025 85
Its system card adds 42 evaluations 6 dropped 42 results in all
System Card: Claude Opus 4 & Claude Sonnet 4Safety
Bias
Alignment
Bio and chem risk
- Bioweapons acquisition uplift trial (Anthropic) 1.7
- Long-form virology tasks
- Virology Capabilities Test (VCT)
- Bioweapons knowledge questions 51.5
- DNA Synthesis Screening Evasion
- LAB-Bench FigQA
- LAB-Bench ProtocolQA
- LAB-Bench SeqQA
- LAB-Bench CloningScenarios
- Creative biology 37
- Short-horizon computational biology tasks
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 36.7
- METR data deduplication 27.6
- Kernels task (Anthropic) 41.7
- Time series forecasting task (Anthropic) 5.8
- Text-based reinforcement learning task (Anthropic) 0.675
- LLM training task (Anthropic) 1.88
- Quadruped reinforcement learning task (Anthropic) 0.35
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.365
Dropped since the lab’s previous system card: Agentic misalignment suite (Anthropic), Reasoning behavior classifiers (Anthropic), Many-shot jailbreaking evaluation (Anthropic), SHADE-Arena, Internal model use survey (Anthropic), Cyber-harness network challenges (Incalmo cyber ranges)
- May 22, 2025
- SWE-bench Verified 79.4
- Terminal-Bench 50
- GPQA Diamond 83.3
- tau-bench 70.5
- MMMLU 88.8
- MMMU 76.5
- AIME 2025 90
Dropped since the last post: IFEval, MATH-500, AIME 2024
Its system card adds 48 evaluations 24 new 4 dropped 48 results in all
System Card: Claude Opus 4 & Claude Sonnet 4Safety
- Single-turn harmful request evaluations (Anthropic) 98.4
- Single-turn benign request evaluations (Anthropic) 0.07
- Ambiguous context evaluations (Anthropic)
- Multi-turn testing (Anthropic)
- Child safety evaluations (Anthropic)
- Malicious computer use (Anthropic)
- Malicious use of Claude Code (Anthropic) 88
Robustness
Bias
Alignment
- Agentic misalignment suite (Anthropic) 84
- SycophancyEval (Sharma et al.)
- Reasoning behavior classifiers (Anthropic)
- SHADE-Arena 0
- Anthropic automated behavioral audit
- Average in-eval reward hacking rate (Anthropic) 15.2
- Reward-hack-prone coding tasks (Anthropic) 9
- Impossible tasks (Anthropic) 47
- Training distribution hack rate (Anthropic)
Bio and chem risk
- Bioweapons acquisition uplift trial (Anthropic) 2.53
- Long-form virology tasks
- Virology Capabilities Test (VCT)
- Bioweapons knowledge questions 51.5
- DNA Synthesis Screening Evasion
- LAB-Bench FigQA
- LAB-Bench ProtocolQA
- LAB-Bench SeqQA
- LAB-Bench CloningScenarios
- Creative biology 45
- Short-horizon computational biology tasks
Cyber risk
AI R&D risk
- SWE-bench Verified (hard subset) 39.5
- METR data deduplication 32.6
- Kernels task (Anthropic) 72.7
- Time series forecasting task (Anthropic) 6.15
- Text-based reinforcement learning task (Anthropic) 0.625
- LLM training task (Anthropic) 2.99
- Quadruped reinforcement learning task (Anthropic) 1.25
- Novel compiler task (Anthropic)
- Internal AI research evaluation suite 2 0.355
- Internal model use survey (Anthropic) 0
Dropped since the lab’s previous system card: Appropriate harmlessness (Anthropic), Chain-of-thought faithfulness with clues (Anthropic), Alignment faking evaluation, helpful-only setting (Anthropic), RE-Bench subset (Anthropic)
- May 21, 2025 Falcon-Arabic-7B Technology Innovation Institute Falcon-Arabic: A Breakthrough in Arabic Language Models (Falcon blog)
- Open Arabic LLM Leaderboard v2 68
- ArabicMMLU 68.2
- AraTrust 82.5
Its technical report adds 12 evaluations 15 results in all
Adapting Falcon3-7B Language Model for Arabic: Methods, Challenges, and OutcomesCapabilities
- AlGhafa 72.4
- MadinahQA 73.6
- ALRAGE 72.3
- ArabCulture 65.2
- AraDiCE
- IFEval 57.6
- GSM8K 67.7
- HellaSwag 71.4
- ARC-Challenge 53.5
- WinoGrande 68.4
- MMLU 63.3
Hallucination
- TruthfulQA 31.5
- May 21, 2025
- SWE-bench Verified 46.8
Dropped since the last post: MMLU, MMLU-Pro, MATH, GPQA Main, GPQA Diamond, MBPP, HumanEval, SimpleQA, MMMU, MMMU-Pro, MathVista, ChartQA, DocVQA, AI2D, MM-MT-Bench, LongBench v2, RULER
Its technical report adds 0 evaluations 10 dropped 1 result in all
Devstral: Fine-tuning Language Models for Coding Agent ApplicationsDropped since the lab’s previous technical report: AIME 2024, AIME 2025, MATH-500, GPQA Diamond, LiveCodeBench v5, LiveCodeBench v6, Humanity's Last Exam, MMMU, MathVista, MMMU-Pro
- May 20, 2025 Falcon-H1-34B Technology Innovation Institute Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance (Falcon blog)
- BIG-Bench Hard 70.7
- ARC-Challenge 61
- TruthfulQA 65.3
- HellaSwag 81.9
- MMLU 84
- GSM8K 83.6
- MATH-500 83.8
- AIME 2024 23.8
- AIME 2025 16.7
- GPQA Main 41.5
- GPQA Diamond 49.7
- MMLU-Pro 58.7
- HumanEval 87.2
- HumanEval+ 81.7
- MBPP 83.9
- MBPP+ 71.4
- CRUXEval 73.1
- IFEval 89.4
- MT-Bench 9.2
- LiveBench 46.3
Its technical report adds 6 evaluations 26 results in all
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and PerformanceCapabilities
- AMC 69.4
- MMLU-stem 83.6
- AlpacaEval 2.0 48.3
- Okapi MMLU 77.8
- MGSM 76.3
- HELMET
- May 20, 2025
- MME 2388
- MMBench 85
- MMMU 55.3
- MM-Vet 67.2
- MathVista 73.1
- GenEval 88
- GEdit-Bench 6.52
- IntelligentBench 55.3
Dropped since the last post: HumanEval, MBPP, MHPP, BigCodeBench, BigCodeBench-Hard, LiveCodeBench
Its technical report adds 4 evaluations 12 results in all
Emerging Properties in Unified Multimodal PretrainingCapabilities
- MMVP 69.3
- WISE 0.7
- RISEBench 11.9
- KRIS-Bench 60.2
- May 16, 2025
- SWE-bench Verified 72.1
Dropped since the last post: AIME 2024, AIME 2025, Codeforces rating, GPQA Diamond, Humanity's Last Exam, MMMU, MathVista, CharXiv, SWE-Lancer, Aider Polyglot, MultiChallenge, BrowseComp, tau-bench
Its system card addendum adds 6 evaluations 4 new 6 results in all
- May 15, 2025
- May 12, 2025
- MMMU 77.9
- MMMU-Pro 67.6
- MATH-Vision (MathVision) 68.7
- OlympiadBench 65
- MathVista 85.6
- V* 89
- VLMs Are Blind 92.1
- ZeroBench 2
- VisuLogic 35
- RealWorldQA 78.4
- SimpleVQA 63.4
- MMStar 77.8
- MMBench 89.9
- MMVP 69.3
- HallusionBench 60.3
- TextVQA 81.8
- AI2D 87.3
- ChartQA 89.1
- InfographicVQA 91.2
- DocVQA 96.9
- CharXiv 60.2
- BLINK 72.1
- VisualWebBench 87.3
- RefCOCO / RefCOCO+ / RefCOCOg 91.3
- CountBench 93.7
- All-Angles Bench 58.6
- MotionBench 68.4
- MVBench 74.4
- TOMATO 44.7
- TVBench 63.6
- TempCompass 83.7
- LongVideoBench 74
- LVBench 64.6
- MLVU 82.1
- Video-MME 77.9
- OVBench 60
- OVO-Bench 72.3
- StreamingBench 68
- Video-MMMU 81.4
- MMVU 70.1
- Charades-STA 64
- ScreenSpot-v2 95.2
- ScreenSpot-Pro 60.9
- OSWorld 36.7
- Windows Agent Arena 39.6
- WebVoyager 87.2
- Online-Mind2Web 76.4
- AndroidWorld 62.1
- OCRBench 86.1
- LVIS-MG 72.5
- DREAM-1K 43.9
- TemporalBench 79.8
- StreamBench (StreamChat) 72.8
- TACoS 49.6
Its technical report adds 3 evaluations 3 new 14 dropped 57 results in all
Seed1.5-VL Technical ReportCapabilities
Dropped since the lab’s previous technical report: AIME 2025, AIME 2024, Beyond AIME, GPQA Diamond, SuperGPQA, MMLU-Pro, Codeforces rating, LiveCodeBench v5, Aider Polyglot, SWE-bench Verified, ARC-AGI, SimpleQA, COLLIE, IFEval
- May 11, 2025
- AIME 2024 78.8
- AIME 2025 64.9
- LiveCodeBench v5 67.8
- GPQA Diamond 66.8
- IFEval 81.5
Its technical report adds 0 evaluations 5 results in all
- May 8, 2025
- HumanEval 84.8
- MBPP 85.2
- MHPP 36.2
- BigCodeBench 53.3
- BigCodeBench-Hard 26.4
- LiveCodeBench
Its technical report adds 11 evaluations 11 new 12 dropped 17 results in all
Seed-Coder: Let the Code Model Curate Data for ItselfCapabilities
- HumanEval+ 78.7
- MBPP+ 71.2
- MBXP 75.3
- NaturalCodeBench 49.6
- FullStackBench 55.8
- CRUXEval
- Aider-Edit 57.1
- CanItEdit 50.5
- CodeEditorBench
- SWE-bench Verified 19.2
- Multi-SWE-bench mini 4
Dropped since the lab’s previous technical report: MME, MMBench, MMMU, MM-Vet, MathVista, MMVP, GenEval, WISE, GEdit-Bench, IntelligentBench, RISEBench, KRIS-Bench
- May 7, 2025
- HumanEval 92.1
- LiveCodeBench v6 30.3
- MultiPL-E 81.4
- Arena-Hard-Auto 97.1
- IFEval 89.4
- MATH-500 91
- GPQA Diamond 57.1
- MMLU-Pro 77.2
- RULER 96
- MMMU 66.1
- DocVQA 95.3
- AI2D 93.7
- ChartQA 82.6
- May 6, 2025 Gemini 2.5 Pro Google DeepMind Gemini 2.5 Pro Preview: even better coding performance (Google Developers Blog)
- WebDev Arena 1420
- Video-MME 84.8
Dropped since the last post: Humanity's Last Exam, GPQA Diamond, AIME 2025, AIME 2024, LiveCodeBench v5, Aider Polyglot, SimpleQA, MMMU, Vibe-Eval, MRCR (Michelangelo, original), Global MMLU Lite
- Apr 30, 2025
- AIME 2024 75.3
- AIME 2025 62.9
- Omni-MATH 76.6
- GPQA Diamond 65.8
- LiveCodeBench v5 53.8
- FlenQA 97.7
- IFEval 83.4
- Arena-Hard-Auto 73.3
- HumanEval+ 92.9
- MMLU-Pro 74.3
- Kitab
- PhiBench 70.6
Dropped since the last post: MATH-500
Its technical report adds 9 evaluations 21 results in all
Phi-4-reasoning Technical ReportCapabilities
- HMMT 2025 43.8
- Codeforces rating 1736
- AIME 1983-2024 83.1
- BA-Calendar 67.7
- 3SAT-Search 78
- TSP-Opt 37.5
- SpatialEval Maze-Nav 55.1
- SpatialEval Spatial-Map 73.7
Bias
- Apr 30, 2025
- MATH-500 94.6
- GPQA Diamond 52
Dropped since the last post: AIME 2024, AIME 2025, Omni-MATH, LiveCodeBench v5, FlenQA, IFEval, Arena-Hard-Auto, HumanEval+, MMLU-Pro, Kitab, PhiBench
Its technical report adds 1 evaluation 1 new 15 dropped 3 results in all
Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in MathCapabilities
- AIME 2024 57.5
Dropped since the lab’s previous technical report: ARC-Challenge, ARC-Easy, OpenBookQA, BoolQ, HellaSwag, PIQA, WinoGrande, CommonSenseQA, TruthfulQA, TriviaQA, MMLU, HumanEval+, GSM8K, IFEval, MT-Bench
- Apr 30, 2025 Amazon Nova Premier Amazon Amazon Nova Premier: Our most capable model for complex tasks and teacher for model distillation (AWS News Blog)
- MMLU 87.4
- GPQA Diamond 57.1
- AIME 2025 16
- MATH-500 82
- BigCodeBench-Hard 28.1
- MBXP 78.4
- IFEval 91.5
- MMMU 68
- OCRBench v2 56.9
- CharXiv 48.8
- EgoSchema 73.8
- TallyQA 61.5
- SimpleQA 86.3
- Berkeley Function Calling Leaderboard (BFCL) 63.7
- ScreenSpot 87.8
- SWE-bench Verified 42.4
Dropped since the last post: GroundUI
Its technical report adds 1 evaluation 17 results in all
Amazon Nova Premier: Technical Report and Model CardCapabilities
- CharXiv (DQ) 84.6
- Apr 30, 2025
- miniF2F 88.9
- PutnamBench 7.4
- ProverBench (DeepSeek) 40
Dropped since the last post: POPE, MME, MMBench, SEED-Bench, GQA, MMMU, MM-Vet, GenEval, DPG-Bench
Its technical report adds 3 evaluations 3 new 9 dropped 6 results in all
DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal DecompositionCapabilities
- ProofNet 37.1
- CombiBench 10
- FormalMATH 28.3
Dropped since the lab’s previous technical report: POPE, MME, MMBench, SEED-Bench, GQA, MMMU, MM-Vet, GenEval, DPG-Bench
- Apr 30, 2025
- AIME 2024 81.3
- AIME 2025 78
- Omni-MATH 81.9
- GPQA Diamond 68.9
- LiveCodeBench v5 53.1
- FlenQA 97.9
- IFEval 84.9
- Arena-Hard-Auto 79
- HumanEval+ 92.3
- MMLU-Pro 76
- Kitab
- PhiBench 74.2
Dropped since the last post: ARC-Challenge, ARC-Easy, OpenBookQA, BoolQ, HellaSwag, PIQA, WinoGrande, CommonSenseQA, TruthfulQA, TriviaQA, MMLU, GSM8K, MATH-500, MT-Bench
Its technical report adds 9 evaluations 9 new 1 dropped 21 results in all
Phi-4-reasoning Technical ReportCapabilities
- HMMT 2025 53.6
- Codeforces rating 1723
- AIME 1983-2024 89.4
- BA-Calendar 65.6
- 3SAT-Search 70.9
- TSP-Opt 42.6
- SpatialEval Maze-Nav 53.4
- SpatialEval Spatial-Map 73.3
Bias
Dropped since the lab’s previous technical report: MATH-500
- Apr 29, 2025
- AlpacaEval 2.0 9.1
- BIG-Bench Hard 35
- DROP 34.6
- GSM8K 68.3
- IFEval 70.1
- MATH 20.7
- MMLU 40
- Tulu 3 Safety Eval 87.6
- PopQA 12.9
- TruthfulQA 48.7
Its technical report adds 0 evaluations 10 results in all
- Apr 29, 2025
- GPQA Diamond 54.4
- SuperGPQA 40.5
- DROP 78.7
- MMLU-Pro 58.6
- IFEval 61
- MATH-500 95.8
- AIME 2024 68.2
- AIME 2025 55.4
- LiveCodeBench v5 57.8
- LiveCodeBench v6 49.3
Its technical report adds 1 evaluation 11 results in all
- Apr 29, 2025
- MMLU-Redux 92.7
- GPQA Diamond 71.1
- C-Eval 89.6
- LiveBench 77.1
- IFEval 83.4
- Arena-Hard-Auto 95.6
- AlignBench 8.94
- Creative Writing v3 84.6
- WritingBench 80.3
- MATH-500 98
- AIME 2024 85.7
- AIME 2025 81.5
- ZebraLogic 80.3
- AutoLogi 89
- Berkeley Function Calling Leaderboard (BFCL) 70.8
- LiveCodeBench v5 70.7
- Codeforces rating 2056
- Multi-IF 71.9
- INCLUDE 78.7
- MMMLU 84.3
- MT-AIME2024 80.8
- PolyMath 54.7
- MLogiQA 77.1
- Aider Polyglot
Dropped since the last post: MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), HallusionBench, MMBench-EN v1.1, AI2D, ChartQA, DocVQA, MMStar, MM-Vet, OCRBench, OCRBench v2, CC-OCR, Video-MME, MVBench, MMBench-Video, LVBench, EgoSchema, Perception Test MCVQA, MLVU, TempCompass, ScreenSpot, ScreenSpot-Pro, Android in the Zoo (AitZ), AndroidControl, AndroidWorld, MobileMiniWoB++, OSWorld
Qwen3-1.7B: Its technical report adds 1 evaluation 1 new 58 dropped 23 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
Dropped since the lab’s previous technical report: MMMU, MMMU-Pro, MathVista, MATH-Vision (MathVision), MathVerse, MEGA-Bench, MMBench, MMBench-EN v1.1, MMStar, MME, MuirBench, BLINK, CRPE, HallusionBench, MTVQA-Test, RealWorldQA, MME-RealWorld, MM-Vet, MM-MT-Bench, MMLU-Pro, MATH, GSM8K, HumanEval, MultiPL-E, CC-OCR, OmniDocBench, AI2D, TextVQA, DocVQA, InfographicVQA, ChartQA, CharXiv, CharXiv (DQ), SEED-Bench-2-Plus, OCRBench, VCR (Visual Caption Restoration), OCRBench v2, RefCOCO / RefCOCO+ / RefCOCOg, ODinW-13, CountBench, Video-MME, Video-MMMU, MMVU, MVBench, MMBench-Video, LongVideoBench, LVBench, EgoSchema, Perception Test MCVQA, MLVU, TempCompass, Charades-STA, ScreenSpot, ScreenSpot-Pro, AndroidControl, AndroidWorld, MobileMiniWoB++, OSWorld
Qwen3-14B: Its technical report adds 2 evaluations 1 new 25 results in all
Its technical report adds 1 evaluation 1 dropped 24 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 95
Dropped since the lab’s previous technical report: Belebele
Qwen3-30B-A3B: Its technical report adds 1 evaluation 24 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 91.6
Qwen3-32B: Its technical report adds 6 evaluations 5 new 29 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 93.7
- Belebele
- LengthCtrl (Qwen) 73.5
- ThinkFollow (Qwen) 98.9
- ToolUse (Qwen) 85.5
Hallucination
- CounterFactQA (Qwen) 68.1
Qwen3-4B: Its technical report adds 2 evaluations 4 dropped 25 results in all
Qwen3 Technical Report (arXiv 2505.09388)Dropped since the lab’s previous technical report: CounterFactQA (Qwen), LengthCtrl (Qwen), ThinkFollow (Qwen), ToolUse (Qwen)
Qwen3-8B: Its technical report adds 1 evaluation 1 dropped 24 results in all
Qwen3 Technical Report (arXiv 2505.09388)Capabilities
- RULER 89.1
Dropped since the lab’s previous technical report: Belebele
- Apr 25, 2025Kimi-Audio-7B-Instruct Moonshot AI Kimi-Audio GitHub README (evaluation section) and technical report
- LibriSpeech 1.85
- FLEURS 2.69
- AISHELL-1 0.6
- AISHELL-2 2.56
- WenetSpeech 5.83
- MMAU 65.2
- Clotho-AQA 71.2
- OpenAudioBench 69.1
- VoiceBench 76.9
Dropped since the last post: MATH-Vision (MathVision), MathVista, MMMU
Its technical report adds 2 evaluations 2 new 3 dropped 11 results in all
Kimi-Audio Technical ReportCapabilities
Dropped since the lab’s previous technical report: MATH-Vision (MathVision), MathVista, MMMU
- Apr 17, 2025
- Humanity's Last Exam 12.1
- GPQA Diamond 78.3
- AIME 2025 78
- AIME 2024 88
- LiveCodeBench v5 63.5
- Aider Polyglot 51.1
- SimpleQA 29.7
- MMMU 76.7
- Vibe-Eval 62
- MRCR (Michelangelo, original) 84.6
- Global MMLU Lite 88.4
Dropped since the last post: SWE-bench Verified
Its model card adds 5 evaluations 15 dropped 16 results in all
Gemini 2.5 Flash Preview Model CardSafety
Dropped since the lab’s previous model card: SWE-bench Verified, SecureBio VMQA, single choice, LAB-Bench ProtocolQA, LAB-Bench CloningScenarios, LAB-Bench SeqQA, WMDP Biology, WMDP Chemistry, CBRN open-ended questions, InterCode-CTF, In-house CTF suite (Phuong et al. 2024), Hack the Box challenge set (Google DeepMind), Cyber key skills benchmark v1, RE-Bench, GDM Stealth, GDM Situational Awareness
- Apr 16, 2025
- AIME 2024 93.4
- AIME 2025 92.7
- Codeforces rating 2719
- GPQA Diamond 81.4
- Humanity's Last Exam 17.7
- MMMU 81.6
- MathVista 84.3
- SWE-Lancer
- SWE-bench Verified 68.1
- Aider Polyglot 68.9
- MultiChallenge 43
- BrowseComp 28.3
- CharXiv 72
- tau-bench 57.4
Dropped since the last post: MMLU, MMMLU, Internal API instruction following (OpenAI), COLLIE, IFEval, Multi-IF, OpenAI MRCR, GraphWalks, ComplexFuncBench
Its system card adds 33 evaluations 15 new 19 dropped 35 results in all
OpenAI o3 and o4-mini System CardCapabilities
- MMMLU 85.2
Hallucination
- SimpleQA 20
- SimpleQA Hallucination Rate 79
- PersonQA 36
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Dropped since the lab’s previous system card: XSTest, Production Jailbreaks, Jailbreak Augmented Examples, Discrim-Eval, Red Teaming Network pairwise safety comparison, Gray Swan Jailbreak Arena, Expert comparisons on biothreat information, Biotool and Wet Lab Actions, BioLP-bench, Tacit Knowledge Brainstorm (Open-Ended), Structured expert probing campaign: radiological and nuclear, Contextual Nuclear Knowledge, Radiological and Nuclear Expert Knowledge, ChangeMyView, Political Persuasion Parallel Generation, MakeMePay, MakeMeSay, Agentic tasks (autonomous replication and adaptation), MLE-bench
- Apr 16, 2025
- Arena-Hard-Auto 57.6
- AlpacaEval 2.0 62.7
- MMLU 65.5
- PopQA 26.2
- TruthfulQA 66.9
- BIG-Bench Hard 69.1
- DROP 59.4
- GSM8K 80.9
- HumanEval 89.7
- HumanEval+ 86.1
- IFEval 74.8
- AttaQ 88.5
- AIME 2024 8.12
- MATH-500 69
- Apr 16, 2025
- OSWorld 42.5
- Windows Agent Arena 42.1
- WebVoyager 84.8
- Online-Mind2Web 75.8
- AndroidWorld 64.2
- ScreenSpot-v2 94.2
- ScreenSpot-Pro 61.6
Dropped since the last post: AIME 2025, AIME 2024, Beyond AIME, GPQA Diamond, SuperGPQA, MMLU-Pro, Codeforces rating, LiveCodeBench v5, Aider Polyglot, SWE-bench Verified, ARC-AGI, SimpleQA, COLLIE, IFEval
- Apr 16, 2025
- AIME 2024 91.6
- AIME 2025 88.9
- Codeforces rating 2706
- GPQA Diamond 83.3
- Humanity's Last Exam 24.9
- MMMU 82.9
- MathVista 86.8
- CharXiv 78.6
- SWE-Lancer
- SWE-bench Verified 69.1
- Aider Polyglot 81.3
- MultiChallenge 56.5
- BrowseComp 49.7
- tau-bench 61.2
Dropped since the last post: IFEval, OpenAI MRCR, GraphWalks, Video-MME
Its system card adds 33 evaluations 11 new 12 dropped 35 results in all
OpenAI o3 and o4-mini System CardCapabilities
- MMMLU 88.8
Hallucination
- SimpleQA 49
- SimpleQA Hallucination Rate 51
- PersonQA 59
Safety
Robustness
Bias
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Dropped since the lab’s previous system card: WildChat (toxic conversations), XSTest, Challenging Red Teaming Evaluation 1 (created for o3-mini), Challenging Red Teaming Evaluation 2 (created for deep research), BioLP-bench, WMDP Biology, Contextual Nuclear Knowledge, Radiological and Nuclear Expert Knowledge, MakeMePay, MakeMeSay, Agentic tasks (autonomous replication and adaptation), MLE-bench
- Apr 15, 2025
- ARC-Challenge 49.9
- ARC-Easy 74.8
- OpenBookQA 41.6
- BoolQ 80.2
- HellaSwag 68.4
- PIQA 77.1
- WinoGrande 71.9
- CommonSenseQA 71.6
- TruthfulQA 45.3
- TriviaQA 33.6
- MMLU 53.2
- HumanEval+ 38.4
- GSM8K 58.4
- MATH-500 43.4
- IFEval 53.5
- MT-Bench 5.85
Dropped since the last post: MMMU, MMBench, MMMU-Pro, ScienceQA, MathVista, Inter-GPS (Geometry3K), AI2D, ChartQA, DocVQA, InfographicVQA, TextVQA, OCRBench, POPE, BLINK, Video-MME
Its technical report adds 0 evaluations 27 dropped 16 results in all
BitNet b1.58 2B4T Technical ReportDropped since the lab’s previous technical report: MMMU, MMMU-Pro, ScienceQA, MathVista, Inter-GPS (Geometry3K), MMBench, POPE, AI2D, ChartQA, TextVQA, DocVQA, InfographicVQA, OCRBench, BLINK, Video-MME, Common Voice, FLEURS, Open ASR Leaderboard, CoVoST2, Golden3 speech summarization (Microsoft), AIR-Bench (Audio Instruction Benchmark), MMAU, Harmful content defect rate (Azure AI Evaluation SDK, Microsoft), XSTest, Microsoft internal multimodal RAI benchmark, RTVLM, VLGuard
- Apr 14, 2025
- AIME 2024 29.4
- GPQA Diamond 50.3
- MMLU 80.1
- MMMLU 66.9
- SWE-Lancer 3.7
- Aider Polyglot 9.8
- Internal API instruction following (OpenAI) 31.6
- MultiChallenge 15
- COLLIE 42.5
- IFEval 74.5
- Multi-IF 57.2
- OpenAI MRCR 36.6
- GraphWalks 25
- MMMU 55.4
- MathVista 56.2
- CharXiv 40.5
- ComplexFuncBench 5.7
- tau-bench 18.3
Dropped since the last post: SWE-bench Verified
- Apr 14, 2025
- AIME 2024 49.6
- GPQA Diamond 65
- MMLU 87.5
- MMMLU 78.5
- SWE-bench Verified 23.6
- SWE-Lancer 13.1
- Aider Polyglot 34.7
- Internal API instruction following (OpenAI) 45.1
- MultiChallenge 35.8
- COLLIE 54.6
- IFEval 84.1
- Multi-IF 67
- OpenAI MRCR 47.2
- GraphWalks 61.7
- MMMU 72.7
- MathVista 73.1
- CharXiv 56.8
- ComplexFuncBench 49.3
- tau-bench 45.9
Dropped since the last post: FrontierMath, Codeforces rating, LiveBench, MATH-500, MGSM, SimpleQA
- Apr 14, 2025
- AIME 2024 80.8
- AIME 2025 63.6
- Omni-MATH 68.4
- LiveCodeBench v5 59.1
- SysBench 81.2
- IFEval 84.5
- Arena-Hard-Auto 90.6
- Berkeley Function Calling Leaderboard (BFCL) 70.5
- SuperGPQA 52.6
- GPQA Diamond 66.1
Dropped since the last post: GSM8K, MATH, BIG-Bench Hard, MMLU, C-Eval, CMMLU, MBPP, AGIEval
- Apr 14, 2025
- SWE-bench Verified 54.6
- Aider Polyglot 52.9
- MultiChallenge 38.3
- IFEval 87.4
- OpenAI MRCR 57.2
- GraphWalks 61.7
- Video-MME 72
- MMMU 74.8
- MathVista 72.2
- CharXiv 56.7
Dropped since the last post: SimpleQA, SimpleQA Hallucination Rate, GPQA Diamond, AIME 2024, MMMLU, SWE-Lancer
- Apr 10, 2025Kimi-VL-A3B-Thinking Moonshot AI Kimi-VL-A3B-Thinking model card (Hugging Face) and Kimi-VL Technical Report
- MATH-Vision (MathVision) 36.8
- MathVista 71.3
- MMMU 61.7
Dropped since the last post: MMLU, MMLU-Pro, BIG-Bench Hard, TriviaQA, HumanEval, MBPP, GSM8K, MATH, CMATH, C-Eval, CMMLU
Its technical report adds 0 evaluations 3 results in all
- Apr 10, 2025
- AIME 2025 74
- AIME 2024 86.7
- Beyond AIME 48
- GPQA Diamond 77.3
- SuperGPQA 62.1
- MMLU-Pro 87
- Codeforces rating 55
- LiveCodeBench v5 64.9
- Aider Polyglot 54.2
- SWE-bench Verified 47
- ARC-AGI 39.9
- SimpleQA 12.9
- COLLIE 73.1
- IFEval 87.4
Its technical report adds 0 evaluations 14 results in all
- Apr 8, 2025
- Apr 7, 2025
- GPQA Diamond 76
- AIME 2025 72.5
- Berkeley Function Calling Leaderboard (BFCL)
- LiveCodeBench v5 66.3
- IFEval 89.5
- MATH-500 97
- JudgeBench 79.1
Dropped since the last post: RoboCasa, RoboCasa GR1 Tabletop Tasks
Its technical report adds 2 evaluations 9 results in all
Llama-Nemotron: Efficient Reasoning ModelsCapabilities
- AIME 2024 80.8
- Arena-Hard-Auto 87
- Apr 5, 2025
- MMMU 69.4
- MMMU-Pro 52.2
- MathVista 70.7
- ChartQA 88.8
- DocVQA 94.4
- LiveCodeBench v5 32.8
- MMLU-Pro 74.3
- GPQA Diamond 57.2
- MGSM 90.6
- MTOB 42.2
- MMLU 79.6
- MATH 50.3
- MBPP 67.8
- TydiQA 31.5
Dropped since the last post: LiveCodeBench
Its model card adds 1 evaluation 14 results in all
- Apr 5, 2025 Llama 4 Maverick Meta Llama 4 model card (Hugging Face meta-llama/Llama-4-Maverick-17B-128E-Instruct)
- MMMU 73.4
- MMMU-Pro 59.6
- MathVista 73.7
- ChartQA 90
- DocVQA 94.4
- LiveCodeBench
- MMLU-Pro 80.5
- GPQA Diamond 69.8
- MGSM 92.3
- MTOB 54
- MMLU 85.5
- MATH 61.2
- MBPP 77.6
- TydiQA 31.7
Dropped since the last post: IFEval, HumanEval, Berkeley Function Calling Leaderboard (BFCL)
Its model card adds 0 evaluations 3 dropped 14 results in all
Llama 4 model card (MODEL_CARD.md, meta-llama/llama-models)Dropped since the lab’s previous model card: IFEval, HumanEval, Berkeley Function Calling Leaderboard (BFCL)
- Apr 5, 2025
- Mar 31, 2025
- ScreenSpot 90.9
- GroundUI 80.5
- Mar 27, 2025
- OmniBench 56.1
- LibriSpeech 2.6
- Common Voice 6.9
- FLEURS 3.55
- CoVoST2 34.7
- MMAU 65.6
- VoiceBench 74.1
- Seed-TTS-eval 2.32
- MMMU 59.2
- MMMU-Pro 36.6
- MathVista 67.9
- MATH-Vision (MathVision) 25
- MMBench-EN v1.1 81.8
- MM-Vet 66.8
- MMStar 64
- MME 2340
- MuirBench 59.2
- CRPE 76.5
- RealWorldQA 70.3
- MME-RealWorld 61.6
- MM-MT-Bench 6
- AI2D 83.2
- TextVQA 84.4
- DocVQA 95.2
- ChartQA 85.3
- OCRBench v2 57.8
- RefCOCO / RefCOCO+ / RefCOCOg 87.7
- ODinW-13 42.4
- Video-MME 72.4
- MVBench 70.3
- EgoSchema 68.6
- MMLU-Pro 47
- MMLU-Redux 71
- LiveBench 29.6
- GPQA Diamond 30.8
- MATH 71.5
- GSM8K 88.7
- HumanEval 78.7
- MBPP 73.2
- MultiPL-E 65.8
- LiveCodeBench 24.6
- WenetSpeech 6.8
Dropped since the last post: AIME 2024, LiveCodeBench v5, IFEval, Berkeley Function Calling Leaderboard (BFCL)
Its technical report adds 1 evaluation 1 new 7 dropped 43 results in all
Qwen2.5-Omni Technical ReportCapabilities
Dropped since the lab’s previous technical report: RULER, LV-Eval, LongBench-Chat, Passkey Retrieval, IFEval, Arena-Hard-Auto, MT-Bench
- Mar 25, 2025
- Humanity's Last Exam 18.8
- GPQA Diamond 84
- AIME 2025 86.7
- AIME 2024 92
- LiveCodeBench v5 70.4
- Aider Polyglot 74
- SWE-bench Verified 63.8
- SimpleQA 52.9
- MMMU 81.7
- Vibe-Eval 69.4
- MRCR (Michelangelo, original) 94.5
- Global MMLU Lite 89.8
Dropped since the last post: MMLU-Pro, Bird-SQL, FACTS Grounding, MATH, HiddenMath, CoVoST2, EgoSchema
Its model card adds 19 evaluations 14 new 7 dropped 31 results in all
Gemini 2.5 Pro Preview Model Card (updated April 28, 2025)Safety
Alignment
- GDM Stealth 20
- GDM Situational Awareness 18.2
Bio and chem risk
Cyber risk
AI R&D risk
Dropped since the lab’s previous model card: MMLU-Pro, Bird-SQL, FACTS Grounding, MATH, HiddenMath, CoVoST2, EgoSchema
- Mar 24, 2025
- MMLU-Pro 81.2
- GPQA Diamond 68.4
- AIME 2024 59.4
- LiveCodeBench 49.2
- MATH-500
Dropped since the last post: MMLU, MMLU-Redux, DROP, IFEval, SimpleQA, FRAMES, AlpacaEval 2.0, Arena-Hard-Auto, Codeforces rating, SWE-bench Verified, Aider Polyglot, CNMO 2024, CLUEWSC, C-Eval, Chinese SimpleQA
- Mar 21, 2025
- MMLU-Pro 87.2
- GPQA Diamond 69.3
- Chinese SimpleQA 67.9
- DROP 93.1
- ZebraLogic 79.6
- MATH-500 96.2
- AIME 2024 78.2
- LiveCodeBench 64.9
- C-Eval 91.8
- CMMLU 90
- Arena-Hard-Auto 91.9
- CFBench 81
- CELLO 76.4
- T-Eval 68.8
Dropped since the last post: MMLU, SimpleQA, BIG-Bench Hard, MATH, HumanEval, LiveBench, IFEval
- Mar 18, 2025 GR00T N1 NVIDIA GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (NVIDIA white paper)
- RoboCasa 32.1
- RoboCasa GR1 Tabletop Tasks 50
Dropped since the last post: Arena-Hard-Auto, MATH-500, AIME 2025, GPQA Diamond, IFEval, Berkeley Function Calling Leaderboard (BFCL), MBPP, MT-Bench
- Mar 18, 2025
- MATH-500 95.7
- AIME 2024 72.1
- AIME 2025 65.8
- CSAT Math 2025 94.5
- GPQA Diamond 66.1
- LiveCodeBench v5 59.5
Its technical report adds 2 evaluations 8 results in all
- Mar 18, 2025
- Arena-Hard-Auto 88.3
- MATH-500 96.6
- AIME 2025 58.4
- GPQA Diamond 66.7
- IFEval 89.2
- Berkeley Function Calling Leaderboard (BFCL)
- MBPP 91.3
- MT-Bench 9.17
Its technical report adds 3 evaluations 9 results in all
Llama-Nemotron: Efficient Reasoning ModelsCapabilities
- AIME 2024 67.5
- LiveCodeBench v5 45.5
- JudgeBench 69.7
- Mar 17, 2025
- MMLU 80.6
- MMLU-Pro 66.8
- MATH 69.3
- GPQA Main 44.4
- GPQA Diamond 46
- MBPP 74.7
- HumanEval 88.4
- SimpleQA 10.4
- MMMU 64
- MMMU-Pro 49.3
- MathVista 68.9
- ChartQA 86.2
- DocVQA 94.1
- AI2D 93.7
- MM-MT-Bench 7.3
- LongBench v2 37.2
- RULER 81.2
Dropped since the last post: MT-Bench, WildBench, Arena-Hard-Auto, IFEval, TriviaQA, ARC-Challenge, GSM8K, AGIEval
- Mar 16, 2025
- Mar 13, 2025
- Cohere enterprise human evaluation 51.7
- MMLU
- MATH
- IFEval
- tau-bench
- Berkeley Function Calling Leaderboard (BFCL)
- MBPP+
- RepoQA
Its technical report adds 35 evaluations 43 results in all
Command A: An Enterprise-Ready Large Language ModelCapabilities
- MMLU-Pro 69.6
- GPQA Diamond 50.8
- InFoBench 94.9
- ChatRAG Bench 72.9
- StrategyQA 76.7
- Bamboogle 76
- DROP 91.1
- HotpotQA 92.1
- NTREX-128 68.8
- FLORES-200 81.2
- m-ArenaHard
- m-tau-bench (Cohere)
- Language Confusion Benchmark 93
- AL-QASIDA
- LiveCodeBench v5 26.9
- BigCodeBench 45.4
- LBPP (Less Basic Python Problems) 51.5
- HumanEvalPack 76.2
- HumanEval 85.4
- HumanEval-COBOL (Cohere) 25.3
- SWE-bench Verified 26.8
- Aider Polyglot 14.7
- Spider 80.2
- Bird-SQL 59.5
- Multi-dialect SQL evaluation (Cohere) 55.3
- AIME 2024 23.3
- Enterprise generative evaluation (Cohere) 94.2
- Enterprise RAG evaluation (Cohere) 4.73
- RULER 95
- LongBench v2 43.4
- MGSM 90.1
- INCLUDE 74.3
- Mar 13, 2025
- AlpacaEval 2.0 42.8
- BIG-Bench Hard 70.6
- DROP 78
- GSM8K 87.6
- IFEval 85.6
- MATH 49.7
- MMLU 77.3
- Tulu 3 Safety Eval 85.9
- PopQA 37.5
- TruthfulQA 73.2
Dropped since the last post: HumanEval, HumanEval+
Its technical report adds 5 evaluations 5 new 2 dropped 15 results in all
2 OLMo 2 FuriousCapabilities
- AGIEval 68.3
- DeepMind Mathematics 34.7
- GPQA Diamond 35.9
- IFEval-OOD 33.1
- MMLU-Pro 52.7
Dropped since the lab’s previous technical report: HumanEval, HumanEval+
- Mar 12, 2025
- GPQA Diamond 42.4
- SimpleQA 10
- FACTS Grounding 74.9
- BIG-Bench Hard 87.6
- BBEH micro average 19.3
- IFEval 90.4
- HellaSwag 85.6
- BoolQ 82.4
- PIQA 83.3
- SIQA 54.9
- TriviaQA 85.5
- Natural Questions 36.1
- ARC-Challenge 70.6
- ARC-Easy 89
- WinoGrande 78.8
- DROP 77.2
- MMLU-Pro 67.5
- LiveCodeBench 29.7
- Bird-SQL 54.4
- MATH 89
- HiddenMath 60.3
- MBPP 74.4
- HumanEval 87.8
- Natural2Code 84.5
- GSM8K 95.9
- MMLU 78.6
- AGIEval 66.2
- Global MMLU Lite 75.7
- ECLeKTic
- WMT24++ 55.7
- MGSM 74.3
- Flores 48.8
- XQuAD 76.8
- IndicGenBench 63.4
- MMMU 64.9
- DocVQA 86.6
- InfographicVQA 70.6
- TextVQA 68.6
- AI2D 84.5
- ChartQA 78
- VQAv2 72.9
- MathVista 67.6
- COCO Captions 116
- RealWorldQA 53.9
- ReMI 44.8
- BLINK 39.6
- OK-VQA 60.2
- TallyQA 54.3
- SpatialSense VQA 59.4
- CountBench 68
Dropped since the last post: LiveCodeBench v5, MRCR (Michelangelo, original), CoVoST2, EgoSchema
Gemma 3 12B: Its technical report adds 4 evaluations 4 new 8 dropped 54 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Capabilities
- RULER 57.1
- MRCR (Michelangelo, original) 49.8
- Perception Test MCVQA 54.9
- ActivityNet-QA 50.4
Dropped since the lab’s previous technical report: LMArena (Elo), RealToxicityPrompts, CrowS-Pairs, BBQ (Bias Benchmark for QA), Winogender, TruthfulQA, WinoBias, ToxiGen
Gemma 3 1B: Its technical report adds 0 evaluations 21 dropped 33 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Dropped since the lab’s previous technical report: MMMU, RULER, MRCR (Michelangelo, original), DocVQA, InfographicVQA, TextVQA, AI2D, ChartQA, VQAv2, MathVista, Perception Test MCVQA, ActivityNet-QA, AGIEval, COCO Captions, RealWorldQA, ReMI, BLINK, OK-VQA, TallyQA, SpatialSense VQA, CountBench
Its technical report adds 5 evaluations 5 new 55 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Capabilities
- LMArena (Elo) 1338
- RULER 66
- MRCR (Michelangelo, original) 59.3
- Perception Test MCVQA 58.1
- ActivityNet-QA 52.8
Gemma 3 4B: Its technical report adds 4 evaluations 1 dropped 54 results in all
Gemma 3 Technical Report (arXiv 2503.19786v1)Capabilities
- RULER 46.8
- MRCR (Michelangelo, original) 44.6
- Perception Test MCVQA 50.6
- ActivityNet-QA 46.3
Dropped since the lab’s previous technical report: LMArena (Elo)
- Mar 10, 2025
- Mar 6, 2025
- Mar 4, 2025 Aya Vision 32B Cohere Cohere For AI Launches Aya Vision, a Multilingual Multimodal AI Model (Cohere blog)
- AyaVisionBench 64
- m-WildVision 72
Its technical report adds 11 evaluations 13 results in all
Aya Vision: Advancing the Frontier of Multilingual MultimodalityCapabilities
- MaXM 62.3
- xMMMU 45.1
- CVQA 74.1
- MTVQA-Test 23.5
- Kaleidoscope 41.7
- xChatBench 70.1
- m-ArenaHard
- Global MMLU Lite 63.6
- MGSM 79.5
- Flores 37.8
- IFEval 78.5
- Feb 27, 2025
OpenVLA-OFT Stanford University Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT) - LIBERO 97.1
- Feb 27, 2025
- MMLU 89.5
- MMLU-Pro 79
- GPQA Diamond 57.5
- SimpleQA 22.8
- Chinese SimpleQA 70.8
- BIG-Bench Hard 92.2
- DROP 91.5
- ZebraLogic 46
- MATH 89.7
- AIME 2024 43.3
- HumanEval 91
- LiveCodeBench 32
- C-Eval 90.9
- CMMLU 90.8
- LiveBench 61
- Arena-Hard-Auto 88.6
- IFEval 88.6
- Feb 27, 2025
- SimpleQA 62.5
- SimpleQA Hallucination Rate 37.1
- GPQA Diamond 71.4
- AIME 2024 36.7
- MMMLU 85.1
- MMMU 74.4
- SWE-Lancer 32.6
- SWE-bench Verified 38
Dropped since the last post: Humanity's Last Exam, GAIA
Its system card adds 33 evaluations 7 new 4 dropped 36 results in all
OpenAI GPT-4.5 System CardHallucination
- PersonQA 78
Safety
Robustness
Bio and chem risk
Cyber risk
Autonomy
Dropped since the lab’s previous system card: Deep research prompt injection evaluation, Red Teaming Network pairwise safety comparison, Personal data evaluations, ChangeMyView
- Feb 26, 2025
- MMMU 55.1
- MMBench 86.7
- MMMU-Pro 38.5
- ScienceQA 97.5
- MathVista 62.4
- Inter-GPS (Geometry3K) 48.6
- AI2D 82.3
- ChartQA 81.4
- DocVQA 93.2
- InfographicVQA 72.7
- TextVQA 75.6
- OCRBench 84.4
- POPE 85.6
- BLINK 61.3
- Video-MME 55
Dropped since the last post: Arena-Hard-Auto, BIG-Bench Hard, MMLU, MMLU-Pro, ARC-Challenge, BoolQ, GPQA Diamond, HellaSwag, OpenBookQA, PIQA, SIQA, TruthfulQA, WinoGrande, MMMLU, MGSM, GSM8K, MATH
Its technical report adds 12 evaluations 10 new 30 dropped 27 results in all
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsCapabilities
- Common Voice 6.8
- FLEURS 4
- Open ASR Leaderboard 6.14
- CoVoST2 39.3
- Golden3 speech summarization (Microsoft) 6.28
- AIR-Bench (Audio Instruction Benchmark) 6.98
- MMAU 55.6
Safety
Dropped since the lab’s previous technical report: BIG-Bench Hard, MMLU, MMLU-Pro, ARC-Challenge, BoolQ, GPQA Diamond, HellaSwag, OpenBookQA, PIQA, SIQA, TruthfulQA, WinoGrande, MMMLU, MGSM, GSM8K, MATH, Qasper, SQuALITY, IFEval, Berkeley Function Calling Leaderboard (BFCL), HumanEval, MBPP, BigCodeBench, HumanEval+, LiveCodeBench, LiveBench, MBPP+, Spider, AIME 2024, MATH-500
- Feb 26, 2025
- Arena-Hard-Auto 32.8
- BIG-Bench Hard 70.4
- MMLU 67.3
- MMLU-Pro 52.8
- ARC-Challenge 83.7
- BoolQ 81.2
- GPQA Diamond 30.4
- HellaSwag 69.1
- OpenBookQA 79.2
- PIQA 77.6
- SIQA 72.5
- TruthfulQA 66.4
- WinoGrande 67
- MMMLU 49.3
- MGSM 63.9
- GSM8K 88.6
- MATH 64
Dropped since the last post: HumanEval, SimpleQA, DROP, LiveBench, IFEval, PhiBench
Its technical report adds 16 evaluations 12 new 11 dropped 32 results in all
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsCapabilities
- Qasper 40.4
- SQuALITY 22.8
- IFEval 70.1
- Berkeley Function Calling Leaderboard (BFCL) 70.3
- HumanEval 74.4
- MBPP 65.3
- BigCodeBench 43
- HumanEval+ 68.3
- LiveCodeBench 19.9
- LiveBench
- MBPP+ 63.8
- Spider 42.2
- AIME 2024 10
- MATH-500 71.8
Safety
Dropped since the lab’s previous technical report: SimpleQA, DROP, Arena-Hard-Auto, PhiBench, AMC 10/12 (November 2024), HELMET, Microsoft RAI benchmark: Grounding, Microsoft RAI benchmark: Third Party Harm (DR-1), Microsoft RAI benchmark: Harmful Content Continuation (DR-3), Microsoft RAI benchmark: Harmful Content Summarization (DR-3), Microsoft RAI benchmark: Jailbreak (DR-1)
- Feb 26, 2025
- Arena-Hard-Auto 55.3
- AlpacaEval 2.0 61.2
- MMLU 66.8
- PopQA 28
- TruthfulQA 66.9
- BIG-Bench Hard 64.8
- DROP 51
- GSM8K 81.7
- HumanEval 89.3
- HumanEval+ 85.7
- IFEval 74.3
- AttaQ 85.4
- Feb 26, 2025Mercury Coder Small Inception Labs Introducing Mercury, the first commercial-scale diffusion large language model (Inception blog)
- HumanEval 90
- MBPP 76.6
- EvalPlus 80.4
- MultiPL-E 76.2
- LiveCodeBench 25
- BigCodeBench 45.5
Its technical report adds 1 evaluation 7 results in all
Mercury: Ultra-Fast Language Models Based on DiffusionCapabilities
- HumanEval Infilling 84.8
- Feb 24, 2025
- GPQA Diamond 84.8
- SWE-bench Verified 70.3
- tau-bench 69.8
- MMMLU 86.1
- MMMU 75
- IFEval 93.2
- MATH-500 96.2
- AIME 2024 80
Dropped since the last post: OSWorld-Verified, MathVista, AI2D, ChartQA, DocVQA, MMLU, MMLU-Pro, MATH, HumanEval, MGSM, DROP, BIG-Bench Hard
Its system card adds 28 evaluations 28 results in all
Claude 3.7 Sonnet System CardSafety
Robustness
Bias
Alignment
Monitorability
Bio and chem risk
Cyber risk
- Feb 22, 2025
- Feb 19, 2025
- AIME 2025 90.8
- AIME 2024 95.8
- GPQA Diamond 84
- LiveCodeBench
- MMLU-Pro 78.9
- LOFT 83.1
- SimpleQA 21.7
- MMMU 69.4
- EgoSchema 74.3
- Feb 19, 2025
- AIME 2025 93.3
- AIME 2024 93.3
- GPQA Diamond 84.6
- LiveCodeBench
- MMMU 78
- MMLU-Pro 79.9
- LOFT 83.3
- SimpleQA 43.6
- EgoSchema 74.5
- LMArena (Elo) 1402
Dropped since the last post: MMLU, MATH, HumanEval, MathVista, DocVQA
- Feb 17, 2025
Dropped since the last post: HumanEval, MBPP, CRUXEval, LiveCodeBench, RepoBench, Spider, CanItEdit, HumanEval Multi
- Feb 11, 2025
- Answer Factuality (Perplexity) 85.1
- Readability (Perplexity) 85.9
- IFEval 86.8
- MMLU 87.1
Dropped since the last post: SimpleQA
- Feb 5, 2025
- MMLU-Pro 71.6
- LiveCodeBench v5 28.9
- Bird-SQL 57.4
- GPQA Diamond 51.5
- SimpleQA 21.7
- FACTS Grounding 83.6
- Global MMLU Lite 78.2
- MATH 86.8
- HiddenMath 55.3
- MRCR (Michelangelo, original) 58
- MMMU 68
- CoVoST2 38.4
- EgoSchema 67.2
Dropped since the last post: MMLU, HellaSwag, PIQA, SIQA, BoolQ, WinoGrande, ARC-Easy, ARC-Challenge, TriviaQA, Natural Questions, HumanEval, MBPP, GSM8K, AGIEval, DROP, BIG-Bench Hard
Its model card adds 5 evaluations 18 results in all
- Feb 5, 2025
- MMLU-Pro 77.6
- LiveCodeBench v5 34.5
- Bird-SQL 58.7
- GPQA Diamond 60.1
- SimpleQA 29.9
- FACTS Grounding 84.6
- Global MMLU Lite 83.4
- MATH 90.9
- HiddenMath 63.5
- MRCR (Michelangelo, original) 70.5
- MMMU 71.7
- CoVoST2 39
- EgoSchema 71.1
Its model card adds 5 evaluations 18 results in all
- Feb 5, 2025
- MMLU-Pro 79.1
- LiveCodeBench v5 36
- Bird-SQL 59.3
- GPQA Diamond 64.7
- SimpleQA 44.3
- FACTS Grounding 82.8
- Global MMLU Lite 86.5
- MATH 91.8
- HiddenMath 65.2
- MRCR (Michelangelo, original) 74.7
- MMMU 72.7
- CoVoST2 40.6
- EgoSchema 71.9
Dropped since the last post: AIME 2024
- Feb 4, 2025π0-FAST Physical Intelligence openpi LIBERO example results, released with Open Sourcing π0 (GitHub)
- LIBERO 85.5
- Feb 4, 2025
- LIBERO 94.2
- Feb 2, 2025
- Humanity's Last Exam 26.6
- GAIA 67.4
Dropped since the last post: OSWorld, WebArena, WebVoyager
Its system card adds 32 evaluations 28 new 9 dropped 32 results in all
Deep Research System CardHallucination
- PersonQA 86
Safety
Robustness
Bio and chem risk
Cyber risk
Autonomy
Persuasion
- ChangeMyView 84.1
- MakeMePay 36
- MakeMeSay 24
Dropped since the lab’s previous system card: Production Jailbreaks, Jailbreak Augmented Examples, Human Sourced Jailbreaks, Agentic harmful task refusals, User Confirmations During Computer Use, Proactive refusals of high-risk tasks, Operator prompt injection scenarios, Biotool and Wet Lab Actions, OSWorld
- Jan 31, 2025
- AIME 2024 87.3
- GPQA Diamond 79.7
- FrontierMath 9.2
- Codeforces rating 2130
- SWE-bench Verified 49.3
- LiveBench 84.6
- MMLU 86.9
- MATH-500 97.9
- MGSM 92
- SimpleQA 13.8
Dropped since the last post: DROP, MATH, HumanEval, MMMU, MathVista
Its system card adds 37 evaluations 38 results in all
OpenAI o3-mini System CardCapabilities
Hallucination
- PersonQA 21.7
Safety
Robustness
Bio and chem risk
- Long-form Biological Risk Questions
- Expert comparisons on biothreat information
- Biotool and Wet Lab Actions
- Multimodal Troubleshooting Virology 53
- BioLP-bench 41
- ProtocolQA Open-Ended 23
- Tacit Knowledge and Troubleshooting 58
- Tacit Knowledge Brainstorm (Open-Ended) 51
- Structured expert probing campaign: radiological and nuclear
- Contextual Nuclear Knowledge 73
- Radiological and Nuclear Expert Knowledge 60
Cyber risk
Autonomy
Persuasion
- ChangeMyView 82.9
- Political Persuasion Parallel Generation 35.5
- MakeMePay 1
- MakeMeSay 62
- Jan 30, 2025
- MMLU 87
- PopQA 55.5
- BIG-Bench Hard 88.6
- MATH 67.3
- GSM8K 95.5
- HumanEval 95.9
- HumanEval+ 92.9
- IFEval 86
- AlpacaEval 2.0 51.4
- Tulu 3 Safety Eval 86.7
Its technical report adds 0 evaluations 10 results in all
- Jan 30, 2025 Mistral Small 3 Mistral AI Mistral Small 3 (launch post) + HF model cards Mistral-Small-24B-Instruct-2501 / Base-2501
- MMLU-Pro 66.3
- GPQA Diamond 45.3
- HumanEval 84.8
- MATH 70.6
- MT-Bench 8.35
- WildBench 52.3
- Arena-Hard-Auto 87.3
- IFEval 82.9
- MMLU 80.7
- TriviaQA 80.3
- ARC-Challenge 91.3
- MBPP 69.6
- GSM8K 80.7
- AGIEval 65.8
Dropped since the last post: HellaSwag, WinoGrande, PIQA, ARC-Easy, Natural Questions
- Jan 28, 2025
- MMMU 70.2
- MMMU-Pro 51.1
- MathVista 74.8
- MATH-Vision (MathVision) 38.1
- HallusionBench 55.2
- MMBench-EN v1.1 88
- AI2D 88.4
- ChartQA 89.5
- DocVQA 96.4
- MMStar 70.8
- MM-Vet 76.2
- OCRBench 88.5
- OCRBench v2 61.5
- CC-OCR 79.8
- Video-MME 79.1
- MVBench 70.4
- MMBench-Video 2.02
- LVBench 47.3
- EgoSchema 76.2
- Perception Test MCVQA 73.2
- MLVU 74.6
- TempCompass 74.8
- ScreenSpot 87.1
- ScreenSpot-Pro 43.6
- Android in the Zoo (AitZ) 83.2
- AndroidControl 67.4
- AndroidWorld 35
- MobileMiniWoB++ 68
- OSWorld 8.83
Dropped since the last post: Arena-Hard-Auto, MMLU-Pro, GPQA Diamond, LiveCodeBench, LiveBench, MMLU, BIG-Bench Hard, C-Eval, CMMLU, HumanEval, MBPP, CRUXEval, GSM8K, MATH
Its technical report adds 34 evaluations 25 new 18 dropped 62 results in all
Qwen2.5-VL Technical ReportCapabilities
- MathVerse 57.6
- MEGA-Bench 51.3
- MMBench 88.6
- MME 2448
- MuirBench 70.7
- BLINK 64.4
- CRPE 79.2
- MTVQA-Test 31.7
- RealWorldQA 75.7
- MME-RealWorld 63.2
- MM-MT-Bench 7.6
- MMLU-Pro 71.2
- MMLU-Redux 85.9
- LiveBench 57
- GPQA Diamond 49
- MATH 83
- GSM8K 95.3
- HumanEval 87.8
- MultiPL-E 79.5
- IFEval 86.3
- OmniDocBench
- TextVQA 83.5
- InfographicVQA 87.3
- CharXiv 49.7
- CharXiv (DQ) 87.4
- SEED-Bench-2-Plus 73
- VCR (Visual Caption Restoration) 79.8
- RefCOCO / RefCOCO+ / RefCOCOg
- ODinW-13 43.1
- CountBench 93.6
- Video-MMMU 60.2
- MMVU 62.9
- LongVideoBench 60.7
- Charades-STA 50.9
Dropped since the lab’s previous technical report: MBPP, LiveCodeBench, Arena-Hard-Auto, MT-Bench, Multilingual IFEval, AMMLU, JMMLU, KMMLU, IndoMMLU, TurkishMMLU, Okapi MMLU, MGSM, BLEnD, Qwen in-house English automatic evaluation, Qwen in-house Chinese automatic evaluation, RULER, LV-Eval, LongBench-Chat
- Jan 28, 2025
- Arena-Hard-Auto
- MMLU-Pro
- GPQA Diamond
- LiveCodeBench
- LiveBench
- MMLU
- BIG-Bench Hard
- C-Eval
- CMMLU
- HumanEval
- MBPP
- CRUXEval
- GSM8K
- MATH
Dropped since the last post: MMLU-Redux, MultiPL-E, IFEval, AlignBench, MT-Bench, Multilingual IFEval, AMMLU, JMMLU, KMMLU, IndoMMLU, TurkishMMLU, Okapi MMLU, MGSM, BLEnD, ARC-Challenge, TruthfulQA, WinoGrande, HellaSwag, TheoremQA, MMLU-stem, Multilingual Exams, Multilingual Understanding, Flores
- Jan 27, 2025
Qwen2.5-14B-Instruct-1M Qwen Qwen2.5-1M: Deploy Your Own Qwen with Context Length up to 1M Tokens (blog) - RULER 95.7
- LV-Eval 49.8
- LongBench-Chat 8.76
- Passkey Retrieval
Its technical report adds 13 evaluations 17 results in all
Qwen2.5-1M Technical ReportCapabilities
- MMLU-Pro 63.3
- MMLU-Redux 80.7
- LiveBench 44.6
- GPQA Diamond 39.9
- MATH 79.5
- GSM8K 94.8
- HumanEval 88.4
- MBPP 80.2
- MultiPL-E 77.1
- LiveCodeBench 38.6
- IFEval 84.3
- Arena-Hard-Auto 70.2
- MT-Bench 8.89
- Jan 27, 2025 Janus-Pro-7B DeepSeek Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling (GitHub README and tech report)
- POPE 87.4
- MME 1567.1
- MMBench 79.2
- SEED-Bench 72.1
- GQA 62
- MMMU 41
- MM-Vet 50
- GenEval 80
- DPG-Bench 84.2
Its technical report adds 0 evaluations 9 results in all
- Jan 23, 2025
- OSWorld 38.1
- WebArena 58.1
- WebVoyager 87
Dropped since the last post: GPQA Diamond, MMLU, SWE-bench Verified, LiveBench, MATH-500, AIME 2024, MGSM, MMMU, MathVista, SimpleQA, tau-bench
Its system card adds 12 evaluations 4 new 39 dropped 13 results in all
Operator System CardSafety
Robustness
Bio and chem risk
Dropped since the lab’s previous system card: WildChat (toxic conversations), XSTest, Image input safety evaluations, Regurgitation Evaluations, SimpleQA, SimpleQA Hallucination Rate, PersonQA, BBQ (Bias Benchmark for QA), Discrim-Eval, Instruction Hierarchy Evaluation, CoT Deception Monitoring, Red Teaming Network pairwise safety comparison, Gray Swan Jailbreak Arena, Apollo Research in-context scheming evaluations, METR general autonomy task suite, Capture the Flag (High School), Capture the Flag (Collegiate), Capture the Flag (Professional), Long-form Biological Risk Questions, Expert comparisons on biothreat information, Multimodal Troubleshooting Virology, ProtocolQA Open-Ended, BioLP-bench, Tacit Knowledge and Troubleshooting, Tacit Knowledge Brainstorm (Open-Ended), Structured expert probing campaign: chem-bio novel design, Structured expert probing campaign: radiological and nuclear, Contextual Nuclear Knowledge, Radiological and Nuclear Expert Knowledge, ChangeMyView, Political Persuasion Parallel Generation, MakeMePay, MakeMeSay, OpenAI Research Engineer Interview: Multiple Choice, OpenAI Research Engineer Interview: Coding, SWE-bench Verified, MLE-bench, MMMLU, QuantBench
- Jan 21, 2025
- AIME 2024 73.3
- GPQA Diamond 74.2
- MMMU 75.4
Dropped since the last post: MMLU-Pro, Natural2Code, Bird-SQL, LiveCodeBench, FACTS Grounding, MATH, HiddenMath, MRCR (Michelangelo, original), Vibe-Eval, CoVoST2, EgoSchema
- Jan 21, 2025
- Jan 20, 2025Kimi k1.5 Moonshot AI Kimi k1.5: Scaling Reinforcement Learning with LLMs (tech report, arXiv 2501.12599, Tables 2 and 3)
- MATH-500 96.2
- AIME 2024 77.5
- Codeforces rating 94
- LiveCodeBench 62.5
- MathVista 74.9
- MMMU 70
- MATH-Vision (MathVision) 38.6
- MMLU 87.4
- IFEval 87.2
- CLUEWSC 91.7
- C-Eval 88.3
- HumanEval Multi 81.5
Its technical report adds 0 evaluations 12 results in all
- Jan 20, 2025
- MMLU 90.8
- MMLU-Redux 92.9
- MMLU-Pro 84
- DROP 92.2
- IFEval 83.3
- GPQA Diamond 71.5
- SimpleQA 30.1
- FRAMES 82.5
- AlpacaEval 2.0 87.6
- Arena-Hard-Auto 92.3
- LiveCodeBench 65.9
- Codeforces rating 2029
- SWE-bench Verified 49.2
- Aider Polyglot 53.3
- AIME 2024 79.8
- MATH-500 97.3
- CNMO 2024 78.8
- CLUEWSC 92.8
- C-Eval 91.8
- Chinese SimpleQA 63.7
Dropped since the last post: LongBench v2, HumanEval Multi, Aider-Edit, Pile-test, BIG-Bench Hard, ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande, RACE-M, RACE-H, TriviaQA, Natural Questions, AGIEval, HumanEval, MBPP, CRUXEval, GSM8K, MATH, MGSM, CMATH, CMMLU, CMRC, C3, CCPM, MMMLU, Needle In A Haystack
DeepSeek-R1-Distill-Llama-70B: Its technical report adds 0 evaluations 20 dropped 5 results in all
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948)Dropped since the lab’s previous technical report: MMLU, MMLU-Redux, MMLU-Pro, DROP, IFEval, SimpleQA, FRAMES, LongBench v2, HumanEval Multi, SWE-bench Verified, Aider-Edit, Aider Polyglot, CNMO 2024, CLUEWSC, C-Eval, Chinese SimpleQA, Arena-Hard-Auto, AlpacaEval 2.0, RewardBench, Needle In A Haystack
DeepSeek-R1-Distill-Qwen-1.5B: Its technical report adds 0 evaluations 5 results in all
DeepSeek-R1-Distill-Qwen-14B: Its technical report adds 0 evaluations 5 results in all
DeepSeek-R1-Distill-Qwen-32B: Its technical report adds 0 evaluations 5 results in all
Its technical report adds 12 evaluations 12 new 1 dropped 31 results in all
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948)Capabilities
- LiveCodeBench v5 65.9
- AMC 12 2024 143.7
- AIME 2025
Safety
- SimpleSafetyTests 97.5
- Anthropic Red Team 96.2
- XSTest 95.3
- Do-Not-Answer 94.8
- HarmBench 35
- DeepSeek in-house safety benchmark 25.2
- DeepSeek multilingual safety evaluation 74.2
Robustness
Bias
Dropped since the lab’s previous technical report: LiveCodeBench
- Jan 15, 2025
- MMLU 88.5
- MMLU-Pro 75.7
- SimpleQA 23.7
- Chinese SimpleQA 67.4
- IFEval 89.1
- Arena-Hard-Auto 89.1
- GPQA Diamond 54.4
- DROP 87.8
- GSM8K 94.8
- MATH 77.4
- MBPP 71.7
- HumanEval 86.9
- RULER 91
- LongBench v2 56.5
- MTOB 51.7
Its technical report adds 11 evaluations 25 results in all
MiniMax-01: Scaling Foundation Models with Lightning AttentionCapabilities
- EvalPlus 71.7
- Needle In A Haystack
- MR-NIAH (Multi-Round Needles-In-A-Haystack)
- General Assistant (MiniMax in-house) 73.9
- Hard Capability (MiniMax in-house) 64.8
- Creative Writing (MiniMax in-house) 81.3
- Knowledge Q&A (MiniMax in-house) 78.6
- Instruction Following (MiniMax in-house) 46.3
- Coding (MiniMax in-house) 90.2
- Long Context (MiniMax in-house) 93.8
Safety
- Jan 13, 2025
- HumanEval 86.6
- MBPP 80.2
- CRUXEval 55.5
- LiveCodeBench 37.9
- RepoBench 38
- Spider 66.5
- CanItEdit 50.5
- HumanEval Multi 71.4
Dropped since the last post: MMLU, MT-Bench, WildBench, Arena-Hard-Auto, MultiPL-E, GSM8K, MATH
- Jan 13, 2025
- CMMLU 83.1
- MMLU 76.6
- MMLU-Pro 57.6
- GPQA Diamond 37.4
- DROP 83.1
- HellaSwag 91.2
- KOR-Bench 56.4
- MATH-500 83
- AIME 2024 20
- LiveCodeBench 17.8
- HumanEval 82.3
- IFEval 79.3
- RULER 87.9
- AlpacaEval 2.0 51.1
- WildBench 33.1
- MT-Bench-101 8.59
2024
- Dec 26, 2024
- MMLU 88.5
- MMLU-Redux 89.1
- MMLU-Pro 75.9
- DROP 91.6
- IFEval 86.1
- GPQA Diamond 59.1
- SimpleQA 24.9
- FRAMES 73.3
- LongBench v2 48.7
- HumanEval Multi 82.6
- LiveCodeBench
- Codeforces rating 51.6
- SWE-bench Verified 42
- Aider-Edit 79.7
- Aider Polyglot 49.6
- AIME 2024 39.2
- MATH-500 90.2
- CNMO 2024 43.2
- CLUEWSC 90.9
- C-Eval 86.5
- Chinese SimpleQA 64.8
- Arena-Hard-Auto 85.5
- AlpacaEval 2.0 70
- Pile-test 0.548
- BIG-Bench Hard 87.5
- ARC-Easy 98.9
- ARC-Challenge 95.3
- HellaSwag 88.9
- PIQA 84.7
- WinoGrande 84.9
- RACE-M 67.1
- RACE-H 51.3
- TriviaQA 82.9
- Natural Questions 40
- AGIEval 79.6
- HumanEval 65.2
- MBPP 75.4
- CRUXEval 69.8
- GSM8K 89.3
- MATH 61.6
- MGSM 79.8
- CMATH 90.7
- CMMLU 88.8
- CMRC 76.3
- C3 78.6
- CCPM 92
- MMMLU 79.4
- Needle In A Haystack
Dropped since the last post: AlignBench, MT-Bench, DS-FIM-Eval, DS-Arena-Code
Its technical report adds 1 evaluation 1 new 16 dropped 25 results in all
DeepSeek-V3 Technical Report (arXiv 2412.19437)Capabilities
- RewardBench 87
Dropped since the lab’s previous technical report: TriviaQA, Natural Questions, ARC-Easy, ARC-Challenge, BIG-Bench Hard, AGIEval, HumanEval, MBPP, CRUXEval, GSM8K, MATH, CMATH, CMMLU, MT-Bench, AlignBench, SuperCLUE-Math6
- Dec 17, 2024
- GPQA Diamond 75.7
- MMLU 91.8
- SWE-bench Verified 48.9
- LiveBench 76.6
- MATH-500 96.4
- AIME 2024 79.2
- MGSM 89.3
- MMMU 77.3
- MathVista 71
- SimpleQA 42.6
- tau-bench 63.9
Dropped since the last post: Codeforces rating
Its system card adds 45 evaluations 15 new 7 dropped 47 results in all
OpenAI o1 System CardCapabilities
- MMMLU
- QuantBench 60
Hallucination
Safety
Robustness
Bio and chem risk
- Long-form Biological Risk Questions
- Expert comparisons on biothreat information
- Biotool and Wet Lab Actions
- Multimodal Troubleshooting Virology 59
- ProtocolQA Open-Ended 24
- BioLP-bench 33
- Tacit Knowledge and Troubleshooting 66
- Tacit Knowledge Brainstorm (Open-Ended) 51
- Structured expert probing campaign: chem-bio novel design
- Structured expert probing campaign: radiological and nuclear
- Contextual Nuclear Knowledge 74
- Radiological and Nuclear Expert Knowledge 70
Cyber risk
AI R&D risk
- MLE-bench 14
Autonomy
Persuasion
- ChangeMyView 89.1
- Political Persuasion Parallel Generation 47.1
- MakeMePay 27
- MakeMeSay 42
Dropped since the lab’s previous system card: External red team jailbreaks, BirthdayFacts, Open Ended Questions (hallucination), Apollo Research strategic deception evaluations, RE-Bench, LAB-Bench ProtocolQA, LAB-Bench CloningScenarios
- Dec 12, 2024
- MMLU 84.8
- GPQA Diamond 56.1
- MATH 80.4
- HumanEval 82.6
- MGSM 80.6
- SimpleQA 3
- DROP 75.5
- MMLU-Pro 70.4
- Arena-Hard-Auto 75.4
- LiveBench 47.6
- IFEval 63
- PhiBench 56.2
Dropped since the last post: BIG-Bench Hard, ARC-Challenge, BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, TruthfulQA, WinoGrande, MMMLU, GSM8K, Qasper, SQuALITY, MBPP
Its technical report adds 8 evaluations 4 new 19 dropped 20 results in all
Phi-4 Technical ReportCapabilities
- HumanEval+ 82.8
- AMC 10/12 (November 2024) 91.8
- HELMET
Hallucination
Safety
Robustness
Dropped since the lab’s previous technical report: HellaSwag, ANLI, GSM8K, MedQA, AGIEval, TriviaQA, ARC-Challenge, ARC-Easy, PIQA, SIQA, BIG-Bench Hard, WinoGrande, OpenBookQA, BoolQ, CommonSenseQA, TruthfulQA, MBPP, MT-Bench, Microsoft RAI benchmark: Ungroundedness
- Dec 11, 2024
- MMLU-Pro 76.4
- Natural2Code 92.9
- Bird-SQL 56.9
- LiveCodeBench 35.1
- FACTS Grounding 83.6
- MATH 89.7
- HiddenMath 63
- GPQA Diamond 62.1
- MRCR (Michelangelo, original) 69.2
- MMMU 70.7
- Vibe-Eval 56.3
- CoVoST2 39.2
- EgoSchema 71.5
Dropped since the last post: PhysicsFinals, Functional MATH, AMC, GSM8K, BIG-Bench Hard, DROP, MMLU, HellaSwag, HumanEval, WMT23, MGSM, MathVista, AI2D, ChemicalDiagramQA, ChartQA, BetterChartQA, DocVQA, DUDE, TAT-DQA, InfographicVQA, TextVQA, VQAv2, V*, BLINK, RealWorldQA, VATEX, VATEX ZH, YouCook2, ActivityNet-QA, OpenEQA
- Dec 6, 2024
- MMLU 86
- MMLU-Pro 68.9
- IFEval 92.1
- GPQA Diamond 50.5
- HumanEval 88.4
- MBPP 87.6
- MATH 77
- Berkeley Function Calling Leaderboard (BFCL)
- MGSM 91.1
Dropped since the last post: GSM8K, ARC-Challenge, Nexus, ZeroSCROLLS/QuALITY, InfiniteBench/En.MC, Needle In A Haystack
Its model card adds 0 evaluations 16 dropped 9 results in all
Llama 3.3 model card (MODEL_CARD.md, meta-llama/llama-models)Dropped since the lab’s previous model card: ARC-Challenge, MultiPL-E, GSM8K, API-Bank, Gorilla APIBench, Nexus, Multilingual MMLU (Meta), AGIEval, CommonSenseQA, WinoGrande, BIG-Bench Hard, TriviaQA, SQuAD, QuAC, BoolQ, DROP
- Oct 22, 2024Claude 3.5 Sonnet (October 2024) Anthropic Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet
- OSWorld-Verified 22
- SWE-bench Verified 49
- tau-bench 57.6
- MMMU 70.4
- MathVista 70.7
- AI2D 95.3
- ChartQA 90.8
- DocVQA 94.2
- GPQA Diamond 65
- MMLU 90.5
- MMLU-Pro 78
- MATH 78.3
- HumanEval 93.7
- MGSM 92.5
- DROP 88.3
- BIG-Bench Hard 93.2
- AIME 2024 27.6
- IFEval 90.2
Dropped since the last post: GSM8K
Claude 3.5 Haiku: Its model card adds 7 evaluations 7 dropped 19 results in all
Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 SonnetSafety
Dropped since the lab’s previous model card: GSM8K, MMMU, MathVista, AI2D, ChartQA, DocVQA, Needle In A Haystack
Its model card adds 8 evaluations 1 new 25 results in all
- Sep 19, 2024
- MMLU-Pro 71.1
- MMLU-Redux 86.8
- GPQA Diamond 49
- MATH 83.1
- GSM8K 95.8
- HumanEval 86.6
- MBPP 88.2
- MultiPL-E 75.1
- LiveCodeBench 55.5
- LiveBench 52.3
- IFEval 84.1
- Arena-Hard-Auto 81.2
- AlignBench 8.16
- MT-Bench 9.35
- Multilingual IFEval 87
- AMMLU 72.4
- JMMLU 80.6
- KMMLU 62
- IndoMMLU 69.3
- TurkishMMLU 76.1
- Okapi MMLU 80
- MGSM 88.2
- BLEnD 32.5
- MMLU 86.1
- BIG-Bench Hard 86.3
- ARC-Challenge 72.4
- TruthfulQA 60.4
- WinoGrande 83.9
- HellaSwag 87.6
- TheoremQA 42.4
- MMLU-stem 82.7
- Multilingual Exams 78.7
- Multilingual Understanding 89.6
- Flores 39
Dropped since the last post: EvalPlus, C-Eval, CMMLU, MMMLU, Multilingual human evaluation
Qwen-Turbo: Its technical report adds 6 evaluations 2 new 7 dropped 19 results in all
Qwen2.5 Technical Report (arXiv 2412.15115)Capabilities
Dropped since the lab’s previous technical report: MMLU, TheoremQA, MixEval, AlignBench, NeedleBench, Multilingual human evaluation, Qwen2 multilingual safety evaluation
Qwen2.5-14B: Its technical report adds 5 evaluations 1 dropped 27 results in all
Qwen2.5 Technical Report (arXiv 2412.15115)Capabilities
Dropped since the lab’s previous technical report: Needle In A Haystack
Qwen2.5-32B: Its technical report adds 5 evaluations 27 results in all
Its technical report adds 5 evaluations 27 results in all
Qwen2.5-7B: Its technical report adds 5 evaluations 27 results in all
- Sep 12, 2024
- AIME 2024 56.7
- Codeforces rating 1258
- GPQA Diamond 78.3
- MATH-500 85.5
- MMLU 90.8
- MMMU
- MathVista
Dropped since the last post: MATH, HumanEval, MGSM, DROP, M3Exam, AI2D, ChartQA, DocVQA, ActivityNet-QA, EgoSchema
o1-mini: Its system card adds 37 evaluations 26 new 23 dropped 37 results in all
OpenAI o1 System Card (o1-preview and o1-mini)Capabilities
- MMMLU
- QuantBench 48
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
Autonomy
Persuasion
- ChangeMyView 69.5
- Political Persuasion Parallel Generation 46.8
- MakeMePay 0.7
- MakeMeSay 38.9
Dropped since the lab’s previous system card: Unauthorized voice generation, Speaker identification, Disparate performance on voice inputs, Ungrounded inference and sensitive trait attribution, Biological threat creation uplift study, Text persuasion study on political opinions, Voice persuasion study on party preferences, Select machine learning engineering tasks from METR, Situational Awareness Dataset (SAD), MedQA, MedQA USMLE (5 options), MedQA Taiwan, MedQA Mainland China, MMLU Clinical Knowledge, MMLU Medical Genetics, MMLU Anatomy, MMLU Professional Medicine, MMLU College Biology, MMLU College Medicine, MedMCQA, Uhura ARC-Easy, Uhura TruthfulQA, Uhura-Eval
Its system card adds 39 evaluations 2 new 39 results in all
OpenAI o1 System Card (o1-preview and o1-mini)Capabilities
- MMMLU
- QuantBench 36
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Persuasion
- ChangeMyView 81.8
- Political Persuasion Parallel Generation 53.8
- MakeMePay 11.6
- MakeMeSay 49
- Sep 5, 2024
- LIBERO 76.5
- Sep 5, 2024
- AlpacaEval 2.0 50.5
- Arena-Hard-Auto 76.2
- AlignBench 8.04
- MT-Bench 9.02
- HumanEval 89
- HumanEval Multi 73.8
- LiveCodeBench 41.8
- Aider-Edit 72.2
- SWE-bench Verified 16.8
- DS-FIM-Eval 78.3
- DS-Arena-Code 63.1
Dropped since the last post: MMLU, BIG-Bench Hard, C-Eval, CMMLU, MBPP, GSM8K, MATH, Needle In A Haystack
- Aug 20, 2024
- Arena-Hard-Auto 37.9
- BIG-Bench Hard 79.1
- MMLU 78.9
- MMLU-Pro 54.3
- ARC-Challenge 91
- BoolQ 84.6
- GPQA Diamond 36.8
- HellaSwag 83.8
- OpenBookQA 89.6
- PIQA 88.6
- SIQA 78
- TruthfulQA 77.5
- WinoGrande 81.3
- MMMLU 69.9
- MGSM 58.7
- GSM8K 88.7
- MATH 59.5
- Qasper 40
- SQuALITY 24.1
- HumanEval 70.7
- MBPP 80.8
Dropped since the last post: ANLI, MedQA, AGIEval, TriviaQA, ARC-Easy, CommonSenseQA, MT-Bench
Phi-3.5-mini: Its technical report adds 2 evaluations 23 results in all
Its technical report adds 7 evaluations 5 new 28 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Hallucination
Safety
Robustness
Phi-3.5-vision-instruct: Its technical report adds 14 evaluations 14 new 28 dropped 14 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Capabilities
Hallucination
- POPE 86.1
Safety
- Microsoft internal multimodal RAI benchmark 8.16
- RTVLM 5.44
- VLGuard 9.1
Dropped since the lab’s previous technical report: Arena-Hard-Auto, BIG-Bench Hard, MMLU, MMLU-Pro, ARC-Challenge, BoolQ, GPQA Diamond, HellaSwag, OpenBookQA, PIQA, SIQA, TruthfulQA, WinoGrande, MMMLU, MGSM, GSM8K, MATH, Qasper, SQuALITY, HumanEval, MBPP, RepoQA, RULER, Microsoft RAI benchmark: Ungroundedness, Microsoft RAI benchmark: Third Party Harm (DR-1), Microsoft RAI benchmark: Harmful Content Continuation (DR-3), Microsoft RAI benchmark: Harmful Content Summarization (DR-3), Microsoft RAI benchmark: Jailbreak (DR-1)
- Aug 13, 2024
- GPQA Diamond 56
- MMLU 87.5
- MMLU-Pro 75.5
- MATH 76.1
- HumanEval 88.4
- MMMU 66.1
- MathVista 69
- DocVQA 93.6
- LMArena (Elo)
Dropped since the last post: GSM8K
- Jul 24, 2024 Mistral Large 2 Mistral AI Large Enough (Mistral Large 2 launch post) + HF model card Mistral-Large-Instruct-2407
- MMLU 84
- MT-Bench 8.63
- WildBench 56.3
- Arena-Hard-Auto 73.2
- HumanEval 92
- MBPP 80
- MultiPL-E 76.9
- GSM8K 93
- MATH 71.5
Dropped since the last post: HellaSwag, WinoGrande, ARC-Challenge, TriviaQA, TruthfulQA
- Jul 23, 2024
- MMLU 88.6
- MMLU-Pro 73.3
- IFEval 88.6
- HumanEval 89
- MBPP 88.6
- GSM8K 96.8
- MATH 73.8
- ARC-Challenge 96.9
- GPQA Diamond 51.1
- Berkeley Function Calling Leaderboard (BFCL) 88.5
- Nexus 58.7
- ZeroSCROLLS/QuALITY 95.2
- InfiniteBench/En.MC 83.4
- Needle In A Haystack 98.1
- MGSM 91.6
Dropped since the last post: AGIEval, CommonSenseQA, WinoGrande, BIG-Bench Hard, TriviaQA, SQuAD, QuAC, BoolQ, DROP
Its model card adds 13 evaluations 4 new 25 results in all
Llama 3.1 model card (MODEL_CARD.md, meta-llama/llama-models)Capabilities
- MultiPL-E
- API-Bank 92
- Gorilla APIBench 35.3
- Multilingual MMLU (Meta)
- AGIEval 71.6
- CommonSenseQA 85.8
- WinoGrande 86.7
- BIG-Bench Hard 85.9
- TriviaQA 91.8
- SQuAD 89.3
- QuAC 53.6
- BoolQ 80
- DROP 84.8
Llama 3.1-70B: Its model card adds 13 evaluations 25 results in all
Llama 3.1 model card (MODEL_CARD.md, meta-llama/llama-models)Capabilities
- MultiPL-E
- API-Bank 90
- Gorilla APIBench 29.7
- Multilingual MMLU (Meta)
- AGIEval 64.6
- CommonSenseQA 84.1
- WinoGrande 83.3
- BIG-Bench Hard 81.6
- TriviaQA 89.8
- SQuAD 81.8
- QuAC 51.1
- BoolQ 79.4
- DROP 79.6
Llama 3.1-8B: Its model card adds 13 evaluations 25 results in all
Llama 3.1 model card (MODEL_CARD.md, meta-llama/llama-models)Capabilities
- MultiPL-E
- API-Bank 82.6
- Gorilla APIBench 8.2
- Multilingual MMLU (Meta)
- AGIEval 47.8
- CommonSenseQA 75
- WinoGrande 60.5
- BIG-Bench Hard 64.2
- TriviaQA 77.6
- SQuAD 77
- QuAC 44.9
- BoolQ 75
- DROP 59.5
Its technical report adds 54 evaluations 45 new 10 dropped 69 results in all
The Llama 3 Herd of ModelsCapabilities
- HumanEval+ 82.3
- MultiPL-E
- API-Bank 92.3
- Gorilla APIBench 35.3
- Qasper 49.8
- SQuALITY 15.4
- InfiniteBench/En.QA 30.5
- Multilingual MMLU (Meta) 83.2
- LSAT 81.1
- SAT Reading 74.8
- SAT Math 94.9
- GMAT Quantitative 96
- GMAT Verbal 86.6
- GRE Physics 80
- AP Art History 86.7
- AP Biology 100
- AP Calculus 88.6
- AP Chemistry 90.6
- AP English Language and Composition 94.3
- AP English Literature and Composition 83.3
- AP Environmental Science 93.5
- AP Macroeconomics 98.3
- AP Microeconomics 93.8
- AP Physics 92.9
- AP Psychology 100
- AP Statistics 85.2
- AP US Government 97.6
- AP US History 97.6
- AP World History 100
- GRE Quantitative 162
- GRE Verbal 166
- SQuAD v2 81.8
- QuAC 53.6
- RACE 58.1
- CommonSenseQA 85.8
- PIQA 85.6
- SIQA 53.7
- OpenBookQA 49.2
- WinoGrande 82.2
- DROP 84.8
- WorldSense 63.7
- AGIEval 71.6
- BIG-Bench Hard 85.9
- QuALITY 87.6
- Adversarial SQuAD
- Dynabench SQuAD
- GSM-Plus
- PAWS
- Llama 3 pairwise human evaluation (Meta)
Robustness
Cyber risk
Dropped since the lab’s previous technical report: BoolQ, HellaSwag, ARC-Easy, Natural Questions, TriviaQA, TruthfulQA, ToxiGen, BOLD, Llama 2-Chat helpfulness human evaluation (Meta), Llama 2-Chat safety human evaluation (Meta)
Llama 3.1-70B: Its technical report adds 53 evaluations 1 dropped 68 results in all
The Llama 3 Herd of ModelsCapabilities
- HumanEval+ 74.4
- MultiPL-E
- API-Bank 90
- Gorilla APIBench 29.7
- Qasper 49
- SQuALITY 16.4
- InfiniteBench/En.QA 36.7
- Multilingual MMLU (Meta) 78.2
- LSAT 74.2
- SAT Reading 71.4
- SAT Math 91.9
- GMAT Quantitative 84
- GMAT Verbal 85.1
- GRE Physics 74.7
- AP Art History 84.4
- AP Biology 100
- AP Calculus 54.3
- AP Chemistry 96.9
- AP English Language and Composition 90.6
- AP English Literature and Composition 79.6
- AP Environmental Science 89.1
- AP Macroeconomics 98.3
- AP Microeconomics 91.7
- AP Physics 78.6
- AP Psychology 100
- AP Statistics 59.3
- AP US Government 97.6
- AP US History 97.6
- AP World History 100
- GRE Quantitative 158
- GRE Verbal 166
- SQuAD v2 81.8
- QuAC 51.1
- RACE 59
- CommonSenseQA 84.1
- PIQA 83.8
- SIQA 52.2
- OpenBookQA 47.6
- WinoGrande 83.5
- DROP 79.6
- WorldSense 61.1
- AGIEval 64.6
- BIG-Bench Hard 81.6
- QuALITY 82.8
- Adversarial SQuAD
- Dynabench SQuAD
- GSM-Plus
- PAWS
Robustness
Cyber risk
Dropped since the lab’s previous technical report: Llama 3 pairwise human evaluation (Meta)
Llama 3.1-8B: Its technical report adds 50 evaluations 3 dropped 65 results in all
The Llama 3 Herd of ModelsCapabilities
- HumanEval+ 67.1
- MultiPL-E
- API-Bank 82.6
- Gorilla APIBench 8.2
- Qasper 39.3
- SQuALITY 15.3
- InfiniteBench/En.QA 27.1
- Multilingual MMLU (Meta) 58.6
- LSAT 53.9
- SAT Reading 57.4
- SAT Math 73.3
- GMAT Quantitative 56
- GMAT Verbal 65.7
- GRE Physics 48
- AP Art History 75.6
- AP Biology 91.7
- AP Calculus 57.1
- AP Chemistry 59.4
- AP English Language and Composition 69.8
- AP English Literature and Composition 59.3
- AP Environmental Science 73.9
- AP Macroeconomics 72.4
- AP Microeconomics 70.8
- AP Physics 57.1
- AP Psychology 94.8
- AP Statistics 66.7
- AP US Government 90.2
- AP US History 78
- AP World History 94.1
- GRE Quantitative 152
- GRE Verbal 149
- SQuAD v2 77
- QuAC 44.9
- RACE 54.3
- CommonSenseQA 75
- PIQA 81
- SIQA 49.5
- OpenBookQA 45
- WinoGrande 75.7
- DROP 59.5
- WorldSense 45.5
- AGIEval 47.8
- BIG-Bench Hard 64.2
- QuALITY 56
- Adversarial SQuAD
- Dynabench SQuAD
- GSM-Plus
- PAWS
Robustness
Dropped since the lab’s previous technical report: CyberSecEval Code Interpreter Abuse, CyberSecEval Spear Phishing, Llama internal safety benchmarks (Meta)
- Jul 18, 2024
- Jun 27, 2024
- MMLU 75.2
- HellaSwag 86.4
- PIQA 83.2
- SIQA 53.7
- BoolQ 84.8
- WinoGrande 83.7
- ARC-Easy 88.6
- ARC-Challenge 71.4
- TriviaQA 83.7
- Natural Questions 34.5
- HumanEval 51.8
- MBPP 62.6
- GSM8K 74
- MATH 42.3
- AGIEval 55.1
- DROP 72.2
- BIG-Bench Hard 74.9
Dropped since the last post: CommonSenseQA, OpenBookQA
Its technical report adds 20 evaluations 13 new 3 dropped 37 results in all
Gemma 2: Improving Open Language Models at a Practical Size (arXiv 2408.00118v1)Capabilities
- LMArena (Elo) 1218
Hallucination
- TruthfulQA 51.6
Safety
- RealToxicityPrompts 8.84
- ToxiGen 38.4
Bias
Cyber risk
Persuasion
Dropped since the lab’s previous technical report: CommonSenseQA, OpenBookQA, BOLD
Gemma 2 9B: Its technical report adds 8 evaluations 12 dropped 25 results in all
Gemma 2: Improving Open Language Models at a Practical Size (arXiv 2408.00118v1)Capabilities
- LMArena (Elo) 1187
Hallucination
- TruthfulQA 50.3
Safety
- RealToxicityPrompts 8.25
- ToxiGen 39.3
Bias
Dropped since the lab’s previous technical report: InterCode-CTF, In-house CTF suite (Phuong et al. 2024), Hack the Box challenge set (Google DeepMind), PrimeVul, DiverseVul, SPI (security patch identification), SecretPatch, Self-proliferation challenges (Phuong et al. 2024), Charm offensive, Hidden agenda, Money talks, Web of lies
- Jun 20, 2024
- GPQA Diamond 67.2
- MMLU 90.4
- MATH 71.1
- HumanEval 92
- MGSM 91.6
- DROP 87.1
- BIG-Bench Hard 93.1
- GSM8K 96.4
- MMMU 68.3
- MathVista 67.7
- AI2D 94.7
- ChartQA 90.8
- DocVQA 95.2
Dropped since the last post: ARC-Challenge, HellaSwag, PubMedQA, WinoGrande, RACE-H, APPS, MBPP
Its model card adds 8 evaluations 1 new 21 dropped 21 results in all
Claude 3.5 Sonnet Model Card AddendumCapabilities
Safety
Dropped since the lab’s previous model card: GPQA Main, GPQA Extended, ARC-Challenge, HellaSwag, PubMedQA, WinoGrande, RACE-H, APPS, MBPP, LSAT, MBE (Multistate Bar Examination), AMC 12 2023, AMC 10 2023, AMC 8 2023, Okapi MMLU, 100Q-Hard (Anthropic), Multi-factual (Anthropic), QuALITY, Multimodal policy red-teaming (Anthropic), Discrim-Eval, BBQ (Bias Benchmark for QA)
- Jun 7, 2024
- MMLU 82.3
- MMLU-Pro 64.4
- GPQA Diamond 42.4
- TheoremQA 44.4
- MT-Bench 9.12
- Arena-Hard-Auto 48.1
- IFEval 77.6
- HumanEval 86
- MBPP 80.2
- MultiPL-E 69.2
- EvalPlus 79
- LiveCodeBench 35.7
- GSM8K 91.1
- MATH 59.7
- C-Eval 83.8
- AlignBench 8.27
- BIG-Bench Hard 82.4
- HellaSwag 87.6
- WinoGrande 85.1
- ARC-Challenge 68.9
- TruthfulQA 54.8
- CMMLU 90.1
- Multilingual Exams 76.6
- Multilingual Understanding 80.7
- Flores 37.8
- MMMLU 78
- MGSM 86.6
- Multilingual human evaluation 3.93
Dropped since the last post: AlpacaEval 2.0, L-Eval, RGB, T-Eval, Chinese Tool-Use Benchmark, Code Interpreter Benchmark Math
Its technical report adds 7 evaluations 7 new 10 dropped 22 results in all
Qwen2 Technical Report (arXiv 2407.10671)Capabilities
Dropped since the lab’s previous technical report: C-Eval, BIG-Bench Hard, HumanEvalPack, MATH-500, Math401, Math23K, Chinese Tool-Use Benchmark, Code Interpreter Benchmark, HuggingFace Agent Benchmark, Qwen Chinese human evaluation
- May 14, 2024 Gemini 1.5 Pro Google DeepMind Gemini 1.5 technical report, May 2024 update with 1.5 Flash (arXiv 2403.05530v3)
- GPQA Diamond 46.2
- MATH 77.9
- PhysicsFinals 63.9
- HiddenMath 20.1
- Functional MATH 64.6
- AMC 46.4
- GSM8K 90.8
- BIG-Bench Hard 89.2
- DROP 74.9
- MMLU 91.7
- HellaSwag 93.3
- HumanEval 84.1
- Natural2Code 82.6
- WMT23 75.3
- MGSM 87.5
- MMMU 62.2
- MathVista 63.9
- AI2D 94.4
- ChemicalDiagramQA 69.7
- ChartQA 87.2
- BetterChartQA 65.8
- DocVQA 93.1
- DUDE 46
- TAT-DQA 37.8
- InfographicVQA 81
- TextVQA 78.7
- VQAv2 80.2
- V* 71.7
- BLINK 61.4
- RealWorldQA 70.4
- VATEX 64.6
- VATEX ZH 55.3
- YouCook2 106.5
- ActivityNet-QA 57.5
- EgoSchema 72.2
- OpenEQA 57.9
- May 13, 2024
- MMLU 88.7
- GPQA Diamond 53.6
- MATH 76.6
- HumanEval 90.2
- MGSM 90.5
- DROP 83.4
- M3Exam
- MMMU 69.1
- MathVista 63.8
- AI2D 94.2
- ChartQA 85.7
- DocVQA 92.8
- ActivityNet-QA 61.9
- EgoSchema 72.2
Its system card adds 34 evaluations 34 results in all
- May 7, 2024
- MMLU 77.8
- BIG-Bench Hard 79.7
- C-Eval 78
- CMMLU 81.6
- HumanEval 81.1
- MBPP 72
- LiveCodeBench 32.5
- GSM8K 92.2
- MATH 53.9
- Needle In A Haystack
- AlpacaEval 2.0
- MT-Bench
- AlignBench 7.91
Dropped since the last post: HellaSwag, TriviaQA, ChineseQA, Hungarian National High-School Exam, IFEval, LeetCode Weekly Contest
Its technical report adds 10 evaluations 2 new 16 dropped 22 results in all
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv 2405.04434)Capabilities
- TriviaQA 86.7
- Natural Questions 53.4
- ARC-Easy 98.1
- ARC-Challenge 92.3
- AGIEval 61.4
- IFEval 63.8
- CRUXEval 61.5
- CMATH 81.9
- CLUEWSC 89.9
- SuperCLUE-Math6 83.3
Dropped since the lab’s previous technical report: HellaSwag, PIQA, WinoGrande, RACE-M, RACE-H, DROP, OpenBookQA, CHID, C3, CCPM, HumanEval Multi, MGSM, LeetCode Weekly Contest, Hungarian National High-School Exam, DeepSeek LLM in-house safety evaluation, Do-Not-Answer
- Apr 22, 2024
- MMLU 78.2
- HellaSwag 83
- ANLI 58.7
- GSM8K 90.3
- MedQA 69.4
- AGIEval 48.4
- TriviaQA 75.6
- ARC-Challenge 91
- ARC-Easy 97.8
- PIQA 87.7
- SIQA 80.2
- BIG-Bench Hard 81.3
- WinoGrande 81.4
- OpenBookQA 87.2
- BoolQ 86.6
- CommonSenseQA 82.6
- TruthfulQA 75.7
- HumanEval 55.5
- MBPP 74.5
- GPQA Diamond
- MT-Bench 8.91
Dropped since the last post: SQuAD v2
Its technical report adds 6 evaluations 6 new 14 dropped 26 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Capabilities
- MATH 53.1
Hallucination
Safety
Robustness
Dropped since the lab’s previous technical report: MMMU, ScienceQA, MathVista, Inter-GPS (Geometry3K), MMBench, POPE, AI2D, ChartQA, TextVQA, BLINK, Video-MME, Microsoft internal multimodal RAI benchmark, RTVLM, VLGuard
phi-3-mini 3.8B: Its technical report adds 7 evaluations 1 new 28 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Hallucination
Safety
Robustness
phi-3-small 7.4B: Its technical report adds 6 evaluations 1 dropped 27 results in all
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4)Capabilities
- MATH 34.6
Hallucination
Safety
Robustness
Dropped since the lab’s previous technical report: MMMLU
- Apr 18, 2024
- MMLU 82
- GPQA Diamond 39.5
- HumanEval 81.7
- GSM8K 93
- MATH 50.4
- AGIEval 63
- CommonSenseQA 83.8
- WinoGrande 83.1
- BIG-Bench Hard 81.3
- ARC-Challenge 93
- TriviaQA 89.7
- SQuAD 85.6
- QuAC 51.1
- BoolQ 79
- DROP 79.7
Its model card adds 0 evaluations 2 dropped 15 results in all
Llama 3 model card (MODEL_CARD.md, meta-llama/llama3)Dropped since the lab’s previous model card: TruthfulQA, ToxiGen
Llama 3-8B: Its model card adds 0 evaluations 15 results in all
- Apr 11, 2024
Dropped since the last post: HellaSwag, ARC-Challenge, WinoGrande, GSM8K, VQAv2, TextVQA, ChartQA, AI2D, DocVQA, InfographicVQA, TVQA, LSMDC
- Mar 28, 2024
Dropped since the last post: Hungarian National High-School Exam
- Mar 4, 2024
- MMLU 88.2
- MATH 73.7
- GSM8K 95
- HumanEval 84.9
- GPQA Diamond 59.5
- MGSM 90.7
- DROP 83.1
- BIG-Bench Hard 86.8
- ARC-Challenge 96.4
- HellaSwag 95.4
- PubMedQA 75.8
- WinoGrande 88.5
- RACE-H 92.9
- APPS 70.2
- MBPP 86.4
- MMMU 59.4
- DocVQA 89.3
- MathVista 50.5
- AI2D 88.1
- ChartQA 80.8
Dropped since the last post: TriviaQA, QuALITY
Claude 3 Haiku: Its model card adds 17 evaluations 14 new 12 dropped 37 results in all
The Claude 3 Model Family: Opus, Sonnet, HaikuCapabilities
- GPQA Main 35.2
- GPQA Extended 34.7
- LSAT 156.3
- MBE (Multistate Bar Examination) 64
- AMC 12 2023 48
- AMC 10 2023 54
- AMC 8 2023 36
- Okapi MMLU 65.2
- QuALITY 80.2
- Needle In A Haystack 95.9
Hallucination
Dropped since the lab’s previous model card: Human feedback win rates (Anthropic), TruthfulQA, Harmfulness scores on held-out prompts (Anthropic), Combined HHH evals (Anthropic), FLORES-200, TriviaQA, GRE Verbal, GRE Quantitative, GRE Analytical Writing, USMLE Step 1 (sample questions), USMLE Step 2 (sample questions), USMLE Step 3 (sample questions)
Its model card adds 23 evaluations 6 new 43 results in all
The Claude 3 Model Family: Opus, Sonnet, HaikuCapabilities
- GPQA Main 49.1
- GPQA Extended 48.8
- LSAT 161
- MBE (Multistate Bar Examination) 85
- AMC 12 2023 63
- AMC 10 2023 72
- AMC 8 2023 84
- GRE Quantitative 159
- GRE Verbal 166
- GRE Analytical Writing 5
- Okapi MMLU 79.1
- QuALITY 90.5
- Needle In A Haystack 99.4
Hallucination
Safety
Claude 3 Sonnet: Its model card adds 21 evaluations 3 new 5 dropped 41 results in all
The Claude 3 Model Family: Opus, Sonnet, HaikuCapabilities
- GPQA Main 38.5
- GPQA Extended 38
- LSAT 158.3
- MBE (Multistate Bar Examination) 71
- AMC 12 2023 27
- AMC 10 2023 24
- AMC 8 2023 54
- Human feedback win rates (Anthropic)
- Okapi MMLU 69
- QuALITY 85.9
- Needle In A Haystack 95.4
Hallucination
Safety
Dropped since the lab’s previous model card: GRE Quantitative, GRE Verbal, GRE Analytical Writing, Autonomous Replication and Adaption (ARA) evaluations (Anthropic), Expert vulnerability discovery and exploit development evaluations (Anthropic)
- Feb 26, 2024
- MMLU 81.2
- HellaSwag 89.2
- WinoGrande 86.7
- ARC-Challenge 94
- TriviaQA 82.7
- TruthfulQA 50.5
- HumanEval 45.1
- MBPP 73.1
- MATH 45
- GSM8K 91.2
- Feb 21, 2024
- MMLU 64.3
- HellaSwag 81.2
- PIQA 81.2
- SIQA 51.8
- BoolQ 83.2
- WinoGrande 72.3
- CommonSenseQA 71.3
- OpenBookQA 52.8
- ARC-Easy 81.5
- ARC-Challenge 53.2
- TriviaQA 63.4
- Natural Questions 23
- HumanEval 32.3
- MBPP 44.4
- GSM8K 46.4
- MATH 24.3
- AGIEval 41.7
- BIG-Bench Hard 55.1
Gemma 2B: Its technical report adds 8 evaluations 26 results in all
Gemma: Open Models Based on Gemini Research and Technology (arXiv 2403.08295v1)Hallucination
- TruthfulQA 44.8
Safety
- RealToxicityPrompts 6.86
- ToxiGen 29.8
Bias
- BOLD 45.6
- CrowS-Pairs 45.8
- BBQ (Bias Benchmark for QA)
- Winogender 51.3
- WinoBias
Its technical report adds 8 evaluations 26 results in all
Gemma: Open Models Based on Gemini Research and Technology (arXiv 2403.08295v1)Hallucination
- TruthfulQA 31.8
Safety
- RealToxicityPrompts 7.9
- ToxiGen 39.6
Bias
- BOLD 49.1
- CrowS-Pairs 51.3
- BBQ (Bias Benchmark for QA)
- Winogender 54.2
- WinoBias
- Feb 15, 2024 Gemini 1.5 Pro Google DeepMind Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (technical report, Feb 2024 version = arXiv 2403.05530v1)
- HellaSwag 92.5
- MMLU 81.9
- GSM8K 91.7
- GPQA Diamond 41.5
- MATH 59.4
- PhysicsFinals 60.7
- AMC 37.2
- BIG-Bench Hard 84
- DROP 78.9
- HumanEval 71.9
- Natural2Code 77.7
- WMT23 75.2
- MGSM 88.7
- MMMU 58.5
- AI2D 80.3
- MathVista 52.1
- ChartQA 81.3
- VQAv2 73.2
- TextVQA 73.5
- DocVQA 86.5
- InfographicVQA 72.7
- VATEX 63
- VATEX ZH 54.9
- YouCook2 134.2
- ActivityNet-QA 56.7
- EgoSchema 63.2
- Feb 5, 2024
- MMLU 77.5
- C-Eval 84.1
- GSM8K 79.5
- MATH 34.1
- HumanEval 41.5
- MBPP 53.4
- BIG-Bench Hard 65.5
- CMMLU 83.5
- MT-Bench 8.61
- AlpacaEval 2.0 27.2
- Multilingual Exams 66.3
- Multilingual Understanding 78.2
- MGSM 61.7
- Flores 35.6
- L-Eval 78.1
- RGB
- T-Eval 76.7
- Chinese Tool-Use Benchmark 95.1
- Code Interpreter Benchmark Math 73.1
Dropped since the last post: VCSUM, HuggingFace Agent Benchmark
- Jan 16, 2024GLM-4 Z.ai GLM-4 launch at Zhipu DevDay (official announcement text and slides, reposted on Volcengine developer community)
- MMLU 81.5
- GSM8K 87.6
- MATH 47.9
- BIG-Bench Hard 82.3
- HellaSwag 85.4
- HumanEval 72
- IFEval 67.7
- AlignBench 7.88
- LongBench 81.1
- Needle In A Haystack 100
Its technical report adds 3 evaluations 9 results in all
2023
- Dec 12, 2023
- Dec 11, 2023
- MMLU 72.5
- HellaSwag 86.7
- ARC-Challenge 85.8
- WinoGrande 81.2
- MBPP 60.7
- GSM8K 58.4
- MT-Bench 8.3
- PIQA 83.6
- ARC-Easy 83.1
- Natural Questions 30.6
- TriviaQA 71.5
- HumanEval 40.2
- MATH 28.4
Its technical report adds 7 evaluations 4 new 3 dropped 20 results in all
Mixtral of ExpertsCapabilities
Bias
Dropped since the lab’s previous technical report: LLMBoxing, Mistral 175 unsafe prompts, Mistral self-reflection moderation set
- Dec 6, 2023 Gemini 1.0 Ultra Google DeepMind Gemini: A Family of Highly Capable Multimodal Models (technical report, arXiv 2312.11805v1)
- MMLU 90
- GSM8K 94.4
- MATH 53.2
- BIG-Bench Hard 83.6
- HumanEval 74.4
- Natural2Code 74.9
- DROP 82.4
- HellaSwag 87.8
- WMT23 74.4
- MMMU 62.4
- TextVQA 82.3
- DocVQA 90.9
- ChartQA 80.8
- InfographicVQA 80.3
- MathVista 53
- AI2D 79.5
- VQAv2 77.8
Dropped since the last post: TriviaQA, Natural Questions, WebQuestions, LAMBADA, StoryCloze, WSC, WinoGrande, Winograd, SQuAD v2, RACE-H, RACE-M, PIQA, ARC-Challenge, ARC-Easy, OpenBookQA, BoolQ, COPA, RTE, WiC, MultiRC, ReCoRD, CB, ANLI, StrategyQA, CommonSenseQA, XCOPA, MGSM
- Nov 29, 2023
- HellaSwag 84
- TriviaQA 81.5
- MMLU 71.1
- GSM8K 84.1
- HumanEval 73.8
- BIG-Bench Hard 71.7
- C-Eval 65.2
- CMMLU 67.8
- ChineseQA 85.1
- Hungarian National High-School Exam
- IFEval
- LeetCode Weekly Contest
Its technical report adds 23 evaluations 34 results in all
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (arXiv 2401.02954)Capabilities
- PIQA 82.6
- WinoGrande 76
- RACE-M 70.9
- RACE-H 56
- Natural Questions 47
- ARC-Easy 81.6
- ARC-Challenge 64.1
- MATH 32.6
- MBPP 61.4
- DROP 71.9
- OpenBookQA 63.2
- AGIEval 46.4
- CLUEWSC 60
- CHID 72.6
- CMATH 80.3
- C3 77
- CCPM 84.9
- HumanEval Multi 53.3
- MGSM 74
- AlignBench 6.43
- MT-Bench 8.35
Safety
- Nov 3, 2023
- GSM8K 62.9
- MMLU 73
- HumanEval 63.2
- MATH 23.9
- Hungarian National High-School Exam 59
- Oct 27, 2023
Its technical report adds 10 evaluations 16 results in all
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All ToolsCapabilities
- GPQA Diamond 26.8
- HumanEval 58.5
- BoolQ 87.9
- CommonSenseQA 86.5
- HellaSwag 79.7
- PIQA 80.1
- DROP 70.9
- GAOKAO-Bench 67.3
- C3 73.9
- Berkeley Function Calling Leaderboard (BFCL) 57.9
- Sep 27, 2023
- MMLU 60.1
- HellaSwag 81.3
- WinoGrande 75.3
- PIQA 83
- ARC-Easy 80
- ARC-Challenge 55.5
- Natural Questions 28.8
- TriviaQA 69.9
- HumanEval 30.5
- MBPP 47.5
- MATH 13.1
- GSM8K 52.1
- MT-Bench 6.84
Its technical report adds 6 evaluations 19 results in all
Mistral 7BCapabilities
- Sep 25, 2023
- C-Eval 71.7
- MMLU 66.5
- HumanEval 43.9
- GSM8K 60.1
- MATH 24.8
- MBPP 40.8
- BIG-Bench Hard 53.4
- CMMLU 71
- VCSUM 17.3
- Chinese Tool-Use Benchmark 98
- Code Interpreter Benchmark Math 58.4
- HuggingFace Agent Benchmark 93.5
Qwen-1_8B: Its technical report adds 1 evaluation 8 results in all
Qwen Technical Report (arXiv 2309.16609)Capabilities
Its technical report adds 6 evaluations 5 new 14 results in all
Qwen Technical Report (arXiv 2309.16609)Capabilities
- HumanEvalPack 33.7
- MATH-500 18.4
- Math401 70.1
- Math23K 67
- Code Interpreter Benchmark 81.7
- Qwen Chinese human evaluation 30.7
Qwen-7B: Its technical report adds 6 evaluations 14 results in all
Qwen Technical Report (arXiv 2309.16609)Capabilities
- HumanEvalPack 24.2
- MATH-500 6.8
- Math401 57.4
- Math23K 51.2
- Code Interpreter Benchmark 70.2
- Qwen Chinese human evaluation 28.4
- Jul 18, 2023
- MMLU 68.9
- BIG-Bench Hard 51.2
- AGIEval 54.2
Llama 2-13B: Its model card adds 2 evaluations 5 results in all
Llama 2 model card (MODEL_CARD.md, meta-llama/llama)Safety
- TruthfulQA 41.9
- ToxiGen 26.1
Its model card adds 2 evaluations 5 results in all
Llama 2 model card (MODEL_CARD.md, meta-llama/llama)Safety
- TruthfulQA 50.2
- ToxiGen 24.6
Llama 2-7B: Its model card adds 2 evaluations 5 results in all
Llama 2 model card (MODEL_CARD.md, meta-llama/llama)Safety
- TruthfulQA 33.3
- ToxiGen 21.3
Llama 2-13B: Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 18.3
- MBPP 30.6
- BoolQ 81.7
- PIQA 80.5
- SIQA 50.3
- HellaSwag 80.7
- WinoGrande 72.8
- ARC-Easy 77.3
- ARC-Challenge 49.4
- OpenBookQA 57
- CommonSenseQA 67.3
- Natural Questions 31.2
- TriviaQA 79.6
- SQuAD v2 72.9
- QuAC 42.7
- GSM8K 28.7
- MATH 3.9
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
Llama 2-34B: Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 22.6
- MBPP 33
- BoolQ 83.7
- PIQA 81.9
- SIQA 50.9
- HellaSwag 83.3
- WinoGrande 76.7
- ARC-Easy 79.4
- ARC-Challenge 54.5
- OpenBookQA 58.2
- CommonSenseQA 74.3
- Natural Questions 32.8
- TriviaQA 84.5
- SQuAD v2 77.4
- QuAC 42.9
- GSM8K 42.2
- MATH 6.24
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 29.9
- MBPP 45
- BoolQ 85
- PIQA 82.8
- SIQA 50.7
- HellaSwag 85.3
- WinoGrande 80.2
- ARC-Easy 80.2
- ARC-Challenge 57.4
- OpenBookQA 60.2
- CommonSenseQA 78.5
- Natural Questions 39.5
- TriviaQA 87.6
- SQuAD v2 80.7
- QuAC 42.4
- GSM8K 56.8
- MATH 13.5
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
Llama 2-7B: Its technical report adds 22 evaluations 25 results in all
Llama 2: Open Foundation and Fine-Tuned Chat ModelsCapabilities
- HumanEval 12.8
- MBPP 20.8
- BoolQ 77.4
- PIQA 78.8
- SIQA 48.3
- HellaSwag 77.2
- WinoGrande 69.2
- ARC-Easy 75.2
- ARC-Challenge 45.9
- OpenBookQA 58.6
- CommonSenseQA 57.8
- Natural Questions 25.7
- TriviaQA 72.1
- SQuAD v2 67.2
- QuAC 39.4
- GSM8K 14.6
- MATH 2.5
- Llama 2-Chat helpfulness human evaluation (Meta)
Safety
Bias
- Jul 11, 2023
Its model card adds 13 evaluations 20 results in all
- May 10, 2023
- TriviaQA 86.1
- Natural Questions 37.5
- WebQuestions 28.2
- LAMBADA 86.9
- HellaSwag 86.8
- StoryCloze 87.4
- WSC 86.9
- WinoGrande 90.9
- Winograd 89.5
- SQuAD v2 80.5
- RACE-H 62.3
- RACE-M 77
- PIQA 85
- ARC-Challenge 95.1
- ARC-Easy 89.7
- OpenBookQA 58.5
- BoolQ 90.9
- COPA 96
- RTE 79.3
- WiC 66.8
- MultiRC 88.2
- ReCoRD 93.8
- CB 87.5
- ANLI 73.1
- DROP 85
- StrategyQA 90.4
- CommonSenseQA 90.4
- XCOPA 94.4
- BIG-Bench Hard 78.1
- MATH 48.8
- GSM8K 91
- MGSM 87
- Mar 14, 2023
- MMLU 86.4
- HellaSwag 95.3
- ARC-Challenge 96.3
- WinoGrande 87.5
- HumanEval 67
- DROP 80.9
- GSM8K 92
- VQAv2 77.2
- TextVQA 78
- ChartQA 78.5
- AI2D 78.2
- DocVQA 88.4
- InfographicVQA 75.1
- TVQA 87.3
- LSMDC 45.7
Its technical report adds 43 evaluations 50 results in all
GPT-4 Technical Report (with appended GPT-4 System Card)Capabilities
- Uniform Bar Exam (MBE+MEE+MPT) 298
- LSAT 163
- SAT Evidence-Based Reading & Writing 710
- SAT Math 700
- GRE Quantitative 163
- GRE Verbal 169
- GRE Analytical Writing 4
- USABO Semifinal Exam 2020 87
- USNCO Local Section Exam 2022 36
- Medical Knowledge Self-Assessment Program (MKSAP) 75
- Codeforces rating 392
- AP Art History 5
- AP Biology 5
- AP Calculus BC 4
- AP Chemistry 4
- AP English Language and Composition 2
- AP English Literature and Composition 2
- AP Environmental Science 5
- AP Macroeconomics 5
- AP Microeconomics 5
- AP Physics 2 4
- AP Psychology 5
- AP Statistics 5
- AP US Government 5
- AP US History 5
- AP World History 4
- AMC 10 2022 30
- AMC 12 2022 60
- Introductory Sommelier (theory knowledge) 92
- Certified Sommelier (theory knowledge) 86
- Advanced Sommelier (theory knowledge) 77
- Leetcode (easy) 75.6
- Leetcode (medium) 26.3
- Leetcode (hard) 6.7
- MMLU translated with Azure Translate
- Hindsight Neglect
- Preference on ChatGPT and API prompts 70.2
Hallucination