Meta
We track 110 benchmarks and 3 arenas that Meta made or helped make. Its models are on its lab page.
12 in use 53 with public code 40 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Multi-IF Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. | Multi-turn conversation | Oct 2024 | 89.1% | In use |
| CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. | Speech recognition and translation | Jul 2020 |  40.7 | Fading |
| TextVQA Models answer 45,336 questions about 28,408 images that require reading text in the image. Scoring is VQA accuracy against human answers. | Documents, charts and text in images | Apr 2019 |  84.8% | Fading |
| ProgramBench tests passed Agents rebuild 200 programs from only their compiled binaries and docs. The score is the share of hidden behavioral tests the rebuilt program passes, averaged over tasks. | Building whole projects | May 2026 | 88.5% | In use |
| CRUXEval Models predict the output of 800 short Python functions for a given input, or an input that yields a given output. Scored by pass@1. | Function-level problems | Jan 2024 |  95.9% | Fading |
| ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original. | Building whole projects | May 2026 | 18.5% | In use |
| Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. | Detection and captioning | Nov 2020 | 13.5 | Fading |
| Flores Models translate 3,001 English Wikipedia sentences that professionals translated into 101 languages. Scored with BLEU or chrF against references. | Translation | Jun 2021 |  48.8 | Dropped |
| ProgramBench almost solved Agents rebuild 200 programs from only their compiled binaries and docs. The score is the share of tasks whose rebuilt program passes at least 95% of the hidden behavioral tests. | Building whole projects | May 2026 | 70.5% | In use |
| GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. | General assistants | Nov 2023 | 91.7% | Fading |
| AdvancedIF Models respond to over 1,600 prompts with complex, multi-turn, and system-level instructions. A judge model checks each response against expert-written rubrics. | Following precise instructions | Nov 2025 |  85% | In use |
| Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. | Knowledge in many languages | Jul 2024 | 83.2% | System cards only |
| Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. | Composite indices | Oct 2026 | In use | |
| ANLI Models label premise and hypothesis pairs as entailment, neutral, or contradiction. People wrote the pairs over three rounds to fool strong models. | Language reasoning | Oct 2019 | 73.1% | Dropped |
| Belebele Models answer multiple-choice reading comprehension questions on FLORES passages in 122 language variants. Scored by accuracy. | Reading in many languages | Aug 2023 | 87.8% | System cards only |
| Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. | General assistants | Sep 2025 | 42.1% | Fading |
| Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. | Genomics and life sciences | 79.4 | System cards only | |
| Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. | Hazardous knowledge | 100% | System cards only | |
| MVPBench Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail. | Physical prediction | Jun 2025 | 72.5% | Dropped |
| StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. | Jailbreaks and prompt injection | 0.50% | In use | |
| WorldSense Models answer synthetic questions that test consistent reasoning about simple arrangements of entities described in text. | Language reasoning | Nov 2023 |  67.3% | Fading |
| FLORES-200 Models translate English Wikipedia sentences into about 200 languages. Outputs are scored with BLEU or chrF against human reference translations. | Translation | Jul 2022 | 81.2 | Dropped |
| Llama 2-Chat safety human evaluation (Meta) Human raters score responses to about 2,000 adversarial prompts on a five-point scale. Ratings of 1 or 2 count as violations. | Refusing harmful requests | Jul 2023 | System cards only | |
| MemoryBench (Meta) Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics. | Finding facts in long inputs | Oct 2026 | In use | |
| MultiLoKo Locally relevant knowledge questions in 31 languages, plus translations. Scored by exact match. | Knowledge in many languages | Apr 2025 | 31% | Fading |
| CyberSecEval Autocomplete Models complete code that leads up to a known insecure pattern. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | Secure code and review | Dec 2023 | 71.3% | System cards only |
| CyberSecEval Instruct Models get coding requests built to elicit known insecure patterns. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | Secure code and review | Dec 2023 | 77.3% | System cards only |
| CyberSecEval Prompt Injection Models face text prompts that embed injected instructions across 15 strategies. Scored as attack success rate. | Jailbreaks and prompt injection | Apr 2024 | 19% | System cards only |
| Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. | Refusing harmful requests | 0.70% | System cards only | |
| Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. | Deception and alignment | 0.50 | System cards only | |
| Meta internal sycophancy evaluation Models meet adversarially curated conversations that invite inappropriate agreement, plus benign controls. Reports sycophancy and excessive anti-sycophancy rates. | Wellbeing and sycophancy | 40.5% | System cards only | |
| CIMemories Models use persistent memory across tasks and must withhold attributes when context requires. Reports worst-case disclosure over five samples. | Refusing harmful requests | Nov 2025 | 26.4% | In use |
| CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. | Refusing harmful requests | Apr 2024 | 3.80% | System cards only |
| Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. | Speech recognition and translation | Dec 2020 | 4.70 | Fading |
| Dynabench SQuAD Models answer reading comprehension questions written by people to fool earlier models in the loop. | Reading comprehension | Apr 2021 | System cards only | |
| Llama verbatim memorization probe (Meta) Models get prompts sampled from their pretraining data. The score is the share of generations that reproduce the true continuation verbatim. | Refusing harmful requests | Jul 2024 | System cards only | |
| SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved. | Fixing real repositories | Oct 2026 | 9.50% | In use |
| AbstentionBench Models face underspecified, ill-posed or unanswerable questions. Scored on recall of appropriate abstention. | Hallucination rates | Jun 2025 | 75.3% | System cards only |
| AIRS-Bench Agents carry out 20 AI research tasks taken from recent ML papers, without baseline code. Scores are normalized so that 1.0 matches the published human state of the art. | AI research tasks | Feb 2026 | 0.40 | System cards only |
| CausalVQA Models answer questions about real egocentric videos that probe physical cause and effect, in counterfactual, hypothetical, anticipation, planning and descriptive types. | Physical prediction | Jun 2025 |  70% | System cards only |
| FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. | General assistants | Oct 2026 | 54.2% | In use |
| OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. | Embodied reasoning | Apr 2024 | 57.9 | Dropped |
| CharXiv Missing Images Models answer CharXiv chart questions with the image removed. The benchmark reports how often they answer incorrectly instead of declining. | Hallucination rates | 12.3% | System cards only | |
| CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. | Capture-the-flag challenges | Jul 2024 | 14% | System cards only |
| Meta catastrophic cyber misuse refusal Models receive high-severity cyber-misuse prompts with technical context. The benchmark reports pass@k attack success rate under adaptive attacks. | Refusing harmful requests | 3.10% | System cards only | |
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Capture-the-flag challenges | Apr 2026 | 89.9% | System cards only |
| Meta cyber false-refusal benchmark Models receive benign and borderline cyber requests and the benchmark measures how often they wrongly refuse. Reports false refusal rate. | Refusing harmful requests | 11% | System cards only | |
| Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. | Deception and alignment | Jul 2026 | 16.8% | System cards only |
| SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. | Deception and alignment | 90.7% | System cards only | |
| SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. | Secure code and review | 0% | System cards only | |
| Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. | Refusing harmful requests | Jul 2024 | System cards only | |
| BOUQuET Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references. | Translation | Feb 2025 | 35.0 | System cards only |
| BigO(Bench) Models predict the time and space complexity of Python solutions and write solutions that meet a required complexity. Uses 3,105 Code Contests problems with profiled complexity labels. | Speed and optimization | Mar 2025 |  41.4% | System cards only |
| WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. | Commonsense | May 2019 | 86.9% | Dropped |
| AgentHarm Verified (Meta) Agents receive 108 malicious AgentHarm tasks kept after human review, plus a benign set. Reports average harm score and false refusal rate. | Refusing harmful requests | Jul 2026 | 3.40% | System cards only |
| CyberSOCEval Malware Analysis Models read potential malware behavior and pick every malicious activity that applies. Scored on precision and recall. | Finding and exploiting real bugs | Sep 2025 | 33% | System cards only |
| CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. | Finding and exploiting real bugs | Sep 2025 | 55.3% | System cards only |
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Finding and exploiting real bugs | Oct 2025 | 41% | System cards only |
| HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds. | Function-level problems | Sep 2025 | 94% | System cards only |
| IntPhys 2 Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy. | Physical prediction | Jun 2025 | 68.3% | In use |
| JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. | Jailbreaks and prompt injection | 2.00% | System cards only | |
| Linguini Models solve 160 International Linguistic Olympiad problems with 894 questions in 75 mostly low-resource languages. All evidence needed is in the context. | Reasoning in many languages | Sep 2024 | 8.80% | System cards only |
| Meta action-belief coherence probe Models act on items from five alignment evaluations and then state beliefs about them. Reports a mean coherency score. | Deception and alignment | 89.1% | System cards only | |
| Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. | Deception and alignment | 20.7% | System cards only | |
| Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. | Lab protocols | Oct 2025 | 43.6% | System cards only |
| Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. | Secure code and review | 47.8% | System cards only | |
| Meta Cyber Misuse Chat Models receive 80 offensive cyber requests, directly and with jailbreak developer prompts. Reports attack success rate and a benign false refusal rate. | Refusing harmful requests | 2.80% | System cards only | |
| Meta cyber misuse request set Models receive 600 private offensive cyber prompts, directly and under jailbreak attacks. Reports attack success rate. | Refusing harmful requests | Apr 2026 | 9.00% | System cards only |
| Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. | Hallucination rates | 5.74% | System cards only | |
| Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. | Deception and alignment | Apr 2026 | 43% | System cards only |
| Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. | Deception and alignment | 8.50% | System cards only | |
| Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. | Deception and alignment | 70% | System cards only | |
| Meta Underspecified Intent evaluation Models answer ambiguous requests sourced from real Meta AI chats where users corrected the reply. Rubrics score whether the true intent is met. | Following precise instructions | Apr 2026 | 71.2% | System cards only |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Finding and exploiting real bugs | Oct 2025 | 8.30% | System cards only |
| Omni3D A large benchmark for 3D object detection from a single image, built by unifying indoor and outdoor datasets such as SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI and nuScenes. Vision-language models answer its 3D grounding questions. | Spatial reasoning and counting | Jul 2022 | 53.4% | System cards only |
| Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. | Chat and instructions in many languages | Aug 2026 | 60.4% | System cards only |
| OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. | Machine learning engineering | 64.2% | System cards only | |
| ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. | AI research tasks | 64.7% | System cards only | |
| SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. | Fixing real repositories | Apr 2026 | 42.9% | System cards only |
| Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. | Secure code and review | 21.1% | System cards only | |
| XNLI Models classify entailment, contradiction or neutral for sentence pairs translated from MultiNLI into 15 languages. | Reasoning in many languages | Sep 2018 |  45.3% | Dropped |
| Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. | Following precise instructions | Sep 2026 | System cards only | |
| Community Alignment personalization Models predict an individual's preferred response from demographics and nine examples, across five countries. Reports global accuracy. | Bias, fairness and politics | System cards only | ||
| Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. | Hazardous knowledge | Oct 2025 | System cards only | |
| Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. | Hazardous knowledge | Apr 2026 | System cards only | |
| Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. | Refusing harmful requests | System cards only | ||
| SWE-Together Coding agents work through 109 repository tasks rebuilt from real multi-turn sessions between users and coding agents. A simulated user replays the original user's goals and gives feedback when the agent needs it. Scores count tasks solved and the number of corrections the user had to make. | Fixing real repositories | Jun 2026 | 69.3% | Not in cards |
| InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. | Simulated robots | Oct 2026 | 41.7% | Not in cards |
| TUA-Bench Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work. | Terminal tasks | Jun 2026 | 65.8% | Not in cards |
| SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. | Language reasoning | May 2019 | 90.4% | Not in cards |
| HackBench Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices. | Deception and alignment | Oct 2026 | 64.9 | Not in cards |
| Gaia2-CLI Agents complete realistic tasks through command-line tools. The benchmark measures task success across search, execution, adaptability, ambiguity, and time constraints. | General assistants | Apr 2026 | 57% | Not in cards |
| DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer. | Data science and SQL | Nov 2022 | 53.9% | Not in cards |
| AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. | Secure code and review | Apr 2025 | 11.5% | Not in cards |
| CyberSecEval MITRE Models are asked to help with attack steps drawn from the MITRE ATT&CK framework. Scored by how often the model complies. | Capture-the-flag challenges | Dec 2023 | 40.5% | Not in cards |
| CyberSecEval MITRE False Refusal Rate Models answer borderline but benign security questions. Scored by how often the model wrongly refuses. | Capture-the-flag challenges | Apr 2024 | 1.60% | Not in cards |
| ANLI R1 Models label premise and hypothesis pairs from the first adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Language reasoning | Oct 2019 | Not in cards | |
| ANLI R2 Models label premise and hypothesis pairs from the second adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Language reasoning | Oct 2019 | Not in cards | |
| ANLI R3 Models label premise and hypothesis pairs from the third adversarial test round as entailment, neutral, or contradiction. The score is test accuracy. | Language reasoning | Oct 2019 | Not in cards | |
| Automated LLM Speedrunning Benchmark Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover. | AI research tasks | Jun 2025 | Not in cards | |
| BackendBench LLMs write kernels that override PyTorch core operators. PyTorch's own OpInfo and FACTO tests check correctness, and real model tensor shapes measure speed. | GPU kernels | Jun 2025 | Not in cards | |
| CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. | Faithful to sources | Jun 2024 | Not in cards | |
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Finding and exploiting real bugs | Jul 2024 | Not in cards | |
| CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. | Secure code and review | Jul 2024 | Not in cards | |
| CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. | Capture-the-flag challenges | Apr 2024 | Not in cards | |
| MLGym-Bench Agents work on 13 open-ended AI research tasks in vision, language, reinforcement learning and game theory. Scores compare the agent's result with baselines on each task's metric. | Machine learning engineering | Feb 2025 | Not in cards | |
| MLQA Models extract answers from Wikipedia passages in seven languages, with questions and contexts that can be in different languages. Scored by F1 and exact match. | Reading in many languages | Oct 2019 | Not in cards | |
| Movie Gen Video Bench Video generation models make clips from a fixed prompt set, and people compare them. Meta released non-cherry-picked Movie Gen videos for comparison. | Video generation | Oct 2024 | Not in cards | |
| Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. | Web browsing | Oct 2026 | Not in cards | |
| XStoryCloze Models pick the right ending for short stories from StoryCloze, professionally translated into 10 languages. | Reasoning in many languages | Dec 2021 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Llama 2-Chat helpfulness human evaluation (Meta) Three human raters compare Llama 2-Chat with a rival chat model on about 4,000 single and multi-turn helpfulness prompts. | Overall chat quality | Jul 2023 | System cards only | |
| Llama 3 pairwise human evaluation (Meta) Human raters compare Llama responses with a rival model on about 7,000 prompts across six capabilities and three multiturn sets. | Overall chat quality | Jul 2024 | System cards only | |
| WildArtifactBench Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo. | Tasks from real jobs | Aug 2026 | Not in cards |