Many languages
Benchmarks in five or more languages. We count 315 benchmarks here, 5 of them arenas. A benchmark belongs here when language includes many languages.
How fast it grows
35 were released since July, against 40 in the first half of 2026.
How much is still open
No model has passed 85% on 66 of the 95 benchmarks here that score in percent, 69% of them.
Every benchmark in this view
Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.
| Benchmark | Level | Released | Best score | Status |
|---|---|---|---|---|
| AssemblyAI Speech Benchmarks Speech recognition systems transcribe clean, accented, multilingual and code-switched audio, identify entities and diarize speakers. Error rates compare production systems across public speech sets. | Expert | Nov 2026 | | Not in cards |
| NarrativeBench Models seek, include, exclude, and recover narratives from multilingual news articles. Citation-based F1 or recall scores narrative coverage after checking the response language. | Everyday | Nov 2026 | | Not in cards |
| Omnilingua-AudioMC Models follow requirements across spoken conversations in ten languages. The authors compare each model's spoken and text responses against the same 1,404 rubric items. | Everyday | Nov 2026 | | Not in cards |
| On Evaluating User Simulators for Interactive Agents Models play the user in replays of 351 real sessions. The evaluation compares simulated behavior with observed behavior and measures agreement with human rankings of nine agents. | Everyday | Nov 2026 | Not in cards | |
| Open TTS Leaderboard 1.4.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Everyday | Nov 2026 | | Not in cards |
| Open TTS Leaderboard 1.5.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Everyday | Nov 2026 | | Not in cards |
| Open TTS Leaderboard 1.6.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Everyday | Nov 2026 | | Not in cards |
| Addis Scribe Benchmark Speech models transcribe recordings in seven African languages. Addis AI compares transcripts with human references and measures how quickly some systems return text. | Everyday | Oct 2026 | | Not in cards |
| ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. | Expert | Oct 2026 | Not in cards | |
| BrowseSafe++ Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. | University | Oct 2026 | 91.5% | Not in cards |
| CulturalRiddles Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning. | Everyday | Oct 2026 | | Not in cards |
| ImgMT Models translate text embedded in diagrams, maps, and infographics. Evaluation measures text recognition, region matching, and translation with CER, F1, chrF, and COMET. | Everyday | Oct 2026 | 61.3 | Not in cards |
| Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. | Everyday | Oct 2026 | 100% | Not in cards |
| Open TTS Leaderboard 1.1.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Everyday | Oct 2026 | | Not in cards |
| Open TTS Leaderboard 1.2.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Everyday | Oct 2026 | | Not in cards |
| Open TTS Leaderboard 1.3.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Everyday | Oct 2026 | | Not in cards |
| SEA-HELM Audio Models answer the same reasoning and instruction-following tasks from speech and text in five languages. Identical checks measure the accuracy lost with speech input. | School | Oct 2026 | Not in cards | |
| τ-bench 2.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. | University | Oct 2026 | 79.9% | Not in cards |
| BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. | Expert | Sep 2026 | Not in cards | |
| CoSE-E Models transcribe business conversations that switch between languages. The main metric checks whether transcription errors change the answers to questions about the conversation. | Everyday | Sep 2026 | | Not in cards |
| E2E-SWE Agents build complete software repositories from natural language specifications in empty workspaces. Hidden tests require every generated project to install and run. | Research frontier | Sep 2026 | Not in cards | |
| Hume AI Voice Design Benchmark Listeners rate generated voices for English voice design, multilingual control, accents and voice qualities. | Everyday | Sep 2026 | | In use |
| Indic-KCC-Agri-Advisory-Benchmark Models answer crop questions from India’s Kisan Call Center in English and ten Indian languages. A judge compares each answer with a human advisor’s response. | University | Sep 2026 | | Not in cards |
| Last Translation Benchmark Models translate inputs that other systems struggled with. A model checks each translation against human-written rules. A translation passes only if it meets every rule. | Research frontier | Sep 2026 | 45.3% | Not in cards |
| Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. | University | Sep 2026 | Not in cards | |
| Open TTS Leaderboard 1.0.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. | Everyday | Sep 2026 | | Not in cards |
| Q2D-Web Retrievers search 190 million web documents for queries that agents reformulate from real conversations. | University | Sep 2026 | | Not in cards |
| SLATE Models generate slides that teach unfamiliar language patterns. Simulated learners take tests before and after the lesson and apply the rules to new examples. | University | Sep 2026 | 5.46% | Not in cards |
| Voice Controllability Leaderboard Text to speech systems create requested voices and follow delivery, emotion and role instructions. Blind human ratings and objective prosody measures score controllability. | Expert | Sep 2026 | | Not in cards |
| Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. | University | Aug 2026 | | System cards only |
| UniVerseBench Models answer questions about music across more than 38 cultural and linguistic groups. The authors report accuracy by language and skill. | University | Aug 2026 | | Not in cards |
| Bible Accuracy Benchmark Models quote verses from a named Bible translation. They earn points for accurate quotations and lose points for inventing verses. | School | Jul 2026 | Not in cards | |
| Indic DiarBench Models transcribe conversations in 22 Indian languages and identify who spoke each word. Human annotators supply speaker labels and timestamps, including for overlapping speech. | Everyday | Jul 2026 | 36.2% | Not in cards |
| Inspect India Evals Models answer questions about Indian culture and reason across Indian languages. Other tasks test social bias, digital public infrastructure safeguards and refusal behavior. | University | Jul 2026 | Not in cards | |
| PazaBench v2 Models transcribe speech in 61 African languages. The leaderboard measures character error rate, word error rate and transcription speed. | Everyday | Jul 2026 | 73.3% | Not in cards |
| BhashaBench-Multi (Krishi) Models answer agricultural exam questions in 22 Indian languages. The evaluation checks the selected answer against a key for each translated question. | University | Jun 2026 | Not in cards | |
| IndicContextEval Models transcribe speech in eight Indian languages using hints about the topic or names. One prompt deliberately supplies names from an unrelated field. | Everyday | Jun 2026 | | Not in cards |
| IOL-AI 2026 Models infer linguistic rules from Olympiad problems and answer individual questions. The official competition combines exact match and character-level similarity. | Competition | Jun 2026 | Not in cards | |
| MSQA Models answer culture-specific knowledge questions in 11 major languages. | University | Jun 2026 | | In use |
| RealityTest Conversational models answer identity questions in text and speech across five languages. A judge classifies explicit, ambiguous and missing identity disclosure. | Everyday | Jun 2026 | Not in cards | |
| Vaani Benchmark V1.0 Models transcribe speakers describing pictures. The scorer compares each prediction with three human transcripts and allows wording that matches any reference. | Everyday | Jun 2026 | Not in cards | |
| WMT26 Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments. | University | Jun 2026 | | In use |
| ACII DaiKon Challenge Models predict interpersonal influence, turn taking and rapport trajectories from multimodal dyadic conversations across five languages. | Expert | May 2026 | Not in cards | |
| Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. | University | May 2026 | 95.7% | In use |
| IFMTBench Models translate text while following explicit instructions, such as keeping JSON or HTML structure, using a glossary, using provided context and matching a register. It covers seven languages with 4,506 single-constraint and 2,838 multi-constraint items. Deterministic checkers verify some constraints, and a rubric-based model judge scores the rest. | University | May 2026 | | Not in cards |
| MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. | Competition | May 2026 | | Fading |
| StereoTales Models write multilingual stories about characters with controlled demographic attributes. Statistical tests find overrepresented associations, and people judge their harmfulness. | Everyday | May 2026 | Not in cards | |
| Deepgram Voice AI Benchmarks Speech models are compared on word error rate across production audio in 20 languages, while speech synthesis models are compared on read-aloud and hard-prompt sets. | Expert | Apr 2026 | Not in cards | |
| Dolly Machine Translated v2 Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Everyday | Apr 2026 | Not in cards | |
| m-ArenaHard v2.1 Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | University | Apr 2026 | Not in cards | |
| MU-Bench Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts. | Everyday | Apr 2026 | | Not in cards |
| PulseBench-Tab Models convert 1,820 table images in nine languages into HTML tables. T-LAG represents adjacent cells as directed edges and matches their text and structure against human annotations. The reported score is F1 over matched edge weights. | Everyday | Apr 2026 | 0.89 | Not in cards |
| WMT26 Video Subtitle Translation Models translate timed Chinese subtitles into five target languages using a synopsis for context. Human reviewers check the model judge’s assessments. | University | Apr 2026 | Not in cards | |
| BIBLE Translation Benchmark (Meta) Models translate verses from the Gospel of John into English. Meta scores the translations against reference verses using chrF++ and xCOMET. | University | Mar 2026 | Not in cards | |
| GlobalMGSM Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number. | School | Mar 2026 | Not in cards | |
| IndiVibe Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise. | Everyday | Mar 2026 | 90% | Fading |
| MDPBench Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage. | Everyday | Mar 2026 | | Not in cards |
| Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. | Everyday | Feb 2026 | System cards only | |
| GEBench Image generation models create interactive GUI states from prompts. A judge scores goal, logic, consistency, interface and quality across five task types. | Everyday | Feb 2026 | 63.2 | Not in cards |
| IndicIFEval Models follow instructions in 14 Indic languages. Rule-based checks test constraints such as required words, paragraph structure and word counts. | Everyday | Feb 2026 | 82.4% | Not in cards |
| Kiji PII Detection Models locate and label personal information in multilingual text. Dataiku publishes a test split, and independent evaluators compare predicted spans with its annotations. | Everyday | Feb 2026 | Not in cards | |
| MAEB Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages. | Everyday | Feb 2026 | | System cards only |
| Malt Freelancer Project Matching Models rank synthetic freelancer profiles against a search query or a project brief. Each task has 200 English prompts and 4,019 profiles across five languages. | University | Feb 2026 | | Not in cards |
| MultiClinSum-2 Models summarize 1,000 clinical cases per language across 15 languages. Organizers report four automatic metrics and four language model judgments separately. | University | Feb 2026 | Not in cards | |
| PazaBench Models transcribe speech in 39 African languages. The leaderboard measures character error rate, word error rate and transcription speed. | Everyday | Feb 2026 | Not in cards | |
| Synthetic Contextual ASR Benchmark (Indic) Models transcribe a synthetic caller’s response with or without the preceding conversation. A language model scores whether each transcript preserves the caller’s intent and named entities. | Everyday | Feb 2026 | Not in cards | |
| ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. | Everyday | Feb 2026 | 8.57 | System cards only |
| ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. | University | Feb 2026 | 1050.0 | System cards only |
| MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. | Expert | Jan 2026 | 42.2% | Not in cards |
| Multilingual Entity Linking of Skills Models match skill names from Belgian, German, and Swedish classifications to European skill entries. Official mappings identify the correct matches. | University | Jan 2026 | Not in cards | |
| Qwen3 forced alignment benchmarks Models place word or character boundaries in audio. Qwen measures timestamp shifts against automatic and human reference alignments. | School | Jan 2026 | Not in cards | |
| Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. | Everyday | Jan 2026 | Not in cards | |
| ViDoRe V3 Retrieval and RAG pipelines answer queries over visually rich documents, such as reports with tables, charts and images. The benchmark has 10 datasets from professional domains, about 26,000 pages and 3,099 human-verified queries, each in 6 languages. It scores retrieval relevance, bounding-box grounding and the final answers. | Everyday | Jan 2026 | | Not in cards |
| WAXAL Speech Recognition Test Speech models transcribe people describing pictures in African languages. Researchers compare the transcripts with human references using word error rate. | Everyday | Jan 2026 | Not in cards | |
| WMT26 Multilingual Instruction Shared Task Models answer questions from passages, summarize papers across languages and follow writing instructions. The overall ranking averages ranks across the three tasks. | University | Jan 2026 | Not in cards | |
| Global MMLU Lite 2.0 Models answer 400 human-translated MMLU questions per language in 18 languages, adding Albanian, Burmese and Welsh to the first 15. Accuracy. | University | Dec 2025 | | System cards only |
| TokSuite Models complete questions under alternative tokenizations. The benchmark measures the change in accuracy relative to the canonical representation. | School | Dec 2025 | Not in cards | |
| IndicParam Models answer multiple-choice questions from Indian competitive exams. The benchmark tests knowledge in low-resource Indic languages. | University | Nov 2025 | 45% | Not in cards |
| MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. | School | Nov 2025 | 99.3% | Not in cards |
| MultiClinAI Models identify diseases, symptoms, and procedures in seven languages. A second task transfers annotated Spanish mentions to the corresponding spans in six translations. | University | Nov 2025 | Not in cards | |
| NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. | University | Nov 2025 | | Fading |
| Omnilingual ASR Corpus Models transcribe recordings from speakers of many languages. Meta compares character error rates on held-out speech, including languages with little existing transcription data. | School | Nov 2025 | Not in cards | |
| RoCS-MT v2 Models translate Reddit posts with revised spelling annotations and corrected references. The authors combine COMETKiwi and MetricX rankings for translations of the original text. | Everyday | Nov 2025 | 7.00 | Not in cards |
| SMM4H-HeaRD 2026 Multilingual Adverse Drug Event Detection Models flag personal reports of drug side effects in seven languages. The organizers withheld Farsi from training and added 15,184 Farsi posts at test time. | Everyday | Nov 2025 | Not in cards | |
| Swivuriso Models transcribe read and spontaneous speech in seven South African languages. The authors measure word error rates and keep speakers separate across splits. | Everyday | Nov 2025 | 15% | Not in cards |
| Global PIQA Models pick the correct solution to everyday commonsense prompts in over 100 languages, many about local foods and customs. Scoring is accuracy. | Everyday | Oct 2025 | | Fading |
| HalloMTBench Models translate English text into one of 11 languages. Judges count translations that add unsupported material, use the wrong language, leave text untranslated or repeat material. | University | Oct 2025 | Not in cards | |
| PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. | School | Oct 2025 | | Not in cards |
| RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. | University | Oct 2025 | | Not in cards |
| Talking Points: When Chatbots Surface Russian State Media Chatbots answer questions about Ukraine in five languages. Analysts check whether each response cites an official Russian channel, a state-controlled outlet, or a state-linked source. | Everyday | Oct 2025 | Not in cards | |
| DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. | Everyday | Sep 2025 | | System cards only |
| Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. | Everyday | Sep 2025 | | Dropped |
| SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. | Everyday | Sep 2025 | | System cards only |
| FilBench Models perform 12 tasks in Philippine languages. The tasks test cultural knowledge, language processing, reading comprehension and generation. | School | Aug 2025 | Not in cards | |
| General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. | Aug 2025 | 0% | Fading | |
| LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. | Everyday | Aug 2025 | 12.0 | System cards only |
| CS-FLEURS Models transcribe speech that switches languages within sentences. Four test sets cover 113 language pairs. People read one set aloud, and synthesizers generate the others. | Everyday | Jul 2025 | Not in cards | |
| Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. | Expert | Jul 2025 | System cards only | |
| SHROOM-CAP Models read answers to questions about scientific papers in nine languages. They flag factual errors and fluency errors separately. | University | Jul 2025 | 0.61 | Not in cards |
| Uganda Cultural and Cognitive Benchmark Models answer questions about Ugandan daily life, institutions and cultural practices. A grader checks answers across 24 domains. | Everyday | Jul 2025 | Not in cards | |
| AfroBench-Lite Models complete language understanding, question answering, reasoning, and generation tasks across 14 African languages. Seven datasets use task-specific metrics, including accuracy, F1, character-level translation scores, and exact match. | School | Jun 2025 | 77.7 | Not in cards |
| Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. | Everyday | Jun 2025 | 64.7% | Fading |
| m-ArenaHard v2.0 Models answer 498 English prompts from Arena-Hard v2.0, translated into 22 more languages for 11,454 prompts in all. An LLM judge compares answers pairwise and reports win rates. | University | Jun 2025 | Not in cards | |
| MAXIFE Models follow 1,667 verifiable instructions in 23 languages and in cross-lingual settings. Rule-based checks and model judges score compliance. | Everyday | Jun 2025 | | Fading |
| MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. | University | Jun 2025 | | Not in cards |
| Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. | Competition | Jun 2025 | System cards only | |
| NTCIR-19 MedNLP-CALL Triage Classification Models classify synthetic emergency calls into three urgency levels from dispatcher and caller dialogue. Medical experts supplied the reference labels for the six-language task. | University | Jun 2025 | Not in cards | |
| WMT25 Systems translate the WMT25 general task test sets in 16 directions, mostly out of English. New pairs include English to Bhojpuri, Korean and Maasai. Humans evaluate the systems. | University | Jun 2025 | | System cards only |
| CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. | Everyday | May 2025 | System cards only | |
| HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. | Expert | May 2025 | 67.2% | In use |
| MiniMax TTS Multilingual Test Set Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity. | Everyday | May 2025 | | In use |
| MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. | University | May 2025 | | Not in cards |
| RESPIN-S1.0 Models transcribe dialect speakers reading sentences about farming and finance in nine Indian languages. The test holds out speakers and reports errors by dialect. | Everyday | May 2025 | Not in cards | |
| BRIDGE Models work on 87 tasks built from real clinical text, such as electronic health records, in nine languages. The tasks span triage, consultation, information extraction, diagnosis, prognosis and billing codes across 14 specialties. Each task uses its own automatic metric, and the board reports the average. | Expert | Apr 2025 | | Not in cards |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | University | Apr 2025 | | System cards only |
| LMArena Search Arena People compare answers from two anonymous search-augmented models on their own questions and vote for the better one. Votes become Bradley-Terry ratings. | Everyday | Apr 2025 | 1,255 | Fading |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Everyday | Apr 2025 | | Dropped |
| MADASR 2.0 Models transcribe read speech and spontaneous conversations across eight Indian languages. The organizers rank each track by character error and also score language and dialect identification. | Everyday | Apr 2025 | Not in cards | |
| MIEB Image and image-text embedding models are scored on 130 tasks in 38 languages, grouped into 8 task types such as retrieval, clustering and classification. | Everyday | Apr 2025 | Not in cards | |
| MultiBLiMP Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence. | Everyday | Apr 2025 | | Not in cards |
| Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. | Everyday | Apr 2025 | System cards only | |
| MultiLoKo Locally relevant knowledge questions in 31 languages, plus translations. Scored by exact match. | Everyday | Apr 2025 | | Fading |
| PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. | Competition | Apr 2025 | | In use |
| AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. | Everyday | Mar 2025 | | Dropped |
| Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. | Everyday | Mar 2025 | | System cards only |
| LingOly-TOO Models solve Linguistics Olympiad problems whose spelling patterns have been systematically changed while preserving the solution logic. Answers receive exact-match credit with no partial credit. The evaluation compares performance on original and obfuscated problems. | Competition | Mar 2025 | 46.7% | Not in cards |
| m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. | Everyday | Mar 2025 | | Dropped |
| MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. | University | Mar 2025 | | In use |
| WMT25 Multilingual Instruction Shared Task Models translate, solve language puzzles, generate text, summarize across languages and judge other answers. The shared task covers 30 languages. | University | Mar 2025 | Not in cards | |
| BenchMAX Models follow instructions, write code, reason, use tools, translate, and retrieve information across 17 languages. Ten tasks use rule checks, model-judged wins, code tests, answer matching, or translation scores. | University | Feb 2025 | | Not in cards |
| BOUQuET Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references. | Everyday | Feb 2025 | | System cards only |
| ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. | Everyday | Feb 2025 | 56.3% | Dropped |
| INJONGO Models identify a speaker’s intent and extract information from utterances in 16 African languages. | Everyday | Feb 2025 | 65.1% | Not in cards |
| KuralHub Models classify emotions in speech from 29 languages. | Everyday | Feb 2025 | Not in cards | |
| MT-AIME2024 Models solve the 30 AIME 2024 problems translated into 55 languages. A rule-based verifier checks each final answer. | Competition | Feb 2025 | | Dropped |
| SEA-HELM Models are evaluated in Filipino, Indonesian, Tamil, Thai and Vietnamese across five pillars: NLP classics, LLM-specific tasks, linguistics, culture and safety. | School | Feb 2025 | 79.9 | Not in cards |
| URO-Bench Spoken dialogue models answer speech with speech across Understanding, Reasoning and Oral conversation. It has a basic track of 16 datasets and a pro track of 20. | Everyday | Feb 2025 | | Fading |
| WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. | University | Feb 2025 | | In use |
| AfriHate Models classify hate speech and abusive language across African languages. | Everyday | Jan 2025 | | Not in cards |
| AmericasNLP 2025 Shared Task on Machine Translation into Indigenous Languages Models translate between Spanish and 13 Indigenous languages of the Americas. The organizers score each direction using character and word overlaps with reference translations. | University | Jan 2025 | Not in cards | |
| IberoBench Models perform language understanding and generation tasks in Iberian languages. The suite combines existing evaluations with new tests specific to each language. | University | Jan 2025 | | Not in cards |
| CC-OCR Models read text in 7,058 images across four tracks: scene text, multilingual text, document parsing, and key information extraction. Each track has its own metric. | Everyday | Dec 2024 | | Fading |
| Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. | University | Dec 2024 | 96% | Dropped |
| Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. | University | Dec 2024 | 95.7% | Dropped |
| INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. | University | Nov 2024 | 89.8% | Fading |
| MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. | University | Nov 2024 | 94.1% | Dropped |
| MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. | University | Nov 2024 | | Dropped |
| AIR-Bench 24.05 Retrievers find documents or passages that answer questions. This release covers 69 datasets across 13 languages. | University | Oct 2024 | Not in cards | |
| European LLM Leaderboard (OpenGPT-X) Models answer exam, science, commonsense, truthfulness, and math questions across European languages. The leaderboard also includes multilingual reading comprehension. | School | Oct 2024 | | Not in cards |
| m-ArenaHard Models answer the 500 Arena-Hard-Auto v0.1 prompts machine-translated into 22 more languages, 23 in all. An LLM judge compares answers pairwise and reports win rates. | University | Oct 2024 | | System cards only |
| M-RewardBench Reward models choose preferred responses across chat, safety, reasoning and translation tasks in 23 languages. Accuracy measures preference selection. | University | Oct 2024 | 83.5% | Not in cards |
| MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. | University | Oct 2024 | Not in cards | |
| Multi-IF Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. | Everyday | Oct 2024 | 89.1% | In use |
| Multilingual Entity Linking of Occupations Models map occupational titles from national classifications to European occupation codes. The benchmark scores the rank of the first correct match. | University | Oct 2024 | Not in cards | |
| Prompt injection test suite for LLM-based machine translation Translation systems receive TruthfulQA questions with injected instructions to answer instead of translate, across WMT 2024 language pairs. Scored by the share of outputs that answer rather than translate. | Everyday | Oct 2024 | | System cards only |
| Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. | Everyday | Oct 2024 | 72.6 | Not in cards |
| VHELM Vision-language models are scored across nine aspects, including perception, reasoning, knowledge, bias, fairness, multilinguality, robustness, toxicity and safety, from many existing datasets. | University | Oct 2024 | | Not in cards |
| xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. | Everyday | Oct 2024 | | System cards only |
| xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. | University | Oct 2024 | | System cards only |
| Linguini Models solve 160 International Linguistic Olympiad problems with 894 questions in 75 mostly low-resource languages. All evidence needed is in the context. | Competition | Sep 2024 | 24.1% | System cards only |
| MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. | University | Sep 2024 | 92.7% | In use |
| Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. | Everyday | Sep 2024 | | Dropped |
| Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. | School | Aug 2024 | System cards only | |
| Uhura ARC-Easy Models answer ARC-Easy science questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | School | Aug 2024 | System cards only | |
| Uhura TruthfulQA Models answer TruthfulQA questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | Everyday | Aug 2024 | System cards only | |
| AutoBencher Models answer generated benchmark questions. The framework searches for topics and produces tests of knowledge, multilingual reasoning, and math. | University | Jul 2024 | 55% | Not in cards |
| Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. | Jul 2024 | System cards only | ||
| Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. | University | Jul 2024 | | System cards only |
| Aya Red-teaming Models answer harmful prompts in eight languages across nine harm categories. Judges score refusal and safety behavior against annotated categories. | Everyday | Jun 2024 | Not in cards | |
| BLEnD Models answer 52.6k questions about everyday culture in 16 countries and regions across 13 languages. Scoring is accuracy on short-answer and multiple-choice formats. | Everyday | Jun 2024 | | Dropped |
| CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. | Everyday | Jun 2024 | | System cards only |
| Gretel Synthetic PII Finance Models identify and label personal information in synthetic financial documents across seven languages. Independent evaluations measure how well the predicted spans match the labeled spans. | Everyday | Jun 2024 | Not in cards | |
| IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. | School | Jun 2024 | 59% | Not in cards |
| Language Confusion Benchmark Models answer prompts in 15 languages and must reply in the language the user wants, both in monolingual and crosslingual settings. A language detector scores line-level and word-level pass rates. | Everyday | Jun 2024 | | System cards only |
| LingOly Models infer linguistic rules from examples in unfamiliar languages. They use those rules to solve linguistics olympiad problems. | Competition | Jun 2024 | 46.3% | Not in cards |
| M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. | School | Jun 2024 | | System cards only |
| MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. | School | Jun 2024 | 83% | In use |
| MultiGEC 2025 Models correct whole texts in 12 European languages. One track rewards minimal edits, and the other rewards fluent rewrites. | School | Jun 2024 | | Not in cards |
| Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. | Everyday | Jun 2024 | 4.15 | Dropped |
| SORRY-Bench Models get unsafe requests across a fine-grained taxonomy of topics and linguistic variations. A judge model scores whether each answer refuses or complies. | Everyday | Jun 2024 | 5.23% | Not in cards |
| Vision Arena People send an image with a prompt, compare answers from two anonymous vision-language models, and vote for the better one. Votes become Bradley-Terry ratings. | Everyday | Jun 2024 | 1,309 | Not in cards |
| WMT24 Systems translate the WMT24 general task test sets, mostly English to X plus some X to Y pairs, at paragraph level across several domains. The shared task ranks systems by human ESA judgments. | University | Jun 2024 | Not in cards | |
| ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. | School | May 2024 | Not in cards | |
| MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. | Everyday | May 2024 | | Fading |
| SEAL Multilingual Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo. | Everyday | May 2024 | 1,134 | Not in cards |
| WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. | Everyday | May 2024 | 98% | System cards only |
| B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. | Everyday | Apr 2024 | 47.7% | Not in cards |
| CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. | University | Apr 2024 | | System cards only |
| CyberSecEval Prompt Injection Models face text prompts that embed injected instructions across 15 strategies. Scored as attack success rate. | Everyday | Apr 2024 | | System cards only |
| European Parliament election chatbot audit Chatbots answer ten European election questions in ten languages. Reviewers score procedural answers for accuracy and political answers for neutrality. | Everyday | Apr 2024 | Not in cards | |
| IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. | Everyday | Apr 2024 | | Dropped |
| RTP-LX Models continue or judge toxic prompts hand-translated from RealToxicityPrompts into dozens of languages, with native-speaker labels. Measures toxic output and toxicity detection. | Everyday | Apr 2024 | Not in cards | |
| EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. | School | Mar 2024 | | Not in cards |
| PII-Masking-300k Models locate and classify personal information in multilingual synthetic text. The open portion covers six languages. The authors separately describe a finance and insurance portion. | Everyday | Mar 2024 | Not in cards | |
| SemEval 2024 Task 1 semantic textual relatedness Models rank sentence pairs by relatedness across languages. The evaluation compares their rankings with human judgments. | School | Mar 2024 | | Not in cards |
| LAMBADA Multilingual (Stable LM 2) Models predict missing final words in narrative passages translated for the Stable LM 2 evaluation. The harness provides five language configurations. | Everyday | Feb 2024 | | Not in cards |
| mDolly Models answer open-ended Dolly prompts machine-translated into 101 languages, plus human-edited sets in 6 languages and human-written prompts in 7. An LLM judge compares or rates the answers. | Everyday | Feb 2024 | Not in cards | |
| Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. | University | Feb 2024 | | Dropped |
| Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. | Everyday | Feb 2024 | | Dropped |
| MuTox Models listen to short speech clips in 30 languages and classify whether they contain toxic speech. The harness compares their decisions with human labels. | Everyday | Jan 2024 | Not in cards | |
| DISPLACE Challenge 2024 Models mark speakers and languages in multilingual conversations and transcribe individual speakers. The organizers score timing errors on room recordings and word errors on close microphone recordings. | Everyday | Dec 2023 | Not in cards | |
| AfroBench Models perform language understanding, reasoning, and generation tasks across 64 African languages. The full suite draws on 22 datasets. | School | Nov 2023 | 59.6 | Not in cards |
| MELA Models judge whether sentences are linguistically acceptable across languages. The evaluation compares their judgments with the labeled examples. | Everyday | Nov 2023 | Not in cards | |
| FLORES+ Models translate English Wikipedia sentences that professionals and community contributors translated into over 200 languages. Outputs are scored with metrics such as chrF against the references. | University | Oct 2023 | | System cards only |
| IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. | School | Oct 2023 | | Dropped |
| LIMMITS 2024 Speech models read text in a target voice across seven languages. Native listeners score naturalness and speaker similarity on the same prompts. | Everyday | Sep 2023 | Not in cards | |
| NusaWrites Models interpret and translate texts in underrepresented Indonesian languages. Google reports emotion classification on original paragraphs and retrieval of matching translations. | School | Sep 2023 | Not in cards | |
| Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. | Everyday | Sep 2023 | | Fading |
| ParlaSent Models classify the sentiment of parliamentary sentences as positive, neutral, or negative. Two additional test sets contain 2,600 English and 2,600 South Slavic sentences. | Everyday | Sep 2023 | 0.77 | Not in cards |
| SIB-200 Models predict the topic of a sentence across 205 languages and dialects. The sentences come from FLORES-200. | Everyday | Sep 2023 | 48.7% | Not in cards |
| XTD-200 Models retrieve images and captions across languages using translations of the XTD test set. Results measure how often the matching item ranks first. | School | Sep 2023 | Not in cards | |
| Belebele Models answer multiple-choice reading comprehension questions on FLORES passages in 122 language variants. Scored by accuracy. | Everyday | Aug 2023 | 51.1% | System cards only |
| IndicQA Models extract answers from Wikipedia passages in Indian languages. Native speakers wrote the questions and answers. | School | Jul 2023 | Not in cards | |
| Okapi ARC Models choose answers to translated science questions from ARC-Challenge. | School | Jul 2023 | Not in cards | |
| Okapi HellaSwag Models choose the most plausible next event in a translated description of an everyday activity. | Everyday | Jul 2023 | Not in cards | |
| Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. | University | Jul 2023 | | Dropped |
| Okapi TruthfulQA Models distinguish true answers from common misconceptions in translated multiple-choice questions. | Everyday | Jul 2023 | Not in cards | |
| SympTEMIST Models locate symptoms, signs, and findings in Spanish clinical cases. They also map given mentions to SNOMED CT codes in Spanish and five additional languages. | University | Jul 2023 | Not in cards | |
| WMT23 Models translate the WMT23 general task test sets between English and languages such as German, Chinese, and Japanese. Model cards score output with BLEURT. | University | Jul 2023 | | Dropped |
| AGILE Automatic Genre Identification Benchmark Models assign one of eight genres to web texts, including news, opinion, and advertising. The test covers English and ten other languages. | Everyday | Jun 2023 | 0.82 | Not in cards |
| Babel-ImageNet Models classify ImageNet images using class descriptions in different languages. The benchmark measures how recognition accuracy changes with the prompt language. | Everyday | Jun 2023 | Not in cards | |
| M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. | School | Jun 2023 | Dropped | |
| AfriQA Models retrieve evidence across languages to answer questions written in African languages. | School | May 2023 | Not in cards | |
| clembench Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages. | Everyday | May 2023 | 88.8 | Not in cards |
| IN22-Gen Models translate parallel sentences across English and Indian languages. Google evaluates whether an embedding model can match a sentence with its translation. | School | May 2023 | | Not in cards |
| IndicLID benchmark Models identify Indian languages in native scripts and in Romanized text. | School | May 2023 | | Not in cards |
| LMArena (Elo) People chat with two anonymous models side by side and vote for the better reply. Votes become Elo-style ratings for each model. | May 2023 | | Fading | |
| MasakhaPOS Models assign parts of speech to words in African languages. The authors compare tagging models across languages. | School | May 2023 | 62.8% | Not in cards |
| Massively Multilingual Speech labeled corpus Models transcribe recorded readings across languages. The evaluation compares predictions with reference transcripts using character error rate. | School | May 2023 | Not in cards | |
| NollySenti Models classify the sentiment of Nollywood movie reviews in five languages spoken in Nigeria. | Everyday | May 2023 | Not in cards | |
| Vistaar Models transcribe speech across 59 combinations of language and dataset in 12 Indian languages. Each test has its own word error rate. | Everyday | May 2023 | Not in cards | |
| MasakhaNEWS Models predict the topic of news articles written in African languages. | Everyday | Apr 2023 | Not in cards | |
| Casual Conversations v2 Models classify attributes of speakers in recorded interviews. The HELM audio tasks compare age and gender predictions with participants’ own labels. | Everyday | Mar 2023 | Not in cards | |
| MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. | University | Mar 2023 | System cards only | |
| AfriSenti Models classify sentiment in human-annotated social media posts across 14 African languages. | Everyday | Feb 2023 | Not in cards | |
| SIGMORPHON 2023 Shared Task on Interlinear Glossing Models label each word’s meaningful parts with translations or grammatical tags. The organizers rank submissions by word-level accuracy pooled across languages. | Expert | Feb 2023 | Not in cards | |
| LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. | University | Jan 2023 | Not in cards | |
| DISPLACE Challenge 2023 Models mark who speaks when and which language each person speaks in multilingual conversations. The scorer counts missed speech, false alarms and incorrect assignments, including overlaps. | Everyday | Dec 2022 | Not in cards | |
| EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. | University | Oct 2022 | Not in cards | |
| MasakhaNER 2.0 Models identify named entities in human-annotated text across 20 African languages. | School | Oct 2022 | Not in cards | |
| MGSM 250 GSM8K problems translated into 10 languages. Final-answer accuracy. | School | Oct 2022 | | Fading |
| MTEB Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities. | Oct 2022 | | Fading | |
| NTREX-128 Models translate 1,997 English news sentences into 128 languages, with human reference translations. Scored with metrics such as COMET or chrF. | University | Oct 2022 | | System cards only |
| MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. | Everyday | Sep 2022 | | System cards only |
| mRobust04 Models rank documents for translated queries from the TREC Robust04 collection. Relevance judgments support comparisons across languages. | School | Sep 2022 | Not in cards | |
| SemEval 2023 Task 3: Genre, Framing and Persuasion Techniques in Online News Models classify a news article as reporting, opinion, or satire. They also label its frames and identify persuasion techniques in individual paragraphs. | University | Sep 2022 | Not in cards | |
| FLORES-200 Models translate English Wikipedia sentences into about 200 languages. Outputs are scored with BLEU or chrF against human reference translations. | Everyday | Jul 2022 | | Dropped |
| MAFAND-MT Models translate news text in a benchmark covering 21 African languages. | School | Jul 2022 | Not in cards | |
| Multilingual HateCheck Models distinguish hate speech from statements that quote, reject, or otherwise resemble hate speech. The test covers multiple languages. | Everyday | Jul 2022 | Not in cards | |
| MultiNERD Models locate fine-grained named entities in Wikipedia and Wikinews sentences across ten languages. | School | Jul 2022 | Not in cards | |
| SemEval 2022 Task 8 multilingual news article similarity Models rate the similarity of two news articles, including articles in different languages. The evaluation compares predicted similarities with human judgments. | School | Jul 2022 | | Not in cards |
| Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. | School | May 2022 | | Not in cards |
| FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. | Everyday | May 2022 | | In use |
| GeoMLAMA Models complete sentences about daily life in five countries. The test checks whether models rank locally correct answers above other choices in five languages. | Everyday | May 2022 | Not in cards | |
| NusaX Models predict sentiment in parallel reviews written in Indonesian local languages, Indonesian, and English. | Everyday | May 2022 | Not in cards | |
| IndicXNLI Models compare a premise and hypothesis in Indian languages. They predict whether the premise entails, contradicts or leaves the hypothesis unresolved. | School | Apr 2022 | Not in cards | |
| MASSIVE Models identify an utterance’s intent and label its argument slots across 51 languages. The held-out test reports intent accuracy, slot-filling F1, and exact match accuracy for the complete prediction. | Everyday | Apr 2022 | Not in cards | |
| Hugging Face Speech Bench Speech recognition models transcribe recordings. The board collects error rates from model cards. | Everyday | Mar 2022 | Not in cards | |
| MuMiN Models use social media context to classify claims as factual or misleading. A second task asks which verdict applies to the claim a post discusses. | University | Feb 2022 | Not in cards | |
| CrossSum Models read a news article in one language and summarize it in another. The dataset aligns articles and summaries across more than 1,500 language pairs. | School | Dec 2021 | Not in cards | |
| XStoryCloze Models pick the right ending for short stories from StoryCloze, professionally translated into 10 languages. | Everyday | Dec 2021 | 58.2% | Not in cards |
| WikiNEuRal Models locate named entities in Wikipedia text across nine languages using automatically produced reference annotations. | School | Nov 2021 | Not in cards | |
| CLEF HIPE 2022 Models identify and link names in digitized newspapers and scholarly commentaries on classical texts. Organizers compare performance across six collections in five languages. | University | Sep 2021 | Not in cards | |
| MultiEURLEX Models assign subject labels to European Union legislation written in different languages. | Expert | Sep 2021 | | Not in cards |
| EuroEval Models answer questions, summarize text, and perform language understanding tasks across European languages. EuroEval combines task rankings on its European leaderboard. | School | Aug 2021 | 1.17 | Not in cards |
| mMARCO Models rank passages by relevance to a query. The multilingual benchmark translates the MS MARCO passage ranking collection. | School | Aug 2021 | Not in cards | |
| Flores Models translate 3,001 English Wikipedia sentences that professionals translated into 101 languages. Scored with BLEU or chrF against references. | University | Jun 2021 | | Dropped |
| XWinograd Models resolve ambiguous pronouns in six languages by choosing which noun fits each sentence. | Everyday | Jun 2021 | Not in cards | |
| MInDS-14 Models identify the banking service a caller wants from a spoken request. | Everyday | Apr 2021 | Not in cards | |
| MUCS 2021 Models transcribe speech in Indian languages and mixed-language recordings. NVIDIA reports results on Bengali-English and Hindi-English speech. | Everyday | Apr 2021 | Not in cards | |
| XTREME-R Multilingual models are fine-tuned on English data and tested on 10 tasks in 50 typologically diverse languages, including harder language-agnostic retrieval tasks. It extends XTREME and adds MultiCheckList, a multilingual diagnostic suite. | School | Apr 2021 | Not in cards | |
| MasakhaNER Models recognize named entities in ten African languages. The authors compare supervised and cross-language transfer methods. | School | Mar 2021 | Not in cards | |
| VoxPopuli Models transcribe European Parliament speeches. The corpus also supports speech translation across languages. | Everyday | Jan 2021 | Not in cards | |
| Cross-lingual Test Dataset Models retrieve images from captions written in different languages. The benchmark compares retrieval against paired photographs and descriptions. | School | Dec 2020 | Not in cards | |
| Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. | Everyday | Dec 2020 | | Fading |
| EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. | School | Nov 2020 | Not in cards | |
| Tatoeba Translation Challenge Models translate sentences for many language pairs. The challenge publishes held-out test sets so systems can report comparable translation scores. | University | Nov 2020 | Not in cards | |
| VoxLingua107 Models identify the language in a speech recording. The corpus includes 107 languages. | Everyday | Nov 2020 | Not in cards | |
| WikiLingua Models summarize procedural articles in different languages. Aligned WikiHow articles support both same-language and cross-language evaluation. | School | Nov 2020 | Not in cards | |
| XOR-TyDi QA Models retrieve passages across languages to answer questions written by speakers of seven languages. Tasks require English evidence or answers in the question’s language. | School | Oct 2020 | Not in cards | |
| MTOP Models parse a user's request into its intent and the details needed to act on it. | School | Aug 2020 | Not in cards | |
| CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. | University | Jul 2020 | | Fading |
| MKQA Models answer 10,000 open-domain questions translated into 26 languages. Exact match and token F1 measure agreement with human answers, including unanswerable questions. | Everyday | Jul 2020 | Not in cards | |
| OPUS-100 Models translate between English and other languages. Separate test sets measure performance on supervised directions and language pairs without direct training data. | University | Jul 2020 | Not in cards | |
| XCOPA Models pick the more plausible cause or effect of a premise from two options, in 11 languages. Scoring is accuracy. | Everyday | May 2020 | 94.4% | Dropped |
| TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. | Everyday | Mar 2020 | | Dropped |
| XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. | School | Mar 2020 | Not in cards | |
| Common Voice Crowd-sourced multilingual read-speech corpus. Used here as an ASR test set scored by word error rate. | Everyday | Dec 2019 | | Fading |
| MLQA Models extract answers from Wikipedia passages in seven languages, with questions and contexts that can be in different languages. Scored by F1 and exact match. | Everyday | Oct 2019 | Not in cards | |
| XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. | Everyday | Oct 2019 | | Dropped |
| PAWS-X Models decide whether two sentences with similar wording express the same meaning across languages. | School | Aug 2019 | Not in cards | |
| Tatoeba sentence retrieval Models find the English translation of a sentence among competing sentences using multilingual embeddings. | School | Dec 2018 | | Not in cards |
| XNLI Models classify entailment, contradiction or neutral for sentence pairs translated from MultiNLI into 15 languages. | Everyday | Sep 2018 | | Dropped |
| VoxCeleb2 Models compare two interview recordings and decide whether the same person speaks in both. | Everyday | Jun 2018 | | Not in cards |
| Corpus of Multilingual Code-switched Soap Opera Speech Models transcribe South African television dialogue in which speakers switch between English and four Southern African languages. | Everyday | May 2018 | Not in cards | |
| Opusparcus Models identify paraphrases among movie and television subtitle sentences in six European languages. | School | May 2018 | Not in cards | |
| WMT 2016 News Translation Models translate news between languages. The harness compares translations with human references from the 2016 shared task. | University | Aug 2016 | Not in cards | |
| WMT 2014 News Translation Models translate news text between English and other languages. The shared task compares submitted translations on a common test set. | University | Jun 2014 | Not in cards | |
| Asian Language Treebank Models translate parallel sentences across Asian languages. The llm-jp-eval harness uses the parallel corpus for machine translation. | School | Not in cards | ||
| BUCC 2018 bitext mining shared task Models find translation pairs in two collections of sentences. The evaluation compares the retrieved pairs with a reference list. | School | Not in cards | ||
| CALLHOME diarization Models identify speakers in multilingual telephone conversations. NVIDIA evaluates Part 2 with two to six speakers per call. | Everyday | Not in cards | ||
| Cyrillic Turkic language identification Models identify the language of text written in Cyrillic. The test includes Turkic languages and Russian. | School | Not in cards | ||
| Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. | Not in cards | |||
| Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. | Expert | System cards only | ||
| IARPA Babel speech recognition Models transcribe conversational speech in languages with limited labeled data. Meta measures character error against reference transcripts on the Babel development split. | School | Not in cards | ||
| IWSLT 2017 TED Translation Models translate TED talk transcripts between languages. The campaign includes bilingual translation and multilingual translation with unseen language pairs. | University | Not in cards | ||
| LAMBADA OpenAI Multilingual Models predict the last word in short narrative passages. The harness includes English and four machine-translated test sets. | Everyday | Not in cards | ||
| LLM Round-Trip Translation Benchmark Models translate text into another language and back. The evaluation measures how well the original meaning survives. | University | 8.69 | Not in cards | |
| MAPA EUR-Lex Models mark sensitive entities in EU legal texts. MAPA annotates general and fine-grained categories. The multilingual conversion covers 21 languages. | University | Not in cards | ||
| MIEB Lite Embedding models complete the MIEB Lite task set. The official API reports the mean across task types. | | System cards only | ||
| Multilingual Conversational Speech Language Model Challenge Models recognize conversational speech across languages. Qwen compares transcription error and language identification accuracy on the challenge recordings. | Everyday | Not in cards | ||
| QA4MRE Models read documents and choose answers that require integrating information across the text. The harness includes the 2011, 2012 and 2013 CLEF tests. | University | Not in cards | ||
| Sunbird African Language Technology Models translate parallel sentences between English and six languages spoken in Uganda and East Africa. | School | Not in cards | ||
| Voice Arena Full Duplex Bench Voice agents respond to pauses, changes of speaker, acknowledgments, and interruptions. The board averages success rates across these conversational behaviors. | Everyday | | Not in cards | |
| xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. | Not in cards |