Benchmark Almanac

Many languages

Benchmarks in five or more languages. We count 315 benchmarks here, 5 of them arenas. A benchmark belongs here when language includes many languages.

How fast it grows

35 were released since July, against 40 in the first half of 2026.

  1. Earlier 98
  2. 2024 H1 32
  3. 2024 H2 27
  4. 2025 H1 41
  5. 2025 H2 25
  6. 2026 H1 40
  7. 2026 H2 35

How much is still open

No model has passed 85% on 66 of the 95 benchmarks here that score in percent, 69% of them.

Every benchmark in this view

Newest first. Facets come from each benchmark’s own documents, and our methodology says how we tag them.

BenchmarkLevelReleasedBest scoreStatus
AssemblyAI Speech Benchmarks Speech recognition systems transcribe clean, accented, multilingual and code-switched audio, identify entities and diarize speakers. Error rates compare production systems across public speech sets. Expert Nov 2026 5.60% Not in cards
NarrativeBench Models seek, include, exclude, and recover narratives from multilingual news articles. Citation-based F1 or recall scores narrative coverage after checking the response language. Everyday Nov 2026 90.7% Not in cards
Omnilingua-AudioMC Models follow requirements across spoken conversations in ten languages. The authors compare each model's spoken and text responses against the same 1,404 rubric items. Everyday Nov 2026 64% Not in cards
On Evaluating User Simulators for Interactive Agents Models play the user in replays of 351 real sessions. The evaluation compares simulated behavior with observed behavior and measures agreement with human rankings of nine agents. Everyday Nov 2026 Not in cards
Open TTS Leaderboard 1.4.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Everyday Nov 2026 2.32% Not in cards
Open TTS Leaderboard 1.5.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Everyday Nov 2026 2.32% Not in cards
Open TTS Leaderboard 1.6.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Everyday Nov 2026 2.32% Not in cards
Addis Scribe Benchmark Speech models transcribe recordings in seven African languages. Addis AI compares transcripts with human references and measures how quickly some systems return text. Everyday Oct 2026 29.9% Not in cards
ASTRA-bench Agents use personal context and interactive tools to complete more than 2,000 multi-step scenarios. Automated checks score whether each user intent is satisfied. Expert Oct 2026 Not in cards
BrowseSafe++ Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. University Oct 2026 91.5% Not in cards
CulturalRiddles Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning. Everyday Oct 2026 79.4% Not in cards
ImgMT Models translate text embedded in diagrams, maps, and infographics. Evaluation measures text recognition, region matching, and translation with CER, F1, chrF, and COMET. Everyday Oct 2026 61.3 Not in cards
Navital Gym Agents complete tasks in six simulated Indian service settings. This port covers UPI Desk, which grades 50 payment and dispute tasks using written rules. Everyday Oct 2026 100% Not in cards
Open TTS Leaderboard 1.1.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Everyday Oct 2026 3.38% Not in cards
Open TTS Leaderboard 1.2.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Everyday Oct 2026 3.38% Not in cards
Open TTS Leaderboard 1.3.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Everyday Oct 2026 2.32% Not in cards
SEA-HELM Audio Models answer the same reasoning and instruction-following tasks from speech and text in five languages. Identical checks measure the accuracy lost with speech input. School Oct 2026 Not in cards
τ-bench 2.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. University Oct 2026 79.9% Not in cards
BabelArena Agents complete 699 tasks from four benchmark families adapted across 23 languages and 13 domains. Pass rates and language consistency score each task family. Expert Sep 2026 Not in cards
CoSE-E Models transcribe business conversations that switch between languages. The main metric checks whether transcription errors change the answers to questions about the conversation. Everyday Sep 2026 4.20% Not in cards
E2E-SWE Agents build complete software repositories from natural language specifications in empty workspaces. Hidden tests require every generated project to install and run. Research frontier Sep 2026 Not in cards
Hume AI Voice Design Benchmark Listeners rate generated voices for English voice design, multilingual control, accents and voice qualities. Everyday Sep 2026 71.4 In use
Indic-KCC-Agri-Advisory-Benchmark Models answer crop questions from India’s Kisan Call Center in English and ten Indian languages. A judge compares each answer with a human advisor’s response. University Sep 2026 4 Not in cards
Last Translation Benchmark Models translate inputs that other systems struggled with. A model checks each translation against human-written rules. A translation passes only if it meets every rule. Research frontier Sep 2026 45.3% Not in cards
Omnilingua-Bench Models handle long-audio transcription, multilingual long-audio question answering, multi-speaker attribution and complex spoken instructions. WER, diarization, accuracy and rubric pass rates score the tracks. University Sep 2026 Not in cards
Open TTS Leaderboard 1.0.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Everyday Sep 2026 3.92% Not in cards
Q2D-Web Retrievers search 190 million web documents for queries that agents reformulate from real conversations. University Sep 2026 79.3% Not in cards
SLATE Models generate slides that teach unfamiliar language patterns. Simulated learners take tests before and after the lesson and apply the rules to new examples. University Sep 2026 5.46% Not in cards
Voice Controllability Leaderboard Text to speech systems create requested voices and follow delivery, emotion and role instructions. Blind human ratings and objective prosody measures score controllability. Expert Sep 2026 71.4 Not in cards
Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. University Aug 2026 60.4% System cards only
UniVerseBench Models answer questions about music across more than 38 cultural and linguistic groups. The authors report accuracy by language and skill. University Aug 2026 74.4% Not in cards
Bible Accuracy Benchmark Models quote verses from a named Bible translation. They earn points for accurate quotations and lose points for inventing verses. School Jul 2026 Not in cards
Indic DiarBench Models transcribe conversations in 22 Indian languages and identify who spoke each word. Human annotators supply speaker labels and timestamps, including for overlapping speech. Everyday Jul 2026 36.2% Not in cards
Inspect India Evals Models answer questions about Indian culture and reason across Indian languages. Other tasks test social bias, digital public infrastructure safeguards and refusal behavior. University Jul 2026 Not in cards
PazaBench v2 Models transcribe speech in 61 African languages. The leaderboard measures character error rate, word error rate and transcription speed. Everyday Jul 2026 73.3% Not in cards
BhashaBench-Multi (Krishi) Models answer agricultural exam questions in 22 Indian languages. The evaluation checks the selected answer against a key for each translated question. University Jun 2026 Not in cards
IndicContextEval Models transcribe speech in eight Indian languages using hints about the topic or names. One prompt deliberately supplies names from an unrelated field. Everyday Jun 2026 17.5% Not in cards
IOL-AI 2026 Models infer linguistic rules from Olympiad problems and answer individual questions. The official competition combines exact match and character-level similarity. Competition Jun 2026 Not in cards
MSQA Models answer culture-specific knowledge questions in 11 major languages. University Jun 2026 50.2% In use
RealityTest Conversational models answer identity questions in text and speech across five languages. A judge classifies explicit, ambiguous and missing identity disclosure. Everyday Jun 2026 Not in cards
Vaani Benchmark V1.0 Models transcribe speakers describing pictures. The scorer compares each prediction with three human transcripts and allows wording that matches any reference. Everyday Jun 2026 Not in cards
WMT26 Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments. University Jun 2026 83.6 In use
ACII DaiKon Challenge Models predict interpersonal influence, turn taking and rapport trajectories from multimodal dyadic conversations across five languages. Expert May 2026 Not in cards
Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. University May 2026 95.7% In use
IFMTBench Models translate text while following explicit instructions, such as keeping JSON or HTML structure, using a glossary, using provided context and matching a register. It covers seven languages with 4,506 single-constraint and 2,838 multi-constraint items. Deterministic checkers verify some constraints, and a rubric-based model judge scores the rest. University May 2026 81.1% Not in cards
MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. Competition May 2026 86% Fading
StereoTales Models write multilingual stories about characters with controlled demographic attributes. Statistical tests find overrepresented associations, and people judge their harmfulness. Everyday May 2026 Not in cards
Deepgram Voice AI Benchmarks Speech models are compared on word error rate across production audio in 20 languages, while speech synthesis models are compared on read-aloud and hard-prompt sets. Expert Apr 2026 Not in cards
Dolly Machine Translated v2 Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. Everyday Apr 2026 Not in cards
m-ArenaHard v2.1 Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. University Apr 2026 Not in cards
MU-Bench Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts. Everyday Apr 2026 14.9% Not in cards
PulseBench-Tab Models convert 1,820 table images in nine languages into HTML tables. T-LAG represents adjacent cells as directed edges and matches their text and structure against human annotations. The reported score is F1 over matched edge weights. Everyday Apr 2026 0.89 Not in cards
WMT26 Video Subtitle Translation Models translate timed Chinese subtitles into five target languages using a synopsis for context. Human reviewers check the model judge’s assessments. University Apr 2026 Not in cards
BIBLE Translation Benchmark (Meta) Models translate verses from the Gospel of John into English. Meta scores the translations against reference verses using chrF++ and xCOMET. University Mar 2026 Not in cards
GlobalMGSM Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number. School Mar 2026 Not in cards
IndiVibe Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise. Everyday Mar 2026 90% Fading
MDPBench Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage. Everyday Mar 2026 86.4 Not in cards
Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. Everyday Feb 2026 System cards only
GEBench Image generation models create interactive GUI states from prompts. A judge scores goal, logic, consistency, interface and quality across five task types. Everyday Feb 2026 63.2 Not in cards
IndicIFEval Models follow instructions in 14 Indic languages. Rule-based checks test constraints such as required words, paragraph structure and word counts. Everyday Feb 2026 82.4% Not in cards
Kiji PII Detection Models locate and label personal information in multilingual text. Dataiku publishes a test split, and independent evaluators compare predicted spans with its annotations. Everyday Feb 2026 Not in cards
MAEB Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages. Everyday Feb 2026 49.9 System cards only
Malt Freelancer Project Matching Models rank synthetic freelancer profiles against a search query or a project brief. Each task has 200 English prompts and 4,019 profiles across five languages. University Feb 2026 50.4% Not in cards
MultiClinSum-2 Models summarize 1,000 clinical cases per language across 15 languages. Organizers report four automatic metrics and four language model judgments separately. University Feb 2026 Not in cards
PazaBench Models transcribe speech in 39 African languages. The leaderboard measures character error rate, word error rate and transcription speed. Everyday Feb 2026 Not in cards
Synthetic Contextual ASR Benchmark (Indic) Models transcribe a synthetic caller’s response with or without the preceding conversation. A language model scores whether each transcript preserves the caller’s intent and named entities. Everyday Feb 2026 Not in cards
ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. Everyday Feb 2026 8.57 System cards only
ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. University Feb 2026 1050.0 System cards only
MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. Expert Jan 2026 42.2% Not in cards
Multilingual Entity Linking of Skills Models match skill names from Belgian, German, and Swedish classifications to European skill entries. Official mappings identify the correct matches. University Jan 2026 Not in cards
Qwen3 forced alignment benchmarks Models place word or character boundaries in audio. Qwen measures timestamp shifts against automatic and human reference alignments. School Jan 2026 Not in cards
Qwen3-ASR internal transcription benchmarks Models transcribe accented conversations, noisy speech, tongue twisters, and multilingual news. Qwen reports error rates for each recording condition. Everyday Jan 2026 Not in cards
ViDoRe V3 Retrieval and RAG pipelines answer queries over visually rich documents, such as reports with tables, charts and images. The benchmark has 10 datasets from professional domains, about 26,000 pages and 3,099 human-verified queries, each in 6 languages. It scores retrieval relevance, bounding-box grounding and the final answers. Everyday Jan 2026 0.58 Not in cards
WAXAL Speech Recognition Test Speech models transcribe people describing pictures in African languages. Researchers compare the transcripts with human references using word error rate. Everyday Jan 2026 Not in cards
WMT26 Multilingual Instruction Shared Task Models answer questions from passages, summarize papers across languages and follow writing instructions. The overall ranking averages ranks across the three tasks. University Jan 2026 Not in cards
Global MMLU Lite 2.0 Models answer 400 human-translated MMLU questions per language in 18 languages, adding Albanian, Burmese and Welsh to the first 15. Accuracy. University Dec 2025 74.5% System cards only
TokSuite Models complete questions under alternative tokenizations. The benchmark measures the change in accuracy relative to the canonical representation. School Dec 2025 Not in cards
IndicParam Models answer multiple-choice questions from Indian competitive exams. The benchmark tests knowledge in low-resource Indic languages. University Nov 2025 45% Not in cards
MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. School Nov 2025 99.3% Not in cards
MultiClinAI Models identify diseases, symptoms, and procedures in seven languages. A second task transfers annotated Spanish mentions to the corresponding spans in six translations. University Nov 2025 Not in cards
NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. University Nov 2025 59.1% Fading
Omnilingual ASR Corpus Models transcribe recordings from speakers of many languages. Meta compares character error rates on held-out speech, including languages with little existing transcription data. School Nov 2025 Not in cards
RoCS-MT v2 Models translate Reddit posts with revised spelling annotations and corrected references. The authors combine COMETKiwi and MetricX rankings for translations of the original text. Everyday Nov 2025 7.00 Not in cards
SMM4H-HeaRD 2026 Multilingual Adverse Drug Event Detection Models flag personal reports of drug side effects in seven languages. The organizers withheld Farsi from training and added 15,184 Farsi posts at test time. Everyday Nov 2025 Not in cards
Swivuriso Models transcribe read and spontaneous speech in seven South African languages. The authors measure word error rates and keep speakers separate across splits. Everyday Nov 2025 15% Not in cards
Global PIQA Models pick the correct solution to everyday commonsense prompts in over 100 languages, many about local foods and customs. Scoring is accuracy. Everyday Oct 2025 93.9% Fading
HalloMTBench Models translate English text into one of 11 languages. Judges count translations that add unsupported material, use the wrong language, leave text untranslated or repeat material. University Oct 2025 Not in cards
PISA-Bench Models answer illustrated PISA questions in six languages. Parallel questions let researchers compare language effects on reading, science and mathematical reasoning. School Oct 2025 63.3% Not in cards
RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. University Oct 2025 69.4% Not in cards
Talking Points: When Chatbots Surface Russian State Media Chatbots answer questions about Ukraine in five languages. Analysts check whether each response cites an official Russian channel, a state-controlled outlet, or a state-linked source. Everyday Oct 2025 Not in cards
DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. Everyday Sep 2025 74.2% System cards only
Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. Everyday Sep 2025 60.8% Dropped
SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. Everyday Sep 2025 74.3% System cards only
FilBench Models perform 12 tasks in Philippine languages. The tasks test cultural knowledge, language processing, reading comprehension and generation. School Aug 2025 Not in cards
General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. Aug 2025 0% Fading
LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. Everyday Aug 2025 12.0 System cards only
CS-FLEURS Models transcribe speech that switches languages within sentences. Four test sets cover 113 language pairs. People read one set aloud, and synthesizers generate the others. Everyday Jul 2025 Not in cards
Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. Expert Jul 2025 System cards only
SHROOM-CAP Models read answers to questions about scientific papers in nine languages. They flag factual errors and fluency errors separately. University Jul 2025 0.61 Not in cards
Uganda Cultural and Cognitive Benchmark Models answer questions about Ugandan daily life, institutions and cultural practices. A grader checks answers across 24 domains. Everyday Jul 2025 Not in cards
AfroBench-Lite Models complete language understanding, question answering, reasoning, and generation tasks across 14 African languages. Seven datasets use task-specific metrics, including accuracy, F1, character-level translation scores, and exact match. School Jun 2025 77.7 Not in cards
Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. Everyday Jun 2025 64.7% Fading
m-ArenaHard v2.0 Models answer 498 English prompts from Arena-Hard v2.0, translated into 22 more languages for 11,454 prompts in all. An LLM judge compares answers pairwise and reports win rates. University Jun 2025 Not in cards
MAXIFE Models follow 1,667 verifiable instructions in 23 languages and in cross-lingual settings. Rule-based checks and model judges score compliance. Everyday Jun 2025 89.2% Fading
MultiFinBen Models complete 36 financial datasets across five languages and text, vision, and audio. The board reports a modality-balanced score. University Jun 2025 33.5 Not in cards
Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. Competition Jun 2025 System cards only
NTCIR-19 MedNLP-CALL Triage Classification Models classify synthetic emergency calls into three urgency levels from dispatcher and caller dialogue. Medical experts supplied the reference labels for the six-language task. University Jun 2025 Not in cards
WMT25 Systems translate the WMT25 general task test sets in 16 directions, mostly out of English. New pairs include English to Bhojpuri, Korean and Maasai. Humans evaluate the systems. University Jun 2025 48.9 System cards only
CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. Everyday May 2025 System cards only
HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. Expert May 2025 67.2% In use
MiniMax TTS Multilingual Test Set Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity. Everyday May 2025 0.36% In use
MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. University May 2025 74.1% Not in cards
RESPIN-S1.0 Models transcribe dialect speakers reading sentences about farming and finance in nine Indian languages. The test holds out speakers and reports errors by dialect. Everyday May 2025 Not in cards
BRIDGE Models work on 87 tasks built from real clinical text, such as electronic health records, in nine languages. The tasks span triage, consultation, information extraction, diagnosis, prognosis and billing codes across 14 specialties. Each task uses its own automatic metric, and the board reports the average. Expert Apr 2025 56.0 Not in cards
Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. University Apr 2025 52.9% System cards only
LMArena Search Arena People compare answers from two anonymous search-augmented models on their own questions and vote for the better one. Votes become Bradley-Terry ratings. Everyday Apr 2025 1,255 Fading
m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. Everyday Apr 2025 46.5% Dropped
MADASR 2.0 Models transcribe read speech and spontaneous conversations across eight Indian languages. The organizers rank each track by character error and also score language and dialect identification. Everyday Apr 2025 Not in cards
MIEB Image and image-text embedding models are scored on 130 tasks in 38 languages, grouped into 8 task types such as retrieval, clustering and classification. Everyday Apr 2025 Not in cards
MultiBLiMP Models compare minimally different sentences across languages. The benchmark tests whether the model assigns higher likelihood to the grammatical sentence. Everyday Apr 2025 90.2% Not in cards
Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. Everyday Apr 2025 System cards only
MultiLoKo Locally relevant knowledge questions in 31 languages, plus translations. Scored by exact match. Everyday Apr 2025 31% Fading
PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Competition Apr 2025 86.5% In use
AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. Everyday Mar 2025 64% Dropped
Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. Everyday Mar 2025 1.40% System cards only
LingOly-TOO Models solve Linguistics Olympiad problems whose spelling patterns have been systematically changed while preserving the solution logic. Answers receive exact-match credit with no partial credit. The evaluation compares performance on original and obfuscated problems. Competition Mar 2025 46.7% Not in cards
m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. Everyday Mar 2025 72% Dropped
MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. University Mar 2025 87% In use
WMT25 Multilingual Instruction Shared Task Models translate, solve language puzzles, generate text, summarize across languages and judge other answers. The shared task covers 30 languages. University Mar 2025 Not in cards
BenchMAX Models follow instructions, write code, reason, use tools, translate, and retrieve information across 17 languages. Ten tasks use rule checks, model-judged wins, code tests, answer matching, or translation scores. University Feb 2025 85.2 Not in cards
BOUQuET Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references. Everyday Feb 2025 35.0 System cards only
ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. Everyday Feb 2025 56.3% Dropped
INJONGO Models identify a speaker’s intent and extract information from utterances in 16 African languages. Everyday Feb 2025 65.1% Not in cards
KuralHub Models classify emotions in speech from 29 languages. Everyday Feb 2025 Not in cards
MT-AIME2024 Models solve the 30 AIME 2024 problems translated into 55 languages. A rule-based verifier checks each final answer. Competition Feb 2025 80.8% Dropped
SEA-HELM Models are evaluated in Filipino, Indonesian, Tamil, Thai and Vietnamese across five pillars: NLP classics, LLM-specific tasks, linguistics, culture and safety. School Feb 2025 79.9 Not in cards
URO-Bench Spoken dialogue models answer speech with speech across Understanding, Reasoning and Oral conversation. It has a basic track of 16 datasets and a pro track of 20. Everyday Feb 2025 84.1% Fading
WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. University Feb 2025 90.5% In use
AfriHate Models classify hate speech and abusive language across African languages. Everyday Jan 2025 58.8% Not in cards
AmericasNLP 2025 Shared Task on Machine Translation into Indigenous Languages Models translate between Spanish and 13 Indigenous languages of the Americas. The organizers score each direction using character and word overlaps with reference translations. University Jan 2025 Not in cards
IberoBench Models perform language understanding and generation tasks in Iberian languages. The suite combines existing evaluations with new tests specific to each language. University Jan 2025 30.3 Not in cards
CC-OCR Models read text in 7,058 images across four tracks: scene text, multilingual text, document parsing, and key information extraction. Each track has its own metric. Everyday Dec 2024 83.4% Fading
Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. University Dec 2024 96% Dropped
Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. University Dec 2024 95.7% Dropped
INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. University Nov 2024 89.8% Fading
MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. University Nov 2024 94.1% Dropped
MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. University Nov 2024 77.1% Dropped
AIR-Bench 24.05 Retrievers find documents or passages that answer questions. This release covers 69 datasets across 13 languages. University Oct 2024 Not in cards
European LLM Leaderboard (OpenGPT-X) Models answer exam, science, commonsense, truthfulness, and math questions across European languages. The leaderboard also includes multilingual reading comprehension. School Oct 2024 56.7% Not in cards
m-ArenaHard Models answer the 500 Arena-Hard-Auto v0.1 prompts machine-translated into 22 more languages, 23 in all. An LLM judge compares answers pairwise and reports win rates. University Oct 2024 78.9% System cards only
M-RewardBench Reward models choose preferred responses across chat, safety, reasoning and translation tasks in 23 languages. Accuracy measures preference selection. University Oct 2024 83.5% Not in cards
MMMLU-Lite Models answer 19,950 multilingual questions. OpenCompass sampled 25 questions per subject and language with a fixed random seed. University Oct 2024 Not in cards
Multi-IF Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. Everyday Oct 2024 89.1% In use
Multilingual Entity Linking of Occupations Models map occupational titles from national classifications to European occupation codes. The benchmark scores the rank of the first correct match. University Oct 2024 Not in cards
Prompt injection test suite for LLM-based machine translation Translation systems receive TruthfulQA questions with injected instructions to answer instead of translate, across WMT 2024 language pairs. Scored by the share of outputs that answer rather than translate. Everyday Oct 2024 0.30% System cards only
Shopping MMLU Models complete 57 shopping tasks involving product concepts, reasoning, user preferences, and several languages. Task-specific metrics check classification, ranking, extraction, translation, and generated text. Skill scores average the corresponding task metrics. Everyday Oct 2024 72.6 Not in cards
VHELM Vision-language models are scored across nine aspects, including perception, reasoning, knowledge, bias, fairness, multilinguality, robustness, toxicity and safety, from many existing datasets. University Oct 2024 87.1 Not in cards
xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. Everyday Oct 2024 70.1 System cards only
xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. University Oct 2024 41.4% System cards only
Linguini Models solve 160 International Linguistic Olympiad problems with 894 questions in 75 mostly low-resource languages. All evidence needed is in the context. Competition Sep 2024 24.1% System cards only
MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. University Sep 2024 92.7% In use
Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. Everyday Sep 2024 87% Dropped
Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. School Aug 2024 System cards only
Uhura ARC-Easy Models answer ARC-Easy science questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. School Aug 2024 System cards only
Uhura TruthfulQA Models answer TruthfulQA questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. Everyday Aug 2024 System cards only
AutoBencher Models answer generated benchmark questions. The framework searches for topics and produces tests of knowledge, multilingual reasoning, and math. University Jul 2024 55% Not in cards
Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. Jul 2024 System cards only
Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. University Jul 2024 83.2% System cards only
Aya Red-teaming Models answer harmful prompts in eight languages across nine harm categories. Judges score refusal and safety behavior against annotated categories. Everyday Jun 2024 Not in cards
BLEnD Models answer 52.6k questions about everyday culture in 16 countries and regions across 13 languages. Scoring is accuracy on short-answer and multiple-choice formats. Everyday Jun 2024 66.3% Dropped
CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. Everyday Jun 2024 74.1% System cards only
Gretel Synthetic PII Finance Models identify and label personal information in synthetic financial documents across seven languages. Independent evaluations measure how well the predicted spans match the labeled spans. Everyday Jun 2024 Not in cards
IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. School Jun 2024 59% Not in cards
Language Confusion Benchmark Models answer prompts in 15 languages and must reply in the language the user wants, both in monolingual and crosslingual settings. A language detector scores line-level and word-level pass rates. Everyday Jun 2024 93% System cards only
LingOly Models infer linguistic rules from examples in unfamiliar languages. They use those rules to solve linguistics olympiad problems. Competition Jun 2024 46.3% Not in cards
M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. School Jun 2024 62.4% System cards only
MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. School Jun 2024 83% In use
MultiGEC 2025 Models correct whole texts in 12 European languages. One track rewards minimal edits, and the other rewards fluent rewrites. School Jun 2024 56.8 Not in cards
Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. Everyday Jun 2024 4.15 Dropped
SORRY-Bench Models get unsafe requests across a fine-grained taxonomy of topics and linguistic variations. A judge model scores whether each answer refuses or complies. Everyday Jun 2024 5.23% Not in cards
Vision Arena People send an image with a prompt, compare answers from two anonymous vision-language models, and vote for the better one. Votes become Bradley-Terry ratings. Everyday Jun 2024 1,309 Not in cards
WMT24 Systems translate the WMT24 general task test sets, mostly English to X plus some X to Y pairs, at paragraph level across several domains. The shared task ranks systems by human ESA judgments. University Jun 2024 Not in cards
ARC Challenge MT Models answer science questions from ARC-Challenge in translation. The harness scores answers in the target language against the original choices. School May 2024 Not in cards
MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. Everyday May 2024 56.6% Fading
SEAL Multilingual Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo. Everyday May 2024 1,134 Not in cards
WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. Everyday May 2024 98% System cards only
B-MoCA Agents use Android apps such as Instagram on phones with different languages, font sizes, and icon positions. Task-specific checks decide whether each attempt succeeded. Everyday Apr 2024 47.7% Not in cards
CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. University Apr 2024 16.4% System cards only
CyberSecEval Prompt Injection Models face text prompts that embed injected instructions across 15 strategies. Scored as attack success rate. Everyday Apr 2024 19% System cards only
European Parliament election chatbot audit Chatbots answer ten European election questions in ten languages. Reviewers score procedural answers for accuracy and political answers for neutrality. Everyday Apr 2024 Not in cards
IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. Everyday Apr 2024 77.1 Dropped
RTP-LX Models continue or judge toxic prompts hand-translated from RealToxicityPrompts into dozens of languages, with native-speaker labels. Measures toxic output and toxicity detection. Everyday Apr 2024 Not in cards
EXAMS-V Models answer multilingual school examination questions presented as images. Questions can require reading diagrams, maps, tables, or formulas. School Mar 2024 44% Not in cards
PII-Masking-300k Models locate and classify personal information in multilingual synthetic text. The open portion covers six languages. The authors separately describe a finance and insurance portion. Everyday Mar 2024 Not in cards
SemEval 2024 Task 1 semantic textual relatedness Models rank sentence pairs by relatedness across languages. The evaluation compares their rankings with human judgments. School Mar 2024 0.65 Not in cards
LAMBADA Multilingual (Stable LM 2) Models predict missing final words in narrative passages translated for the Stable LM 2 evaluation. The harness provides five language configurations. Everyday Feb 2024 39.8% Not in cards
mDolly Models answer open-ended Dolly prompts machine-translated into 101 languages, plus human-edited sets in 6 languages and human-written prompts in 7. An LLM judge compares or rates the answers. Everyday Feb 2024 Not in cards
Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. University Feb 2024 78.7% Dropped
Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. Everyday Feb 2024 89.6% Dropped
MuTox Models listen to short speech clips in 30 languages and classify whether they contain toxic speech. The harness compares their decisions with human labels. Everyday Jan 2024 Not in cards
DISPLACE Challenge 2024 Models mark speakers and languages in multilingual conversations and transcribe individual speakers. The organizers score timing errors on room recordings and word errors on close microphone recordings. Everyday Dec 2023 Not in cards
AfroBench Models perform language understanding, reasoning, and generation tasks across 64 African languages. The full suite draws on 22 datasets. School Nov 2023 59.6 Not in cards
MELA Models judge whether sentences are linguistically acceptable across languages. The evaluation compares their judgments with the labeled examples. Everyday Nov 2023 Not in cards
FLORES+ Models translate English Wikipedia sentences that professionals and community contributors translated into over 200 languages. Outputs are scored with metrics such as chrF against the references. University Oct 2023 47.5 System cards only
IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. School Oct 2023 35.8% Dropped
LIMMITS 2024 Speech models read text in a target voice across seven languages. Native listeners score naturalness and speaker similarity on the same prompts. Everyday Sep 2023 Not in cards
NusaWrites Models interpret and translate texts in underrepresented Indonesian languages. Google reports emotion classification on original paragraphs and retrieval of matching translations. School Sep 2023 Not in cards
Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. Everyday Sep 2023 5.42% Fading
ParlaSent Models classify the sentiment of parliamentary sentences as positive, neutral, or negative. Two additional test sets contain 2,600 English and 2,600 South Slavic sentences. Everyday Sep 2023 0.77 Not in cards
SIB-200 Models predict the topic of a sentence across 205 languages and dialects. The sentences come from FLORES-200. Everyday Sep 2023 48.7% Not in cards
XTD-200 Models retrieve images and captions across languages using translations of the XTD test set. Results measure how often the matching item ranks first. School Sep 2023 Not in cards
Belebele Models answer multiple-choice reading comprehension questions on FLORES passages in 122 language variants. Scored by accuracy. Everyday Aug 2023 51.1% System cards only
IndicQA Models extract answers from Wikipedia passages in Indian languages. Native speakers wrote the questions and answers. School Jul 2023 Not in cards
Okapi ARC Models choose answers to translated science questions from ARC-Challenge. School Jul 2023 Not in cards
Okapi HellaSwag Models choose the most plausible next event in a translated description of an everyday activity. Everyday Jul 2023 Not in cards
Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. University Jul 2023 80% Dropped
Okapi TruthfulQA Models distinguish true answers from common misconceptions in translated multiple-choice questions. Everyday Jul 2023 Not in cards
SympTEMIST Models locate symptoms, signs, and findings in Spanish clinical cases. They also map given mentions to SNOMED CT codes in Spanish and five additional languages. University Jul 2023 Not in cards
WMT23 Models translate the WMT23 general task test sets between English and languages such as German, Chinese, and Japanese. Model cards score output with BLEURT. University Jul 2023 75.3 Dropped
AGILE Automatic Genre Identification Benchmark Models assign one of eight genres to web texts, including news, opinion, and advertising. The test covers English and ten other languages. Everyday Jun 2023 0.82 Not in cards
Babel-ImageNet Models classify ImageNet images using class descriptions in different languages. The benchmark measures how recognition accuracy changes with the prompt language. Everyday Jun 2023 Not in cards
M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. School Jun 2023 Dropped
AfriQA Models retrieve evidence across languages to answer questions written in African languages. School May 2023 Not in cards
clembench Chat models play rule-based dialogue games in self-play, such as Taboo, Wordle and reference games. The clemscore combines how often a model follows the game rules at all with how well it plays when it does. Version 4.0 runs 14 games in 30 languages. Everyday May 2023 88.8 Not in cards
IN22-Gen Models translate parallel sentences across English and Indian languages. Google evaluates whether an embedding model can match a sentence with its translation. School May 2023 99.5% Not in cards
IndicLID benchmark Models identify Indian languages in native scripts and in Romanized text. School May 2023 95.1% Not in cards
LMArena (Elo) People chat with two anonymous models side by side and vote for the better reply. Votes become Elo-style ratings for each model. May 2023 1,527 Fading
MasakhaPOS Models assign parts of speech to words in African languages. The authors compare tagging models across languages. School May 2023 62.8% Not in cards
Massively Multilingual Speech labeled corpus Models transcribe recorded readings across languages. The evaluation compares predictions with reference transcripts using character error rate. School May 2023 Not in cards
NollySenti Models classify the sentiment of Nollywood movie reviews in five languages spoken in Nigeria. Everyday May 2023 Not in cards
Vistaar Models transcribe speech across 59 combinations of language and dataset in 12 Indian languages. Each test has its own word error rate. Everyday May 2023 Not in cards
MasakhaNEWS Models predict the topic of news articles written in African languages. Everyday Apr 2023 Not in cards
Casual Conversations v2 Models classify attributes of speakers in recorded interviews. The HELM audio tasks compare age and gender predictions with participants’ own labels. Everyday Mar 2023 Not in cards
MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. University Mar 2023 System cards only
AfriSenti Models classify sentiment in human-annotated social media posts across 14 African languages. Everyday Feb 2023 Not in cards
SIGMORPHON 2023 Shared Task on Interlinear Glossing Models label each word’s meaningful parts with translations or grammatical tags. The organizers rank submissions by word-level accuracy pooled across languages. Expert Feb 2023 Not in cards
LEXTREME Models analyze legal text across languages. Tasks include predicting document labels, identifying legal arguments, and labeling words within documents. University Jan 2023 Not in cards
DISPLACE Challenge 2023 Models mark who speaks when and which language each person speaks in multilingual conversations. The scorer counts missed speech, false alarms and incorrect assignments, including overlaps. Everyday Dec 2022 Not in cards
EUR-Lex-Sum Models summarize European Union legislation in the language of the source document. Official summaries provide references across EU languages. University Oct 2022 Not in cards
MasakhaNER 2.0 Models identify named entities in human-annotated text across 20 African languages. School Oct 2022 Not in cards
MGSM 250 GSM8K problems translated into 10 languages. Final-answer accuracy. School Oct 2022 93% Fading
MTEB Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities. Oct 2022 72.2% Fading
NTREX-128 Models translate 1,997 English news sentences into 128 languages, with human reference translations. Scored with metrics such as COMET or chrF. University Oct 2022 68.8 System cards only
MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. Everyday Sep 2022 62.3% System cards only
mRobust04 Models rank documents for translated queries from the TREC Robust04 collection. Relevance judgments support comparisons across languages. School Sep 2022 Not in cards
SemEval 2023 Task 3: Genre, Framing and Persuasion Techniques in Online News Models classify a news article as reporting, opinion, or satire. They also label its frames and identify persuasion techniques in individual paragraphs. University Sep 2022 Not in cards
FLORES-200 Models translate English Wikipedia sentences into about 200 languages. Outputs are scored with BLEU or chrF against human reference translations. Everyday Jul 2022 81.2 Dropped
MAFAND-MT Models translate news text in a benchmark covering 21 African languages. School Jul 2022 Not in cards
Multilingual HateCheck Models distinguish hate speech from statements that quote, reject, or otherwise resemble hate speech. The test covers multiple languages. Everyday Jul 2022 Not in cards
MultiNERD Models locate fine-grained named entities in Wikipedia and Wikinews sentences across ten languages. School Jul 2022 Not in cards
SemEval 2022 Task 8 multilingual news article similarity Models rate the similarity of two news articles, including articles in different languages. The evaluation compares predicted similarities with human judgments. School Jul 2022 0.72 Not in cards
Crossmodal-3600 Models describe images in different languages or retrieve an image from a description. School May 2022 0.65 Not in cards
FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. Everyday May 2022 4.00% In use
GeoMLAMA Models complete sentences about daily life in five countries. The test checks whether models rank locally correct answers above other choices in five languages. Everyday May 2022 Not in cards
NusaX Models predict sentiment in parallel reviews written in Indonesian local languages, Indonesian, and English. Everyday May 2022 Not in cards
IndicXNLI Models compare a premise and hypothesis in Indian languages. They predict whether the premise entails, contradicts or leaves the hypothesis unresolved. School Apr 2022 Not in cards
MASSIVE Models identify an utterance’s intent and label its argument slots across 51 languages. The held-out test reports intent accuracy, slot-filling F1, and exact match accuracy for the complete prediction. Everyday Apr 2022 Not in cards
Hugging Face Speech Bench Speech recognition models transcribe recordings. The board collects error rates from model cards. Everyday Mar 2022 Not in cards
MuMiN Models use social media context to classify claims as factual or misleading. A second task asks which verdict applies to the claim a post discusses. University Feb 2022 Not in cards
CrossSum Models read a news article in one language and summarize it in another. The dataset aligns articles and summaries across more than 1,500 language pairs. School Dec 2021 Not in cards
XStoryCloze Models pick the right ending for short stories from StoryCloze, professionally translated into 10 languages. Everyday Dec 2021 58.2% Not in cards
WikiNEuRal Models locate named entities in Wikipedia text across nine languages using automatically produced reference annotations. School Nov 2021 Not in cards
CLEF HIPE 2022 Models identify and link names in digitized newspapers and scholarly commentaries on classical texts. Organizers compare performance across six collections in five languages. University Sep 2021 Not in cards
MultiEURLEX Models assign subject labels to European Union legislation written in different languages. Expert Sep 2021 0.05 Not in cards
EuroEval Models answer questions, summarize text, and perform language understanding tasks across European languages. EuroEval combines task rankings on its European leaderboard. School Aug 2021 1.17 Not in cards
mMARCO Models rank passages by relevance to a query. The multilingual benchmark translates the MS MARCO passage ranking collection. School Aug 2021 Not in cards
Flores Models translate 3,001 English Wikipedia sentences that professionals translated into 101 languages. Scored with BLEU or chrF against references. University Jun 2021 48.8 Dropped
XWinograd Models resolve ambiguous pronouns in six languages by choosing which noun fits each sentence. Everyday Jun 2021 Not in cards
MInDS-14 Models identify the banking service a caller wants from a spoken request. Everyday Apr 2021 Not in cards
MUCS 2021 Models transcribe speech in Indian languages and mixed-language recordings. NVIDIA reports results on Bengali-English and Hindi-English speech. Everyday Apr 2021 Not in cards
XTREME-R Multilingual models are fine-tuned on English data and tested on 10 tasks in 50 typologically diverse languages, including harder language-agnostic retrieval tasks. It extends XTREME and adds MultiCheckList, a multilingual diagnostic suite. School Apr 2021 Not in cards
MasakhaNER Models recognize named entities in ten African languages. The authors compare supervised and cross-language transfer methods. School Mar 2021 Not in cards
VoxPopuli Models transcribe European Parliament speeches. The corpus also supports speech translation across languages. Everyday Jan 2021 Not in cards
Cross-lingual Test Dataset Models retrieve images from captions written in different languages. The benchmark compares retrieval against paired photographs and descriptions. School Dec 2020 Not in cards
Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. Everyday Dec 2020 4.70 Fading
EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. School Nov 2020 Not in cards
Tatoeba Translation Challenge Models translate sentences for many language pairs. The challenge publishes held-out test sets so systems can report comparable translation scores. University Nov 2020 Not in cards
VoxLingua107 Models identify the language in a speech recording. The corpus includes 107 languages. Everyday Nov 2020 Not in cards
WikiLingua Models summarize procedural articles in different languages. Aligned WikiHow articles support both same-language and cross-language evaluation. School Nov 2020 Not in cards
XOR-TyDi QA Models retrieve passages across languages to answer questions written by speakers of seven languages. Tasks require English evidence or answers in the question’s language. School Oct 2020 Not in cards
MTOP Models parse a user's request into its intent and the details needed to act on it. School Aug 2020 Not in cards
CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. University Jul 2020 40.7 Fading
MKQA Models answer 10,000 open-domain questions translated into 26 languages. Exact match and token F1 measure agreement with human answers, including unanswerable questions. Everyday Jul 2020 Not in cards
OPUS-100 Models translate between English and other languages. Separate test sets measure performance on supervised directions and language pairs without direct training data. University Jul 2020 Not in cards
XCOPA Models pick the more plausible cause or effect of a premise from two options, in 11 languages. Scoring is accuracy. Everyday May 2020 94.4% Dropped
TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. Everyday Mar 2020 31.7% Dropped
XTREME Multilingual models are fine-tuned on English data and then tested on 9 tasks in 40 languages. The tasks cover sentence classification, structured prediction such as named entities, question answering and sentence retrieval. The benchmark measures how well skills transfer across languages. School Mar 2020 Not in cards
Common Voice Crowd-sourced multilingual read-speech corpus. Used here as an ASR test set scored by word error rate. Everyday Dec 2019 5.50% Fading
MLQA Models extract answers from Wikipedia passages in seven languages, with questions and contexts that can be in different languages. Scored by F1 and exact match. Everyday Oct 2019 Not in cards
XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. Everyday Oct 2019 76.8 Dropped
PAWS-X Models decide whether two sentences with similar wording express the same meaning across languages. School Aug 2019 Not in cards
Tatoeba sentence retrieval Models find the English translation of a sentence among competing sentences using multilingual embeddings. School Dec 2018 99.5% Not in cards
XNLI Models classify entailment, contradiction or neutral for sentence pairs translated from MultiNLI into 15 languages. Everyday Sep 2018 45.3% Dropped
VoxCeleb2 Models compare two interview recordings and decide whether the same person speaks in both. Everyday Jun 2018 75.9% Not in cards
Corpus of Multilingual Code-switched Soap Opera Speech Models transcribe South African television dialogue in which speakers switch between English and four Southern African languages. Everyday May 2018 Not in cards
Opusparcus Models identify paraphrases among movie and television subtitle sentences in six European languages. School May 2018 Not in cards
WMT 2016 News Translation Models translate news between languages. The harness compares translations with human references from the 2016 shared task. University Aug 2016 Not in cards
WMT 2014 News Translation Models translate news text between English and other languages. The shared task compares submitted translations on a common test set. University Jun 2014 Not in cards
Asian Language Treebank Models translate parallel sentences across Asian languages. The llm-jp-eval harness uses the parallel corpus for machine translation. School Not in cards
BUCC 2018 bitext mining shared task Models find translation pairs in two collections of sentences. The evaluation compares the retrieved pairs with a reference list. School Not in cards
CALLHOME diarization Models identify speakers in multilingual telephone conversations. NVIDIA evaluates Part 2 with two to six speakers per call. Everyday Not in cards
Cyrillic Turkic language identification Models identify the language of text written in Cyrillic. The test includes Turkic languages and Russian. School Not in cards
Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. Not in cards
Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. Expert System cards only
IARPA Babel speech recognition Models transcribe conversational speech in languages with limited labeled data. Meta measures character error against reference transcripts on the Babel development split. School Not in cards
IWSLT 2017 TED Translation Models translate TED talk transcripts between languages. The campaign includes bilingual translation and multilingual translation with unseen language pairs. University Not in cards
LAMBADA OpenAI Multilingual Models predict the last word in short narrative passages. The harness includes English and four machine-translated test sets. Everyday Not in cards
LLM Round-Trip Translation Benchmark Models translate text into another language and back. The evaluation measures how well the original meaning survives. University 8.69 Not in cards
MAPA EUR-Lex Models mark sensitive entities in EU legal texts. MAPA annotates general and fine-grained categories. The multilingual conversion covers 21 languages. University Not in cards
MIEB Lite Embedding models complete the MIEB Lite task set. The official API reports the mean across task types. 65.7 System cards only
Multilingual Conversational Speech Language Model Challenge Models recognize conversational speech across languages. Qwen compares transcription error and language identification accuracy on the challenge recordings. Everyday Not in cards
QA4MRE Models read documents and choose answers that require integrating information across the text. The harness includes the 2011, 2012 and 2013 CLEF tests. University Not in cards
Sunbird African Language Technology Models translate parallel sentences between English and six languages spoken in Uganda and East Africa. School Not in cards
Voice Arena Full Duplex Bench Voice agents respond to pauses, changes of speaker, acknowledgments, and interruptions. The board averages success rates across these conversational behaviors. Everyday 99.9% Not in cards
xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. Not in cards