Cohere
We track 28 benchmarks and 1 arena that Cohere made or helped make. Its models are on its lab page.
1 in use 3 with public code 6 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. | Knowledge in many languages | Nov 2024 | 89.8% | Fading |
| Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. | Knowledge in many languages | Dec 2024 | 95.7% | Dropped |
| Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. | Knowledge in many languages | Dec 2024 | 96% | Dropped |
| Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. | Knowledge in many languages | May 2026 |  90.1% | In use |
| AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. | General visual questions | Mar 2025 | 64% | Dropped |
| LBPP (Less Basic Python Problems) Models write Python solutions to 161 new problems built to avoid training-set contamination. Hidden tests score pass@1. | Function-level problems | Jul 2024 |  81% | System cards only |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Customer service | Aug 2025 | 46.5% | Dropped |
| m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. | General visual questions | Mar 2025 | 72% | Dropped |
| m-ArenaHard Models answer the 500 Arena-Hard-Auto v0.1 prompts machine-translated into 22 more languages, 23 in all. An LLM judge compares answers pairwise and reports win rates. | Chat and instructions in many languages | Oct 2024 | System cards only | |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | Knowledge in many languages | Apr 2025 |  52.9% | System cards only |
| Default safety evaluation (Cohere) Models answer unsafe prompts in five categories with no system preamble. LLM judges score the share of safe responses, and a jury rates relative safety. | Refusing harmful requests | Apr 2025 | 70.4% | System cards only |
| Enterprise generative evaluation (Cohere) Models complete 22 enterprise writing tasks such as job ads with format rules. Rule checks and LLM judges give a pass rate. | Running a business | Apr 2025 | 94.2% | System cards only |
| Enterprise RAG evaluation (Cohere) Models answer workplace policy and technical questions over long documents, some unanswerable. Llama Index Correctness grades answers from 1 to 5. | Faithful to sources | Apr 2025 | 4.73 | System cards only |
| Global MMLU Lite 2.0 Models answer 400 human-translated MMLU questions per language in 18 languages, adding Albanian, Burmese and Welsh to the first 15. Accuracy. | Knowledge in many languages | Dec 2025 | 74.5% | System cards only |
| HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1. | Function-level problems | Apr 2025 | 25.3% | System cards only |
| Language Confusion Benchmark Models answer prompts in 15 languages and must reply in the language the user wants, both in monolingual and crosslingual settings. A language detector scores line-level and word-level pass rates. | Chat and instructions in many languages | Jun 2024 | 93% | System cards only |
| MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. | Reasoning in many languages | May 2026 | 86% | Fading |
| Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases. | Data science and SQL | Apr 2025 | 55.3% | System cards only |
| Structured Translation (Cohere) Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts. | Translation | Sep 2026 | 93.7% | System cards only |
| AL-QASIDA Models write in eight dialectal Arabic varieties. Scored on fidelity, understanding, quality and diglossia, including the ADI2 dialect identification score. | Chat and instructions in many languages | Dec 2024 | System cards only | |
| Safety mode controllability (Cohere) Models follow contextual or strict safety instructions. Accuracy averages correct answers on an over-refusal set and correct refusals on a control set. | Refusing harmful requests | Apr 2025 | System cards only | |
| CulturalRiddles Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning. | Reasoning in many languages | Oct 2026 | 79.4% | Not in cards |
| Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. | Documents and spreadsheets | May 2026 | Not in cards | |
| Dolly Machine Translated v2 Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Chat and instructions in many languages | Apr 2026 | Not in cards | |
| GlobalMGSM Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number. | Reasoning in many languages | Mar 2026 | Not in cards | |
| m-ArenaHard v2.0 Models answer 498 English prompts from Arena-Hard v2.0, translated into 22 more languages for 11,454 prompts in all. An LLM judge compares answers pairwise and reports win rates. | Chat and instructions in many languages | Jun 2025 | Not in cards | |
| m-ArenaHard v2.1 Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Chat and instructions in many languages | Apr 2026 | Not in cards | |
| mDolly Models answer open-ended Dolly prompts machine-translated into 101 languages, plus human-edited sets in 6 languages and human-written prompts in 7. An LLM judge compares or rates the answers. | Chat and instructions in many languages | Feb 2024 | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Cohere enterprise human evaluation Trained annotators blindly compare two models' answers on enterprise tasks such as HR FAQs, technical FAQs and workplace assistants. They judge accuracy, instruction following and style. Results are win, tie and loss rates. | Running a business | Mar 2025 | 51.7% | Dropped |