Benchmark Almanac

Hugging Face

We track 35 benchmarks and 1 arena that Hugging Face made or helped make. Its models are on its lab page.

1 in use 19 with public code 19 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. General assistants Nov 2023 66.1% Fading
Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. Speech recognition and translation Sep 2023 5.42% Fading
HuggingFace Agent Benchmark Agents write Python code that calls Hugging Face Transformers tools to solve short tasks. Scores cover tool selection, tools used, and code correctness. Function calling May 2023 93.5% Dropped
HumanEvalPack Models write functions for HumanEval problems translated into Python, JavaScript, Java, Go, C++ and Rust. Hidden tests give pass@1. Function-level problems Aug 2023 76.2% System cards only
CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. CAD models May 2026 45.1% System cards only
Decision Index Public v0.3 Decision models return probabilities over labeled options in public tasks covering knowledge, language, retrieval, tools, and arts. A chance-corrected weighted aggregate combines the task scores. Composite indices 51.3 In use
Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. Composite indices Jun 2024 48.0 Not in cards
Open LLM Leaderboard v1 Language models answer tasks from the original Open LLM Leaderboard suite. Composite indices Apr 2023 77.4 Not in cards
Open Japanese LLM Leaderboard Models perform Japanese language tasks drawn from more than 20 datasets. Knowledge in many languages Nov 2024 0.67 Not in cards
LLM-Perf Leaderboard Language models generate tokens on a specified hardware and software configuration. The board reports latency, throughput, memory and energy alongside quality scores. Composite indices 29.6 Not in cards
TimeScope Models answer questions about short clips inserted into videos lasting up to eight hours. Accuracy measures retrieval, synthesis and temporal perception. Long videos Jul 2025 97.3% Not in cards
Far-Field ASR Leaderboard Models transcribe speech under simulated room acoustics. The board compares word error rates. Speech recognition and translation Jun 2026 13.4% Not in cards
Open TTS Leaderboard 1.6.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Audio understanding and voice Nov 2026 2.32% Not in cards
Decision Index Decision models read a state and a typed question and return a decision. The index weights private same-skill sets and new-domain sets, equated on 21 base models. Composite indices Sep 2026 67.6 Not in cards
Open TTS Leaderboard 1.0.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Audio understanding and voice Sep 2026 3.92% Not in cards
Open TTS Leaderboard 1.1.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Audio understanding and voice Oct 2026 3.38% Not in cards
Open TTS Leaderboard 1.2.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Audio understanding and voice Oct 2026 3.38% Not in cards
Open TTS Leaderboard 1.3.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Audio understanding and voice Oct 2026 2.32% Not in cards
Open TTS Leaderboard 1.4.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Audio understanding and voice Nov 2026 2.32% Not in cards
Open TTS Leaderboard 1.5.0 Speech models read multilingual text aloud. The board compares transcriptions with the reference text and measures voice similarity. Audio understanding and voice Nov 2026 2.32% Not in cards
RoboChallenge Fine-tuned policies run 10 real rollouts on each of 30 tabletop tasks across four robot types. Scored by success rate and progress. Real robots Oct 2025 42.7% Not in cards
Decision Index Vision Models make decisions over images and screens. The score is the published equal-weight mean of public and private vision scores. Composite indices Sep 2026 63.4 Not in cards
RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. Other Oct 2025 69.4% Not in cards
ScreenSuite Models use screenshots to locate controls, select actions and complete computer tasks. The suite measures perception, single actions and complete interactions. Desktop apps Jun 2025 56% Not in cards
RoboChallenge Table30 v2 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. Real robots May 2026 0.73% Not in cards
DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. Data science and SQL Jun 2025 Not in cards
Enterprise PII Models respond to 100 prompts that request sensitive business information. The evaluation checks whether the responses disclose it. Deception and alignment Jan 2024 Not in cards
Enterprise Scenarios Creative Writing Models write stories in response to 100 prompts. A model grades how engaging the stories are. Creative writing Jan 2024 Not in cards
Enterprise Scenarios Customer Support Dialogue Models answer customer support questions using product information and conversation history. A judge scores their responses. Multi-turn conversation Jan 2024 Not in cards
Enterprise Scenarios Leaderboard Models complete six tasks drawn from workplace settings. The leaderboard combines their scores across all six tasks. Running a business Jan 2024 Not in cards
Enterprise Scenarios Toxicity Models respond to 100 prompts. The Perspective API scores how often their responses contain toxic language. Deception and alignment Jan 2024 Not in cards
FilBench Models perform 12 tasks in Philippine languages. The tasks test cultural knowledge, language processing, reading comprehension and generation. Knowledge in many languages Aug 2025 Not in cards
Hugging Face Speech Bench Speech recognition models transcribe recordings. The board collects error rates from model cards. Speech recognition and translation Mar 2022 Not in cards
Object Detection Leaderboard Models locate and classify objects in images. The board compares average precision. Detection and captioning Sep 2023 Not in cards
Red-Teaming Resistance Benchmark Models respond to prompts from multiple safety datasets. Judges assess whether the responses comply with harmful requests. Jailbreaks and prompt injection Feb 2024 Not in cards

Arenas

Name Kind Released Best score Status
3D Arena Users compare two 3D models generated from the same input and vote for the better one. A live leaderboard ranks the generators. Spatial reasoning and counting Not in cards