Benchmark Almanac

Qatar Computing Research Institute

We track 10 benchmarks that Qatar Computing Research Institute made or helped make.

0 in use 7 with public code 1 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
AraDiCE Models are tested on Arabic dialect understanding and generation and on cultural awareness of the Gulf, Egypt and the Levant. Datasets were machine translated into dialects and post-edited by people. Knowledge in many languages Sep 2024 System cards only
IslamicFaithQA Models answer short Islamic knowledge questions in Arabic and English. A model judges their answers against reference answers. Expert-written questions Jan 2026 21.6% Not in cards
xCodeEval Models understand, generate, translate, and retrieve contest code across seven tasks and up to 17 programming languages. Generated programs run against test cases through ExecEval. Competitive programming Mar 2023 3.81% Not in cards
AraBench Models translate dialectal Arabic into English. The suite covers travel conversations, media transcripts and religious texts. Researchers score translations against references with BLEU. Translation Dec 2020 Not in cards
IslamicEval 2025 Quotation Correction Models correct inaccurate Quran and Hadith quotations against the supplied sources. The organizers score exact corrections and accept a designated answer for uncorrectable text. Faithful to sources Jun 2025 Not in cards
IslamicEval 2025 Quotation Span Identification Models mark the character spans of intended Quran and Hadith quotations in Arabic model answers. Organizers score character labels with macro F1. Faithful to sources Jun 2025 Not in cards
IslamicEval 2025 Quotation Validation Models check whether marked Quran and Hadith quotations agree with their sources. The organizers report accuracy. Faithful to sources Jun 2025 Not in cards
IslamicEval 2025 Quran and Hadith Passage Retrieval Models rank up to 20 passages from the Quran and Sahih al-Bukhari for an Arabic question. Organizers score mean average precision at ten. Questions that need search Jun 2025 Not in cards
PalmX 2025 General Culture Models choose answers about customs, history and daily life across Arab countries. The test contains 2,000 questions in Modern Standard Arabic. Knowledge in many languages Jun 2025 Not in cards
PalmX 2025 Islamic Culture Models choose answers about Islamic practices, scripture and history in Modern Standard Arabic. The organizers score exact answer choices. Knowledge in many languages Jun 2025 Not in cards