Benchmark Almanac

QIMMA

Applied-Innovation-Center/Karnak-Th-41B-v1.0 73.7% tops its leaderboard Qwen2.5-72B 68.5% leads the models we track
Not in model cards

Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review.

The suite includes cleaned versions of ArabicMMLU, AraTrust, ArabCulture, AraDiCE culture, 3LM STEM, medical and legal sets, plus code tasks. The results dataset held 83 models on August 26th, 2026. Per-sample outputs are public.

Released
January 2026
Kind
Knowledge in many languages
Level
School
Scored
Checked automatically
Answers
Multiple choice
Tasks
All public
Strongest model tested
Qwen 3.8 27B #58 on the capability index
Last model added
August 2026
Scores refreshed
November 12th, 2026

In the authors’ words

We present QIMMA, a quality-assured Arabic LLM leaderboard that places systematic benchmark validation at its core. Rather than aggregating existing resources as-is, QIMMA applies a multi-model assessment pipeline combining automated LLM judgment with human review to surface and resolve systematic quality issues in well-established Arabic benchmarks before evaluation. The result is a curated, multi-domain, multi-task evaluation suite of over 52k samples, grounded predominantly in native Arabic content; code evaluation tasks are the sole exception, as they are inherently language-agnostic. Transparent implementation via LightEval, EvalPlus and public release of per-sample inference outputs make QIMMA a reproducible and community-extensible foundation for Arabic NLP evaluation.

Abstract of Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation (arXiv:2604.03395, CC0)
88
Entries on its own leaderboard, 25 models we track

Applied-Innovation-Center/Karnak-Th-41B-v1.0 tops the leaderboard with 73.7%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen2.5-72B, at 68.5%.

The newest model with a score here came out in August 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

88 entries. Scroll sideways for the rest.

  1. 73.7 Applied-Innovation-Center/Karnak-Th-41B-v1.0
  2. 69.2 Qwen/Qwen3.5-397B-A17B-FP8
  3. 69.1 Cloudflare clef
  4. 69.0 Applied-Innovation-Center/Karnak-40B-v1.0
  5. 68.5 Qwen2.5-72B
  6. 67.8 Applied-Innovation-Center/AIC-1
  7. 67.8 inceptionai/Jais-2-70B-Chat
  8. 67.1 Llama 3.3 70B Instruct
  9. 67.0 Sakalti Ultiima-72B
  10. 66.8 Qwen/Qwen3.5-122B-A10B-FP8
  11. 66.3 Qwen3.5-122B-A10B
  12. 66.1 Qwen/Qwen2.5-32B-Instruct
  13. 64.2 Applied-Innovation-Center/Karnak-6B-v1.0
  14. 63.8 Gemma 3 27B
  15. 63.0 FreedomIntelligence AceGPT-v2-32B-Chat
  16. 62.7 MBZUAI-Paris Atlas-Chat-27B
  17. 62.6 google/gemma-3-12b-it
  18. 62.0 Qwen3.5-27B
  19. 61.2 Qwen3.5-35B-A3B
  20. 60.6 QCRI/Fanar-2-27B-Instruct
  21. 60.4 Gemma 4 31B IT
  22. 59.3 inceptionai/Jais-2-8B-Chat
  23. 59.3 Qwen/Qwen2.5-14B-Instruct
  24. 59.1 Qwen3.5-9B
  25. 59.0 Qwen/Qwen3.5-35B-A3B-Base
  26. 59.0 hammh0a Hala-9B
  27. 58.7 LiquidAI LFM2-24B-A2B
  28. 58.6 UBC-NLP NileChat-3B
  29. 58.6 QCRI/Fanar-1-9B-Instruct
  30. 58.2 Navid-AI Yehia-7B-preview
  31. 57.8 airev-ai emirati-14b-v2
  32. 57.7 humain-ai ALLaM-7B-Instruct-preview
  33. 57.4 google/gemma-3-27b-pt
  34. 57.3 MBZUAI-Paris Atlas-Chat-9B
  35. 56.9 CohereLabs/c4ai-command-r7b-arabic-02-2025
  36. 56.7 Qwen/Qwen3.5-9B-Base
  37. 56.6 Phi-4
  38. 56.5 ibm-research granite-4.0-h-3b-ar
  39. 56.3 Qwen/Qwen3-8B-Base
  40. 56.0 Command R7B
  41. 55.4 Qwen 3.6 35B-A3B
  42. 53.9 inceptionai/jais-adapted-70b-chat
  43. 53.2 google/gemma-3-12b-pt
  44. 53.1 QCRI/Fanar-1-9B
  45. 52.9 silma-ai SILMA-9B-Instruct-v1.0
  46. 52.9 Qwen3-VL-235B-A22B-Instruct
  47. 52.8 Qwen/Qwen3-235B-A22B-Instruct-2507
  48. 52.6 ibm-granite/granite-4.0-h-micro
  49. 52.3 Qwen/Qwen3.5-4B-Base
  50. 52.0 Qwen/Qwen2.5-7B-Instruct
  51. 51.8 Gemma 4 26B A4B
  52. 50.4 LiquidAI LFM2-8B-A1B
  53. 49.4 inceptionai/jais-adapted-13b-chat
  54. 49.3 Qwen/Qwen2.5-3B-Instruct
  55. 49.2 FreedomIntelligence AceGPT-v2-32B
  56. 49.1 google/gemma-3-4b-it
  57. 48.8 UBC-NLP NileChat-3B-Base
  58. 48.5 Tiny Aya Global
  59. 48.4 microsoft/Phi-4-mini-instruct
  60. 47.8 Gemma 4 E4B
  61. 47.5 Qwen3.5-4B
  62. 47.1 MBZUAI-Paris Nile-Chat-4B
  63. 46.5 Granite 3.3 8B Instruct
  64. 46.2 MBZUAI-Paris Atlas-Chat-2B
  65. 45.6 Qwen3-VL-30B-A3B-Instruct
  66. 45.4 google/gemma-3-4b-pt
  67. 45.3 Llama 3.1-8B
  68. 44.5 ibm-granite/granite-4.0-micro-base
  69. 44.5 Qwen3.6 27B
  70. 44.0 Qwen/Qwen3.5-2B-Base
  71. 43.6 Qwen/Qwen3-30B-A3B-Instruct-2507
  72. 43.4 google/gemma-4-26B-A4B-it
  73. 42.3 Qwen 3.8 27B
  74. 42.2 Qwen3-8B
  75. 41.7 LiquidAI LFM2.5-1.2B-Instruct
  76. 40.5 openai gpt-oss-120b
  77. 38.5 google/gemma-3-1b-it
  78. 38.1 Gemma 4 E2B
  79. 36.9 Qwen/Qwen3.5-0.8B-Base
  80. 36.8 CohereLabs/command-a-plus-05-2026-bf16
  81. 36.1 meta-llama/Llama-3.2-3B-Instruct
  82. 35.8 CohereLabs/aya-23-8B
  83. 35.7 openai gpt-oss-20b
  84. 33.9 Qwen3.5-0.8B
  85. 33.8 Qwen3.5-2B
  86. 29.4 openbmb MiniCPM5-1B
  87. 27.4 google/gemma-3-1b-pt
  88. 27.4 google/gemma-3-270m-it
  1. 1 Applied-Innovation-Center/Karnak-Th-41B-v1.0 instruct; Missing; checkpoint 098de69a98440c272db1ef9cb3a55deda243fa3f 73.7%
  2. 2 Qwen/Qwen3.5-397B-A17B-FP8 instruct; Missing; checkpoint ea5b4f81096f3901c91dea97f81324302495781d 69.2%
  3. 3 Cloudflare clef instruct; Missing; checkpoint 171287b2571042abec9e7b99752aee58216b0ed0 69.1%
  4. 4 Applied-Innovation-Center/Karnak-40B-v1.0 instruct; Missing; checkpoint 4bed61923a31eaf23dc0771576d19f6a2b86fa5f 69%
  5. 5 Qwen2.5-72B instruct; Missing; checkpoint 495f39366efef23836d0cfae4fbe635880d2be31 68.5%
  6. 6 Applied-Innovation-Center/AIC-1 instruct; Missing; checkpoint 49f9abaa2ffa413c01b6463464a3fee7fba2fc28 67.8%
  7. 7 inceptionai/Jais-2-70B-Chat instruct; Missing; checkpoint 8bcbd3a66b34e59424a0e5eb0329cf022df4482e 67.8%
  8. 8 Llama 3.3 70B Instruct instruct; Missing; checkpoint 6f6073b423013f6a7d4d9f39144961bfbfbc386b 67.1%
  9. 9 Sakalti Ultiima-72B instruct; Missing; checkpoint f194962be52eca89314b841ede0858994c194263 67%
  10. 10 Qwen/Qwen3.5-122B-A10B-FP8 instruct; Missing; checkpoint a099dee70ccfcd8d5dda56aaa0b60cb8ecadabc9 66.8%
  11. 11 Qwen3.5-122B-A10B instruct; Missing; checkpoint dc4d348443bc740c68e2d77492492c11606384d5 66.3%
  12. 12 Qwen/Qwen2.5-32B-Instruct instruct; Missing; checkpoint 5ede1c97bbab6ce5cda5812749b4c0bdf79b18dd 66.1%
  13. 13 Applied-Innovation-Center/Karnak-6B-v1.0 instruct; Missing; checkpoint 3537b6709ddda3ff5cff45f61a8b974dd32ed2c1 64.2%
  14. 14 Gemma 3 27B instruct; Missing; checkpoint 005ad3404e59d6023443cb575daa05336842228a 63.8%
  15. 15 FreedomIntelligence AceGPT-v2-32B-Chat instruct; Missing; checkpoint 1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4 63%
  16. 16 MBZUAI-Paris Atlas-Chat-27B instruct; Missing; checkpoint 350132dd433d7e3696430a068dc98a56f9178851 62.7%
  17. 17 google/gemma-3-12b-it instruct; Missing; checkpoint 96b6f1eccf38110c56df3a15bffe176da04bfd80 62.6%
  18. 18 Qwen3.5-27B instruct; Missing; checkpoint fc05daec18b0a78c049392ed2e771dde82bdf654 62%
  19. 19 Qwen3.5-35B-A3B instruct; Missing; checkpoint 59d61f3ce65a6d9863b86d2e96597125219dc754 61.2%
  20. 20 QCRI/Fanar-2-27B-Instruct instruct; Missing; checkpoint 074005040e9c3511b1c43b4298bef09908285534 60.6%
  21. 21 Gemma 4 31B IT instruct; Missing; checkpoint 842da3794eaa0b77d5f08bae87a17459d91ff475 60.4%
  22. 22 inceptionai/Jais-2-8B-Chat instruct; Missing; checkpoint da0e1639cd92b508b24120f7f77f5270a8465dc4 59.3%
  23. 23 Qwen/Qwen2.5-14B-Instruct instruct; Missing; checkpoint cf98f3b3bbb457ad9e2bb7baf9a0125b6b88caa8 59.3%
  24. 24 Qwen3.5-9B instruct; Missing; checkpoint c202236235762e1c871ad0ccb60c8ee5ba337b9a 59.1%
  25. 25 Qwen/Qwen3.5-35B-A3B-Base base; Missing; checkpoint 0f0813072d2358973511097385626f21fcb6d422 59%
  26. 26 hammh0a Hala-9B instruct; Missing; checkpoint 40412e872cd118860955ae43346ecaea110345fb 59%
  27. 27 LiquidAI LFM2-24B-A2B instruct; Missing; checkpoint 20052c41d2a7ccca3d5325331dd54de0c4d9021e 58.7%
  28. 28 UBC-NLP NileChat-3B instruct; Missing; checkpoint af10fe47f6fb389097fba9904c9892ba65a5854b 58.6%
  29. 29 QCRI/Fanar-1-9B-Instruct instruct; Missing; checkpoint d30938d0efb1fd251727ee96df6987802ff84662 58.6%
  30. 30 Navid-AI Yehia-7B-preview instruct; Missing; checkpoint 1367e7de5b4769cade39f3be9d3a7da67b5be22e 58.2%
  31. 31 airev-ai emirati-14b-v2 instruct; Missing; checkpoint 4e110e8fb532a52328a9eb3e534b1a6be375daa0 57.8%
  32. 32 humain-ai ALLaM-7B-Instruct-preview instruct; Missing; checkpoint a28dd1e67420cde72d3629c8633a974cf7d9c366 57.7%
  33. 33 google/gemma-3-27b-pt base; Missing; checkpoint 9fe3c4ebc93fbadb14913801536d022054ef11cc 57.4%
  34. 34 MBZUAI-Paris Atlas-Chat-9B instruct; Missing; checkpoint ca3a230bc58cbc37e9f23d682c15a86c999c0681 57.3%
  35. 35 CohereLabs/c4ai-command-r7b-arabic-02-2025 instruct; Missing; checkpoint 2d5440c7cea22410006fb461a1b24c067683da77 56.9%
  36. 36 Qwen/Qwen3.5-9B-Base base; Missing; checkpoint 68c46c4b3498877f3ef123c856ecfde50c39f404 56.7%
  37. 37 Phi-4 instruct; Missing; checkpoint 2db69c1c3e91a05d2c64a3185acfbaf36f744e25 56.6%
  38. 38 ibm-research granite-4.0-h-3b-ar instruct; Missing; checkpoint b32260f6a427527d0c38a463c656b8d3b8560c26 56.5%
  39. 39 Qwen/Qwen3-8B-Base base; Missing; checkpoint 49e3418fbbbca6ecbdf9608b4d22e5a407081db4 56.3%
  40. 40 Command R7B instruct; Missing; checkpoint 4f3d0aa6856e322f2f4480fe65420d5d53d297b8 56%
  41. 41 Qwen 3.6 35B-A3B instruct; Missing; checkpoint 345d467b117d1fd5eda9b879a5988de9ae5f47a7 55.4%
  42. 42 inceptionai/jais-adapted-70b-chat instruct; Missing; checkpoint 07c93d6799cba82e240633e5fc9bb4cceea6feb2 53.9%
  43. 43 google/gemma-3-12b-pt base; Missing; checkpoint 295efb63d01a7017928f273a94ebb86105c9526f 53.2%
  44. 44 QCRI/Fanar-1-9B base; Missing; checkpoint d33480ed13158518a0902a1d4ac00d66cf4086a9 53.1%
  45. 45 silma-ai SILMA-9B-Instruct-v1.0 instruct; Missing; checkpoint d8f4d2b11a71e10f4231d38f0a343ee49715f392 52.9%
  46. 46 Qwen3-VL-235B-A22B-Instruct instruct; Missing; checkpoint 710c13861be6c466e66de3f484069440b8f31389 52.9%
  47. 47 Qwen/Qwen3-235B-A22B-Instruct-2507 instruct; Missing; checkpoint ac9c66cc9b46af7306746a9250f23d47083d689e 52.8%
  48. 48 ibm-granite/granite-4.0-h-micro instruct; Missing; checkpoint d5f01a3ea75f088947be3aae039f4ad52837dfde 52.6%
  49. 49 Qwen/Qwen3.5-4B-Base base; Missing; checkpoint 1001bb4d826a52d1f399e183466143f4da7b741b 52.3%
  50. 50 Qwen/Qwen2.5-7B-Instruct instruct; Missing; checkpoint a09a35458c702b33eeacc393d103063234e8bc28 52%
  51. 51 Gemma 4 26B A4B base; Missing; checkpoint 24548b62aa021d562695c04aaf7758a1ea47990b 51.8%
  52. 52 LiquidAI LFM2-8B-A1B instruct; Missing; checkpoint 7ed1404fdf502a5446c1b67d4fe1533158d6f2ad 50.4%
  53. 53 inceptionai/jais-adapted-13b-chat instruct; Missing; checkpoint ee47988c252bba70001d697afb666bbb4c9fd5aa 49.4%
  54. 54 Qwen/Qwen2.5-3B-Instruct instruct; Missing; checkpoint aa8e72537993ba99e69dfaafa59ed015b17504d1 49.3%
  55. 55 FreedomIntelligence AceGPT-v2-32B base; Missing; checkpoint d07e8ea20b3cb0ccab624615b5e0d5e2da9a1378 49.2%
  56. 56 google/gemma-3-4b-it instruct; Missing; checkpoint 093f9f388b31de276ce2de164bdc2081324b9767 49.1%
  57. 57 UBC-NLP NileChat-3B-Base base; Missing; checkpoint 6ab0364bb527a136bd833bd653179cfc40687517 48.8%
  58. 58 Tiny Aya Global instruct; Missing; checkpoint af89d219b53ed9b13b8a4645f9c8028973510324 48.5%
  59. 59 microsoft/Phi-4-mini-instruct instruct; Missing; checkpoint cfbefacb99257ffa30c83adab238a50856ac3083 48.4%
  60. 60 Gemma 4 E4B base; Missing; checkpoint 11b8250ef59eb3ebc5dae7b21144d7b84f2c06dd 47.8%
  61. 61 Qwen3.5-4B instruct; Missing; checkpoint 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a 47.5%
  62. 62 MBZUAI-Paris Nile-Chat-4B instruct; Missing; checkpoint 2a365667659bbd6786c2760d868e37c506d76ce3 47.1%
  63. 63 Granite 3.3 8B Instruct instruct; Missing; checkpoint 51dd4bc2ade4059a6bd87649d68aa11e4fb2529b 46.5%
  64. 64 MBZUAI-Paris Atlas-Chat-2B instruct; Missing; checkpoint a6244a4c50bc32b65c6c8dc4dc1dab218b580088 46.2%
  65. 65 Qwen3-VL-30B-A3B-Instruct instruct; Missing; checkpoint 9c4b90e1e4ba969fd3b5378b57d966d725f1b86c 45.6%
  66. 66 google/gemma-3-4b-pt base; Missing; checkpoint cc012e0a6d0787b4adcc0fa2c4da74402494554d 45.4%
  67. 67 Llama 3.1-8B base; Missing; checkpoint d04e592bb4f6aa9cfee91e2e20afa771667e1d4b 45.3%
  68. 68 ibm-granite/granite-4.0-micro-base base; Missing; checkpoint 98e1d641451818f6c8e6b17c1c359b5b4ac495d0 44.5%
  69. 69 Qwen3.6 27B instruct; Missing; checkpoint 8f33b98937c6031bb49e713dee3e8e185aafd26f 44.5%
  70. 70 Qwen/Qwen3.5-2B-Base base; Missing; checkpoint b1485b2fa6dfa1287294f269f5fb618e03d52d7c 44%
  71. 71 Qwen/Qwen3-30B-A3B-Instruct-2507 instruct; Missing; checkpoint 0d7cf23991f47feeb3a57ecb4c9cee8ea4a17bfe 43.6%
  72. 72 google/gemma-4-26B-A4B-it instruct; Missing; checkpoint 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 43.4%
  73. 73 Qwen 3.8 27B instruct; Missing; checkpoint 37561816e66b5b9153b1b64c10968c18c1e6f4a0 42.3%
  74. 74 Qwen3-8B instruct; Missing; checkpoint b968826d9c46dd6066d109eabc6255188de91218 42.2%
  75. 75 LiquidAI LFM2.5-1.2B-Instruct instruct; Missing; checkpoint b9d816ab510f9a28dca37cb182e9da8b0226e13c 41.7%
  76. 76 openai gpt-oss-120b instruct; Missing; checkpoint b5c939de8f754692c1647ca79fbf85e8c1e70f8a 40.5%
  77. 77 google/gemma-3-1b-it instruct; Missing; checkpoint dcc83ea841ab6100d6b47a070329e1ba4cf78752 38.5%
  78. 78 Gemma 4 E2B instruct; Missing; checkpoint 4adf15748de9e1a0097b6274cee9c15836c41c37 38.1%
  79. 79 Qwen/Qwen3.5-0.8B-Base base; Missing; checkpoint dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 36.9%
  80. 80 CohereLabs/command-a-plus-05-2026-bf16 instruct; Missing; checkpoint 5fb6fde5fd12ff89356aae552e11883bc49f069b 36.8%
  81. 81 meta-llama/Llama-3.2-3B-Instruct instruct; Missing; checkpoint 0cb88a4f764b7a12671c53f0838cd831a0843b95 36.1%
  82. 82 CohereLabs/aya-23-8B instruct; Missing; checkpoint 89da1a0ed02d6130f93ae0ffdbedb63b760c0471 35.8%
  83. 83 openai gpt-oss-20b instruct; Missing; checkpoint 6cee5e81ee83917806bbde320786a8fb61efebee 35.7%
  84. 84 Qwen3.5-0.8B instruct; Missing; checkpoint 2fc06364715b967f1860aea9cf38778875588b17 33.9%
  85. 85 Qwen3.5-2B instruct; Missing; checkpoint 15852e8c16360a2fea060d615a32b45270f8a8fc 33.8%
  86. 86 openbmb MiniCPM5-1B instruct; Missing; checkpoint ac83f813ba7da263c51a9f3ec37c2072597e54af 29.4%
  87. 87 google/gemma-3-1b-pt base; Missing; checkpoint fcf18a2a879aab110ca39f8bffbccd5d49d8eb29 27.4%
  88. 88 google/gemma-3-270m-it instruct; Missing; checkpoint ac82b4e820549b854eebf28ce6dedaf9fdfa17b3 27.4%

Source: Official QIMMA leaderboard

#EntryScoreSettingDateSource
1 Applied-Innovation-Center/Karnak-Th-41B-v1.0 73.7% instruct; Missing; checkpoint 098de69a98440c272db1ef9cb3a55deda243fa3f Official leaderboard
2 Qwen/Qwen3.5-397B-A17B-FP8 69.2% instruct; Missing; checkpoint ea5b4f81096f3901c91dea97f81324302495781d Official leaderboard
3 Cloudflare clef 69.1% instruct; Missing; checkpoint 171287b2571042abec9e7b99752aee58216b0ed0 Official leaderboard
4 Applied-Innovation-Center/Karnak-40B-v1.0 69% instruct; Missing; checkpoint 4bed61923a31eaf23dc0771576d19f6a2b86fa5f Official leaderboard
5 Qwen2.5-72B 68.5% instruct; Missing; checkpoint 495f39366efef23836d0cfae4fbe635880d2be31 Sep 2024 Official leaderboard
6 Applied-Innovation-Center/AIC-1 67.8% instruct; Missing; checkpoint 49f9abaa2ffa413c01b6463464a3fee7fba2fc28 Official leaderboard
7 inceptionai/Jais-2-70B-Chat 67.8% instruct; Missing; checkpoint 8bcbd3a66b34e59424a0e5eb0329cf022df4482e Official leaderboard
8 Llama 3.3 70B Instruct 67.1% instruct; Missing; checkpoint 6f6073b423013f6a7d4d9f39144961bfbfbc386b Dec 2024 Official leaderboard
9 Sakalti Ultiima-72B 67% instruct; Missing; checkpoint f194962be52eca89314b841ede0858994c194263 Official leaderboard
10 Qwen/Qwen3.5-122B-A10B-FP8 66.8% instruct; Missing; checkpoint a099dee70ccfcd8d5dda56aaa0b60cb8ecadabc9 Official leaderboard
11 Qwen3.5-122B-A10B 66.3% instruct; Missing; checkpoint dc4d348443bc740c68e2d77492492c11606384d5 Feb 2026 Official leaderboard
12 Qwen/Qwen2.5-32B-Instruct 66.1% instruct; Missing; checkpoint 5ede1c97bbab6ce5cda5812749b4c0bdf79b18dd Official leaderboard
13 Applied-Innovation-Center/Karnak-6B-v1.0 64.2% instruct; Missing; checkpoint 3537b6709ddda3ff5cff45f61a8b974dd32ed2c1 Official leaderboard
14 Gemma 3 27B 63.8% instruct; Missing; checkpoint 005ad3404e59d6023443cb575daa05336842228a Mar 2025 Official leaderboard
15 FreedomIntelligence AceGPT-v2-32B-Chat 63% instruct; Missing; checkpoint 1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4 Official leaderboard
16 MBZUAI-Paris Atlas-Chat-27B 62.7% instruct; Missing; checkpoint 350132dd433d7e3696430a068dc98a56f9178851 Official leaderboard
17 google/gemma-3-12b-it 62.6% instruct; Missing; checkpoint 96b6f1eccf38110c56df3a15bffe176da04bfd80 Official leaderboard
18 Qwen3.5-27B 62% instruct; Missing; checkpoint fc05daec18b0a78c049392ed2e771dde82bdf654 Feb 2026 Official leaderboard
19 Qwen3.5-35B-A3B 61.2% instruct; Missing; checkpoint 59d61f3ce65a6d9863b86d2e96597125219dc754 Feb 2026 Official leaderboard
20 QCRI/Fanar-2-27B-Instruct 60.6% instruct; Missing; checkpoint 074005040e9c3511b1c43b4298bef09908285534 Official leaderboard
21 Gemma 4 31B IT 60.4% instruct; Missing; checkpoint 842da3794eaa0b77d5f08bae87a17459d91ff475 Apr 2026 Official leaderboard
22 inceptionai/Jais-2-8B-Chat 59.3% instruct; Missing; checkpoint da0e1639cd92b508b24120f7f77f5270a8465dc4 Official leaderboard
23 Qwen/Qwen2.5-14B-Instruct 59.3% instruct; Missing; checkpoint cf98f3b3bbb457ad9e2bb7baf9a0125b6b88caa8 Official leaderboard
24 Qwen3.5-9B 59.1% instruct; Missing; checkpoint c202236235762e1c871ad0ccb60c8ee5ba337b9a Feb 2026 Official leaderboard
25 Qwen/Qwen3.5-35B-A3B-Base 59% base; Missing; checkpoint 0f0813072d2358973511097385626f21fcb6d422 Official leaderboard
26 hammh0a Hala-9B 59% instruct; Missing; checkpoint 40412e872cd118860955ae43346ecaea110345fb Official leaderboard
27 LiquidAI LFM2-24B-A2B 58.7% instruct; Missing; checkpoint 20052c41d2a7ccca3d5325331dd54de0c4d9021e Official leaderboard
28 UBC-NLP NileChat-3B 58.6% instruct; Missing; checkpoint af10fe47f6fb389097fba9904c9892ba65a5854b Official leaderboard
29 QCRI/Fanar-1-9B-Instruct 58.6% instruct; Missing; checkpoint d30938d0efb1fd251727ee96df6987802ff84662 Official leaderboard
30 Navid-AI Yehia-7B-preview 58.2% instruct; Missing; checkpoint 1367e7de5b4769cade39f3be9d3a7da67b5be22e Official leaderboard
31 airev-ai emirati-14b-v2 57.8% instruct; Missing; checkpoint 4e110e8fb532a52328a9eb3e534b1a6be375daa0 Official leaderboard
32 humain-ai ALLaM-7B-Instruct-preview 57.7% instruct; Missing; checkpoint a28dd1e67420cde72d3629c8633a974cf7d9c366 Official leaderboard
33 google/gemma-3-27b-pt 57.4% base; Missing; checkpoint 9fe3c4ebc93fbadb14913801536d022054ef11cc Official leaderboard
34 MBZUAI-Paris Atlas-Chat-9B 57.3% instruct; Missing; checkpoint ca3a230bc58cbc37e9f23d682c15a86c999c0681 Official leaderboard
35 CohereLabs/c4ai-command-r7b-arabic-02-2025 56.9% instruct; Missing; checkpoint 2d5440c7cea22410006fb461a1b24c067683da77 Official leaderboard
36 Qwen/Qwen3.5-9B-Base 56.7% base; Missing; checkpoint 68c46c4b3498877f3ef123c856ecfde50c39f404 Official leaderboard
37 Phi-4 56.6% instruct; Missing; checkpoint 2db69c1c3e91a05d2c64a3185acfbaf36f744e25 Dec 2024 Official leaderboard
38 ibm-research granite-4.0-h-3b-ar 56.5% instruct; Missing; checkpoint b32260f6a427527d0c38a463c656b8d3b8560c26 Official leaderboard
39 Qwen/Qwen3-8B-Base 56.3% base; Missing; checkpoint 49e3418fbbbca6ecbdf9608b4d22e5a407081db4 Official leaderboard
40 Command R7B 56% instruct; Missing; checkpoint 4f3d0aa6856e322f2f4480fe65420d5d53d297b8 Dec 2024 Official leaderboard
41 Qwen 3.6 35B-A3B 55.4% instruct; Missing; checkpoint 345d467b117d1fd5eda9b879a5988de9ae5f47a7 Apr 2026 Official leaderboard
42 inceptionai/jais-adapted-70b-chat 53.9% instruct; Missing; checkpoint 07c93d6799cba82e240633e5fc9bb4cceea6feb2 Official leaderboard
43 google/gemma-3-12b-pt 53.2% base; Missing; checkpoint 295efb63d01a7017928f273a94ebb86105c9526f Official leaderboard
44 QCRI/Fanar-1-9B 53.1% base; Missing; checkpoint d33480ed13158518a0902a1d4ac00d66cf4086a9 Official leaderboard
45 silma-ai SILMA-9B-Instruct-v1.0 52.9% instruct; Missing; checkpoint d8f4d2b11a71e10f4231d38f0a343ee49715f392 Official leaderboard
46 Qwen3-VL-235B-A22B-Instruct 52.9% instruct; Missing; checkpoint 710c13861be6c466e66de3f484069440b8f31389 Sep 2025 Official leaderboard
47 Qwen/Qwen3-235B-A22B-Instruct-2507 52.8% instruct; Missing; checkpoint ac9c66cc9b46af7306746a9250f23d47083d689e Official leaderboard
48 ibm-granite/granite-4.0-h-micro 52.6% instruct; Missing; checkpoint d5f01a3ea75f088947be3aae039f4ad52837dfde Official leaderboard
49 Qwen/Qwen3.5-4B-Base 52.3% base; Missing; checkpoint 1001bb4d826a52d1f399e183466143f4da7b741b Official leaderboard
50 Qwen/Qwen2.5-7B-Instruct 52% instruct; Missing; checkpoint a09a35458c702b33eeacc393d103063234e8bc28 Official leaderboard
51 Gemma 4 26B A4B 51.8% base; Missing; checkpoint 24548b62aa021d562695c04aaf7758a1ea47990b Apr 2026 Official leaderboard
52 LiquidAI LFM2-8B-A1B 50.4% instruct; Missing; checkpoint 7ed1404fdf502a5446c1b67d4fe1533158d6f2ad Official leaderboard
53 inceptionai/jais-adapted-13b-chat 49.4% instruct; Missing; checkpoint ee47988c252bba70001d697afb666bbb4c9fd5aa Official leaderboard
54 Qwen/Qwen2.5-3B-Instruct 49.3% instruct; Missing; checkpoint aa8e72537993ba99e69dfaafa59ed015b17504d1 Official leaderboard
55 FreedomIntelligence AceGPT-v2-32B 49.2% base; Missing; checkpoint d07e8ea20b3cb0ccab624615b5e0d5e2da9a1378 Official leaderboard
56 google/gemma-3-4b-it 49.1% instruct; Missing; checkpoint 093f9f388b31de276ce2de164bdc2081324b9767 Official leaderboard
57 UBC-NLP NileChat-3B-Base 48.8% base; Missing; checkpoint 6ab0364bb527a136bd833bd653179cfc40687517 Official leaderboard
58 Tiny Aya Global 48.5% instruct; Missing; checkpoint af89d219b53ed9b13b8a4645f9c8028973510324 Feb 2026 Official leaderboard
59 microsoft/Phi-4-mini-instruct 48.4% instruct; Missing; checkpoint cfbefacb99257ffa30c83adab238a50856ac3083 Official leaderboard
60 Gemma 4 E4B 47.8% base; Missing; checkpoint 11b8250ef59eb3ebc5dae7b21144d7b84f2c06dd Mar 2026 Official leaderboard
61 Qwen3.5-4B 47.5% instruct; Missing; checkpoint 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a Feb 2026 Official leaderboard
62 MBZUAI-Paris Nile-Chat-4B 47.1% instruct; Missing; checkpoint 2a365667659bbd6786c2760d868e37c506d76ce3 Official leaderboard
63 Granite 3.3 8B Instruct 46.5% instruct; Missing; checkpoint 51dd4bc2ade4059a6bd87649d68aa11e4fb2529b Apr 2025 Official leaderboard
64 MBZUAI-Paris Atlas-Chat-2B 46.2% instruct; Missing; checkpoint a6244a4c50bc32b65c6c8dc4dc1dab218b580088 Official leaderboard
65 Qwen3-VL-30B-A3B-Instruct 45.6% instruct; Missing; checkpoint 9c4b90e1e4ba969fd3b5378b57d966d725f1b86c Oct 2025 Official leaderboard
66 google/gemma-3-4b-pt 45.4% base; Missing; checkpoint cc012e0a6d0787b4adcc0fa2c4da74402494554d Official leaderboard
67 Llama 3.1-8B 45.3% base; Missing; checkpoint d04e592bb4f6aa9cfee91e2e20afa771667e1d4b Jul 2024 Official leaderboard
68 ibm-granite/granite-4.0-micro-base 44.5% base; Missing; checkpoint 98e1d641451818f6c8e6b17c1c359b5b4ac495d0 Official leaderboard
69 Qwen3.6 27B 44.5% instruct; Missing; checkpoint 8f33b98937c6031bb49e713dee3e8e185aafd26f Apr 2026 Official leaderboard
70 Qwen/Qwen3.5-2B-Base 44% base; Missing; checkpoint b1485b2fa6dfa1287294f269f5fb618e03d52d7c Official leaderboard
71 Qwen/Qwen3-30B-A3B-Instruct-2507 43.6% instruct; Missing; checkpoint 0d7cf23991f47feeb3a57ecb4c9cee8ea4a17bfe Official leaderboard
72 google/gemma-4-26B-A4B-it 43.4% instruct; Missing; checkpoint 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 Official leaderboard
73 Qwen 3.8 27B 42.3% instruct; Missing; checkpoint 37561816e66b5b9153b1b64c10968c18c1e6f4a0 Aug 2026 Official leaderboard
74 Qwen3-8B 42.2% instruct; Missing; checkpoint b968826d9c46dd6066d109eabc6255188de91218 Apr 2025 Official leaderboard
75 LiquidAI LFM2.5-1.2B-Instruct 41.7% instruct; Missing; checkpoint b9d816ab510f9a28dca37cb182e9da8b0226e13c Official leaderboard
76 openai gpt-oss-120b 40.5% instruct; Missing; checkpoint b5c939de8f754692c1647ca79fbf85e8c1e70f8a Official leaderboard
77 google/gemma-3-1b-it 38.5% instruct; Missing; checkpoint dcc83ea841ab6100d6b47a070329e1ba4cf78752 Official leaderboard
78 Gemma 4 E2B 38.1% instruct; Missing; checkpoint 4adf15748de9e1a0097b6274cee9c15836c41c37 Mar 2026 Official leaderboard
79 Qwen/Qwen3.5-0.8B-Base 36.9% base; Missing; checkpoint dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 Official leaderboard
80 CohereLabs/command-a-plus-05-2026-bf16 36.8% instruct; Missing; checkpoint 5fb6fde5fd12ff89356aae552e11883bc49f069b Official leaderboard
81 meta-llama/Llama-3.2-3B-Instruct 36.1% instruct; Missing; checkpoint 0cb88a4f764b7a12671c53f0838cd831a0843b95 Official leaderboard
82 CohereLabs/aya-23-8B 35.8% instruct; Missing; checkpoint 89da1a0ed02d6130f93ae0ffdbedb63b760c0471 Official leaderboard
83 openai gpt-oss-20b 35.7% instruct; Missing; checkpoint 6cee5e81ee83917806bbde320786a8fb61efebee Official leaderboard
84 Qwen3.5-0.8B 33.9% instruct; Missing; checkpoint 2fc06364715b967f1860aea9cf38778875588b17 Feb 2026 Official leaderboard
85 Qwen3.5-2B 33.8% instruct; Missing; checkpoint 15852e8c16360a2fea060d615a32b45270f8a8fc Feb 2026 Official leaderboard
86 openbmb MiniCPM5-1B 29.4% instruct; Missing; checkpoint ac83f813ba7da263c51a9f3ec37c2072597e54af Official leaderboard
87 google/gemma-3-1b-pt 27.4% base; Missing; checkpoint fcf18a2a879aab110ca39f8bffbccd5d49d8eb29 Official leaderboard
88 google/gemma-3-270m-it 27.4% instruct; Missing; checkpoint ac82b4e820549b854eebf28ce6dedaf9fdfa17b3 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Gemma 4 31B IT
base; Missi… instruct; M…
60.4%
Gemma 4 E4B
instruct; M… base; Missi…
47.8%
Gemma 4 E2B
base; Missi…
38.1%

Source: Official QIMMA leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
Qwen2.5-72B instruct; Missing; checkpoint 495f39366efef23836d0cfae4fbe635880d2be31 68.5%
Llama 3.3 70B Instruct instruct; Missing; checkpoint 6f6073b423013f6a7d4d9f39144961bfbfbc386b 67.1%
Qwen3.5-122B-A10B instruct; Missing; checkpoint dc4d348443bc740c68e2d77492492c11606384d5 66.3%
Gemma 3 27B instruct; Missing; checkpoint 005ad3404e59d6023443cb575daa05336842228a 63.8%
Qwen3.5-27B instruct; Missing; checkpoint fc05daec18b0a78c049392ed2e771dde82bdf654 62%
Qwen3.5-35B-A3B instruct; Missing; checkpoint 59d61f3ce65a6d9863b86d2e96597125219dc754 61.2%
Gemma 4 31B IT instruct; Missing; checkpoint 842da3794eaa0b77d5f08bae87a17459d91ff475 60.4%
Qwen3.5-9B instruct; Missing; checkpoint c202236235762e1c871ad0ccb60c8ee5ba337b9a 59.1%
Phi-4 instruct; Missing; checkpoint 2db69c1c3e91a05d2c64a3185acfbaf36f744e25 56.6%
Gemma 4 31B IT base; Missing; checkpoint 5bbc2fb1c1b2c611d06e3d9f23c170ba21659d89 56.5%
Command R7B instruct; Missing; checkpoint 4f3d0aa6856e322f2f4480fe65420d5d53d297b8 56%
Qwen 3.6 35B-A3B instruct; Missing; checkpoint 345d467b117d1fd5eda9b879a5988de9ae5f47a7 55.4%
Qwen3-VL-235B-A22B-Instruct instruct; Missing; checkpoint 710c13861be6c466e66de3f484069440b8f31389 52.9%
Gemma 4 26B A4B base; Missing; checkpoint 24548b62aa021d562695c04aaf7758a1ea47990b 51.8%
Tiny Aya Global instruct; Missing; checkpoint af89d219b53ed9b13b8a4645f9c8028973510324 48.5%
Gemma 4 E4B base; Missing; checkpoint 11b8250ef59eb3ebc5dae7b21144d7b84f2c06dd 47.8%
Qwen3.5-4B instruct; Missing; checkpoint 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a 47.5%
Granite 3.3 8B Instruct instruct; Missing; checkpoint 51dd4bc2ade4059a6bd87649d68aa11e4fb2529b 46.5%
Qwen3-VL-30B-A3B-Instruct instruct; Missing; checkpoint 9c4b90e1e4ba969fd3b5378b57d966d725f1b86c 45.6%
Llama 3.1-8B base; Missing; checkpoint d04e592bb4f6aa9cfee91e2e20afa771667e1d4b 45.3%
Qwen3.6 27B instruct; Missing; checkpoint 8f33b98937c6031bb49e713dee3e8e185aafd26f 44.5%
Qwen 3.8 27B instruct; Missing; checkpoint 37561816e66b5b9153b1b64c10968c18c1e6f4a0 42.3%
Gemma 4 E4B instruct; Missing; checkpoint 4ecc2ba63bcac23ba4a767fc87025b1450f84c9c 42.3%
Qwen3-8B instruct; Missing; checkpoint b968826d9c46dd6066d109eabc6255188de91218 42.2%
Gemma 4 E2B instruct; Missing; checkpoint 4adf15748de9e1a0097b6274cee9c15836c41c37 38.1%
Gemma 4 E2B base; Missing; checkpoint 433813a669639fd7eaa83155828e9000ed357d53 37.4%
Qwen3.5-0.8B instruct; Missing; checkpoint 2fc06364715b967f1860aea9cf38778875588b17 33.9%
Qwen3.5-2B instruct; Missing; checkpoint 15852e8c16360a2fea060d615a32b45270f8a8fc 33.8%

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen
Own leaderboard Not tested since Aug 2026
Qwen2.5-72B 68.5%

Source: Official QIMMA leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen2.5-72B Llama 3.1-8B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen2.5-72B Llama 3.1-8B

Source: Official QIMMA leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Multilingual

All 120 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Nemotron 3 Ultra 550B-A55B Base 90.1%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%