HalachaBench
Models answer 60 Hebrew rabbinical qualification exam questions. A model judge awards rubric credit on a scale from 0 to 100.
Each model answers every question three times. AI generated the rubrics and judged the answers. No rabbi reviewed the evaluation, and scores are not qualification grades. The public result bundle withholds question text and rubric text.
- Made by
- HalachaBench
- Released
- November 2026
- Kind
- Other professions
- Level
- Expert
- Scored
- Graded by a model
- Answers
- Open-ended
- Tasks
- Kept private
- Measures
- Capability
- Subject
- Philosophy, ethics and religion, Law
- Language
- Hebrew
- Interaction
- Single turn
- Items from
- Exams and contests
- Refresh
- One fixed set
- Run by
- A host
- Occupations
- Community and social service
- In views
- Law and finance
- Strongest model tested
- Claude Fable 5.2 #2 on the capability index
- Last model added
- November 2026
- Scores refreshed
- November 18th, 2026
Claude Fable 5.2 leads with 85.8 Claude Fable 5.2 leads with 85.8
Best score per entry from the benchmark's own leaderboard, read on November 18th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
- 85.8 Claude Fable 5.2
- 82.3
Gemini 4 Carbon
- 79.7 Claude Fable 5.1
- 78.2
Gemini 4 Argon
- 77.3 GPT-5.6 Sol
- 73.3 GPT-6 Astra
- 72.8 GPT-6.1 Sol
- 69.1 Claude Opus 5.5
- 64.4
Gemini 3.8 Flash
- 60.8
Muse Wonder Full
- 48.7 GPT-6 Sol
- 48.2
Muse Wonder
- 48.0 Claude Opus 4.8
- 36.2
Muse Spark 1.3
- 34.6 GPT-6.1 Terra
- 32.4 GPT-6 Luna
- 29.9 Grok 4.7
- 29.6 Claude Sonnet 4.6
- 28.7 Claude Sonnet 5.5
- 15.3
Muse Ember
- 10.8 Grok 4.3
- 10.5
Muse Glimmer
- 8.60 Claude Haiku 5.5
- 6.10
Ling-3.0-flash
- 6.00
Qwen 3.8 27B
- 4.40 gpt-oss-120b
- 4.30 DictaLM 3.0 24B Thinking
- 4.30
Gemma 4 31B IT
- 4.20 Llama 4 Scout 17B-16E
- 1.20 gpt-oss-20b
- 1 Claude Fable 5.2 provisional; 60 test questions; effort high 85.8
- 2
Gemini 4 Carbon provisional; 60 test questions; effort high 82.3
- 3 Claude Fable 5.1 provisional; 60 test questions; effort high 79.7
- 4
Gemini 4 Argon provisional; 60 test questions; effort high 78.2
- 5 GPT-5.6 Sol provisional; 60 test questions; effort high 77.3
- 6 GPT-6 Astra provisional; 60 test questions; effort high 73.3
- 7 GPT-6.1 Sol provisional; 60 test questions; effort high 72.8
- 8 Claude Opus 5.5 provisional; 60 test questions; effort high 69.1
- 9
Gemini 3.8 Flash provisional; 60 test questions; effort high 64.4
- 10
Muse Wonder Full provisional; 60 test questions; effort high 60.8
- 11 GPT-6 Sol provisional; 60 test questions; effort high 48.7
- 12
Muse Wonder provisional; 60 test questions; effort high 48.2
- 13 Claude Opus 4.8 provisional; 60 test questions; effort high 48.0
- 14
Muse Spark 1.3 provisional; 60 test questions; effort high 36.2
- 15 GPT-6.1 Terra provisional; 60 test questions; effort high 34.6
- 16 GPT-6 Luna provisional; 60 test questions; effort high 32.4
- 17 Grok 4.7 provisional; 60 test questions; effort high 29.9
- 18 Claude Sonnet 4.6 provisional; 60 test questions; effort high 29.6
- 19 Claude Sonnet 5.5 provisional; 60 test questions; effort high 28.7
- 20
Muse Ember provisional; 60 test questions; effort high 15.3
- 21 Grok 4.3 provisional; 60 test questions; effort high 10.8
- 22
Muse Glimmer provisional; 60 test questions; llama.cpp; Q4_K_M 10.5
- 23 Claude Haiku 5.5 provisional; 60 test questions; effort high 8.60
- 24
Ling-3.0-flash provisional; 60 test questions; llama.cpp; Q4_K_M 6.10
- 25
Qwen 3.8 27B provisional; 60 test questions; llama.cpp; UD-Q4_K_M 6.00
- 26 gpt-oss-120b provisional; 60 test questions; llama.cpp; MXFP4 4.40
- 27 DictaLM 3.0 24B Thinking provisional; 60 test questions; llama.cpp; Q4_K_M 4.30
- 28
Gemma 4 31B IT provisional; 60 test questions; llama.cpp; Q4_K_M 4.30
- 29 Llama 4 Scout 17B-16E provisional; 60 test questions; llama.cpp; Q4_K_M 4.20
- 30 gpt-oss-20b provisional; 60 test questions; llama.cpp; MXFP4 1.20
Source: Official HalachaBench leaderboard
- 85.8 Claude Fable 5.2
- 82.3
Gemini 4 Carbon
- 79.7 Claude Fable 5.1
- 78.2
Gemini 4 Argon
- 77.3 GPT-5.6 Sol
- 73.3 GPT-6 Astra
- 72.8 GPT-6.1 Sol
- 69.1 Claude Opus 5.5
- 64.4
Gemini 3.8 Flash
- 60.8
Muse Wonder Full
- 48.7 GPT-6 Sol
- 48.2
Muse Wonder
- 48.0 Claude Opus 4.8
- 36.2
Muse Spark 1.3
- 34.6 GPT-6.1 Terra
- 32.4 GPT-6 Luna
- 29.9 Grok 4.7
- 29.6 Claude Sonnet 4.6
- 28.7 Claude Sonnet 5.5
- 15.3
Muse Ember
- 10.8 Grok 4.3
- 10.5
Muse Glimmer
- 8.60 Claude Haiku 5.5
- 6.10
Ling-3.0-flash
- 6.00
Qwen 3.8 27B
- 4.40 gpt-oss-120b
- 4.30
Gemma 4 31B IT
- 1.20 gpt-oss-20b
- 1 Claude Fable 5.2 provisional; 60 test questions; effort high 85.8
- 2
Gemini 4 Carbon provisional; 60 test questions; effort high 82.3
- 3 Claude Fable 5.1 provisional; 60 test questions; effort high 79.7
- 4
Gemini 4 Argon provisional; 60 test questions; effort high 78.2
- 5 GPT-5.6 Sol provisional; 60 test questions; effort high 77.3
- 6 GPT-6 Astra provisional; 60 test questions; effort high 73.3
- 7 GPT-6.1 Sol provisional; 60 test questions; effort high 72.8
- 8 Claude Opus 5.5 provisional; 60 test questions; effort high 69.1
- 9
Gemini 3.8 Flash provisional; 60 test questions; effort high 64.4
- 10
Muse Wonder Full provisional; 60 test questions; effort high 60.8
- 11 GPT-6 Sol provisional; 60 test questions; effort high 48.7
- 12
Muse Wonder provisional; 60 test questions; effort high 48.2
- 13 Claude Opus 4.8 provisional; 60 test questions; effort high 48.0
- 14
Muse Spark 1.3 provisional; 60 test questions; effort high 36.2
- 15 GPT-6.1 Terra provisional; 60 test questions; effort high 34.6
- 16 GPT-6 Luna provisional; 60 test questions; effort high 32.4
- 17 Grok 4.7 provisional; 60 test questions; effort high 29.9
- 18 Claude Sonnet 4.6 provisional; 60 test questions; effort high 29.6
- 19 Claude Sonnet 5.5 provisional; 60 test questions; effort high 28.7
- 20
Muse Ember provisional; 60 test questions; effort high 15.3
- 21 Grok 4.3 provisional; 60 test questions; effort high 10.8
- 22
Muse Glimmer provisional; 60 test questions; llama.cpp; Q4_K_M 10.5
- 23 Claude Haiku 5.5 provisional; 60 test questions; effort high 8.60
- 24
Ling-3.0-flash provisional; 60 test questions; llama.cpp; Q4_K_M 6.10
- 25
Qwen 3.8 27B provisional; 60 test questions; llama.cpp; UD-Q4_K_M 6.00
- 26 gpt-oss-120b provisional; 60 test questions; llama.cpp; MXFP4 4.40
- 27
Gemma 4 31B IT provisional; 60 test questions; llama.cpp; Q4_K_M 4.30
- 28 gpt-oss-20b provisional; 60 test questions; llama.cpp; MXFP4 1.20
Source: Official HalachaBench leaderboard
| # | Entry | Score ± | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude Fable 5.2 | 85.8 82.7 to 88.5 | provisional; 60 test questions; effort high, Claude Code CLI | Nov 2026 | Official leaderboard |
| 2 | | 82.3 78.4 to 86 | provisional; 60 test questions; effort high, Antigravity CLI | Oct 2026 | Official leaderboard |
| 3 | Claude Fable 5.1 | 79.7 75.8 to 83.4 | provisional; 60 test questions; effort high, Claude Code CLI | Sep 2026 | Official leaderboard |
| 4 | | 78.2 74.3 to 81.9 | provisional; 60 test questions; effort high, Antigravity CLI | Sep 2026 | Official leaderboard |
| 5 | GPT-5.6 Sol | 77.3 72.4 to 81.3 | provisional; 60 test questions; effort high, Codex | Jul 2026 | Official leaderboard |
| 6 | GPT-6 Astra | 73.3 69.4 to 77 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 7 | GPT-6.1 Sol | 72.8 69.1 to 76.2 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 8 | Claude Opus 5.5 | 69.1 64.7 to 73.3 | provisional; 60 test questions; effort high, Claude Code CLI | Sep 2026 | Official leaderboard |
| 9 | | 64.4 59.7 to 68.9 | provisional; 60 test questions; effort high, Antigravity CLI | Sep 2026 | Official leaderboard |
| 10 | | 60.8 ±4.4 | provisional; 60 test questions; effort high, Muse Code CLI | Oct 2026 | Official leaderboard |
| 11 | GPT-6 Sol | 48.7 ±4.6 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 12 | | 48.2 ±4.9 | provisional; 60 test questions; effort high, Muse Code CLI | Oct 2026 | Official leaderboard |
| 13 | Claude Opus 4.8 | 48.0 ±4.7 | provisional; 60 test questions; effort high, Claude Code CLI | May 2026 | Official leaderboard |
| 14 | | 36.2 32.2 to 40.4 | provisional; 60 test questions; effort high, Muse Code CLI | Sep 2026 | Official leaderboard |
| 15 | GPT-6.1 Terra | 34.6 ±3.9 | provisional; 60 test questions; effort high, Codex | Oct 2026 | Official leaderboard |
| 16 | GPT-6 Luna | 32.4 ±3.9 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 17 | Grok 4.7 | 29.9 ±3.7 | provisional; 60 test questions; effort high, xAI API | Sep 2026 | Official leaderboard |
| 18 | Claude Sonnet 4.6 | 29.6 ±4.3 | provisional; 60 test questions; effort high, Claude Code CLI | Feb 2026 | Official leaderboard |
| 19 | Claude Sonnet 5.5 | 28.7 ±4.5 | provisional; 60 test questions; effort high, Claude Code CLI | Sep 2026 | Official leaderboard |
| 20 | | 15.3 12 to 18.8 | provisional; 60 test questions; effort high, Muse Code CLI | Nov 2026 | Official leaderboard |
| 21 | Grok 4.3 | 10.8 ±2.8 | provisional; 60 test questions; effort high, xAI API | Apr 2026 | Official leaderboard |
| 22 | | 10.5 7.6 to 13.7 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Aug 2026 | Official leaderboard |
| 23 | Claude Haiku 5.5 | 8.60 6 to 11.5 | provisional; 60 test questions; effort high, Claude Code CLI | Oct 2026 | Official leaderboard |
| 24 | | 6.10 4.4 to 8 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Aug 2026 | Official leaderboard |
| 25 | | 6.00 4.1 to 8.2 | provisional; 60 test questions; llama.cpp; UD-Q4_K_M, local | Aug 2026 | Official leaderboard |
| 26 | gpt-oss-120b | 4.40 2.6 to 6.4 | provisional; 60 test questions; llama.cpp; MXFP4, local | Aug 2025 | Official leaderboard |
| 27 | DictaLM 3.0 24B Thinking | 4.30 2.8 to 6 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Official leaderboard | |
| 28 | | 4.30 2.5 to 6.4 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Apr 2026 | Official leaderboard |
| 29 | Llama 4 Scout 17B-16E | 4.20 2.6 to 6 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Official leaderboard | |
| 30 | gpt-oss-20b | 1.20 0.5 to 2.1 | provisional; 60 test questions; llama.cpp; MXFP4, local | Aug 2025 | Official leaderboard |
| # | Model | Score ± | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude Fable 5.2 | 85.8 82.7 to 88.5 | provisional; 60 test questions; effort high, Claude Code CLI | Nov 2026 | Official leaderboard |
| 2 | | 82.3 78.4 to 86 | provisional; 60 test questions; effort high, Antigravity CLI | Oct 2026 | Official leaderboard |
| 3 | Claude Fable 5.1 | 79.7 75.8 to 83.4 | provisional; 60 test questions; effort high, Claude Code CLI | Sep 2026 | Official leaderboard |
| 4 | | 78.2 74.3 to 81.9 | provisional; 60 test questions; effort high, Antigravity CLI | Sep 2026 | Official leaderboard |
| 5 | GPT-5.6 Sol | 77.3 72.4 to 81.3 | provisional; 60 test questions; effort high, Codex | Jul 2026 | Official leaderboard |
| 6 | GPT-6 Astra | 73.3 69.4 to 77 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 7 | GPT-6.1 Sol | 72.8 69.1 to 76.2 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 8 | Claude Opus 5.5 | 69.1 64.7 to 73.3 | provisional; 60 test questions; effort high, Claude Code CLI | Sep 2026 | Official leaderboard |
| 9 | | 64.4 59.7 to 68.9 | provisional; 60 test questions; effort high, Antigravity CLI | Sep 2026 | Official leaderboard |
| 10 | | 60.8 ±4.4 | provisional; 60 test questions; effort high, Muse Code CLI | Oct 2026 | Official leaderboard |
| 11 | GPT-6 Sol | 48.7 ±4.6 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 12 | | 48.2 ±4.9 | provisional; 60 test questions; effort high, Muse Code CLI | Oct 2026 | Official leaderboard |
| 13 | Claude Opus 4.8 | 48.0 ±4.7 | provisional; 60 test questions; effort high, Claude Code CLI | May 2026 | Official leaderboard |
| 14 | | 36.2 32.2 to 40.4 | provisional; 60 test questions; effort high, Muse Code CLI | Sep 2026 | Official leaderboard |
| 15 | GPT-6.1 Terra | 34.6 ±3.9 | provisional; 60 test questions; effort high, Codex | Oct 2026 | Official leaderboard |
| 16 | GPT-6 Luna | 32.4 ±3.9 | provisional; 60 test questions; effort high, Codex | Sep 2026 | Official leaderboard |
| 17 | Grok 4.7 | 29.9 ±3.7 | provisional; 60 test questions; effort high, xAI API | Sep 2026 | Official leaderboard |
| 18 | Claude Sonnet 4.6 | 29.6 ±4.3 | provisional; 60 test questions; effort high, Claude Code CLI | Feb 2026 | Official leaderboard |
| 19 | Claude Sonnet 5.5 | 28.7 ±4.5 | provisional; 60 test questions; effort high, Claude Code CLI | Sep 2026 | Official leaderboard |
| 20 | | 15.3 12 to 18.8 | provisional; 60 test questions; effort high, Muse Code CLI | Nov 2026 | Official leaderboard |
| 21 | Grok 4.3 | 10.8 ±2.8 | provisional; 60 test questions; effort high, xAI API | Apr 2026 | Official leaderboard |
| 22 | | 10.5 7.6 to 13.7 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Aug 2026 | Official leaderboard |
| 23 | Claude Haiku 5.5 | 8.60 6 to 11.5 | provisional; 60 test questions; effort high, Claude Code CLI | Oct 2026 | Official leaderboard |
| 24 | | 6.10 4.4 to 8 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Aug 2026 | Official leaderboard |
| 25 | | 6.00 4.1 to 8.2 | provisional; 60 test questions; llama.cpp; UD-Q4_K_M, local | Aug 2026 | Official leaderboard |
| 26 | gpt-oss-120b | 4.40 2.6 to 6.4 | provisional; 60 test questions; llama.cpp; MXFP4, local | Aug 2025 | Official leaderboard |
| 27 | | 4.30 2.5 to 6.4 | provisional; 60 test questions; llama.cpp; Q4_K_M, local | Apr 2026 | Official leaderboard |
| 28 | gpt-oss-20b | 1.20 0.5 to 2.1 | provisional; 60 test questions; llama.cpp; MXFP4, local | Aug 2025 | Official leaderboard |
2 models have held number one on the official leaderboard since its release in November 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
Source: Official HalachaBench leaderboard
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official HalachaBench leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Law, finance and medicine
- MedXpertQA-MM In use Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy.
Qwen 3.8 Max 80.4%
- HealthBench Professional In use 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. Claude Sonnet 5.5 69.2%
- HealthBench In use 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. GPT-5 67.2%
- FinSearchComp In use Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy.
Ling-3.1-flash 81.7%
- KorMedMCQA In use Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. o1-preview 92.7%
- Harvey LAB-AA v1.1 In use Artificial Analysis's run of Harvey's Legal Agent Benchmark on Harvey's newer private set of 120 tasks across 24 practice areas. A deliverable with a material hallucination scores zero, and the headline is the share of judges who find every rubric criterion passed. GPT-6.2 Astra 15%
Data audit
All 10 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access The tasks are private.
- Pass Confidence Checked against a primary source.
Scores 4 pass 1 note
- Pass Independent runs Scores from the official leaderboard (30 entries).
- Pass Models scored 28 models we track, plus 2 other entries on its leaderboard.
- Pass Newest score The newest is from November 2026, for Claude Fable 5.2.
- Pass Frontier models 9 of the top 10 have a score, led by Claude Fable 5.2 (#2).
- Note Headroom Scored as a plain number, with no fixed ceiling.
Leaderboard import 3 pass
- Pass Importer We read the board on github.com: 28 rows for models we track and 2 other entries at the last read.
- Pass Last read November 18th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.