Benchmark Almanac

COLIEE organizing committee

We track 3 benchmarks that COLIEE organizing committee made or helped make.

0 in use 0 with public code 0 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
COLIEE 2024 Models retrieve Canadian precedents and supporting paragraphs. They also retrieve Japanese Civil Code articles and answer yes or no on bar exam questions. Law Oct 2023 Not in cards
COLIEE 2025 Models retrieve Canadian cases and answer Japanese bar exam questions using statutes. A new pilot asks which tort claims succeed and which arguments courts accept. Law Jan 2025 Not in cards
COLIEE 2026 Models retrieve Canadian precedents and apply Japanese statutes. The statute retrieval track also requires a correct yes or no answer with sufficient supporting articles. Law Dec 2025 Not in cards