Benchmark Almanac

Sakana AI

We track 10 benchmarks that Sakana AI made or helped make.

0 in use 7 with public code 5 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Sudoku-Bench Models solve Sudoku variants with unusual rule sets that need creative, multi-step deduction. Scored by the share of puzzles solved. Abstract puzzles and logic May 2025 14% Dropped
ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants. Competitive programming Jun 2025 3,164 Not in cards
JFinBench Models answer questions from six Japanese certification exams in accounting, tax, financial planning, and banking. The authors report scores by exam. Finance Nov 2026 97.3% Not in cards
EDINET-Bench Models read Japanese financial disclosures to detect fraud, forecast earnings and identify industries. Finance Jun 2025 0.73 Not in cards
ChouBun Models answer questions about long Japanese documents and summarize company securities reports. Reasoning over long documents Oct 2024 17.4 Not in cards
JA-VLM-Bench-In-the-Wild Models answer Japanese questions about images of places, objects and culture in Japan. General visual questions Mar 2024 41.1 Not in cards
CoffeeBench Agents manage a coffee roaster and trade with other firms for 90 simulated days. Running a business Jun 2026 Not in cards
robust-kbench Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel. GPU kernels Sep 2025 Not in cards
Sakana Japanese-English translation evaluation Models translate Japanese text into English while handling Japan-specific proper nouns and honorifics. Translation Aug 2026 Not in cards
Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. Documents, charts and text in images Jun 2026 Not in cards