Benchmark Almanac

Toloka

We track 6 benchmarks and 2 arenas that Toloka made or helped make.

0 in use 3 with public code 4 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
Toloka Arena Enterprise Tool Use (November 2026) Agents complete private workflows across 11 industries. They must leave the right records and follow the required process in all five attempts. Customer service Nov 2026 81% Not in cards
DPAI Arena Spring Benchmark Coding agents implement changes in Java Spring repositories. DPAI Arena scores each run and publishes its duration. Fixing real repositories Oct 2025 64.9 Not in cards
Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. Customer service Apr 2026 33% Not in cards
Beemo Detectors distinguish human writing from model outputs, including texts revised by experts or other models. Composite indices Nov 2024 74.1% Not in cards
JEEM Vision-language models caption images and answer visual questions in four Arabic dialects. The evaluation compares outputs with reference annotations. Knowledge in many languages Apr 2025 Not in cards
Toloka Visual Question Answering Benchmark Models answer questions about images by locating the object that answers each question. The benchmark compares predicted and reference bounding boxes. Spatial reasoning and counting Not in cards

Arenas

Name Kind Released Best score Status
VIST Image models generate a new image in a reference brand style. Human evaluators compare pairs of outputs. Image generation and editing Dec 2025 Not in cards
VOX-DUB Dubbing systems speak translated lines in the original speaker’s voice. Listeners compare pairs of recordings on five aspects of dubbing quality. Speech recognition and translation Sep 2025 Not in cards