Toloka
We track 6 benchmarks and 2 arenas that Toloka made or helped make.
0 in use 3 with public code 4 with a leaderboard here
Benchmarks
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Toloka Arena Enterprise Tool Use (November 2026) Agents complete private workflows across 11 industries. They must leave the right records and follow the required process in all five attempts. | Customer service | Nov 2026 | | Not in cards |
| DPAI Arena Spring Benchmark Coding agents implement changes in Java Spring repositories. DPAI Arena scores each run and publishes its duration. | Fixing real repositories | Oct 2025 | 64.9 | Not in cards |
| Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. | Customer service | Apr 2026 | 33% | Not in cards |
| Beemo Detectors distinguish human writing from model outputs, including texts revised by experts or other models. | Composite indices | Nov 2024 | | Not in cards |
| JEEM Vision-language models caption images and answer visual questions in four Arabic dialects. The evaluation compares outputs with reference annotations. | Knowledge in many languages | Apr 2025 | Not in cards | |
| Toloka Visual Question Answering Benchmark Models answer questions about images by locating the object that answers each question. The benchmark compares predicted and reference bounding boxes. | Spatial reasoning and counting | Not in cards |
Arenas
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| VIST Image models generate a new image in a reference brand style. Human evaluators compare pairs of outputs. | Image generation and editing | Dec 2025 | Not in cards | |
| VOX-DUB Dubbing systems speak translated lines in the original speaker’s voice. Listeners compare pairs of recordings on five aspects of dubbing quality. | Speech recognition and translation | Sep 2025 | Not in cards |