Benchmark Almanac

Meituan

We track 19 benchmarks that Meituan made or helped make. Its models are on its lab page.

1 in use 12 with public code 10 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
VITA-Bench Agents handle multi-turn requests in simulated food delivery, in-store, and travel services with 66 tools. A rubric-based judge scores task success. Customer service Sep 2025 32.5% Fading
AMO-Bench Models solve 50 original math problems at or above IMO difficulty that need only a final answer. Answers are graded automatically for accuracy. Competition problems with answers Oct 2025 52.4% Fading
LongCat safety evaluation (Meituan) Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category. Refusing harmful requests Sep 2025 95.7% Dropped
Meeseeks Models follow complex instructions over several turns and get feedback on failed requirements after each turn. Scored by how many requirements they meet. Multi-turn conversation Apr 2025 43% Dropped
UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. Audio understanding and voice Oct 2025 76.5% Dropped
Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. Apps and MCP servers Jan 2026 35.8% Fading
tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Customer service Jan 2026 67.1% Fading
VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. Apps and MCP servers Jan 2026 20.5% Fading
Math-VR Models answer 5,000 English and Chinese math questions whose solutions need visual reasoning, such as drawing auxiliary lines or plotting functions. Answers are checked against ground truth. Visual math and diagrams Oct 2025 66.8% System cards only
VitaBench 2.0 Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics. Customer service May 2026 50.3% In use
Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. Speech recognition and translation Oct 2025 1.37 System cards only
General365 Models solve 365 hand-written reasoning problems and 1,095 variants in eight categories that need only K-12 knowledge. Scoring mixes rule-based and model-based checks. Abstract puzzles and logic Apr 2026 62.8% Not in cards
LocalSearchBench Agents search merchant listings to answer 900 questions about local services across nine Chinese cities. A model judges each answer against a reference. Questions that need search Dec 2025 35.6% Not in cards
Meeseeks Utility Rate Models follow complex instructions with feedback for up to 20 turns. The score measures fully usable responses as a percentage. Multi-turn conversation Apr 2025 90.5% Not in cards
WBench Interactive video world models run 289 multi-turn cases with navigation, subject actions, event edits and perspective switches. 22 metrics cover quality, adherence, consistency and physics. Interactive world models May 2026 79.0 Not in cards
PRISM-Bench (audio and video generation) Models generate speech, music, or environmental sounds with video. A judge compares outputs with reference clips across 35 criteria, including synchronization and prompt following. Video generation May 2026 91.9 Not in cards
LARYBench Models answer expert questions in the LARYBench dataset released by Meituan LongCat. Expert-written questions Mar 2026 Not in cards
OutboundEval Agents conduct outbound calls with simulated users. The benchmark scores compliance with the professional workflow and the quality of the interaction. Multi-turn conversation Oct 2025 Not in cards
PosterBench Agents turn research papers into posters. Programmatic checks and visual model judgments score seven content and design dimensions, with ceilings for serious rendering or layout failures. Documents and spreadsheets Aug 2026 Not in cards