Benchmark Almanac

ByteDance Seed

We track 78 benchmarks that ByteDance Seed made or helped make. Its models are on its lab page.

13 in use 27 with public code 41 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Expert-written questions Feb 2025 73.9% In use
NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. Building whole projects Dec 2025 68.2% In use
WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Questions that need search Aug 2025 83.9% In use
Beyond AIME Models answer 100 original competition-style math problems at or above the hardest AIME level. Each answer is a positive integer checked by exact match. Competition problems with answers Apr 2025 87% Fading
Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests. Fixing real repositories Apr 2025 52.7% Fading
FullStackBench Models write code for 3,374 problems in 16 programming languages across domains like data analysis and web development. Unit tests score each answer. Function-level problems Nov 2024 68.3% Fading
FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. Finance Sep 2025 81.7% In use
Seed-TTS-eval Models clone a voice from a short prompt and speak new text in English and Mandarin. Scored by word error rate from an ASR model and by speaker similarity. Speech recognition and translation Jun 2024 0.77% Fading
MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. Documents, charts and text in images May 2024 56.6% Fading
MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. Deep research Aug 2025 24.3% Fading
Inverse IFEval Models follow 1,012 Chinese and English instructions that conflict with habits learned in fine-tuning, such as writing code without comments. A model judge scores compliance. Following precise instructions Sep 2025 76.1% Fading
VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. Video captioning Apr 2019 71.3 Fading
AetherCode Models write solutions to problems from premier programming contests such as IOI and ICPC. Expert-validated test suites judge each solution. Competitive programming Aug 2025 35.5% Fading
MARS-Bench Models handle long multi-turn dialogues built from play-by-play sports commentary. Tasks test ultra multi-turn, interactive multi-turn and cross-turn reasoning. Multi-turn conversation May 2025 62.3% Fading
DiscoX Models translate 200 expert texts between Chinese and English at discourse level, averaging over 1,700 tokens each. Metric-S, a reference-free judge, scores accuracy, fluency and appropriateness, and the overall score is their sum. Translation Nov 2025 76.7 Fading
WorkSpaceBench Agent completes 388 tasks in realistic worker file workspaces (20k files) needing cross-file dependency reasoning. Scored against 7,399 rubrics. Documents and spreadsheets May 2026 67.7% In use
XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. Tasks from real jobs Dec 2025 64.5% Fading
LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. Expert-written questions Nov 2025 52.6% Fading
MME-CC Models answer 1,173 expert-annotated vision-grounded questions in 11 task types grouped as spatial, geometric and visual knowledge reasoning. Spatial reasoning and counting Nov 2025 40.3% Fading
BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. Genomics and life sciences Feb 2026 52.3% Fading
WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. General assistants Dec 2025 32.7% Fading
AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. Scientific code Dec 2025 47.7% Fading
BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. Genomics and life sciences Dec 2025 53.5% Fading
Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. Following precise instructions Dec 2025 64.7% Fading
Customer Support Q&A (ByteDance Seed in-house) Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom. Customer service Dec 2025 72.4% Fading
Education (ByteDance Seed in-house) Models solve, grade, explain and write K to 12 problems across core subjects in teaching scenarios. Other professions Dec 2025 62.8% Fading
Information Extraction (ByteDance Seed in-house) Models extract words, sentences or fields from meeting records, legal texts, contracts, product descriptions and knowledge bases. Documents and spreadsheets Dec 2025 52% Fading
Encyclo-K Models judge questions composed at test time from knowledge statements extracted from authoritative textbooks. Random composition is meant to resist contamination and test several facts at once. Expert-written questions Dec 2025 62.1% Fading
DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. Video captioning Jul 2024 34.4% Dropped
Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing. Fixing real repositories Apr 2025 25.8% Dropped
DeR2 Bench Models reason over supplied scientific documents in a sandbox that separates evidence access from reasoning. Four regimes range from instructions only to relevant documents plus distractors. Deep research Jan 2026 83.8% Fading
Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. Running a business Jun 2026 68.8% In use
SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. General assistants Jun 2026 66.6% In use
CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. Desktop apps Jun 2026 42.5% In use
Doubao Multi-Turn Bench Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer. Multi-turn conversation Jun 2026 52.5% In use
Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. Spatial reasoning and counting Jun 2026 48.0 In use
xDailyBench Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness. Overall chat quality Jun 2026 61% In use
IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. Video generation May 2025 55.3 Dropped
ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. Long videos Dec 2025 14.5% System cards only
Information Processing (ByteDance Seed in-house) Models analyze large collections of user-generated content, for example summarizing viewpoints and judging overall sentiment. Documents and spreadsheets Dec 2025 53.9% Fading
Intention Recognition (ByteDance Seed in-house) Models classify intent from dialogues, meeting transcripts, social posts and articles, such as purchase intent in customer service chats. Running a business Dec 2025 63.2% Fading
LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. Detection and captioning May 2025 72.5% Dropped
MSQA Models answer culture-specific knowledge questions in 11 major languages. Knowledge in many languages Jun 2026 50.2% In use
U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need. Web and app front ends Dec 2025 49.2% Fading
ViVerBench Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX. General visual questions Oct 2025 74.4% System cards only
Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing. Fixing real repositories Jul 2025 25.3% Dropped
PatchEval Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. Finding and exploiting real bugs Nov 2025 83.9% In use
Chinese Complex Instruction Following (ByteDance Seed in-house) Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions. Following precise instructions Feb 2026 75.3% System cards only
Compositional Tasks (ByteDance Seed in-house) Models identify several elements of a business text in one inference, such as user intent and slot values. Running a business Feb 2026 59.1% System cards only
FinSearchComp Global Agents search for financial information about global assets across three task types. The score is the published average accuracy for the Global subset. Finance Sep 2025 65.5% Dropped
GDPVal-Diamond Models complete a ByteDance Seed subset of GDPval occupational tasks. Rubrics grade each deliverable automatically. Tasks from real jobs Feb 2026 21.3% System cards only
Past IMO (Seed-Prover formal set) Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F. Formal proofs Jul 2025 78.1% Dropped
Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. General visual questions May 2025 59.3% System cards only
Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. Running a business Feb 2026 69% System cards only
UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. Video games Sep 2025 59.8 Dropped
SeedKernelBench Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code. GPU kernels Jun 2026 System cards only
Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. Fixing real repositories Jun 2026 System cards only
AncientDoc Models transcribe historical Chinese pages, translate passages into modern Chinese, and answer questions about meaning, knowledge, and literary style. Documents, charts and text in images Aug 2025 5.77 Not in cards
ReSpaceBench Models recognize rooms across separate visits and track objects that move between visits. The authors test 32 models on synthetic and real homes. Embodied reasoning Nov 2026 50.5% Not in cards
SafePyramid Models identify rules violated by a conversation under a supplied policy. Rule matching rate averages four thresholds for false positive and false negative counts. Refusing harmful requests Jun 2026 54.2% Not in cards
Workspace-Bench-Lite Agents complete 100 workspace tasks. The leaderboard reports the percentage of 1,856 rubric checks passed, with separate runs for each model and harness. Documents and spreadsheets May 2026 66.7% Not in cards
AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. Scientific code Dec 2025 44% Not in cards
FutureX Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more. Real-world events Aug 2025 27.9 Not in cards
RoughCut-Bench Models select and order video clips or mark speech to remove from a transcript. The evaluation compares cut plans with professional editors’ annotations. Video understanding Jul 2026 0.53 Not in cards
DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. Data science and SQL Dec 2025 47% Not in cards
HarnessDev Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score. AI research tasks Sep 2026 67.8% Not in cards
EdgeBench Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment. Long terminal sessions Jul 2026 51.3 Not in cards
SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions. Fixing real repositories Aug 2026 41.2% Not in cards
PatchEval-Verified Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. Finding and exploiting real bugs Jul 2026 83.9% Not in cards
ReportBench Agents write literature surveys. The evaluator compares their references with published surveys and checks whether cited sources support their claims. Deep research Aug 2025 0.39 Not in cards
Aspire Agent gets only a vague capability goal and must choose data, training and self-checks to evolve weights or harness. Scored on 520 hidden items. AI research tasks Aug 2026 25.6% Not in cards
ScribbleEdit Models edit images using a scribble as the only instruction. The evaluator checks whether the change matches the intended operation and location. Image generation and editing Oct 2026 0.26% Not in cards
BM-Bench Image editing models perform instruction-guided non-rigid image manipulations on the ByteMorph benchmark dataset. Image generation and editing Jun 2025 Not in cards
EvaLearn Models solve 648 problems in 182 sequences to measure learning from feedback across six task types. Task length Jun 2025 Not in cards
FinSearchComp Greater China Agents search for financial information about Greater China assets across three task types. The score is the published average accuracy for the Greater China subset. Finance Sep 2025 Not in cards
M3-Bench Multimodal agents answer questions about long videos while using long-term memory and multimodal retrieval. Long videos Aug 2025 Not in cards
RivaBench Models answer questions about stand-up routines and academic talks using speech and visual cues. They also classify clips as real or synthetic. Video understanding Feb 2025 Not in cards
S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. AI research tasks Aug 2026 Not in cards