Benchmark Almanac

Patronus AI

We track 14 benchmarks that Patronus AI made or helped make.

0 in use 5 with public code 4 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
GlobeBench Models act as text world models. Given an agent's past actions and observations, they predict the environment's next observation across code, enterprise, games, math and science, and dialogue. Interactive world models Oct 2026 49.6% Not in cards
BLUR Models identify a remembered item from incomplete clues in several languages and media. Questions that need search Mar 2025 54% Not in cards
TRACE Models identify reward hacking in agent trajectories from coding environments. Fixing real repositories Jan 2026 63% Not in cards
TRAIL Models read long agent workflow traces and must find and classify the errors in them, using a formal taxonomy of agent error types. The set has 148 large human-annotated traces from established agent benchmarks. General assistants May 2025 0.12 Not in cards
HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. Hallucination rates Jul 2024 78.8% Not in cards
FinanceBench Models answer open-book questions about public companies using their filings, such as 10-Ks and earnings reports. The full set has 10,231 questions with answers and evidence strings, and 150 of them are public. The makers graded answers by hand, and later users often grade with a model judge. Finance Nov 2023 41% Not in cards
MEMTRACK Agents track changes across software platforms and use earlier information to complete tasks. Reasoning over long documents Oct 2025 61% Not in cards
DETOUR Agents ask a simulated user for clues and identify a partially remembered item. Questions that need search Feb 2026 Not in cards
Enterprise PII Models respond to 100 prompts that request sensitive business information. The evaluation checks whether the responses disclose it. Deception and alignment Jan 2024 Not in cards
Enterprise Scenarios Creative Writing Models write stories in response to 100 prompts. A model grades how engaging the stories are. Creative writing Jan 2024 Not in cards
Enterprise Scenarios Customer Support Dialogue Models answer customer support questions using product information and conversation history. A judge scores their responses. Multi-turn conversation Jan 2024 Not in cards
Enterprise Scenarios Leaderboard Models complete six tasks drawn from workplace settings. The leaderboard combines their scores across all six tasks. Running a business Jan 2024 Not in cards
Enterprise Scenarios Toxicity Models respond to 100 prompts. The Perspective API scores how often their responses contain toxic language. Deception and alignment Jan 2024 Not in cards
SpeedrunBench Agents try to finish goals in 10 video games as fast as possible. Score is time to goal in frames or seconds, lower is better. Video games Sep 2026 Not in cards