Benchmark Almanac

Writer

We track 5 benchmarks that Writer made or helped make. Its models are on its lab page.

0 in use 0 with public code 1 with a leaderboard here

Benchmarks

Name Kind Released Best score Status
FailSafeQA Models answer finance questions over long documents while the query or the document is degraded, irrelevant or missing. An LLM judge scores robustness, context grounding and compliance. Finance Feb 2025 0.76 Not in cards
Current-facts set Models answer questions about current facts. Writer evaluates responses as part of its neutrality and safety study. Short factual questions Aug 2026 Not in cards
OmniACT Models generate executable scripts from a screenshot and a user request. The evaluation compares the actions needed to complete desktop and web tasks. Desktop apps Feb 2024 Not in cards
Paired-symmetry suite Models respond to paired political prompts. Writer compares refusal behavior across the paired requests. Bias, fairness and politics Aug 2026 Not in cards
Palmyra X6 Enterprise Workflow Evaluations Agents perform production workflows that require retrieval, tool use, content generation, or delegation. Writer compares capability and cost. Running a business Aug 2026 Not in cards