LLM Games
Models play Connect Four against one another. Ramp publishes match scores and final Elo ratings.
- GPT-5 1,728
- o3 1,682
- Grok 4 1,616
- o4-mini 1,547
-
Gemini 2.5 Flash Jun 2025 1,536
Official leaderboard
We track 11 benchmarks and 1 arena that Ramp Labs made or helped make.
Top five in each of its 1 arena with three or more rated models, the most rated first.
| Name | Kind | Released | Best score | Status |
|---|---|---|---|---|
| Ramp SWE-Bench v1 Agents modify Ramp’s backend code to satisfy engineering requests. Hidden tests check whether each patch solves the task. | Fixing real repositories | Jun 2026 | 88.5% | Not in cards |
| Ramp SWE-Bench v2 Agents solve 500 engineering tasks from Ramp's production backend. A panel of model judges compares each patch with the code engineers merged. | Fixing real repositories | Nov 2026 | 76.4% | Not in cards |
| Accounting Autocoding Models select a bookkeeping category for each expense. Ramp checks the first choice against the assigned category. | Finance | Mar 2026 | | Not in cards |
| Ramp Accounting Bench Agents reconcile accounts and prepare financial records across 22 simulated companies. A model grades their answers and files against accounting rubrics. | Finance | Nov 2026 | | Not in cards |
| Fund Smart Routing Agent Models choose a fund for a transaction. Ramp measures how often a user overrides that choice. | Finance | Mar 2026 | 100% | Not in cards |
| Policy Agent Models approve expenses, reject them, or request human review. Ramp compares their decisions with human expense reviews. | Finance | Mar 2026 | 100% | Not in cards |
| Agentic Credit Analysis Models extract and standardize income statements and balance sheets. Ramp compares their figures with credit analysts’ figures using a five percent tolerance. | Finance | Oct 2026 | 80.2% | Not in cards |
| Ramp Stack Accounting Benchmark Agents perform month-end accounting work across eight simulated businesses. Ramp measures the fraction of grading criteria each agent satisfies. | Finance | Jun 2026 | 62.1% | Not in cards |
| Contextual Invoice OCR Models extract invoice fields using a business’s past bills. Ramp counts invoices where every field matches the submitted bill. | Finance | Mar 2026 | Not in cards | |
| Financial Statement OCR Models extract financial figures from profit and loss statements. Ramp counts values within one percent of a risk operator’s transcription. | Finance | Mar 2026 | Not in cards | |
| Partner Restrictions Compliance Models identify bank partner restrictions that apply to businesses. Ramp scores missed restrictions, coverage across categories, and false positives. | Finance | Mar 2026 | Not in cards |