Benchmark Almanac

AMemGym

Gemini 3 Pro 55.6% leads since January 2026
Not in model cards

Assistants hold long simulated conversations with users whose details and preferences change over time. Simulated users react on policy, and the test checks whether the assistant remembers and applies the right facts.

The paper calls it the first interactive on-policy benchmark for conversational memory. Version v1.base matches the paper.

Released
January 2026
Kind
Learning from the context
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
All public
Strongest model tested
GPT-5.2 #50 on the capability index
Last model added
December 2025
Scores refreshed
November 12th, 2026
28
Entries on its own leaderboard, 11 models we track

Gemini 3 Pro leads with 55.6% among models released by December 2025

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in December 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

28 entries. Scroll sideways for the rest.

  1. 55.6 Gemini 3 Pro
  2. 50.3 GPT-5.2
  3. 47.8 GLM-4.7
  4. 46.5 GPT-5.1
  5. 46.2 Claude Sonnet 4
  6. 44.8 gemini-2.5-flash (snapshot unspecified)
  7. 43.4 AWE-(2,4,30)
  8. 42.6 Mem0-G
  9. 42.2 AWE-(2,8,30)
  10. 41.8 AWE-(2,4,10)
  11. 41.6 Claude Sonnet 4.5
  12. 41.6 Seed1.8
  13. 41.4 Qwen3-Max
  14. 41.1 AWE-(4,4,30)
  15. 41.0 AWE-(2,0,30)
  16. 40.1 AWE-(2,4,50)
  17. 40.0 gemini-2.0-flash (snapshot unspecified)
  18. 39.9 gemini-2.5-flash-lite (snapshot unspecified)
  19. 39.5 GPT-4.1
  20. 39.1 AWE-(8,4,30)
  21. 38.9 Nemori
  22. 38.4 RAG-(2,4,30)
  23. 38.2 A-Mem
  24. 37.0 LLM
  25. 36.7 GPT-4.1 mini
  26. 35.3 AWI
  27. 32.6 deepseek-v3 (snapshot unspecified)
  28. 31.7 GPT-4o mini
  1. 1 Gemini 3 Pro 55.6%
  2. 2 GPT-5.2 xhigh 50.3%
  3. 3 GLM-4.7 47.8%
  4. 4 GPT-5.1 high 46.5%
  5. 5 Claude Sonnet 4 46.2%
  6. 6 gemini-2.5-flash (snapshot unspecified) 44.8%
  7. 7 AWE-(2,4,30) 43.4%
  8. 8 Mem0-G 42.6%
  9. 9 AWE-(2,8,30) 42.2%
  10. 10 AWE-(2,4,10) 41.8%
  11. 11 Claude Sonnet 4.5 41.6%
  12. 12 Seed1.8 41.6%
  13. 13 Qwen3-Max thinking 41.4%
  14. 14 AWE-(4,4,30) 41.1%
  15. 15 AWE-(2,0,30) 41%
  16. 16 AWE-(2,4,50) 40.1%
  17. 17 gemini-2.0-flash (snapshot unspecified) 40%
  18. 18 gemini-2.5-flash-lite (snapshot unspecified) 39.9%
  19. 19 GPT-4.1 39.5%
  20. 20 AWE-(8,4,30) 39.1%
  21. 21 Nemori 38.9%
  22. 22 RAG-(2,4,30) 38.4%
  23. 23 A-Mem 38.2%
  24. 24 LLM 37%
  25. 25 GPT-4.1 mini 36.7%
  26. 26 AWI 35.3%
  27. 27 deepseek-v3 (snapshot unspecified) 32.6%
  28. 28 GPT-4o mini 31.7%

Source: Official AMemGym leaderboard

#EntryScoreSettingDateSource
1 Gemini 3 Pro 55.6% Nov 2025 Official leaderboard
2 GPT-5.2 50.3% xhigh Dec 2025 Official leaderboard
3 GLM-4.7 47.8% Dec 2025 Official leaderboard
4 GPT-5.1 46.5% high Nov 2025 Official leaderboard
5 Claude Sonnet 4 46.2% May 2025 Official leaderboard
6 gemini-2.5-flash (snapshot unspecified) 44.8% Official leaderboard
7 AWE-(2,4,30) 43.4% memory agent Official leaderboard
8 Mem0-G 42.6% memory agent Official leaderboard
9 AWE-(2,8,30) 42.2% memory agent Official leaderboard
10 AWE-(2,4,10) 41.8% memory agent Official leaderboard
11 Claude Sonnet 4.5 41.6% Sep 2025 Official leaderboard
12 Seed1.8 41.6% Dec 2025 Official leaderboard
13 Qwen3-Max 41.4% thinking Sep 2025 Official leaderboard
14 AWE-(4,4,30) 41.1% memory agent Official leaderboard
15 AWE-(2,0,30) 41% memory agent Official leaderboard
16 AWE-(2,4,50) 40.1% memory agent Official leaderboard
17 gemini-2.0-flash (snapshot unspecified) 40% Official leaderboard
18 gemini-2.5-flash-lite (snapshot unspecified) 39.9% Official leaderboard
19 GPT-4.1 39.5% Apr 2025 Official leaderboard
20 AWE-(8,4,30) 39.1% memory agent Official leaderboard
21 Nemori 38.9% memory agent Official leaderboard
22 RAG-(2,4,30) 38.4% memory agent Official leaderboard
23 A-Mem 38.2% memory agent Official leaderboard
24 LLM 37% memory agent Official leaderboard
25 GPT-4.1 mini 36.7% Apr 2025 Official leaderboard
26 AWI 35.3% memory agent Official leaderboard
27 deepseek-v3 (snapshot unspecified) 32.6% Official leaderboard
28 GPT-4o mini 31.7% Jul 2024 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind
Own leaderboard Not tested since Jan 2026
Gemini 3 Pro 55.6%

Source: Official AMemGym leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Gemini 3 Pro GPT-5.1 Claude Sonnet 4 GPT-4.1 GPT-4o mini 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Gemini 3 Pro GPT-5.1

Source: Official AMemGym leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Long context

All 79 in Long context
  • MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
  • LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Inkling 2 Small 70.6%
  • GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 99.96%
  • OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. Gemini 3.7 Flash 89%
  • CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
  • AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. Step 5 Preview 88.3%