Benchmark Almanac

SugarCrepe

Not in model cards

Models choose between a caption that describes an image and a plausible caption with a changed detail.

The alternatives can replace, swap, or add content.

Released
June 2023
Kind
General visual questions
Level
School
Scored
Checked automatically
Answers
Prediction
Tasks
All public
Measures
Capability
Language
English
In and out
Text, Images → Text
Interaction
Single turn
Refresh
One fixed set
Run by
Anyone
In views
Vision
Last model added
July 2024
Scores refreshed
November 20th, 2026
58
Entries on SOTA2’s leaderboard, 2 models we track

InternVL2-8B leads with 95.2% InternVL2-8B leads with 95.2%

Best score per entry from SOTA2’s leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.

  1. 1 InternVL2-8B Model=InternVL2-8B 95.2%
  2. 2 Llama-90B Model=Llama-90B 94.7%
  3. 3 OneVision-4B Model=OneVision-4B 94.4%
  4. 4 OneVision-8B Model=OneVision-8B 94.1%
  5. 5 OneVision-4B-LN Model=OneVision-4B-LN 89.6%
  6. 6 ReasonCLIP + READ ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 88.5%
  7. 7 TinyLLaVA LLM=Qwen2.5, Size=3B 88.1%
  8. 8 LLaVA-13B Model=LLaVA-13B 87.9%
  9. 9 CE-CLIP+ Evaluation Protocol=Zero-shot 87.5%
  10. 10 TRAG LLM=Qwen2.5, Size=1.5B 87.3%
  11. 11 SPARCL Training Source=COCO, Num Synthetic Images=820K, Num Synthetic Captions=820K 87.1%
  12. 12 READ-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 87%
  13. 13 COSMOS Data Size=30M 86.6%
  14. 14 Cambrian-1-8B Model=Cambrian-8B 86.5%
  15. 15 DAC Evaluation Protocol=finetuned 86.4%
  16. 16 CE-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 86%
  17. 17 AMR-NegCLIP Training Source=COCO, Num Synthetic Captions=500K 85.2%
  18. 18 CLOVE Training Source=LAION-COCO, Num Synthetic Captions=>1B 85.1%
  19. 19 FSC-CLIP Training Source=COCO, Num Synthetic Captions=1.5M 85.1%
  20. 20 ReasonSigLIP ViT=So/14, Resolution=384, Data=+58M 84.1%
  21. 21 NegCLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 83.6%
  22. 22 OpenCLIP Data Size=2B 83.5%
  23. 23 SigLIP ViT=So/14, Resolution=384, Data=10.0B 83%
  24. 24 SigLIP2 ViT=G/16, Resolution=384, Data=10.0B 83%
  25. 25 Long-CLIP ViT=L/14, Resolution=224, Data=0.4B 82.9%
  26. 26 CounterCurate Training Source=Flickr, Num Synthetic Images=150K, Num Synthetic Captions=150K 82.8%
  27. 27 Triplet-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 82.7%
  28. 28 TripletCLIP Evaluation Protocol=finetuned 82.5%
  29. 29 DreamLIP Data Size=30M 81.8%
  30. 30 LLaVA-7B Model=LLaVA-7B 81.8%
  31. 31 CompoDistill LLM=Qwen1.5, Size=1.8B 81.7%
  32. 32 Eva-CLIP-02 ViT=L/14, Resolution=224, Data=2.0B 81.6%
  33. 33 MetaCLIP ViT=L/14, Resolution=224, Data=2.5B 80.9%
  34. 34 CLIP Training Source=COCO, Training Protocol=Finetune 79.9%
  35. 35 ReasonCLIP ViT=L/14, Resolution=224, Data=+58M 78.2%
  36. 36 GNM-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 77.9%
  37. 37 Baseline Evaluation Protocol=finetuned 77.8%
  38. 38 TSVLC (RB) Evaluation Protocol=finetuned 77.8%
  39. 39 CLIP-FT Evaluation Protocol=Fine-tuned on COCO 77.2%
  40. 40 ReasonSigLIP2 ViT=G/16, Resolution=384, Data=+58M 76.8%
  41. 41 LLaVA-KD LLM=Qwen1.5, Size=1.8B 76.6%
  42. 42 KD LLM=Qwen1.5, Size=1.8B 76.1%
  43. 43 CON-CLIP Evaluation Protocol=finetuned 75.6%
  44. 44 CLIPS (ViT-L/14) Backbone=ViT-L/14 75.2%
  45. 45 C^2LIP Backbone=ViT-B/16 75%
  46. 46 FIGCLIP Training Source=VidSitu, Num Real Images=20K videos 74.6%
  47. 47 SigLIP2-ViT-B/16 Backbone=ViT-B/16 74.4%
  48. 48 TSVLC (LLM+RB) Evaluation Protocol=finetuned 73.6%
  49. 49 SigLIP ViT-L/16-res256 Backbone=ViT-L/16, Input Resolution=256x256 73.4%
  50. 50 NegCLIP++ Pre-training Dataset=CC12M 73%
  51. 51 BLIP-B Backbone=ViT-B 71.3%
  52. 52 SF-CLIP Training Source=YFCC15M 71.2%
  53. 53 CLIP-A (ViT-L/14) Backbone=ViT-L/14 70.5%
  54. 54 IL-CLIP Training Source=CC12M 70.3%
  55. 55 FLAVA Backbone=ViT-B 69.9%
  56. 56 LaCLIP Pre-training Dataset=CC12M 67.2%
  57. 57 TripletCLIP++ Pre-training Dataset=CC3M 66.1%
  58. 58 LaCLIP + HN Pre-training Dataset=CC3M 53.9%

Source: SOTA2’s leaderboard

#EntryScoreSettingDateSource
1 InternVL2-8B 95.2% Model=InternVL2-8B Jul 2024 Official leaderboard
2 Llama-90B 94.7% Model=Llama-90B Official leaderboard
3 OneVision-4B 94.4% Model=OneVision-4B Official leaderboard
4 OneVision-8B 94.1% Model=OneVision-8B Official leaderboard
5 OneVision-4B-LN 89.6% Model=OneVision-4B-LN Official leaderboard
6 ReasonCLIP + READ 88.5% ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true Official leaderboard
7 TinyLLaVA 88.1% LLM=Qwen2.5, Size=3B Official leaderboard
8 LLaVA-13B 87.9% Model=LLaVA-13B Official leaderboard
9 CE-CLIP+ 87.5% Evaluation Protocol=Zero-shot Official leaderboard
10 TRAG 87.3% LLM=Qwen2.5, Size=1.5B Official leaderboard
11 SPARCL 87.1% Training Source=COCO, Num Synthetic Images=820K, Num Synthetic Captions=820K Official leaderboard
12 READ-CLIP 87% ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true Official leaderboard
13 COSMOS 86.6% Data Size=30M Official leaderboard
14 Cambrian-1-8B 86.5% Model=Cambrian-8B Jun 2024 Official leaderboard
15 DAC 86.4% Evaluation Protocol=finetuned Official leaderboard
16 CE-CLIP 86% ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true Official leaderboard
17 AMR-NegCLIP 85.2% Training Source=COCO, Num Synthetic Captions=500K Official leaderboard
18 CLOVE 85.1% Training Source=LAION-COCO, Num Synthetic Captions=>1B Official leaderboard
19 FSC-CLIP 85.1% Training Source=COCO, Num Synthetic Captions=1.5M Official leaderboard
20 ReasonSigLIP 84.1% ViT=So/14, Resolution=384, Data=+58M Official leaderboard
21 NegCLIP 83.6% ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true Official leaderboard
22 OpenCLIP 83.5% Data Size=2B Official leaderboard
23 SigLIP 83% ViT=So/14, Resolution=384, Data=10.0B Official leaderboard
24 SigLIP2 83% ViT=G/16, Resolution=384, Data=10.0B Official leaderboard
25 Long-CLIP 82.9% ViT=L/14, Resolution=224, Data=0.4B Official leaderboard
26 CounterCurate 82.8% Training Source=Flickr, Num Synthetic Images=150K, Num Synthetic Captions=150K Official leaderboard
27 Triplet-CLIP 82.7% ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true Official leaderboard
28 TripletCLIP 82.5% Evaluation Protocol=finetuned Official leaderboard
29 DreamLIP 81.8% Data Size=30M Official leaderboard
30 LLaVA-7B 81.8% Model=LLaVA-7B Official leaderboard
31 CompoDistill 81.7% LLM=Qwen1.5, Size=1.8B Official leaderboard
32 Eva-CLIP-02 81.6% ViT=L/14, Resolution=224, Data=2.0B Official leaderboard
33 MetaCLIP 80.9% ViT=L/14, Resolution=224, Data=2.5B Official leaderboard
34 CLIP 79.9% Training Source=COCO, Training Protocol=Finetune Official leaderboard
35 ReasonCLIP 78.2% ViT=L/14, Resolution=224, Data=+58M Official leaderboard
36 GNM-CLIP 77.9% ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true Official leaderboard
37 Baseline 77.8% Evaluation Protocol=finetuned Official leaderboard
38 TSVLC (RB) 77.8% Evaluation Protocol=finetuned Official leaderboard
39 CLIP-FT 77.2% Evaluation Protocol=Fine-tuned on COCO Official leaderboard
40 ReasonSigLIP2 76.8% ViT=G/16, Resolution=384, Data=+58M Official leaderboard
41 LLaVA-KD 76.6% LLM=Qwen1.5, Size=1.8B Official leaderboard
42 KD 76.1% LLM=Qwen1.5, Size=1.8B Official leaderboard
43 CON-CLIP 75.6% Evaluation Protocol=finetuned Official leaderboard
44 CLIPS (ViT-L/14) 75.2% Backbone=ViT-L/14 Official leaderboard
45 C^2LIP 75% Backbone=ViT-B/16 Official leaderboard
46 FIGCLIP 74.6% Training Source=VidSitu, Num Real Images=20K videos Official leaderboard
47 SigLIP2-ViT-B/16 74.4% Backbone=ViT-B/16 Official leaderboard
48 TSVLC (LLM+RB) 73.6% Evaluation Protocol=finetuned Official leaderboard
49 SigLIP ViT-L/16-res256 73.4% Backbone=ViT-L/16, Input Resolution=256x256 Official leaderboard
50 NegCLIP++ 73% Pre-training Dataset=CC12M Official leaderboard
51 BLIP-B 71.3% Backbone=ViT-B Official leaderboard
52 SF-CLIP 71.2% Training Source=YFCC15M Official leaderboard
53 CLIP-A (ViT-L/14) 70.5% Backbone=ViT-L/14 Official leaderboard
54 IL-CLIP 70.3% Training Source=CC12M Official leaderboard
55 FLAVA 69.9% Backbone=ViT-B Official leaderboard
56 LaCLIP 67.2% Pre-training Dataset=CC12M Official leaderboard
57 TripletCLIP++ 66.1% Pre-training Dataset=CC3M Official leaderboard
58 LaCLIP + HN 53.9% Pre-training Dataset=CC3M Official leaderboard
#ModelScoreSettingDateSource
1 InternVL2-8B 95.2% Model=InternVL2-8B Jul 2024 Official leaderboard
2 Cambrian-1-8B 86.5% Model=Cambrian-8B Jun 2024 Official leaderboard

Cambrian-1-8B on the official leaderboard passed 85% in June 2024

That was 12 months after its release. By the same age, 17% of the benchmarks from 2023 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.

On SOTA2’s leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) InternVL2-8B Cambrian-1-8B 0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) InternVL2-8B Cambrian-1-8B

Source: SOTA2’s leaderboard

No lab model card reports it yet

We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.

More in Vision

All 636 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. Qwen 3.6 Plus 86%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. Seed2.1 Pro 90.7%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%

Data audit

All 8 checks pass. Every benchmark

Catalog record 2 pass
  • Pass Task access All tasks are public.
  • Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
  • Pass Independent runs Scores from the official leaderboard (58 entries).
  • Pass Models scored 2 models we track, plus 56 other entries on its leaderboard.
  • Note Newest score The newest came 2 years ago, in July 2024, for InternVL2-8B. That’s expected for a saturated test.
  • Note Frontier models None of its models are on the capabilities index.
  • Note Headroom Saturated. The best is 95.2%, past 85% since June 2024.
Leaderboard import 3 pass
  • Pass Importer We read the board on SOTA2: 2 rows for models we track and 56 other entries at the last read.
  • Pass Last read November 20th, 2026.
  • Pass Read worked The last read worked.
Consistency 1 pass 1 note
  • Note Lab claims No model has both a lab number and an independent run.
  • Pass Known issues No published audit found problems.

How we check