SugarCrepe
Models choose between a caption that describes an image and a plausible caption with a changed detail.
The alternatives can replace, swap, or add content.
- Made by
- Cheng-Yu Hsieh and colleagues
- Released
- June 2023
- Kind
- General visual questions
- Level
- School
- Scored
- Checked automatically
- Answers
- Prediction
- Tasks
- All public
- Measures
- Capability
- Language
- English
- Interaction
- Single turn
- Refresh
- One fixed set
- Run by
- Anyone
- In views
- Vision
- Last model added
- July 2024
- Scores refreshed
- November 20th, 2026
InternVL2-8B leads with 95.2% InternVL2-8B leads with 95.2%
Best score per entry from SOTA2’s leaderboard, read on November 20th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. This view shows only models we track, each at its best.
- 95.2
InternVL2-8B
- 94.7 Llama-90B
- 94.4 OneVision-4B
- 94.1 OneVision-8B
- 89.6 OneVision-4B-LN
- 88.5 ReasonCLIP + READ
- 88.1 TinyLLaVA
- 87.9 LLaVA-13B
- 87.5 CE-CLIP+
- 87.3 TRAG
- 87.1 SPARCL
- 87.0 READ-CLIP
- 86.6 COSMOS
- 86.5
Cambrian-1-8B
- 86.4 DAC
- 86.0 CE-CLIP
- 85.2 AMR-NegCLIP
- 85.1 CLOVE
- 85.1 FSC-CLIP
- 84.1 ReasonSigLIP
- 83.6 NegCLIP
- 83.5 OpenCLIP
- 83.0 SigLIP
- 83.0 SigLIP2
- 82.9 Long-CLIP
- 82.8 CounterCurate
- 82.7 Triplet-CLIP
- 82.5 TripletCLIP
- 81.8 DreamLIP
- 81.8 LLaVA-7B
- 81.7 CompoDistill
- 81.6 Eva-CLIP-02
- 80.9 MetaCLIP
- 79.9 CLIP
- 78.2 ReasonCLIP
- 77.9 GNM-CLIP
- 77.8 Baseline
- 77.8 TSVLC (RB)
- 77.2 CLIP-FT
- 76.8 ReasonSigLIP2
- 76.6 LLaVA-KD
- 76.1 KD
- 75.6 CON-CLIP
- 75.2 CLIPS (ViT-L/14)
- 75.0 C^2LIP
- 74.6 FIGCLIP
- 74.4 SigLIP2-ViT-B/16
- 73.6 TSVLC (LLM+RB)
- 73.4 SigLIP ViT-L/16-res256
- 73.0 NegCLIP++
- 71.3 BLIP-B
- 71.2 SF-CLIP
- 70.5 CLIP-A (ViT-L/14)
- 70.3 IL-CLIP
- 69.9 FLAVA
- 67.2 LaCLIP
- 66.1 TripletCLIP++
- 53.9 LaCLIP + HN
- 1
InternVL2-8B Model=InternVL2-8B 95.2%
- 2 Llama-90B Model=Llama-90B 94.7%
- 3 OneVision-4B Model=OneVision-4B 94.4%
- 4 OneVision-8B Model=OneVision-8B 94.1%
- 5 OneVision-4B-LN Model=OneVision-4B-LN 89.6%
- 6 ReasonCLIP + READ ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 88.5%
- 7 TinyLLaVA LLM=Qwen2.5, Size=3B 88.1%
- 8 LLaVA-13B Model=LLaVA-13B 87.9%
- 9 CE-CLIP+ Evaluation Protocol=Zero-shot 87.5%
- 10 TRAG LLM=Qwen2.5, Size=1.5B 87.3%
- 11 SPARCL Training Source=COCO, Num Synthetic Images=820K, Num Synthetic Captions=820K 87.1%
- 12 READ-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 87%
- 13 COSMOS Data Size=30M 86.6%
- 14
Cambrian-1-8B Model=Cambrian-8B 86.5%
- 15 DAC Evaluation Protocol=finetuned 86.4%
- 16 CE-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 86%
- 17 AMR-NegCLIP Training Source=COCO, Num Synthetic Captions=500K 85.2%
- 18 CLOVE Training Source=LAION-COCO, Num Synthetic Captions=>1B 85.1%
- 19 FSC-CLIP Training Source=COCO, Num Synthetic Captions=1.5M 85.1%
- 20 ReasonSigLIP ViT=So/14, Resolution=384, Data=+58M 84.1%
- 21 NegCLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 83.6%
- 22 OpenCLIP Data Size=2B 83.5%
- 23 SigLIP ViT=So/14, Resolution=384, Data=10.0B 83%
- 24 SigLIP2 ViT=G/16, Resolution=384, Data=10.0B 83%
- 25 Long-CLIP ViT=L/14, Resolution=224, Data=0.4B 82.9%
- 26 CounterCurate Training Source=Flickr, Num Synthetic Images=150K, Num Synthetic Captions=150K 82.8%
- 27 Triplet-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 82.7%
- 28 TripletCLIP Evaluation Protocol=finetuned 82.5%
- 29 DreamLIP Data Size=30M 81.8%
- 30 LLaVA-7B Model=LLaVA-7B 81.8%
- 31 CompoDistill LLM=Qwen1.5, Size=1.8B 81.7%
- 32 Eva-CLIP-02 ViT=L/14, Resolution=224, Data=2.0B 81.6%
- 33 MetaCLIP ViT=L/14, Resolution=224, Data=2.5B 80.9%
- 34 CLIP Training Source=COCO, Training Protocol=Finetune 79.9%
- 35 ReasonCLIP ViT=L/14, Resolution=224, Data=+58M 78.2%
- 36 GNM-CLIP ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true 77.9%
- 37 Baseline Evaluation Protocol=finetuned 77.8%
- 38 TSVLC (RB) Evaluation Protocol=finetuned 77.8%
- 39 CLIP-FT Evaluation Protocol=Fine-tuned on COCO 77.2%
- 40 ReasonSigLIP2 ViT=G/16, Resolution=384, Data=+58M 76.8%
- 41 LLaVA-KD LLM=Qwen1.5, Size=1.8B 76.6%
- 42 KD LLM=Qwen1.5, Size=1.8B 76.1%
- 43 CON-CLIP Evaluation Protocol=finetuned 75.6%
- 44 CLIPS (ViT-L/14) Backbone=ViT-L/14 75.2%
- 45 C^2LIP Backbone=ViT-B/16 75%
- 46 FIGCLIP Training Source=VidSitu, Num Real Images=20K videos 74.6%
- 47 SigLIP2-ViT-B/16 Backbone=ViT-B/16 74.4%
- 48 TSVLC (LLM+RB) Evaluation Protocol=finetuned 73.6%
- 49 SigLIP ViT-L/16-res256 Backbone=ViT-L/16, Input Resolution=256x256 73.4%
- 50 NegCLIP++ Pre-training Dataset=CC12M 73%
- 51 BLIP-B Backbone=ViT-B 71.3%
- 52 SF-CLIP Training Source=YFCC15M 71.2%
- 53 CLIP-A (ViT-L/14) Backbone=ViT-L/14 70.5%
- 54 IL-CLIP Training Source=CC12M 70.3%
- 55 FLAVA Backbone=ViT-B 69.9%
- 56 LaCLIP Pre-training Dataset=CC12M 67.2%
- 57 TripletCLIP++ Pre-training Dataset=CC3M 66.1%
- 58 LaCLIP + HN Pre-training Dataset=CC3M 53.9%
Source: SOTA2’s leaderboard
- 1
InternVL2-8B Model=InternVL2-8B 95.2%
- 2
Cambrian-1-8B Model=Cambrian-8B 86.5%
Source: SOTA2’s leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 95.2% | Model=InternVL2-8B | Jul 2024 | Official leaderboard |
| 2 | Llama-90B | 94.7% | Model=Llama-90B | Official leaderboard | |
| 3 | OneVision-4B | 94.4% | Model=OneVision-4B | Official leaderboard | |
| 4 | OneVision-8B | 94.1% | Model=OneVision-8B | Official leaderboard | |
| 5 | OneVision-4B-LN | 89.6% | Model=OneVision-4B-LN | Official leaderboard | |
| 6 | ReasonCLIP + READ | 88.5% | ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true | Official leaderboard | |
| 7 | TinyLLaVA | 88.1% | LLM=Qwen2.5, Size=3B | Official leaderboard | |
| 8 | LLaVA-13B | 87.9% | Model=LLaVA-13B | Official leaderboard | |
| 9 | CE-CLIP+ | 87.5% | Evaluation Protocol=Zero-shot | Official leaderboard | |
| 10 | TRAG | 87.3% | LLM=Qwen2.5, Size=1.5B | Official leaderboard | |
| 11 | SPARCL | 87.1% | Training Source=COCO, Num Synthetic Images=820K, Num Synthetic Captions=820K | Official leaderboard | |
| 12 | READ-CLIP | 87% | ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true | Official leaderboard | |
| 13 | COSMOS | 86.6% | Data Size=30M | Official leaderboard | |
| 14 | | 86.5% | Model=Cambrian-8B | Jun 2024 | Official leaderboard |
| 15 | DAC | 86.4% | Evaluation Protocol=finetuned | Official leaderboard | |
| 16 | CE-CLIP | 86% | ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true | Official leaderboard | |
| 17 | AMR-NegCLIP | 85.2% | Training Source=COCO, Num Synthetic Captions=500K | Official leaderboard | |
| 18 | CLOVE | 85.1% | Training Source=LAION-COCO, Num Synthetic Captions=>1B | Official leaderboard | |
| 19 | FSC-CLIP | 85.1% | Training Source=COCO, Num Synthetic Captions=1.5M | Official leaderboard | |
| 20 | ReasonSigLIP | 84.1% | ViT=So/14, Resolution=384, Data=+58M | Official leaderboard | |
| 21 | NegCLIP | 83.6% | ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true | Official leaderboard | |
| 22 | OpenCLIP | 83.5% | Data Size=2B | Official leaderboard | |
| 23 | SigLIP | 83% | ViT=So/14, Resolution=384, Data=10.0B | Official leaderboard | |
| 24 | SigLIP2 | 83% | ViT=G/16, Resolution=384, Data=10.0B | Official leaderboard | |
| 25 | Long-CLIP | 82.9% | ViT=L/14, Resolution=224, Data=0.4B | Official leaderboard | |
| 26 | CounterCurate | 82.8% | Training Source=Flickr, Num Synthetic Images=150K, Num Synthetic Captions=150K | Official leaderboard | |
| 27 | Triplet-CLIP | 82.7% | ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true | Official leaderboard | |
| 28 | TripletCLIP | 82.5% | Evaluation Protocol=finetuned | Official leaderboard | |
| 29 | DreamLIP | 81.8% | Data Size=30M | Official leaderboard | |
| 30 | LLaVA-7B | 81.8% | Model=LLaVA-7B | Official leaderboard | |
| 31 | CompoDistill | 81.7% | LLM=Qwen1.5, Size=1.8B | Official leaderboard | |
| 32 | Eva-CLIP-02 | 81.6% | ViT=L/14, Resolution=224, Data=2.0B | Official leaderboard | |
| 33 | MetaCLIP | 80.9% | ViT=L/14, Resolution=224, Data=2.5B | Official leaderboard | |
| 34 | CLIP | 79.9% | Training Source=COCO, Training Protocol=Finetune | Official leaderboard | |
| 35 | ReasonCLIP | 78.2% | ViT=L/14, Resolution=224, Data=+58M | Official leaderboard | |
| 36 | GNM-CLIP | 77.9% | ViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true | Official leaderboard | |
| 37 | Baseline | 77.8% | Evaluation Protocol=finetuned | Official leaderboard | |
| 38 | TSVLC (RB) | 77.8% | Evaluation Protocol=finetuned | Official leaderboard | |
| 39 | CLIP-FT | 77.2% | Evaluation Protocol=Fine-tuned on COCO | Official leaderboard | |
| 40 | ReasonSigLIP2 | 76.8% | ViT=G/16, Resolution=384, Data=+58M | Official leaderboard | |
| 41 | LLaVA-KD | 76.6% | LLM=Qwen1.5, Size=1.8B | Official leaderboard | |
| 42 | KD | 76.1% | LLM=Qwen1.5, Size=1.8B | Official leaderboard | |
| 43 | CON-CLIP | 75.6% | Evaluation Protocol=finetuned | Official leaderboard | |
| 44 | CLIPS (ViT-L/14) | 75.2% | Backbone=ViT-L/14 | Official leaderboard | |
| 45 | C^2LIP | 75% | Backbone=ViT-B/16 | Official leaderboard | |
| 46 | FIGCLIP | 74.6% | Training Source=VidSitu, Num Real Images=20K videos | Official leaderboard | |
| 47 | SigLIP2-ViT-B/16 | 74.4% | Backbone=ViT-B/16 | Official leaderboard | |
| 48 | TSVLC (LLM+RB) | 73.6% | Evaluation Protocol=finetuned | Official leaderboard | |
| 49 | SigLIP ViT-L/16-res256 | 73.4% | Backbone=ViT-L/16, Input Resolution=256x256 | Official leaderboard | |
| 50 | NegCLIP++ | 73% | Pre-training Dataset=CC12M | Official leaderboard | |
| 51 | BLIP-B | 71.3% | Backbone=ViT-B | Official leaderboard | |
| 52 | SF-CLIP | 71.2% | Training Source=YFCC15M | Official leaderboard | |
| 53 | CLIP-A (ViT-L/14) | 70.5% | Backbone=ViT-L/14 | Official leaderboard | |
| 54 | IL-CLIP | 70.3% | Training Source=CC12M | Official leaderboard | |
| 55 | FLAVA | 69.9% | Backbone=ViT-B | Official leaderboard | |
| 56 | LaCLIP | 67.2% | Pre-training Dataset=CC12M | Official leaderboard | |
| 57 | TripletCLIP++ | 66.1% | Pre-training Dataset=CC3M | Official leaderboard | |
| 58 | LaCLIP + HN | 53.9% | Pre-training Dataset=CC3M | Official leaderboard |
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 95.2% | Model=InternVL2-8B | Jul 2024 | Official leaderboard |
| 2 | | 86.5% | Model=Cambrian-8B | Jun 2024 | Official leaderboard |
Cambrian-1-8B on the official leaderboard passed 85% in June 2024
That was 12 months after its release. By the same age, 17% of the benchmarks from 2023 had passed 85%. All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: SOTA2’s leaderboard
No lab model card reports it yet
We read the headline results of 540 launch posts from 57 labs, and none of them list this benchmark.
More in Vision
- MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy.
Qwen 3.6 Plus 86%
- MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. GPT-5.6 Sol 84.6%
- CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
- MathVista In use Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy.
Seed2.1 Pro 90.7%
- MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy.
Qwen 3.8 Max 97.7%
- MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy.
Qwen 3.8 Max 85.9%
Data audit
All 8 checks pass. Every benchmark
Catalog record 2 pass
- Pass Task access All tasks are public.
- Pass Confidence Checked against a primary source.
Scores 2 pass 3 notes
- Pass Independent runs Scores from the official leaderboard (58 entries).
- Pass Models scored 2 models we track, plus 56 other entries on its leaderboard.
- Note Newest score The newest came 2 years ago, in July 2024, for InternVL2-8B. That’s expected for a saturated test.
- Note Frontier models None of its models are on the capabilities index.
- Note Headroom Saturated. The best is 95.2%, past 85% since June 2024.
Leaderboard import 3 pass
- Pass Importer We read the board on SOTA2: 2 rows for models we track and 56 other entries at the last read.
- Pass Last read November 20th, 2026.
- Pass Read worked The last read worked.
Consistency 1 pass 1 note
- Note Lab claims No model has both a lab number and an independent run.
- Pass Known issues No published audit found problems.