Benchmark.ai Image Benchmark
Built by Rapidata for Benchmark.ai.
This dataset contains 6,123,853 human responses, collected with the
Rapidata Python SDK, comparing 43 text-to-image models on
1,514 prompts. Each row is a head-to-head comparison between two models' images for the same
prompt, judged by human annotators on up to four questions — Alignment, Preference, Coherence and Text coherence.
Every image is ranked purely by human judgement, no automated metrics. This is the data behind the
Benchmark.ai image leaderboard.
If you get value from this dataset and would like to see more in the future, please consider liking it ❤️
To evaluate your own models and create a leaderboard, check out our MRI.
At a glance
| |
|---|
| Head-to-head comparisons (rows) | 680,548 |
| Human votes | 6,123,853 (Alignment 1,730,190 · Preference 1,731,352 · Coherence 1,730,112 · Text coherence 932,199) |
| Models compared | 43 |
| Prompts | 1,514 (616 of them ask for rendered text) |
| Unique generated images | 64,266 |
| Annotator countries | 148 |
The leaderboards
Every model is compared on the same kind of image pairs across four independent questions, run as four
leaderboards on one Rapidata benchmark:
| Leaderboard | Question shown to annotators | Prompt shown? | Measures |
|---|
| Alignment | "Which image matches the description better?" | Yes | how faithfully the image depicts the prompt |
| Preference | "Which image looks higher quality and more visually appealing?" | No | overall visual appeal, independent of the prompt |
| Coherence | "Which image has more glitches and is more likely to be AI generated?" | No | visual soundness — the image with fewer artifacts wins |
| Text coherence | "In which image does the text have glitches or is harder to read?" | No | legibility of rendered text — the image with fewer text glitches wins (text prompts only) |
Alignment
The alignment score measures how well the image matches its prompt. Annotators saw the prompt and were asked: "Which image matches the description better?"
Preference
The preference score reflects how visually appealing annotators found each image. Without seeing the prompt, they were asked: "Which image looks higher quality and more visually appealing?"
Coherence
The coherence score measures whether the image is free of artifacts and visual glitches. Without seeing the prompt, annotators were asked: "Which image has more glitches and is more likely to be AI generated?" The image picked less often wins.
Text coherence
The text coherence score measures how cleanly a model renders text. It runs only on the 616 prompts that ask for text. Without seeing the prompt, annotators were asked: "In which image does the text have glitches or is harder to read?" The image picked less often wins.
Rankings
Overall ranking (ELO, aggregated across all four leaderboards)
| # | Model | Lab | Overall | Alignment | Preference | Coherence | Text coherence |
|---|
| 1 | GPT Image 2.5 (Sunburst) | OpenAI | 1238.3 | 1326 | 1250 | 1246 | 1181 |
| 2 | GPT Image 2.5 (Flare) | OpenAI | 1207.1 | 1341 | 1196 | 1219 | 1136 |
| 3 | GPT Image 2.0 | OpenAI | 1167.3 | 1242 | 1135 | 1113 | 1186 |
| 4 | Reve 2.1 | Reve AI | 1147.5 | 1149 | 1096 | 1157 | 1193 |
| 5 | Grok Image Imagine 2.0 (Medium) | SpaceXAI | 1136.8 | 1192 | 1099 | 1096 | 1165 |
| 6 | GPT Image 1.5 (high) | OpenAI | 1131.9 | 1247 | 1242 | 896 | 1154 |
| 7 | nano-banana-2 | Google | 1128.1 | 1169 | 1242 | 1050 | 1051 |
| 8 | Grok Imagine Image 2.0 | SpaceXAI | 1107.5 | 1128 | 1086 | 1076 | 1155 |
| 9 | Muse Image | Meta | 1096.3 | 1171 | 1108 | 1062 | 1034 |
| 10 | Qwen-Image-3.0 | Alibaba Cloud | 1083.1 | 1110 | 1205 | 972 | 1031 |
| 11 | nano-banana-pro | Google | 1071.6 | 1070 | 1044 | 1104 | 1049 |
| 12 | Seedream 4.5 | ByteDance | 1046.7 | 1032 | 999 | 948 |
A model shows "—" where it has no standing on that leaderboard.
By output type and capability
OpenAI's GPT Image 2.5 tops every category the benchmark slices by, so the last column shows the strongest model
from any other lab. The full per-category standings, including subject and prompt source, are on
Benchmark.ai.
| Output type | Prompts | #1 overall (ELO) | Best non-OpenAI model (ELO) |
|---|
| General Image | 524 | GPT Image 2.5 (Sunburst) (1265) | Reve 2.1 — Reve AI (1151) |
| Photorealism | 243 | GPT Image 2.5 (Flare) (1242) | Reve 2.1 — Reve AI (1141) |
| UI / UX | 182 | GPT Image 2.5 (Sunburst) (1213) | Muse Image — Meta (1174) |
| Illustration & Anime | 147 | GPT Image 2.5 (Sunburst) (1237) | Reve 2.1 — Reve AI (1161) |
| Graphic Design & Marketing | 137 | GPT Image 2.5 (Sunburst) (1249) | Reve 2.1 — Reve AI (1222) |
| Logos & Branding | 134 | GPT Image 2.5 (Flare) (1209) | nano-banana-2 — Google (1151) |
| Infographics | 131 | GPT Image 2.5 (Sunburst) (1235) | nano-banana-2 — Google (1154) |
| 3D Modeling & Rendering | 19 | GPT Image 2.5 (Sunburst) (1270) | Qwen-Image-3.0 — Alibaba Cloud (1176) |
| Capability | Prompts | #1 overall (ELO) | Best non-OpenAI model (ELO) |
|---|
| Attribute Binding | 758 | GPT Image 2.5 (Sunburst) (1214) | Reve 2.1 — Reve AI (1153) |
| Text | 616 | GPT Image 2.5 (Sunburst) (1236) | Reve 2.1 — Reve AI (1159) |
| Scene Composition | 428 | GPT Image 2.5 (Flare) (1230) | Reve 2.1 — Reve AI (1163) |
| Knowledge & Reasoning | 384 | GPT Image 2.5 (Sunburst) (1227) | nano-banana-2 — Google (1141) |
| Spatial & Size | 264 | GPT Image 2.5 (Sunburst) (1228) | Reve 2.1 — Reve AI (1129) |
| Number of Objects | 252 | GPT Image 2.5 (Sunburst) (1265) | Reve 2.1 — Reve AI (1169) |
| Surreal | 244 | GPT Image 2.5 (Flare) (1206) | nano-banana-2 — Google (1142) |
| Emotion & Mood | 130 | GPT Image 2.5 (Sunburst) (1254) | Reve 2.1 — Reve AI (1177) |
| Noisy Input | 20 | GPT Image 2.5 (Sunburst) (1377) | Grok Image Imagine 2.0 (Medium) — SpaceXAI (1238) |
How the dataset was built
1. Prompts (1,514)
The prompts are drawn from established text-to-image evaluation sets, real user prompts and prompts written for this
benchmark:
| Source | Prompts |
|---|
| Rapidata Original | 328 |
| DiffusionDB | 325 |
| PartiPrompts | 243 |
| Community | 235 |
| MONET | 187 |
| DrawBench | 87 |
| ABC-6K | 40 |
| T2I-CompBench | 31 |
| DALLE3-EVAL | 20 |
| HRS-Bench | 18 |
Every prompt is tagged on three more axes, which the live leaderboard can slice by:
- Capability — attribute binding, text, scene composition, knowledge & reasoning, spatial & size, number of
objects, surreal, emotion & mood, noisy input (a prompt can carry several)
- Output type — general image, photorealism, UI / UX, illustration & anime, graphic design & marketing, logos &
branding, infographics, 3D modeling & rendering
- Subject — objects & abstract, people & portraits, animals, architecture & spaces, vehicles, sci-fi & fantasy,
food & drink, nature & landscapes
2. Image generation (43 models)
Each prompt was sent to every model, and the resulting images were uploaded to a
Rapidata MRI benchmark as one participant per model. Every model has images for
between 1,410 and 1,514 of the 1,514 prompts.
Alibaba Cloud — Qwen-Image, Qwen-Image 2.0 Pro, Qwen-Image-3.0, Wan 2.7 Pro · Baidu — ERNIE Image · Black Forest Labs — FLUX.2 dev, FLUX.2 flex, FLUX.2 klein 9B, FLUX.2 max, FLUX.2 pro · ByteDance — Seedream 4.5, Seedream 5.0 Lite, Seedream 5.0 Pro · Google — Imagen 3, Imagen 4, Imagen 4 Fast, Imagen 4 Ultra, nano-banana-2, nano-banana-pro · HiDream AI — HiDream O1 · Ideogram — Ideogram 4.0 Quality, Ideogram 4.5 Quality · Luma Labs — Luma UNI 1.1 Max · Meta — Muse Image · Microsoft AI — MAI-Image 2.5 · OpenAI — GPT Image 1.5 (high), GPT Image 2.0, GPT Image 2.5 (Flare), GPT Image 2.5 (Sunburst) · Pruna AI — Pruna P-Image, Pruna-P-Ideogram (High), Pruna-P-Ideogram (Low), Pruna-P-Ideogram (Medium), Pruna-P-Ideogram (Very Low) · Recraft — Recraft V4.1 Utility Pro · Reve AI — Reve 2.1 · SpaceXAI — Grok Image Imagine 2.0 (Medium), Grok Image Imagine 2.0 (Thinking), Grok Imagine Image, Grok Imagine Image (quality), Grok Imagine Image 2.0 · Stability AI — Stable Diffusion 3.5 Large · Tencent — HunyuanImage 3.0
3. Human evaluation
The four leaderboards were run over pairwise matchups between the models' images for the same prompt. Alignment,
Coherence and Text coherence were answered by dedicated, qualified Rapidata audiences; Preference by the open global
pool. Each pair is judged on a subset of the questions, with at least 5 votes per question it was judged on; Text
coherence runs only on the 616 text prompts. Per-annotator detail — chosen side, country, language, gender, age
bucket, occupation and the annotator's userScore — is preserved in the detailed_results_* columns. Votes came from
annotators in 148 countries.
Dataset structure
One row per head-to-head image pair generated from the same prompt.
| Column | Type | Description |
|---|
prompt | string | the prompt both images were generated from |
image1 | string | public URL of the image from model1 |
image2 | string | public URL of the image from model2 |
model1 | string | name of the model that produced image1 |
model2 | string | name of the model that produced image2 |
weighted_results_image1_<q> | float32 | sum of the userScore weights of the votes for image1 on question <q> |
weighted_results_image2_<q> | float32 | sum of the userScore weights of the votes for image2 on question <q> |
detailed_results_<q> | string | every vote on question <q> as a JSON array (votedFor is A for image1 and B for image2, plus annotator demographics, userScore and votedAt) |
<q> is one of alignment, preference, coherence and text_coherence. A row's columns for a question it was
not judged on are empty.
- The
weighted_results_* values are userScore-weighted vote sums, not probabilities — they do not sum to 1.
Divide by their sum for a normalised win share.
- Coherence and Text coherence are already flipped. Annotators were asked which image has more glitches, but in
this dataset
votedFor and weighted_results_* point to the image with fewer glitches, so a higher value is
better on every question.
import json
from datasets import load_dataset
ds = load_dataset("Rapidata/benchmark-ai-image-benchmark", split="train")
row = ds[0]
print(row["prompt"], "|", row["model1"], "vs", row["model2"])
for q in ["alignment", "preference", "coherence", "text_coherence"]:
w1, w2 = row[f"weighted_results_image1_{q}"], row[f"weighted_results_image2_{q}"]
if w1 is None or w1 + w2 == 0:
continue # this pair was not judged on q
print(f"{q}: {row['model1']} wins {w1 / (w1 + w2):.0%} of the weighted vote")
Licensing & attribution
This dataset combines material under different terms:
- Prompts. Prompts taken from public evaluation sets (PartiPrompts, DrawBench, DiffusionDB, MONET, T2I-CompBench,
HRS-Bench, ABC-6K, DALLE3-EVAL) remain under their original licenses; Rapidata-original and community prompts are
released under CC-BY-4.0.
- Generated images. Produced by third-party models. Model outputs are governed by the terms of use of each
respective model provider.
- Human annotations. Collected via Rapidata, released under CC-BY-4.0.
Citation
If you use this dataset, please cite it:
@misc{vikat2026texttoimage,
title = {{text-to-image-human-feedback-benchmark}},
author = {Vikat, Karl and Giger, Lino and M'Hamdi, Fayçal and Saadi, Mohamed Sadek and Teikari, Elias and Tollis, Manuel and Kuhlmann-Joergensen, Mads},
year = {2026},
publisher = {Hugging Face},
doi = {10.57967/hf/11068},
url = {https://huggingface.co/datasets/Rapidata/text-to-image-human-feedback-benchmark}
}
Vikat, K., Giger, L., M'Hamdi, F., Saadi, M. S., Teikari, E., Tollis, M., & Kuhlmann-Joergensen, M. (2026).
text-to-image-human-feedback-benchmark [Data set]. Hugging Face.
https://doi.org/10.57967/hf/11068
About Rapidata
Rapidata's technology makes collecting human feedback at scale faster and more accessible than ever before. Visit
rapidata.ai to learn more about how we're revolutionizing human feedback collection for
AI development.
Explore all of our live benchmarks on Benchmark.ai.
Related datasets
