Left: how the 32 apps and their 10-request sequences were built. Right: how an agent is evaluated on one app, request after request.
EvoWeb gives a coding agent 32 web apps to build and then extend, request by request: a first build and nine follow-up requests per app, each adding a feature that may depend on earlier ones. After every request a browser agent tests the running app with five checks. Grade summarises how much working functionality survives the whole sequence; Survival counts how many requests in a row stay usable.
The harness that runs the benchmark (OpenHands as the agent, WebVoyager as the judge), the scoring code and the released traces of the paper's runs are at github.com/two-delta/EvoWeb. This dataset is the benchmark itself, as one row per app.
| config | rows | what |
|---|---|---|
default | 32 | the benchmark: 32 apps x 10 requests, with the judge's checks |
multilingual (splits fr de zh hi he) | 32 each | the same apps with the user's requests translated; the app's UI labels, seeded data and the judge's checks stay in English |
from datasets import load_dataset
evoweb = load_dataset("TD-Labs/EvoWeb", split="test") # the benchmark
french = load_dataset("TD-Labs/EvoWeb", "multilingual", split="fr")
Row order matters. A sample index in a run or a trace is the 1-based row position in the file the run used, and the scorer keys task counts by that position. Never re-sort a configuration.
id "000001" zero-padded WebGen-Bench app id (not the sample index)
instruction the first (build) request
Category, application_type WebGen-Bench metadata, unused by the harness
ui_instruct [ {task, expected_result, task_category} ] the judge's checks for the first request
follow_ups [ 9 x { request k = follow_ups[k-2], k = 2..10
archetype derived-state | multi-surface-workflow | interactive-tool | design-restyle
instruction the user's request
source_website, source_feature the real product feature the request was grounded in
ui_instruct [ 5 x {kind, task, expected_result, task_category, [copied_from_request]} ]
} ]
dependency_graph {nodes: [{id: "R1".."R10", label, archetype}], edges: [{from, to, what}]}
global_rules [ 3 x {rule, introduced_in_request} ] app-wide rules the requests introduce
pinned_seed {entity, introduced_in_request, records_gist, [records]}
the fixed demo data every exact-value check is written against
(`records`, present for 3 apps, is a JSON-encoded string here)
target_app_summary prose description of the app the 10 requests build towards
Every follow-up request carries exactly five checks in a fixed order: three functional
(exact-value or feature), one integration (the new feature combined with an earlier one,
along an edge of dependency_graph, or one of the global_rules when the request has no incoming
edge), and one backward (a check copied from an earlier request).
The 32 apps are WebGen-Bench seeds. For each, an LLM workflow grounded in real similar websites
designed the grown app and its dependency DAG, wrote the nine follow-up requests with their
checks, and was audited and repaired; the generator, its audit rules and the assembly scripts are
in the repository under data/data_gen/.
@inproceedings{evoweb2027,
title = {EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications},
author = {Anonymous authors},
booktitle = {International Conference on Learning Representations (ICLR), under review},
year = {2027}
}