ProductWebBench: Evaluating Website Continuity Beyond Task Completion
Frontend coding agents are usually graded only on whether the requested target gets built, but real engineering demands that the rest of the site keeps working. ProductWebBench evaluates this second obligation—website continuity—through 400 frozen tasks where a simulated user reveals requirements stage by stage and every cleared gate remains armed for the rest of the run. Across 8,737 runs from 13 models, 28.2% of Change attempts that pass every requested-work check still fail a continuity check, showing that scoring a patch against the target alone is insufficient.
The paper makes a strong methodological contribution by formalizing website continuity and grounding it in exact regression provenance via two independent channels: an untouched-repository baseline and protected-state replay. The headline finding—that 28.2% of completing Change runs still fail a continuity check—is supported by careful cumulative evaluation and pre-admission controls, though the authors are candid that a 0.190 false-fail rate on completion gates leaves the completing subset compositionally biased.
The cumulative-gate design is rigorous: once a milestone clears, its assertions join a protected set replayed at every later checkpoint, so pass-to-fail transitions are mechanically identified without post-hoc labelling. Admission controls are equally careful—every task enters the pool only after a reference passes every gate and plausible wrong implementations fail—so satisfiability and gate discrimination are settled before any model runs. The resulting split between completion and continuity verification backed by deterministic browser checks, rather than static labels, meaningfully advances how frontend agents are evaluated.
The verifier audit in Appendix D reveals a 0.190 false-fail rate on completion-family gates, and critically all twelve false-fail cases in the blind consensus sample sit on completion gates rather than continuity gates. Because the 674 completing runs are defined by passing completion gates, compositional bias from these false fails could shift the measured 28.2% gap in an unsigned direction, and the paper correctly warns against reading post-completion figures as conservative bounds.
The benchmark is modest in scale—400 tasks, only 234 Change slots with capturable untouched baselines—and the main text provides no public URL for code, containers, or task packages, deferring all release specifics to appendices. Several cited works carry 2025/2026 dates that are not currently publicly verifiable, slightly weakening confidence in the related-work baseline.
The positioning against SWE-bench, WebArena, and frontend generation benchmarks is accurate and well-scoped. However, the paper slightly overstates the prompt-scale conclusion: while the claim that 'what the solver is told moves the score; what it is shown, and how the request is worded, do not' is headline-grabbing, Appendix F reports that screenshots significantly affect the experience facet (+0.0144, p=0.001) and broad continuity (−0.0124, p=0.009), contradicting the zero-effect framing. The scaffold-versus-backbone grid is also limited to two backbones with uncontrolled time and token budgets, so the conclusion that scaffold matters as much as backbone is suggestive rather than definitively causal.
The protocol is described in detail—12 turns per stage, 200 per run, deterministic 19-gate verifier, frozen manifests—but no public repository link, container specification, or task-package download URL appears in the main text. The authors candidly report a per-cell noise floor of $\sigma \approx 0.09$ at $T=0$ and commit to releasing the 8,737-cell manifest, which is laudable, yet without the frozen toolchain and harness code an independent group cannot reconstruct the evaluation environment. Scaffold-specific system prompts and the edit-application harness are additionally underspecified outside the appendices.
Frontend coding agents are usually graded on one question: did the requested target get built. Real engineering changes carry a second obligation, because the rest of the site has to keep working. We call that second obligation website continuity, and introduce ProductWebBench, a benchmark that scores both on living frontend repositories: 320 Change tasks over existing code and 80 long-horizon Build tasks. Every task freezes a repository snapshot, staged requirements, browser evidence and deterministic checks before evaluation, and enters only once a reference passes every check and plausible-wrong ones fail. A simulated user reveals the requirements one stage at a time, and every check cleared earlier stays armed for the rest of the run. Across 8,737 runs from 13 models the two come apart: of the Change runs passing every requested-work check, 28.2% still fail a continuity check -- a blank route, horizontal overflow, console errors -- and 10.8% push a browser state they had already passed back to failing. Because the same checks also run on the untouched repository, each regression carries exact provenance, not a post-hoc label: on the 586 runs where both readings exist, 18.3% are caught breaking a verified state. Interaction is a further blind spot, failing in 29.5% of runs whose static DOM already passes. ProductWebBench therefore scores a patch against the living system it lands in, not the target alone.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.