UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.

Flow-A11y: Flow-Aware Accessibility Testing

Modern web applications increasingly expose accessibility barriers through interaction flows rather than static page snapshots. Keyboard traps, focus loss, modal leakage, delayed status updates, dynamic controls, and changing page regions often become observable only after users perform concrete actions. These behaviors are directly related to dynamic WCAG criteria, yet they remain difficult to automate because their assessment depends on runtime interaction evidence and is still commonly performed through manual inspection. We present Flow-A11y, a flow-aware accessibility testing system for interaction-dependent WCAG criteria. Given a target page and a natural-language scenario, Flow-A11y executes the flow in a real browser, records an ordered runtime trace, constructs criterion-specific evidence packets, gates unsupported judgments, and emits auditable findings grounded in resolvable runtime evidence. Evaluated on 19 real public-web scenarios covering 45 dynamic WCAG criteria, Flow-A11y achieves over ten times higher oracle agreement than a generic browser-agent audit, while its evidence-calibration layer improves fail precision from 23.5% to 41.4% and eliminates invalid evidence references. These results show that runtime traces provide actionable evidence for assessing interaction-dependent accessibility behavior. They demonstrate a practical path toward automating dynamic WCAG criteria that page-level scanners cannot assess and that have traditionally required manual evaluation.