Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training
Trace2World tackles the bottleneck of building faithful GUI training environments from sparse real-app traces. It separates global semantic reconstruction—resolving page identities, state fields, and conditional effects across traces—from browser implementation, then validates by independently replaying every source workflow. On 200 workflows across ten apps, this contract reaches 91.1% end-to-end fidelity versus 59.8% for the strongest task-local baseline and retains 94.0% of simulated training gains when transferred to real devices.
The paper presents a conceptually clean architecture and strong empirical argument that reconciling source-linked semantics before coding yields substantially higher fidelity than coupled coding-agent pipelines. The 91.1% workflow fidelity and the self-validation control (79.1%) together show that the separation of reconstruction from implementation is not merely theoretical: it measurably prevents hard-to-detect state and effect errors. However, the evaluation is restricted to a single backbone (Qwen3.5-9B) and a 100-task real-device panel, and the browser-based environment target differs in native interaction modality from the mobile apps that PhoneWorld and AndroidWorld address. The core claim—that the bottleneck is no longer producing runnable mocks— is well supported within its scope, but the broader scalability of sim-to-real transfer for native mobile remains partially open.
Two design choices receive unusually strong empirical defense. First, the six-check workflow-fidelity metric directly exposes that endpoint reachability is a misleading proxy: 'Reachability alone overstates reconstruction quality,' with PhoneWorld-style and GUI-GENESIS-style reaching 50.4–94.8% of endpoints yet preserving only 16.7–59.8% of workflows (Table 1, Section 4.1). Second, the self-validation ablation isolates the value of source-trace validation specifically: 'Replacing source-trace validation with self-validation lowers workflow fidelity from 91.1% to 79.1% while Reach stays near 98%,' and real-device gain retention drops from 94.0% to 54.7% (Section 4.2, Table 3). The concrete case in Meituan, where 'tapping Disable quick pay opens a confirmation without changing the saved setting; our PhoneWorld-style reconstruction disables it immediately yet passes a final-value check,' makes the failure mode vivid and traceable (Introduction, Figure 2).
The evaluation, while internally consistent, carries important scope limitations. All downstream training uses 'Qwen3.5-9B trained with DAPO across three frozen stages,' so the 20.2-point benchmark gain and external transfer claims are model-dependent and lack replication across scales or RL algorithms (Section 4). The real-device transfer panel is just 'the shared 100-task panel' with matched goals and action budgets, leaving robustness to app diversity and long-tail state configurations largely unaudited (Section 4.2). Additionally, Trace2World builds browser-hosted DOM packages rather than native Android apps, which means the 'real-device' evaluation still measures transfer from browser mocks to real mobile apps—a different modality gap from the 'visual or dynamics gaps' the paper explicitly warns against (Section 1). The learner-guided augmentation gains are modest (+4.2 pp over static generation at matched horizon $H=45$) and depend on a fixed external solver and a multi-stage freeze schedule whose sensitivity is not fully explored.
The comparison against reimplemented baselines on the same substrate and recordings is methodologically fair, but the chosen metric favors Trace2World because workflow fidelity to source traces is literally what competitors do not optimize for. The paper correctly identifies that 'CUA-Gym codes mocks from topic specifications' and 'GUI-GENESIS codes a task-local site from one trace,' while 'PhoneWorld codes reusable Android pages from frequency-ranked screen types... then checks representative flows with human QA' (Introduction). However, GUI-GENESIS's actual objective is 'the first framework to automatically synthesize efficient GUI training environments with verifiable rewards' with an emphasis on latency reduction and code-native determinism (Cao et al., 2026), and CUA-Gym's is to 'co-generate task instructions, environment states, and reward functions jointly from a shared topic specification' at scale (Wang et al., 2026). Evaluating them by trace fidelity is valid for the specific claim of source adherence but does not establish dominance on their own scaling axes. The 90.2% human-automated reviewer agreement on 1,200 packets lends credibility to the fidelity scores (Appendix E.1.4).
The authors promise release of source recordings, trace records, environments, and evaluation artifacts (Ethics Statement), and hyperparameters are detailed in Appendix D.4. However, full reproduction faces three barriers. First, the semantic reconciliation and repair rely on 'GPT-5.2' and 'GPT-5.5,' which appear to be proprietary or future model versions without public availability; the paper notes that 'compared builders receive GPT-5.5 in every role they define' while Trace2World uses the cheaper model for reconstruction—a capability gap that may not persist as models evolve (Section 4.1). Second, complete prompt templates are only 'condensed' in Appendix G, with full versions residing in unreleased artifacts. Third, the multi-stage pipeline involves 'an estimated 3.6 hours and 1,160 model calls per app' and 'the largest model-visible call is 33.2K tokens under a 40K-token budget,' making independent replication costly even with artifact access (Section 3.2, Appendix E.2.4).
Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce \textbf{Trace2World}, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4--16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.