PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents

The ray-aligned cross-attention is the strongest technical idea. Mapping each token's viewing ray to rotary phases so that tokens observing the same world direction have zero relative phase by construction is clean, well-motivated, and validated both qualitatively and quantitatively. Learned attention concentrates up to 57% of mass inside the POV frustum versus 10% by chance (Table 12), and training without cross-attention causes the model to 'increasingly collapse late in the rollout into over-scaled avatars and uniform views.' The quadratic-phase player encoding, adapted from mutually-unbiased basis constructions, theoretically justifies growing the identity pool from $m+1$ to $m^2$ codes and is borne out at 16 players. The dataset pipeline decoupling CPU simulation from GPU rendering is a pragmatic advance over client-side capture.

First, the main quantitative comparisons are methodologically uneven. Because no baseline accepts panoramas, PAWA receives strictly more conditioning information than the compared models; the paper then notes that PAWA still outperforms baselines even with cross-attention disabled, but this still leaves PAWA benefiting from a panorama-initialized training run. Second, PAWA's POV tower is initialized from Gamma-World and then jointly trained, so reporting that PAWA 'halves' Gamma-World FVD conflates task-specific joint training with architectural novelty. Third, Table 2 reports that PAWA improves on FVD and LPIPS as player count grows from 4 to 16—a counterintuitive trend that receives no explanation and is not controlled for episode difficulty or activity variation across tiers. Finally, the limitation that 'its POV and panoramic forecasts diverge gradually since the cross-attention aligns directions without enforcing pixel agreement' undercuts the promise of using the panorama as a true shared world state.

The internal ablations supporting ray-aligned cross-attention are thorough: disabling the seven sites at inference raises POV FVD from 138.1 to 179.0 and LPIPS from 0.465 to 0.510 (Table 9), training without them from the same checkpoint widens the gap to 0.056 LPIPS after 3000 updates (Table 11), and Table 7 shows that panoramic conditioning specifically helps when co-players re-enter the POV after being hidden. However, the paper never isolates the dataset contribution: no baseline is retrained on the PAWA data without panoramic inputs, so it remains unclear how much the large margins stem from cleaner or more diverse data versus the architecture. The related work discussion is fair and accurate, correctly noting that prior multiplayer models generate only perspective video even when they use panoramic data for training.

Both the dataset (Hugging Face) and code (GitHub) are openly released, which is commendable. The architecture details are fairly specific: 28 DiT blocks, width 2048, 16 heads, 7 cross-attention sites, equal-area equirectangular rotary encoding, and flow matching with co-player weighting ($\lambda$ schedule in Eq. 12). Training hyperparameters including AdamW settings, learning rates ($3\times 10^{-5}$ to $10^{-5}$), clip lengths (93 or 45 frames), and distillation timesteps (1000, 937, 833, 625) are documented in Appendix 8.3. Yet three critical barriers to independent reproduction remain: the total GPU-hours or training hardware is never stated, the prompts and VLM criteria used by coding agents for data validation are not provided, and the VAE/preprocessing specifics for video latents are buried in appendices without clear main-text pointers. Without the compute budget or the exact data filtering recipes, reproducing the distilled four-step sampler from scratch would require substantial guesswork.