Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE

The hierarchical architecture is pragmatic: a slow VLM sets referents, a fast perception module tracks them, and the RGT generates motion, which keeps the VLM off the critical control path. The autoregressive fine-tuning stage is a real strength; the authors observe that Flow Matching inverts its ranking between single-step and closed-loop evaluation, and they correctly note that 'single-step accuracy does not transfer to autoregressive deployment, motivating our use of long-horizon closed-loop rollout as the primary evaluation in the main paper' (Appendix A.1). The latent-bottleneck design without skip connections also successfully forces the model to compress full-head dynamics into a 64-D code, as argued: 'Because the decoder has no skip connection to X_t, all information about the recent past must pass through z, forcing it to summarize both kinematic state—inertia, head overshoot, microsaccadic patterns—and motion style' (Sec. 3.4).

The quantitative evidence is weaker than the kinematic MSE table suggests. The benchmark values come from 'one stochastic evaluation run; evaluation seeds and confidence intervals were not recorded for that run' (Sec. 4.1), so even small differences between RGT-Base, RGT-Large, and the RNN baseline are unvalidated. MSE also measures joint-space reproduction, not perceptual quality; as the authors concede, 'MSE measures robot-space kinematic agreement and APD measures stochastic diversity; neither is an observer rating' (Sec. 6). Additionally, the VLM-guided targeting pipeline is never tested quantitatively because the benchmark supplies every controller with the same ground-truth target, leaving open-vocabulary referent accuracy and fixation robustness unmeasured.
A deeper limitation is data scope and generalizability. Pretraining rests on six behavior sequences from a single participant, and adaptation is shown on only five more. The authors state that 'the small, non-clinical cohort does not establish performance across broader populations or atypical ocular kinematics' (Sec. 4.2), and the 2-second rollout horizon leaves long-term drift uncharacterized. Blink templates are replayed from a small extracted library rather than synthesized, constraining eyelid variety.

The paper does compare learned baselines under matched parameter counts and identical protocols, and the rule-based controllers provide a sensible lower bound. The reversal between single-step and closed-loop rankings is used effectively to argue for autoregressive fine-tuning. However, there is no quantitative comparison with prior dedicated gaze controllers such as Pan et al. (IROS 2020), only narrative discussion; without this, it is hard to judge whether the learning-based approach outperforms existing engineered solutions on the physical robot. The participant-adaptation results show a consistent fivefold MSE reduction, but again lack confidence intervals.

The authors link an anonymous repository with implementation and simulation materials, yet state that 'the complete dataset is not currently public' (Appendix A.4). Most key hyperparameters are disclosed—optimizer settings, layer depths, loss weights $\lambda_{\text{KL}}=10^{-4}$ and $\lambda_{\text{BCE}}=0.1$, and autoregressive rollout horizon—but evaluation seeds are missing, preventing exact reproduction of the stochastic MSE table. Hardware control gains are given in Appendix A.2, though full end-to-end latency numbers for the physical pipeline are not reported: 'the reported latency applies to RGT inference, not to the entire perception, communication, and actuation pipeline' (Sec. 4.1).