VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning

The framework's generality is its main strength: it decouples learning-history construction from policy integration, supporting simulation RL, human-in-the-loop curricula, and hindsight relabeling within a single training pipeline. The use of Transformer-XL recurrent memory to handle long-horizon trials across episodes is well motivated for robotic deployment, and the real-world cross-episode memory task demonstrates that the model can leverage binary reward feedback to resolve ambiguous instructions without parameter updates. These design choices are grounded in prior work on algorithm distillation, which the authors correctly describe as training a causal sequence model on ordered learning histories so that the model 'could perform in-context RL and gradually improve the task return by conditioning on its own interaction history in unseen tasks, without parameter updates.'

Several issues temper the empirical claims. First, the comparison with RoboTTT is confounded by deliberately withholding reward and termination signals from the baseline, so the reported gap reflects the value of reward information rather than a fair framework-to-framework comparison. Second, real-world fruit-placement results show an initial decline in success relative to the frozen baseline—'After a slight initial decline relative to $\pi_{0.5}$, VLRLA steadily improves'—and performance on grapes is unstable ('Performance on grape follows a less consistent trend, fluctuating across adaptation stages'), suggesting limited robustness across objects. Third, the training objectives for both VLRLA-simp ($\mathcal{L}(\theta_{\mathrm{RLA}})$) and VLRLA-bridge ($\mathcal{L}(\theta_{\mathrm{RL}})$) are essentially supervised losses on historical trajectories; the paper offers no theoretical argument that minimizing these losses yields a policy improvement operator in the RL sense, and the absence of an in-context imitation baseline without rewards makes it impossible to isolate the contribution of RL feedback.

The simulation evidence supports in-context adaptation, but it is incomplete. Success rates are reported as averages over 128 environment instances without confidence intervals or statistical tests, so it is unclear whether differences like the 7.3-point unseen-task gain are significant. The comparison to GRPO is also mixed: VLRLA-bridge trails GRPO on unseen tasks and only narrowly leads RoboTTT overall (63.4% vs. 58.2%), despite GRPO requiring far more compute. The authors note that 'This gap reflects the absence of reward signals in RoboTTT hinders the model’s adaptation capability,' confirming that the comparison is more about reward conditioning than architectural superiority. A missing baseline is in-context behavior cloning on the same histories without reward labels, which would clarify whether the RL formulation is necessary or if simple imitation of improving trajectories suffices.

The authors provide unusually detailed hyperparameters in Appendix A (Tables S1–S5), including PPO fine-tuning configs, network dimensions, and optimizer settings, which helps reproducibility. However, the code and datasets are not publicly referenced, and the real-world experiments rely on proprietary base models ($\pi_{0.5}$), specialized hardware, and small-scale human-in-the-loop collection (80 teleoperation episodes plus 60 corrected episodes). Complex design choices such as the learned summarization module for KV-cache compression and the low-rank residual adapter in Eq. (12) are described but not ablated, so reproducing the exact real-world behavior would be difficult without access to the same infrastructure and training pipelines.