Divide-and-Remember: Recursive Action-Relevant Memory for Long-Horizon VLA Policies
Vision-language-action policies often fail on partially observable, long-horizon manipulation because the current image alone does not determine the correct action. This paper reformulates robotic memory as an optimisation problem: the optimal memory should maximise the conditional mutual information $I(a_t; m_t \mid o_t)$ between action and memory given the current observation. To realise this, the authors propose Divide-and-Remember (D&R), a recursive lightweight selector that retains the top-$K$ most action-relevant tokens from an unbounded history while staying in the pretrained VLA token space. On the RoboMME benchmark and in real-robot experiments, D&R achieves strong gains with budgets as small as 64 tokens, offering both theoretical clarity and empirical progress on memory-augmented visuomotor control.
D&R is a compelling and theoretically grounded memory mechanism for VLAs: the derivation shows that "maximising $I(a_t; m_t \mid o_t)$ is equivalent to minimising the discarded information" and the optimal memory is "$\mathcal{M}^\star = \arg\max_{\mathcal{M}} I(a_t; \mathcal{M}(h_t) \mid o_t)$". The recursive selector scales to long histories with fixed parameters, yielding consistent gains on RoboMME and transferring to real-robot trials. Yet the paper itself notes that "On state-like facts D&R only ties latent memory, and on StopCube and ButtonUnmask it falls behind, since D&R stores and uses history tokens directly, whereas a latent memory can reformulate or aggregate knowledge in a more compact way."
The information-theoretic framing in Section 4 is the paper's strongest conceptual asset, giving a unified objective that exposes why perceptual heuristics are loose and why latent memories sacrifice spatial fidelity. Empirically, D&R dominates on a tight 64-token budget: "Overall, D&R reaches the highest average success rate over the four suites (38.6%, against 32.2% for HAMLET, 27.9% for FrameSamp, and 17.9% for the memoryless $\pi_{0.5}$)". The real-robot evaluation is equally persuasive: "D&R succeeds in 35 of 40 trials, against 22 for FrameSamp and 3 for $\pi_{0.5}$", showing robustness to human intervention and perceptual noise.
The derivation of the optimal memory relies on the expert policy $\pi^\star$, which is unknown; the tractable objective replaces the true conditional $p(a_t \mid o_t, m_t)$ with the current parametric policy $\pi_\theta$, creating a joint optimisation that may cycle if either module is weak. A second concern is the discrete candidate pool: the selector can only choose from "a candidate pool of $|\mathcal{H}|$ tokens, $|\mathcal{H}|/16$ frames sampled uniformly from the history with 16 pooled patch tokens each", so critical moments missed by the subsampling are irrecoverably lost. Finally, D&R's strength is uneven: as the authors concede, "On state-like facts D&R only ties latent memory, and on StopCube and ButtonUnmask it falls behind, since D&R stores and uses history tokens directly, whereas a latent memory can reformulate or aggregate knowledge in a more compact way."
The evidence supports the core claims within the RoboMME benchmark: all methods use the same $\pi_{0.5}$ backbone and 64-token budget, and D&R tops every suite average. The ablation in Section 6.3 is particularly telling: the authors report that asking D&R to also predict future states means "the prediction loss brings no gain: asking the memory to predict the future spends capacity that the action does not buy back". However, the benchmark is small (16 tasks), and scalability is not yet fully demonstrated because "its curve flattens at 256 tokens (50.1%) and dips at 512 (46.4%), which we attribute to convergence: a larger budget needs more training steps".
The authors publish hyperparameters in Appendix B.1 — "Detailed hyperparameters are provided in Table 3" — and a project website, but several architectural details remain underspecified. For instance, the selector is described as "A transformer of $N$ self-attention and FFN blocks then scores each token", yet the value of $N$ and the hidden dimension are not stated, nor are the schedules for the Gumbel temperature or straight-through threshold. Training costs are reported transparently — "Training takes about 20 hours for the perceptual and latent memory variants, 45 hours for TTT, and 10 days for RB-VLA" — but the reliance on cached SigLIP tokens and specific baseline re-implementations means reproducibility hinges on code release fidelity. The real-robot protocol is sketched without controller gains or exact intervention scripts.
Vision-language-action (VLA) models struggle on history-dependent manipulation tasks, where the current observation alone does not determine the action, and the policy needs a memory of the history. Existing memory methods decide what to remember by design, for example, keeping frames with large pixel changes, and show inconsistent gains across tasks. We view what to remember as an optimisation problem. From the POMDP formulation of imitation learning, we show that the optimal memory maximises the conditional mutual information $I(a_t; m_t \mid o_t)$ between the action and the memory given the current observation. Intuitively, this means preserving the action-relevant information in the history that is not already contained in the current observation. Based on our analysis, we propose Divide-and-Remember (D&R), a recursive memory method that learns a memory function $m_t = M(h_t)$ and scales to long contexts while staying compute-light. It involves two strategies: (1) the selection over the full history is divided recursively into subproblems of top-$K$ selection over $2K$ tokens, so that fixed-size, lightweight selectors learned end-to-end support an unbounded history; (2) all recursion blocks share one selector, which captures the selection rule common to every block and keeps the method efficient. On RoboMME, a benchmark of 16 long-horizon manipulation tasks that require remembering when, where, what, and how to act, D&R achieves a state-of-the-art average success rate with consistent gains across all four suites under a budget of only 64 tokens; real-robot experiments show the same gain. Code, checkpoints and more results are at https://dnr-memory.github.io/
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.