Memory before Goal: Role of Pre-History Memory for Humanoid Loco-Manipulation
Whole-body vision-language-action (VLA) policies for humanoid loco-manipulation usually condition only on the current view, so they fail when brief object cues vanish before the robot must act. This paper proposes WholeBodyMem, which uses a frozen SAM3 to track task objects, detects object events (appear, disappear, move, change shape) to causally select up to $B_{\max}=7$ key frames, and feeds those native tokens together with mask-pooled object tokens into a flow-matching action expert via cross-attention and FiLM. On the new HuMeGo benchmark and standard SIMPLE tasks, the method raises the $\Psi_0$ base policy's simulated success from $29.1\%$ to $72.6\%$ and real-world Unitree G1 success from $36.7\%$ to $88.3\%$, while also improving reactive tasks to $86.9\%$.
The paper presents a well-motivated and empirically strong contribution. The core idea—that causal, event-driven frame selection inside a fixed native-token budget matters more than uniform memory or larger capacity—is convincingly supported by granular ablations and real-world Unitree G1 experiments. While the reliance on an external frozen tracker introduces single-point fragility and the benchmark tasks are somewhat toy-like, the overall claim that pre-history memory should be object-guided and adaptive holds up.
The ablations isolate the value of each component at a fixed seven-frame, 539-token budget: simply switching from uniform sampling to event-based key frames without object tokens or the decision head lifts average success from $44.4\%$ to $60.4\%$, and the full system reaches $72.6\%$. The separation of memory-decision accuracy ($94\%$) from manipulation success also cleanly isolates the memory problem from control noise.
The method depends on a frozen SAM3 tracker with hand-tuned thresholds—$3\%$ image motion, $65\%$ mask overlap, $0.35$ confidence—shared across all tasks; a missed detection is irrecoverable because the memory module never hallucinates. The real-world evaluation rests on only 15 teleoperated demonstrations per task and 10 trials each, which is too narrow to claim robustness. Moreover, the 'memory as context' ablation is arguably a straw-man: feeding long native visual tokens into a frozen VLM is known to perform poorly, so the large gap does not rule out other memory integration schemes.
Simulated evidence strongly supports the headline claim that memory-decision accuracy rises from near-chance ($39\%$) to $94\%$, but comparisons to prior memory methods are incomplete. The best competing baseline is an adapted RoboMME uniform sampler at the same token budget, while methods such as MemoryVLA and KEMO are discussed but not directly evaluated. The real-world gains are promising yet based on only six tasks and 10 trials each, leaving unresolved how the method scales to cluttered, contact-rich settings.
Training hyperparameters (AdamW, $10^{-4}$ learning rate, $40$k steps, frozen Qwen3-VL-2B backbone) and the LeRobot v2.1 data format are specified, but no code, weights, or task prompts are released at submission. Independent reproduction is feasible in principle yet would require replicating the SAM3 event detection pipeline, the proprietary teleoperation stack (PICO headset, SONIC controller), and an RTX 5090 for real-time action chunking. Exact SAM3 prompts are generated by Claude Opus 5.5 and fixed before evaluation, but their text is not provided.
Humanoid loco-manipulation in dynamic scenes is non-Markovian: cues vanish as objects change or are hidden, or leave the view as the robot turns and walks, yet whole-body vision-language-action (VLA) policies see only the present. We present WholeBodyMem, an object-guided perceptual memory that keeps the pre-history of object events seen before the robot acts on its goal. SAM3 tracks the task objects, and their events causally select key frames within an adaptive budget, which the action expert reads by cross-attention together with object tokens encoding appearance, geometry and time. We also contribute HuMeGo, a humanoid memory benchmark with six simulated and six real-world Unitree G1 tasks. Keeping the right frames rather than more frames, WholeBodyMem surpasses memoryless and uniform-memory baselines, raising its base policy's success from 29.1% to 72.6% in simulation and from 36.7% to 88.3% on the real robot, while also improving the standard SIMPLE tasks (70.2% to 86.9%).
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.