Benchmarking Robot Memory Under Interference

Contributions

• A cross-session interference benchmark. We extend RoboMME with a controllable history buffer: a relevant prior demonstration followed by k ∈ {0,1,3,7} unrelated sessions, turning memory distance into a measured variable.
• A complete, released result grid. Nine RoboMME task families × nine released memory systems × five history conditions × 50 episodes = 18,450 rollouts.
• A measurable failure mode. Perceptual memory delivers large near-session gains (FrameSamp-Modul: +27 pp at k0) that erode steadily under interference (−26 pp by k7), while recurrent variants stay near baseline. Systems that look strong at short range can collapse across sessions.

The Benchmark

Every query episode is run through the same policy interface, varying only the external history buffer. The relevant lesson, a prior demonstration of the task, always appears first; the history conditions differ in how many unrelated sessions are inserted before the query reads it.

Why these nine tasks. The protocol requires the task-relevant information to live in a self-contained prior session that can be lifted into the history buffer. We therefore use the nine RoboMME tasks that deliver this information as a prior demonstration video, distinct from execution; the remaining tasks embed their cue inside a single episode and have no separable prior session to place under interference.

Why unrelated distractors. Distractor sessions are drawn from different task families, so they cannot inject contradictory same-family facts. The benchmark therefore measures interference from genuinely unrelated robot experience, the deployment-like failure mode, rather than constructed wrong priors.

Tasks

Nine task families from RoboMME, each presenting a prior demonstration the policy must use during a separate execution episode. They span three memory types.

Task definitions and assets are from RoboMME (Dai et al., 2026).

Memory Systems

We run RoboMME's released memory-augmented variants through the benchmark. Each pairs a memory representation (frames, tokens, or hidden states) with a mechanism for integrating that memory into the base π_0.5 policy: Context concatenates memory tokens with the input, Modulator conditions the policy via adaptive LayerNorm, and Expert routes memory through a dedicated expert with block-wise causal attention. The perceptual variants (FrameSamp, TokenDrop) sample or drop visual-history tokens; the recurrent ones (TTT, RMT) keep a compressed hidden state. The baseline is the π_0.5 backbone with no memory.

✓ evaluated: eight memory variants plus the π_0.5 baseline (nine systems).
– not evaluated: RMT and TTT-Modulator had no released checkpoints at evaluation time.

Results

1 Relevant memory helps — when it is near

At k0, with only the relevant session in history, perceptual memory lifts success well above each system's own no-history floor: FrameSamp-Modul from 18.2% to 45.3% (+27.1 pp) and TokenDrop-Modul from 17.1% to 35.3% (+18.2 pp), against the 17.3% π_0.5 baseline.

Full grid

Success rates across all nine families.

Reproducibility & Artifacts

Everything needed to reproduce the grid and build on the protocol is released.

• Paper — the full method and protocol on arXiv.
• GitHub repository — code, analysis scripts, and figures.
• Canonical rollout CSV — all 18,450 rollouts, the source of truth.
• Main success table — per-system rates with confidence intervals.
• Results summary — headline numbers and effects.

Citation

@misc{rathi2026robommeinterference,
  title         = {Benchmarking Robot Memory Under Interference},
  author        = {Rathi, Soumil},
  year          = {2026},
  eprint        = {2606.22338},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2606.22338}
}

This benchmark builds on RoboMME; please cite it as well.

@article{dai2026robomme,
  title  = {RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies},
  author = {Dai, Yinpei and Fu, Hongze and Lee, Jayjun and Liu, Yuejiang and
            Zhang, Haoran and Yang, Jianing and Finn, Chelsea and Fazeli, Nima and Chai, Joyce},
  journal = {arXiv preprint arXiv:2603.04639},
  year   = {2026}
}