MemBodied. Recurrent associative memory for vision-language-action models.
Vision-language-action models turn observations and instructions into actions. In a multi-step job, the next action may depend on evidence that has already disappeared.
Here, MemBodied recovers from a failed grasp and continues folding. The robot retries, completes the remaining folds, and sets the garment aside.
Imagine returning a tool to its original tray after use. The current view can be identical, but the correct destination depends on where the tool began.
Or an inspection signal disappears after a failed test. Memory tells the robot which attempt failed. These tasks are non-Markovian from the current observation: the right action depends on history.
MemBodied combines two memory pathways. A fixed anchor preserves the starting scene. Associative matrices record subsequent interactions.
The current policy state queries the associative memory. A gated readout enters a dedicated memory token, where the action expert can attend to it.
The current observation also queries the fixed scene anchor. This separate pathway conditions the action tokens on a compact reference to the beginning.
After the action is executed, the next observation reveals its outcome. MemBodied pairs the action chunk with that observed consequence.
The write links the policy state to the action and its observed outcome. Learned gates retain existing state and control a correction to the stored association.
First, read memory and generate the action. Execute it, observe the outcome, then write back. All pathways learn through the policy's action objective.
Across five RMBench tasks, MemBodied achieves seven point eight one times the stateless success rate, and two point nine eight times vanilla recurrent memory, the strongest non-video stateful baseline in this evaluation.
MemBodied also complements video-history memory. Adding it to NativeMEM raises mean task success from thirty-eight point four to forty-five point two percent. Standalone MemBodied reaches fifty percent.
On three physical robot tasks, MemBodied achieves eight times the baseline’s mean success rate: sixteen successful trials out of sixty, compared with two.
On the same A100, average policy-cycle time falls from about one point six seconds with NativeMEM, to one hundred twenty-nine milliseconds with MemBodied.
That is ninety-one point nine percent lower latency and nine point five percent less peak GPU memory. Remember actions and outcomes. Read them into the next decision.
Narration uses a synthetic voice. Robot footage shows selected demonstrations; aggregate results are reported separately.
When earlier evidence leaves the frame, a robot must remember. Compare MemBodied and stateless π₀ on five history-dependent tasks.
π₀-StatelessCurrent observation only
Task success2.0%
Clip complete
MemBodiedCurrent observation + memory
Task success92.0%
Clip complete
M(1) / OBJECT RECALL
Which block was here first?
After the button press, both mats contain identical blocks. Only the initial layout tells the robot which block to return to the gap.
ACROSS THE ANNOTATED ROLLOUTS
50/50 correct block selections with MemBodied, compared with 20/40 for stateless π₀ when it moves a block.
The stateless policy moves no block in 10 further rollouts. Correct selection and complete task success are distinct measures.
Paired examples use the same seed (100000). Success rates are measured over 50 rollouts per policy and task, not estimated from these selected clips. Playback uses a shared elapsed-time clock; shorter clips hold their final frame.
REARRANGE BLOCKS
Remembering which block to move
Bars show the annotated failure rate with marginal 95% Wilson intervals. P-values are Holm-corrected.
Associative matrices retain evolving interactions. A fixed anchor preserves the initial scene. Their readouts inform action generation while the episode memory stays the same size.
PERSISTENT STATEAssociative matrices
Queried by the current state.
→
CONTEXT FOR THE POLICYMemory token
Retrieved content enters self-attention.
→
CONTROLNext action chunk
Conditioned on the current input and memory.
↳
After execution: the action and its observed consequence update the matrices through a gated delta rule.
The matrices persist across calls. The contextual token is reintroduced at each call. The episode anchor stays fixed.
NARRATED TECHNICAL WALKTHROUGH / 7:23
Inside the architecture.
Follow the original diagram through the associative read, scene anchor, action generation and delayed memory update.
Chapter: Why a robot needs episode memoryPress play for narration · 14 chapters
A robot can arrive at the same visible scene through different histories. If it must return a block to its original position, the current image may no longer reveal where that position was. MemBodied addresses this problem by carrying a compact episode state between policy calls. This walkthrough follows the actual architecture: an associative read, a fixed scene anchor, action generation, and a delayed write.
Two complementary memory pathways
There are two distinct memory pathways. Each action-network layer has an associative matrix, S, with rank r by r. The experiments use rank one hundred and twenty-eight. These matrices change as interactions arrive. The episode anchor, A, instead stays fixed. At a new episode, the matrices return to learned initial states, and the anchor is rebuilt from the new first observation. The diagram shows one action-expert layer.
Build the first-scene anchor
Start with the purple anchor pathway. The first camera observation is encoded into the frozen visual tokens used by the policy prefix. Each camera's sixteen by sixteen patch grid is average-pooled to four by four. The pooled tokens are concatenated across cameras to form A. This preserves a compact reference to the initial scene without retaining the raw image. A is not updated by subsequent interactions.
Use the anchor to condition actions
At each call, state-conditioned pooling produces the current visual representation, z. It supplies the query for cross-attention over A, whose projected tokens supply keys and values. The anchor attention uses rank sixty-four. A projected attention result is combined with z and mapped to a conditioning vector, b. That vector is repeated over the action horizon and concatenated with each action embedding through the conditioner. The policy's visual prefix is not extended.
Retrieve an association
Now follow the blue associative read. The state-token representation entering a layer is projected to a query. A hyperbolic tangent followed by L two normalization bounds and normalizes that query. Multiplying the existing matrix by the query retrieves a memory vector. This is an associative lookup in a fixed-size matrix, rather than a search through an ever-growing list of earlier observations. Each layer reads its own matrix.
Inject the retrieved memory
The retrieved vector is projected back into the action expert's hidden space and scaled by alpha memory divided by rank. The main configuration uses two hundred and fifty-six divided by one hundred and twenty-eight, giving a multiplier of two. A sigmoid gate from the current state controls the contribution. This gated vector is added to a dedicated memory token before self-attention. The learned token is inserted anew at each call; the associative matrices carry information between calls.
Generate, execute, then observe
Self-attention operates on the state, memory, and action tokens to generate an action chunk. The robot executes actions, and the environment supplies the next image and robot state. Only then can the memory write use the observed consequence. The action at the current call never receives that future image. At inference, the implementation caches layer states and the sampled action chunk from the final denoising step for the later write.
Construct the transition value
The green write pathway begins with the observed transition. A query derived from the projected robot state attends to each camera's next-observation patch tokens. The camera encodings are averaged, then concatenated with a sum over the preceding action chunk to form y. The value projection maps y into memory space. Gradients from this value path stop at the visual patch tokens. The stored association can therefore represent both an action and its visual consequence.
Choose a key and two gates
The write key comes from the cached state-token representation after the action-network layer, again using hyperbolic tangent and L two normalization. Two sigmoid gate vectors depend on that hidden state together with y. Alpha controls how much of each row of the old matrix is retained. Beta controls the strength of the new correction. These are learned, input-dependent decisions about how to update the existing state.
Correct the recalled association
The update has two steps. First, the retention gate produces S bar from the previous matrix. Second, compare the new value, v, with the value that S bar already recalls for the write key, k. This difference is the prediction error for that association. The write gate scales the error, and its outer product with k corrects the matrix. Its entries change, while its dimensions stay fixed. The animated cells are a schematic, not measured activations.
Keep the write causally delayed
This ordering is essential. Call t reads the previous state and generates an action. After the new observation arrives, the transition-conditioned update creates the next memory state. That state is read by a subsequent call. The initial-scene anchor remains unchanged throughout this process. Repeating this cycle allows information from earlier interactions to affect later control decisions without appending every observation to the policy context.
Learn memory through action prediction
Training must preserve this sequence. Observation-action endpoints are sampled one action horizon apart. Image encoding can run in parallel, but memory propagates sequentially, with gradients passing through the full sampled sequence. Later action losses can therefore train earlier reads and writes. There is no separate memory-prediction loss. For LIBERO's five-step replanning, ten fixed memory slots are used in rotation, each revisited after fifty steps. That write summarizes the full predicted chunk, including actions beyond the five actually executed.
Why storage is O(1)
For a fixed architecture, the recurrent state has a fixed number of rank-by-rank matrices, and the anchor has a fixed number of pooled tokens. With a fixed number of memory slots, neither grows with episode length. That is the O of one storage claim. It does not imply unlimited recall, constant total computation, or immunity to interference. The paper reports fifty percent mean success on five R M Bench tasks, versus six point four percent for stateless pi zero, while longer searches and motor failures remain challenging.
Memory that evolves within a fixed space
MemBodied combines a changing associative state with a fixed reference to the first scene. Together, they let earlier interactions influence later actions within a constant episode-memory footprint. The paper provides the full equations, evaluation protocol, and limitations. Explore the code and demonstrations to see the method in practice.
Narration uses a synthetic voice. Animated matrix values illustrate the update; they are not recorded model activations.
EPISODIC STORAGE
More experience. The same memory footprint.
MemBodied updates a fixed-size state as the episode unfolds.
Relative storage
Episode startMore policy calls →
O(T)Full observation history
O(1)MemBodied episodic state
Fixed capacity. Changing contents. The matrices update while their size stays fixed. The episode anchor stays fixed within the episode.
Scaling schematic at a fixed architecture, memory rank and camera configuration; not measured GPU allocation. Bounded frame stacking and μVLA can also use O(1) storage. MemBodied’s contribution is its memory pathways and read/write formulation.
01
Read before acting
Layer-wise queries retrieve information from associative matrices. The readout enters a dedicated memory token before self-attention.
02
Keep an initial-scene anchor
A compact, pooled reference to the first observation stays fixed. The current scene queries this anchor to condition action embeddings.
03
Write after the outcome
The gated delta rule retains part of the existing state and corrects the association for the current key using the action and its observed consequence.
STEP THROUGH A POLICY CALL
Changing contents. Constant size.
EPISODE ANCHOR
Initial scene · fixed
RECURRENT STATE · M₀0 interactions written
Read the existing state.
The current observation queries the associative matrices. Their readout conditions the next action.
O(1) episodic storage
Schematic cells, not measured activations. The model keeps one 128 × 128 matrix per action-network layer, plus the fixed anchor. Storage stays constant with episode length.
THE MEMORY UPDATE
Retain the state. Correct the association.
The gated delta rule updates each layer’s existing matrix after observing an action’s consequence.
Equivalent to the paper’s update, with layer and batch indices omitted. Diag turns a gate vector into a diagonal matrix.
St, S̄t
Updated memory and retained memory.
αt, βt
Learned retention and write gates.
kt
Normalised key from the action-network layer state.
vt
Value combining the action and its observed visual consequence.
Each layer keeps the same r × r matrix (r = 128 in the experiments). New interactions change its contents, keeping storage O(1) in episode length. The fixed initial-scene anchor is stored separately.
LEARNED THROUGH ACTION
One training objective.
Later action losses train earlier memory reads and writes through the sampled sequence. MemBodied learns its memory parameters with the policy’s native action objective, without a separate memory-prediction loss.
Explore the full architecture +The associative matrices carry temporal information; the contextual token is reintroduced at each call. At episode boundaries, matrices return to their learned initial state and the anchor is rebuilt. Method details
03 / EXPERIMENTAL EVIDENCE
Memory changes the next decision.
Evaluated on memory-dependent simulation, general manipulation, and physical robots. Results cover task success, inference latency and GPU memory allocation.
Mean success on five RMBench tasks
SUCCESS (%) ↑
0255075100
NativeMEM’s video-history encoding is evaluated in the same setup, alone and combined with MemBodied. No method is strongest on every task.
Across the five memory tasks.
Select a task to watch its paired demonstration.
Exact task results +
RMBench task success over 50 rollouts per policy
Task
π₀-Stateless
MemBodied
Gain
Success over 50 rollouts per task and policy. M(1) requires recall from one past event; M(n) requires information from multiple events. Gains are percentage points.
Inspect task-level results +
BEYOND VIDEO HISTORY
A stronger memory. Even alongside video.
NativeMEM compresses past camera observations into history tokens. Adding MemBodied supplies a recurrent state that updates after each interaction, improving mean success in the same evaluation setup.
+6.8pp
mean success when MemBodied is added to NativeMEM. 38.4% → 45.2% across five RMBench tasks.
Task success (%) · higher is better
Put Back Block · Rearrange Blocks · Swap Blocks · Battery Try · Block Ranking. 50 rollouts per task and policy. Every task has equal weight in the mean.
Standalone MemBodied reaches 50.0%, exceeding NativeMEM by 11.6 percentage points. The combined model is strongest on Put Back Block; no method leads every task.
NativeMEM encodes incoming video history and extends the policy context. MemBodied reads and updates fixed-size associative matrices. The measured comparison includes each method’s memory processing and action generation.
MODEL-SIDE LATENCY
91.9%
lower inference latency.
129.2 ms per cycle with MemBodied. 1,593.7 ms with NativeMEM.
MemBodied129.2 ms
NativeMEM1,593.7 ms
0 ms1,600 ms
Steady-state mean cycle time · lower is better. Pooled across four tasks after excluding warm-up queries.
PEAK GPU ALLOCATION
9.5%
less peak GPU memory.
18.61 GiB with MemBodied. 20.57 GiB with NativeMEM.
MemBodied18.61 GiB
NativeMEM20.57 GiB
0 GiB24 GiB
Peak allocation · lower is better. Maximum allocator usage across calls and tasks, relative to NativeMEM.
Measurement protocol and exact values +
Current paper’s inference efficiency measurements
Metric
NativeMEM
MemBodied
Reduction
Mean policy cycle
1,593.7 ms
129.2 ms
91.9%
Peak GPU allocation
20.57 GiB
18.61 GiB
9.5%
Matched hardware and tasks
One NVIDIA A100 with 80 GB VRAM per run, BF16 for both methods. Put Back Block, Rearrange Blocks, Swap Blocks and Battery Try; 50 episodes per task and policy, or 200 episodes each.
What the clock includes
Model-side computation only. MemBodied includes input preprocessing, current-image encoding, memory operations and 50-step action generation with 10 denoising steps. NativeMEM includes all video-history encoding since the previous query, queue updates and policy inference.
How cycles are aggregated
The first three policy queries of each task-policy run are excluded as warm-up. The arithmetic mean pools 2,548 NativeMEM cycles and 2,563 MemBodied cycles. Peak memory is the maximum JAX allocator usage across calls and tasks.
Which models are compared
Standalone MemBodied and NativeMEM. The NativeMEM + MemBodied combination is evaluated for success above; it is not part of this latency comparison.
NVIDIA A100 80 GB · BF16 for both methods. Full-model GPU allocation and O(1) episodic-state scaling describe different quantities.
EVALUATION DETAILS
RMBench: memory-dependent manipulation +
Five tasks, 50 rollouts per task. The reported mean gives every task equal weight. Episodic state resets at the beginning of each rollout.
MemBodied trains for 10,000 steps with batch size 8, a 50-step action horizon and associative rank 128. Training sequences contain 8, 14 or 18 endpoints, depending on the task. The main π₀ comparisons share the backbone and adaptation recipe. Matching optimiser steps does not imply equal compute cost; NativeMEM uses a separate staged training procedure.
LIBERO: general manipulation +
Four suites, ten tasks per suite, 50 rollouts per task: 2,000 rollouts per checkpoint. Results use 30,000-step full fine-tuning and five-step replanning.
Ten recurrent slots are revisited in a fixed round-robin schedule to preserve a 50-step memory-update interval. Each write uses the new observation and the previously predicted 50-step action chunk, including predictions beyond the five executed steps. The number of slots stays fixed throughout an episode.
04 / BEYOND SIMULATION
Memory in the physical world.
8×
the π₀ baseline’s mean success rate on three real-robot tasks
π₀ baseline3.33%
2 / 60 successful trials
MemBodied26.67%
16 / 60 successful trials
0%100%
Put Back Block · Rearrange Blocks · Swap Blocks.
20 trials per task and policy. Absolute success remains limited; memory does not remove manipulation failures.
MemBodied
Put Back Block
π₀ baseline1/20 · 5%
MemBodied7/20 · 35%
0%100%
Rearrange Blocks
π₀ baseline1/20 · 5%
MemBodied7/20 · 35%
0%100%
Swap Blocks
π₀ baseline0/20 · 0%
MemBodied2/20 · 10%
0%100%
Charts show complete-task success over 20 trials per task and policy; all use a shared 0–100% scale. Videos are selected physical-robot demonstrations. Use the policy buttons to view the supplied baseline clips for Rearrange Blocks and Swap Blocks.
Robot platform, training data and trial protocol +
Two AgileX PiPER arms, an Orbbec Gemini 336L top camera and two Intel RealSense D405 wrist cameras. The robot state and action each have 14 values: six joints and one gripper value per arm.
Fifty demonstrations were collected per task at 30 Hz. After cleaning, training retained 41 Put Back Block, 50 Rearrange Blocks and 49 Swap Blocks trajectories. Policies were trained separately for each task.
Each policy is evaluated on 20 physical rollouts per task. Success requires the complete instructed task; partial completion receives no credit. The associative state and initial-scene anchor reset before every rollout.
05 / DESIGN INSIGHTS
How memory becomes action.
The analyses examine how the policy receives memory, which information it retains, and how much capacity it needs.
01 / THE READOUT
Give the policy memory as context.
A dedicated token brings the retrieved association into self-attention. This is the strongest of the three anchor-free memory designs evaluated.
All three use associative memory. Attention steering alters attention representations; the hierarchical variant adds a recurrent cell and changes the state update. These are comparisons of memory designs, not a readout-only ablation.
02 / THE INITIAL SCENE
Keep a reference that never gets rewritten.
A fixed visual anchor complements the changing associative state. Adding it raises mean success from 37.6% to 50.0% and improves four of five tasks.
3.5×
the anchor-free success rate on Swap Blocks
16% → 56% with the initial-scene anchor. The largest gain among the five tasks.
Block Ranking falls from 32% to 22%. The initial scene is not equally useful for every task, particularly when the policy must track evolving progress.
Adding the episode anchor
Task success (%)
Without anchorWith anchor
Without anchor37.6%
With anchor50.0%
0%100%
Same memory-token model, without and with the fixed anchor. Five RMBench tasks, 50 rollouts per task and model.
03 / THE WRITE VALUE
Remember the action and what followed.
Memory stores an interaction: an action summary paired with its observed visual consequence. Combining both gives the highest success on all three evaluated tasks.
1.09×
the mean success rate of vision-only writes
Vision + action: 46.7%. Vision only: 42.7%.
On Battery Try, removing action information reduces success from 30% to 24%; removing vision reduces it to 22%.
What gets written into memory
Three-task mean success (%)
Anchor-free memory-token model. Put Back Block, Rearrange Blocks and Battery Try; only the write content changes.
See each task +
04 / THE CAPACITY
More capacity. Higher success.
Increasing the associative rank improves both evaluated tasks. The memory size is chosen once; it stays fixed as the episode grows.
2.04×
mean task success
55% vs 27%Rank 128 vs 32
Success at each memory rank
Anchor-free memory-token model
100%50%0
27%
Rank 321,024 entries / layer
42%
Rank 644,096 entries / layer
55%
Rank 12816,384 entries / layer
Fixed through time. Chosen by design. Each layer stores an r × r matrix. Doubling r quadruples its entries. At any fixed rank, episodic storage remains O(1) in episode length.
Two tasks, equally weighted. Put Back Block and Rearrange Blocks. The multiplier compares task success at rank 128 with rank 32. The mean weights both tasks equally.
THE PAPER
Recurrent Associative Memory for Vision-Language-Action Models
Method, experimental protocol, task-level results, ablations, and qualitative analysis.
Citation
@misc{pala2026membodied,
title = {{MemBodied}: Recurrent Associative Memory for Vision-Language-Action Models},
author = {Pala, Tej Deep and Majumder, Navonil and Goh, Bryce and Yee, Raphael and
Yang, Jianfei and Chen, Liming and Poria, Soujanya},
year = {2026},
eprint = {2609.28256},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
doi = {10.48550/arXiv.2609.28256},
url = {https://arxiv.org/abs/2609.28256}
}
