MemBodied: a technical walkthrough 0:00 Why a robot needs episode memory A robot can arrive at the same visible scene through different histories. If it must return a block to its original position, the current image may no longer reveal where that position was. MemBodied addresses this problem by carrying a compact episode state between policy calls. This walkthrough follows the actual architecture: an associative read, a fixed scene anchor, action generation, and a delayed write. 0:27 Two complementary memory pathways There are two distinct memory pathways. Each action-network layer has an associative matrix, S, with rank r by r. The experiments use rank one hundred and twenty-eight. These matrices change as interactions arrive. The episode anchor, A, instead stays fixed. At a new episode, the matrices return to learned initial states, and the anchor is rebuilt from the new first observation. The diagram shows one action-expert layer. 0:58 Build the first-scene anchor Start with the purple anchor pathway. The first camera observation is encoded into the frozen visual tokens used by the policy prefix. Each camera's sixteen by sixteen patch grid is average-pooled to four by four. The pooled tokens are concatenated across cameras to form A. This preserves a compact reference to the initial scene without retaining the raw image. A is not updated by subsequent interactions. 1:26 Use the anchor to condition actions At each call, state-conditioned pooling produces the current visual representation, z. It supplies the query for cross-attention over A, whose projected tokens supply keys and values. The anchor attention uses rank sixty-four. A projected attention result is combined with z and mapped to a conditioning vector, b. That vector is repeated over the action horizon and concatenated with each action embedding through the conditioner. The policy's visual prefix is not extended. 2:01 Retrieve an association Now follow the blue associative read. The state-token representation entering a layer is projected to a query. A hyperbolic tangent followed by L two normalization bounds and normalizes that query. Multiplying the existing matrix by the query retrieves a memory vector. This is an associative lookup in a fixed-size matrix, rather than a search through an ever-growing list of earlier observations. Each layer reads its own matrix. 2:31 Inject the retrieved memory The retrieved vector is projected back into the action expert's hidden space and scaled by alpha memory divided by rank. The main configuration uses two hundred and fifty-six divided by one hundred and twenty-eight, giving a multiplier of two. A sigmoid gate from the current state controls the contribution. This gated vector is added to a dedicated memory token before self-attention. The learned token is inserted anew at each call; the associative matrices carry information between calls. 3:04 Generate, execute, then observe Self-attention operates on the state, memory, and action tokens to generate an action chunk. The robot executes actions, and the environment supplies the next image and robot state. Only then can the memory write use the observed consequence. The action at the current call never receives that future image. At inference, the implementation caches layer states and the sampled action chunk from the final denoising step for the later write. 3:34 Construct the transition value The green write pathway begins with the observed transition. A query derived from the projected robot state attends to each camera's next-observation patch tokens. The camera encodings are averaged, then concatenated with a sum over the preceding action chunk to form y. The value projection maps y into memory space. Gradients from this value path stop at the visual patch tokens. The stored association can therefore represent both an action and its visual consequence. 4:06 Choose a key and two gates The write key comes from the cached state-token representation after the action-network layer, again using hyperbolic tangent and L two normalization. Two sigmoid gate vectors depend on that hidden state together with y. Alpha controls how much of each row of the old matrix is retained. Beta controls the strength of the new correction. These are learned, input-dependent decisions about how to update the existing state. 4:34 Correct the recalled association The update has two steps. First, the retention gate produces S bar from the previous matrix. Second, compare the new value, v, with the value that S bar already recalls for the write key, k. This difference is the prediction error for that association. The write gate scales the error, and its outer product with k corrects the matrix. Its entries change, while its dimensions stay fixed. The animated cells are a schematic, not measured activations. 5:07 Keep the write causally delayed This ordering is essential. Call t reads the previous state and generates an action. After the new observation arrives, the transition-conditioned update creates the next memory state. That state is read by a subsequent call. The initial-scene anchor remains unchanged throughout this process. Repeating this cycle allows information from earlier interactions to affect later control decisions without appending every observation to the policy context. 5:36 Learn memory through action prediction Training must preserve this sequence. Observation-action endpoints are sampled one action horizon apart. Image encoding can run in parallel, but memory propagates sequentially, with gradients passing through the full sampled sequence. Later action losses can therefore train earlier reads and writes. There is no separate memory-prediction loss. For LIBERO's five-step replanning, ten fixed memory slots are used in rotation, each revisited after fifty steps. That write summarizes the full predicted chunk, including actions beyond the five actually executed. 6:16 Why storage is O(1) For a fixed architecture, the recurrent state has a fixed number of rank-by-rank matrices, and the anchor has a fixed number of pooled tokens. With a fixed number of memory slots, neither grows with episode length. That is the O of one storage claim. It does not imply unlimited recall, constant total computation, or immunity to interference. The paper reports fifty percent mean success on five R M Bench tasks, versus six point four percent for stateless pi zero, while longer searches and motor failures remain challenging. 6:52 Memory that evolves within a fixed space MemBodied combines a changing associative state with a fixed reference to the first scene. Together, they let earlier interactions influence later actions within a constant episode-memory footprint. The paper provides the full equations, evaluation protocol, and limitations. Explore the code and demonstrations to see the method in practice. Narration uses a synthetic voice. Animated matrix values are schematic, not recorded activations.