Lab note
δ-mem: Giving Large Language Models Lightweight, Online, and Dynamically Evolving Memory
Long-running assistants and agents need to reuse earlier information without placing their entire history in every new prompt. δ-mem tests a small state that updates during inference.
Why memory is not just longer context
Putting more history into a prompt lets a model see the past, but increases attention cost and does not guarantee that the relevant information will be used. The memory problem also includes deciding what to retain, updating it as new information arrives and retrieving it for the current input.
Standard Transformer attention becomes more expensive as context grows. Long contexts can also disperse attention and make relevant details harder to recover.
δ-mem therefore keeps memory outside the visible prompt and updates it while the model is running.
How δ-mem works
δ-mem stores history in an Online State of Associative Memory beside a frozen full-attention backbone.
When a token or interaction segment arrives, the model projects it into a low-dimensional memory space and updates the state with a delta rule. The update is residual: it depends on the state’s prediction error rather than simply adding each new value.
The state changes online without retraining the backbone. In the main setting it is an 8 × 8 matrix.
During generation, the current input reads a signal from the previous state. That signal becomes low-rank corrections to the query and output sides of attention. Earlier information can therefore affect computation without reappearing as prompt tokens.
Comparison with other memory mechanisms
Textual memory methods such as RAG, MemoryBank and prompt compression return memory to the context as text. δ-mem instead writes directly to a numerical state.
Unlike methods with a separate retriever, reader and fusion path, its state directly produces attention corrections.
LoRA, prefix tuning and model editing usually produce parameter changes that are fixed after training or editing. δ-mem also uses a low-rank interface, but its corrections depend on the current memory state and can change from one history to another.
Three write schedules
The paper studies three write granularities.
- Token-State Write (TSW): the state is updated at every token. This captures fine-grained information changes, but it can be sensitive to formatting symbols, repeated expressions, and local noise.
- Sequence-State Write (SSW): the hidden states of a message or segment are averaged and written once per segment. This reduces redundant writes and makes state evolution smoother.
- Multi-State Write (MSW): multiple parallel memory states are maintained, allowing different states to carry different information types and reducing overwriting within a single state.
Reported results
On Qwen3-4B-Instruct, TSW raises the reported overall average from 46.79% for the frozen backbone to 51.66%. Its average is 6.76 points above Context2LoRA in the same table.
The gains are especially visible on memory-intensive evaluations. On MemoryAgentBench, δ-mem increases the average score from 29.54% to a maximum of 38.85%. On LoCoMo, MSW reaches the highest score of 49.12%. On HotpotQA, TSW improves EM/F1 from 42.35% / 56.00% to 49.41% / 63.66%.
The no-context test
The paper also removes the original historical context and asks the model to answer using only the compressed memory state.
In that setting, δ-mem scores above the no-context baseline on HotpotQA and LoCoMo. On HotpotQA, overall EM rises from 0.08% to 6.48% and F1 from 8.27% to 15.20%. The state recovers part, but not all, of the information lost when the context is removed.
Parameter and memory cost
In the main experiments, SSW and TSW add 4.87M trainable parameters, about 0.12% of the Qwen3-4B backbone. The multi-state MSW version uses 19.47M parameters, about 0.48%.
Reported memory use is close to the base model. Decoding is slightly slower because the state must be updated.
What the experiments show
The results show that a small recurrent state can retain useful information on the evaluated tasks and influence a frozen model through attention corrections. They do not establish unlimited memory or complete recovery: the no-context scores show that substantial information is still lost. Testing other backbones, longer histories and interactive tasks remains open work.