δ-mem + Σ-Mem

δ-mem: compress context into an online state

Once earlier text leaves a language model’s context window, the model cannot attend to that text directly. δ-mem writes earlier context into a learned associative memory attached to a frozen Transformer. Hidden states are projected into an eight-dimensional key, query and value space. Before the current item is written, an 8 × 8 state is read; its output produces low-rank corrections on the attention query and output branches. A gated delta rule then writes only the residual between the stored prediction and the current value, while a retention gate controls forgetting.

The paper tests three write schedules. Token-State Write updates on every token; Sequence-State Write averages each message or segment before one update; Multi-State Write maintains four parallel matrices and concatenates their readouts. Only these projections, gates and readout layers are trained. Training uses one epoch on 2,219 QASPER examples: up to 8,192 context tokens are written to memory while the explicit backbone sequence is capped at 512 tokens.

Evaluation covers IFEval, HotpotQA, GPQA-Diamond, MemoryAgentBench and LoCoMo on Qwen3-4B-Instruct, with BM25 retrieval, LLMLingua-2, MemoryBank, Context2LoRA, MemGen and an MLP memory as baselines. The best δ-mem schedule raised the aggregate score from 46.79 to 51.66. On individual memory-heavy tests, MemoryAgentBench rose from 29.54 to 38.85 and LoCoMo from 40.79 to 49.12; HotpotQA improved from 42.35/56.00 exact match/F1 to 49.41/63.66. The gain also appeared on Qwen3-8B (47.20 to 50.86) and SmolLM3-3B (26.08 to 36.96), although the best write schedule differed by backbone.

Removing the explicit context shows how much the small state leaves out. HotpotQA rises from 0.08/8.27 to 6.48/15.20 exact match/F1 and LoCoMo from 3.49 to 8.05, still well below the results with context available. The memory is therefore useful alongside the visible text, but cannot stand in for the original record. It also adds work at each decoding step, making generation slower than the unmodified model. The TSW and SSW adapters add 4.87 million trainable parameters (0.12% of Qwen3-4B); the four-state MSW version adds 19.47 million.

Σ-Mem: remember which peer is reliable

Remembering what was said is different from remembering who was right. In a group of agents, one peer may be good at coding and another at factual questions; their errors may also overlap. Σ-Mem records this task-dependent history. For each peer, it stores a symmetric competence matrix that is queried and updated along a learned direction for the current task. After the system predicts, an external correct/incorrect label adds or subtracts a bounded rank-one update and decays older evidence. A second matrix records whether peers tend to be correct together or in opposition.

At selection time, these states can steer a frozen central model’s yes/no assessment of each answer, route the question before answers are generated, or weight peers that return the same canonical answer.

Training uses 2,963 labelled events: 1,000 GSM8K questions, 1,000 SQuAD questions and 963 APPS programs. Each event contains answers from Gemma-3-4B-it, Phi-4-mini-instruct and Qwen2.5-Coder-7B-Instruct, plus a correctness label for every peer. Five Qwen-family central models are then tested under counterfactual shifts of 0%, 50%, 70% and 90%, with three-, four- and five-peer pools. Out-of-domain tests use PIQA, MMLU, OpenBookQA, SciQ, BBH and SuperGLUE; the extra peer pools introduce Llama-3.2-3B and BitCPM-CANN-3B, which are absent from training.

Under the strongest reliability shift, Σ-Mem raised Qwen3-0.6B selection accuracy from 46.22% to 71.10% and Qwen3.5-9B from 70.99% to 76.95%. It improved the base model in 27 of 30 out-of-domain model–benchmark pairs. On the pooled six-benchmark test, Qwen3-4B rose from 55.98% to 60.54%; direct memory routing scored 60.68% and memory-weighted voting 60.96%, compared with 59.12% for majority voting.

The same history that helps Σ-Mem choose a peer can mislead it when past reliability becomes ambiguous. In the 50% counterfactual stream, Qwen3.5-9B fell from 68.94% to 65.96%, with a similar pattern across larger peer pools. More revealed correctness labels improved the results, which makes the source of those labels an important part of the method. The experiments provide persistent peer identities, canonical answers and delayed ground truth; they leave open how the memory should recover when that feedback is itself unreliable.