What should an AI system remember?

δ-mem: compress context into an online state

δ-mem attaches a learned associative memory to a frozen Transformer. Hidden states are projected into an eight-dimensional key, query and value space. Before the current item is written, an 8 × 8 state is read; its output produces low-rank corrections on the attention query and output branches. A gated delta rule then writes only the residual between the stored prediction and the current value, while a retention gate controls forgetting.

The paper tests three write schedules. Token-State Write updates on every token; Sequence-State Write averages each message or segment before one update; Multi-State Write maintains four parallel matrices and concatenates their readouts. Only these projections, gates and readout layers are trained. Training uses one epoch on 2,219 QASPER examples: up to 8,192 context tokens are written to memory while the explicit backbone sequence is capped at 512 tokens.

Evaluation covers IFEval, HotpotQA, GPQA-Diamond, MemoryAgentBench and LoCoMo on Qwen3-4B-Instruct, with BM25 retrieval, LLMLingua-2, MemoryBank, Context2LoRA, MemGen and an MLP memory as baselines. The best δ-mem schedule raised the aggregate score from 46.79 to 51.66. On individual memory-heavy tests, MemoryAgentBench rose from 29.54 to 38.85 and LoCoMo from 40.79 to 49.12; HotpotQA improved from 42.35/56.00 exact match/F1 to 49.41/63.66. The gain also appeared on Qwen3-8B (47.20 to 50.86) and SmolLM3-3B (26.08 to 36.96), although the best write schedule differed by backbone.

The 8 × 8 state is a lossy summary, not a substitute for the original record. With all explicit context removed, it lifted HotpotQA from 0.08/8.27 to only 6.48/15.20 exact match/F1 and LoCoMo from 3.49 to 8.05. It also decodes more slowly than the unmodified model because every token reads and updates the state. The TSW and SSW adapters add 4.87 million trainable parameters (0.12% of Qwen3-4B); the four-state MSW version adds 19.47 million.

Σ-Mem: remember which peer is reliable

Σ-Mem addresses a different decision. For each peer, it stores a symmetric competence matrix that is queried and updated along a learned direction for the current task. After the system predicts, an external correct/incorrect label adds or subtracts a bounded rank-one update and decays older evidence. A second matrix records whether peers tend to be correct together or in opposition. At selection time, these states can steer a frozen central model’s yes/no assessment of each answer, route the question before answers are generated, or weight peers that return the same canonical answer.

Training uses 2,963 labelled events: 1,000 GSM8K questions, 1,000 SQuAD questions and 963 APPS programs. Each event contains answers from Gemma-3-4B-it, Phi-4-mini-instruct and Qwen2.5-Coder-7B-Instruct, plus a correctness label for every peer. Five Qwen-family central models are then tested under counterfactual shifts of 0%, 50%, 70% and 90%, with three-, four- and five-peer pools. Out-of-domain tests use PIQA, MMLU, OpenBookQA, SciQ, BBH and SuperGLUE; the extra peer pools introduce Llama-3.2-3B and BitCPM-CANN-3B, which are absent from training.

Under the strongest reliability shift, Σ-Mem raised Qwen3-0.6B selection accuracy from 46.22% to 71.10% and Qwen3.5-9B from 70.99% to 76.95%. It improved the base model in 27 of 30 out-of-domain model–benchmark pairs. On the pooled six-benchmark test, Qwen3-4B rose from 55.98% to 60.54%; direct memory routing scored 60.68% and memory-weighted voting 60.96%, compared with 59.12% for majority voting.

Reliability memory can also amplify misleading history. In the ambiguous 50% counterfactual stream, Qwen3.5-9B fell from 68.94% to 65.96%, and the same pattern appeared across larger peer pools. The experiments assume persistent peer identities, canonical answers and delayed ground-truth labels; results improve as more labels are revealed. They do not show that the method can infer reliability without verification or recover when the feedback itself is wrong.