Agents that must learn while acting

ScrambleToolBench: infer the interface

ScrambleToolBench is a deterministic Python terminal environment with 28 file-system, network and data-processing tools, plus three visible control commands. Each episode samples five sequential tasks from 20 procedural templates; targets such as paths, ports and usernames are regenerated for the episode. Tool names, argument names, output fields and success tokens are replaced by random identifiers. A malformed call reveals only the required keys and their types, so the agent has to infer a tool by executing it rather than reading its name.

The evaluation runs 20 episodes per model, with 100 inference steps for each task. It tests a static scrambled condition and three changes, separately and together: 25% of tool identifiers are reassigned between tasks, valid calls time out with probability 0.15, and selected procedures must finish within ten actions after a trigger. The reported metrics are the fraction of episodes in which all five tasks are solved and the mean number of tasks solved.

Across 15 models, mean completion was 0.93 in the named-tool control, 0.32 after scrambling, and 0.03 when drift, timeouts and execution windows were combined. Mean tasks solved fell from 4.88 to 2.31 and then 0.84. The failure is not only initial discovery: under drift, strong models repeatedly probe tools instead of tracing the small permutation implied by their existing map. In retained episodes, their recovery cost on Tasks 2–5 was roughly 5.6–8.9 times the cycle-tracing reference.

The memory baseline keeps two JSON stores: reusable task recipes and a tool dictionary with inferred arguments and confidence. On four paired models in the combined condition, memory added 0.09 to completion and 0.59 tasks solved on average; Gemini 3.1 Pro moved from 0.20/2.35 to 0.50/3.60. That gain is not automatic correction. Memory reduced Qwen 3.6 27B’s stale calls from 3.67 to 2.05 per task, but increased Claude Sonnet 4.5’s from 2.38 to 2.76 when an obsolete mapping remained in the store.

MNIST-PRO: build the scene from glimpses

MNIST-PRO turns digit recognition into an active-perception task. Level 1 upscales one binarized MNIST digit to 224 × 224 pixels; Level 2 concatenates two digits before processing, producing a 224 × 448 image. The agent begins at a random 64 × 64 window, moves it by 32 pixels, and must predict within 36 steps for Level 1 or 78 for Level 2. Each of the ten models is tested on 100 episodes per level; Level 1 is balanced across digits and Level 2 samples both digits independently.

The main comparisons retain the full image history or replace it with a one-step view plus one of three states: free-form text, a coordinate grid, or a programmatically aligned visual canvas. Fully visible control images are usually easy, but partial observation is not. In the natural multi-turn setting, Gemini 3.7 Flash scored 98%/97% on fully visible Level 1/2 images and 75%/47% when it had to navigate the glimpses.

Trajectory audits show where the loss occurs. Among 1,600 Level 2 runs with textual or grid state, 323 predictions contained only one digit; 238 of those runs never exposed a pixel of the other digit, and 266 exposed less than a quarter of it before stopping after about 15 of 78 available steps. More coverage was still insufficient: exact two-digit accuracy was only 26% when agents had seen more than 75% of both digits. With Gemini 3.7 Flash, the grid reduced repeated moves from 45.1% to 13.6%, but missed one digit in 23 of 100 episodes—almost the same as text at 24 of 100.

A fixed-trajectory replay separates exploration from state construction. The same textual-state glimpses that gave Claude 5 Opus 41%/13% on Levels 1/2 produced 76%/67% after the environment assembled them into a coordinate-aligned canvas; Claude 5 Fable rose from 38%/18% to 81%/69%. A tool-enabled Gemini 3.7 Flash agent reached 88%/63%, but persistent procedural notes did not improve it: accuracy became 85%/62%, while Level 2 sensing grew from 41.00 to 52.42 steps. Adding correctness feedback yielded 86%/63% and 63.67 steps.

Both benchmarks are controlled diagnostics rather than replicas of production software or embodied vision. ScrambleToolBench uses a simulator and an intentionally worst-case naming scheme; MNIST-PRO uses clean two-dimensional digits. Their measurements isolate map discovery, revision, spatial integration and stopping, but do not by themselves establish performance in noisier real systems.