From pixels to robot actions

The problem: turn one image and one instruction into control

A vision-language-action policy must locate the right object, understand a verbal goal and emit low-level motion commands quickly enough to control a robot. The hard cases are not only new objects. Distractors, spatial language, long action sequences and a change of robot embodiment can each break the mapping from pixels to movement.

NORA: a 3B autoregressive robot policy

NORA starts from Qwen2.5-VL-3B and trains on 970,000 real-robot demonstrations from Open X-Embodiment. It receives one 224 × 224 third-person image and a language instruction. FAST+ applies a discrete cosine transform and byte-pair encoding to continuous robot actions; NORA adds 2,048 action tokens to the model vocabulary and predicts them autoregressively. The base model replans after one action, while NORA-Long predicts a five-action horizon.

The paper reports about 8.3 GB of GPU memory at inference. Pretraining was still substantial: 1.1 million updates over roughly three weeks on eight H100 GPUs, or about 4,000 H100-hours.

What NORA was tested on

The real-robot study ran nine WidowX-250 tasks ten times each. The tasks covered unseen objects, spatial instructions and placing two objects. NORA averaged 56.7% success, compared with 40.0% for OpenVLA, 11.1% for SpatialVLA and 4.4% for RT-1. Its success was 90% for both “put carrot in pot” and “put banana in pot,” but only 30–40% on the three multi-object tasks.

On LIBERO, the paper evaluated the Spatial, Object, Goal and Long suites over 500 trials. Fine-tuned one-step NORA averaged 73.9%; using five-action chunks at fine-tuning raised this to 79.5%. NORA-Long, pretrained and fine-tuned with a five-action horizon, reached 87.9%, compared with 76.5% for fine-tuned OpenVLA and 78.1% for SpatialVLA.

The same chunking strategy did not transfer cleanly to the lower-frequency WidowX setup. Executing all five predicted actions without replanning caused collisions. Executing only the first action avoided them, but NORA-Long then scored 0% on the tested two-object task and 40% on “move the banana close to the pan.” Added distractor objects also reduced both NORA and OpenVLA performance.

NORA-1.5 keeps two ways to generate actions

NORA-1.5 extends NORA-Long with a roughly 400M-parameter flow-matching expert that reads intermediate keys and values from the vision-language backbone and generates five continuous actions. It does not discard the FAST+ decoder. Training jointly optimizes cross-entropy over FAST+ tokens and flow-matching loss, so the same model can be evaluated through the autoregressive -FAST branch or the continuous expert. The expert and backbone were jointly trained for another 150,000 updates on the same Open X-Embodiment mixture, using about 960 H100-hours.

Turn predicted outcomes into preferences

After supervised fine-tuning on a target robot, NORA-1.5 samples several candidate action chunks for the same image and instruction. A 1.3B V-JEPA2 action-conditioned world model predicts the next visual embedding; distance to an end-goal or immediate subgoal image becomes one reward. A second reward is the L1 distance from the recorded action. The combined score gives the action-distance term half the weight of the visual goal term, ranks candidates into winner-loser pairs, and Direct Preference Optimisation updates the policy from those pairs.

What changed after post-training

On SimplerEnv visual matching, fine-tuned NORA-1.5 averaged 77.9% success before preference training and 82.8% after DPO with the end-goal-plus-action reward. Under variant aggregation, the same comparison was 70.7% to 71.9%. On LIBERO, NORA-1.5 averaged 94.5% before DPO and 95.0% after it; the Object suite moved in the other direction, from 96.4% to 96.0%.

For cross-embodiment transfer, the team collected 1,000 teleoperated episodes on a Galaxea A1 arm, a robot absent from pretraining. Across nine tasks with ten trials each, the FAST branch of NORA-1.5 reached 71.11% full success, compared with 58.88% for NORA and 25.55% for π₀. On an expanded 13-task set with more distractors, subgoal-plus-action DPO raised full success from 56.92% to 70.00%, partial success from 72.30% to 83.84%, and reduced distractor grasps from 16% to 12%.

Where the evidence is still limited

The flow expert underperformed the autoregressive FAST branch on Galaxea’s smaller 50,000-frame adaptation set, even though it performed better in the larger simulation settings. World-model predictions can be noisy; distance from one recorded action can also penalize a different trajectory that would succeed. DPO gains were small and non-uniform on the already saturated LIBERO suites. These rewards rank candidate actions on the evaluated tasks; they are not a guarantee of safe behaviour outside them.