GE-Act 2.0

Learning from recordings with different kinds of labels

A robot demonstration usually pairs camera images with an instruction and a sequence of actions. A video of a person manipulating an object has no robot action labels. A failed robot attempt may have actions but no useful instruction or success annotation. All three recordings show something about how objects move, but a model trained only to imitate labelled actions cannot use them in the same way.

GE-Act 2.0 separates two jobs. A visual planner predicts how a scene could change after an instruction. An inverse dynamics model learns which robot actions connect the observed and future states. They can first learn from different recordings, then be connected and trained together.

The visual planner is pretrained on a 39,000-hour mixture, the inverse dynamics model on a 32,000-hour mixture, and the largest joint-training run uses 30,000 hours. These are overlapping, stage-specific mixtures—not 101,000 unique hours of data.

Predicting a future the robot can act on

The model first compresses each 256 × 384 camera frame into 24 visual tokens. Its control-oriented autoencoder, CoAE, learns both to reconstruct images and to align its features with frozen visual teachers. The aim is to retain the cues needed for an action despite this substantial compression.

The single-step visual planner predicts a sequence of future visual states in one MeanFlow pass. The inverse dynamics model reads those predictions alongside current observations and the robot’s own state to produce an action chunk. Near-term predictions support immediate movement; sparser predictions farther ahead provide context. After executing a short chunk, the robot observes the scene again and replans.

Generating the future in one differentiable pass also changes training. The action loss can reach the visual planner directly, so the two models can adapt to each other rather than treating a generated video as an unchangeable input. The visual generator and action model are initialized from scratch, but not every component is: CoAE transfers compatible autoencoder weights, and the system uses a frozen Qwen3.5-2B vision-language model. CoAE and the vision-language model remain frozen during joint training. Architecture and training details, sections 3.1–3.3.

Matching a predicted future to the recorded action

There is more than one way to complete an instruction. A generated future might show a perfectly reasonable movement that differs from the one in the demonstration. Pairing that future with the demonstration’s action would teach the action model an inconsistent mapping.

Knowledge-aligned selective optimization, or KASO, addresses this mismatch. At each training step, the planner generates four candidate futures. The current inverse dynamics model compares them with the recorded future in action space. KASO keeps the lowest-energy candidate, replays its generator noise with gradients enabled, and uses it for the generated-future action loss. The selection happens again at the next optimizer step; it is not a one-time filtering of a synthetic dataset.

A controlled G2-90D experiment starts each variant from the same pretrained components and a 300-hour connection-stage mixture. KASO and the baseline that retains pretraining losses both contact the instructed target in 95% of the four-object trials. Completed pickup differs: 37.5% with KASO, against 22.5% for that baseline. This is an alignment-stage ablation, not the final model’s overall success rate. Section 3.4.

One checkpoint, 100 real-robot tasks

The main evaluation covers 100 tasks in 20 skill groups on two robots, G1-OP and G2-90D. The tasks include picking, pouring, folding, stacking, insertion and tool use. Scenes, backgrounds, lighting and physical object instances are held out. Each task has ten trials per robot and training-data scale. A single checkpoint and fixed deployment settings are used across the suite, without task-specific fine-tuning.

Increasing the joint-training pool from 300 to 30,000 hours raises overall success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D.

At the largest scale, 19 of 20 G1-OP skill groups and 18 of 20 G2-90D groups improve over the smallest run. G2-90D makes up less than 2% of the joint-training data, so its improvement is consistent with transfer from the broader mixture. The experiment does not isolate that transfer, however. Nor does it isolate data volume at equal compute: each run trains until one epoch or its compute-budget limit. Real-robot protocol and results, section 5.2.

Finding the right object does not guarantee a successful grasp

A separate 295-rollout study distinguishes following the instruction from completing the movement. Across object identity, colour, direct position and shape, the robot contacts the instructed target in at least 90% of trials. Size is less reliable, and ordinal instructions remain difficult.

Across the entire study, target-following is 83.1%, while completed-task success is 72.9%. Some failures therefore occur after the robot has already selected the right target. The distinction matters: improving language grounding alone would not fix a grasp that slips or a placement that fails.

The project also shows revised commands after movement has begun and unusual destinations, such as putting a cup inside a shoebox. These are qualitative demonstrations, separate from the benchmark averages. Instruction-following study, section 5.3.

Tasks the model still cannot complete

The larger model still completes fewer than half the trials in the main suite. Zip tasks have no successes across 240 trials spanning both robots and all four data scales. With only ten trials for an individual task at each setting, small per-task differences also need caution.

GE-Act 2.0 is a low-level manipulation policy, not an explicit system for long-horizon planning, memory or self-correction. The report identifies larger-scale learning from egocentric video and integration with higher-level planning as next steps. Its simulation results use benchmark-specific adaptation and should not be read as the same zero-shot test used on the real robots. Limitations, sections 5.4 and 7.