Analysis

Where the failures come from, how episodes end, and what each episode costs. For GPT-6 Astra, Claude Opus 5.5 and GPT-6.1 Sol, we test the execution skills the tasks are built from in isolation, with the hidden information given, and trace every failure to its first unrepaired error.

Skills in isolation
72–81%
of isolated skill tests succeed when the oracle plan is given
Execution failures
18–21%
of failures: the right action was chosen but failed, such as a missed grasp or a drop
Missing evidence
43–46%
of failures: the model never collected the evidence it needed
Side effects
7–13%
of failures: the model disturbed the scene and never repaired it

Both analyses cover GPT-6 Astra, Claude Opus 5.5 and GPT-6.1 Sol, each with medium thinking.

Execution skills in isolation

We tested each skill the tasks are built from on its own. The hidden information is given in the instruction, or the scene holds a single target. Each test starts in front of the target and ends when the goal holds, or after 50 decisions.

Success by skill

Success rate (%) over 20 scenes per skill, the same scenes for every model. Higher is better.

Pick and place (90–95%) and drive and pick (90–100%) are reliable. Opening a drawer or cabinet (30–60%) and sliding the shim under the short leg (35–60%) are the weakest. In those scenes the gripper touched the handle in 75–85% and the shim came within 3 cm of the leg in 90–100%, so the failures happen in the final, precise motion.

Failure attribution

Every unmet requirement of a failed episode, such as a target missing from the tray or a cube in the wrong bin, is traced to the first error that was never repaired. Deterministic rules on the recorded camera images and simulator state assign it to one of four families.

Missing evidence

The model never collected the evidence that decides the requirement, and not because an attempt to collect it failed: a hiding place never opened, a cube face never turned to a camera, a parcel never weighed.
Wrong decision

The model collected the deciding evidence, visible in at least one of its camera images, but acted against it or never acted on it: a misread label, a stamp printed in the wrong cell after a test print, a target seen but never fetched.
Side effect

The action succeeded but disturbed something the goal depends on, and the damage was never repaired: balls spilled from a tipped mug, the test ball rolled off the stand, a stand knocked over.
Execution failure

The decision was right but the physical action failed: a missed grasp, a dropped object, a placement off its target, a drawer not pulled out far enough.

Failures by family

Share of failures (%), each failed episode weighted equally: 384, 431 and 439 failed episodes with 931, 1,143 and 1,182 unmet requirements.

Together, missing evidence and wrong decisions account for 66–74% of failures. The hatched parts are failures in episodes that ran out of decisions (8%, 1% and 19%). For GPT-6 Astra and GPT-6.1 Sol, more than half of these are hiding places that were never opened.

Failure patterns

Most failures come from exploration that stops too early.

Exploration stops too early

Failures from missing evidence and wrong decisions, split by what became of the object (% of all failures).

In 45–53% of failures an object the task needed was never placed, and in 27–29% it stayed in a drawer, cabinet or box, or under a cover, that was never opened. Hiding places that never appeared in any camera image account for only 1–2% of failures; most had appeared five or more times.

Models also decide before they have the evidence, as in Painted Cubes and Odd Parcel. When objects start out of view or under a cover, success drops by 8–10 points, and every extra failure is a case of missing evidence. Exploring can also disturb the scene, as when balls spill from a tipped mug. Learning by trial and error is hard too. With the Puzzle Box bolts covered, Claude Opus 5.5 falls from 94% to 37%.

How episodes end

An episode ends when the robot presses SUBMIT, when the model stops or replies without an action, or after 200 decisions.

Episode endings

Share of each model’s episodes (%). Stopping on its own includes a stop call and a reply with no action.

No model stopped because the task was done: in every episode a model ended on its own, the goal was still unmet.

Submission precision

Correct submissions out of all SUBMIT presses (%) · Higher is better

GPT-6 Astra and GPT-6.1 Sol press SUBMIT in 85% and 75% of their episodes, but only 27% and 16% of those submissions are correct.

Cost and effort

What one episode takes: money at list prices, tokens, wall-clock time and model decisions.

Cost per success

Total cost divided by episodes solved (USD) · Lower is better

Tokens per episode

Input plus output tokens, in millions · Lower is better

Time per episode

Wall-clock minutes, including model replies and simulation · Lower is better

Decisions per episode

Model replies per episode, at most 200 · Lower is better

Fine-tuned π₀.₅

We also fine-tuned π₀.₅, a vision-language-action model, on 1,000 of our demonstrations, and tested it in kitchens and setups it had never seen.

π₀.₅ on each task

Success rate and task progress (%). Higher is better.

π₀.₅ succeeds only on Puzzle Box, in 2% of episodes. In most other episodes it runs out of time, often stuck repeating the same moves.