Evaluation protocol
- Simulator
- RoboCasa365 kitchens in MuJoCo
- Robot
- Franka Panda arm on a mobile base
- Tasks
- Ten tasks in three families: Search, Inspect and Test
- Instances
- 50 per task, 500 episodes per model
- Scenes
- 18 kitchen layouts. The test kitchens use styles, and some task setups, that never appear in the demonstrations
- Models
- Gemini 4 Argon, GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5 and Claude Fable 5.1 with medium thinking; Gemini 3.8 Flash with high thinking
- Costs
- Official list prices, added up over each model’s 500 episodes. 86–94% of input tokens come from the prompt cache
- Time
- The real time each episode took, including model responses, retries and simulation
- Decisions
- Up to 200 per episode, one tool call each
- Commands
- Each runs for up to 200 ticks; one tick is 0.05 s of simulated time
- Memory
- Each episode starts a fresh conversation. All text is kept, but only the images from the two latest observations
- Episode end
- The first SUBMIT press, 200 decisions, a stop call, or a reply without a tool call
Agent interface
Observations
Before each decision, the model gets three 512 × 512 images, from the left and right scene cameras and the wrist camera, along with the positions of the gripper and the base, the 13 joint positions and velocities, the time elapsed and the number of decisions left. Each observation is built from a fixed list of allowed fields, so no task specification, seed or hidden ground truth reaches the model.
Actions
The model answers with exactly one tool call and decides how long it runs. There are no tools that find or grab objects for it, so to see anything new, the robot has to move itself or move something.
| Tool | Arguments | Default ticks |
|---|---|---|
arm |
Gripper position in world metres, optional orientation as a quaternion, gripper command from −1 (open) to +1 (close) | 120 |
base |
Body-frame forward, left and yaw velocity, each from −0.5 to 0.5 | 20 |
wait |
Optional gripper command; holds the arm and the base | 20 |
stop |
None; abandons the episode, which then fails | – |
Feedback
After each call, the model is told whether the command was accepted, the current tick count, and one fixed error message if the call was invalid. It never sees a score or the state of the task. A reply with no tool call, several tool calls or invalid arguments doesn’t move the robot, but it still uses up a decision.
System prompt for Puzzle Box
You control a simulated mobile manipulation robot. Use only the provided camera images, robot proprioception and action/budget feedback. Determine what the goal requires from the scene. Choose exactly one arm, base, wait or stop tool per response. There are no semantic object-location or grasp tools. Images are upright RGB from three fixed cameras; new views require physical robot or object motion. The arm tool controls the right gripper site in world metres, with optional world orientation as quaternion xyzw. World z is up. Gripper -1 opens, +1 closes and 0 keeps its current setpoint. The base tool takes normalized body-frame forward, left and yaw velocity commands; positive yaw turns left. The arm stays relative to the base during base motion. The torso is held. A tick is 0.05 seconds. Commands consume up to 200 ticks each, ending early on the first physical SUBMIT button press. Check observed outcomes: commanded targets are not guaranteed to be reached. Physically press SUBMIT to commit the result; the first press freezes the score and ends the episode immediately. Wrong submission, timeout without submission, or calling stop without submission fails. The stop tool abandons the episode without advancing time. Invalid or multiple tool calls perform no physics and still consume this model response. No intermediate success is awarded. Goal: Open the wooden puzzle box on the counter, take out the tangerine inside, and stand it inside the tray. The lid slides sideways but is locked by sliding bolts with round black handles: a part moves only when nothing blocks it, so work out the order. Any order that works is fine. Once the tangerine is inside the tray and you have let go of everything, press the red SUBMIT button. Your first submission ends the episode; a wrong submission or no submission fails.
Scoring
Success
An episode succeeds only if everything the goal asks for is true when the robot first presses SUBMIT. Pressing the button freezes the scene. A wrong press, a stop call and running out of decisions all count as failures, and there is no partial success.
Task progress
Progress gives partial credit for how much of the goal the frozen scene shows, in the same stages as the success condition. A stage only counts once things are at rest: the object released, supported and, where it matters, untouched. means the goal was reached. Success also needs a valid press and, in tasks that ask for it, a tidy result. Those tasks take off for an untidy result, and is kept within .
Aggregation
We compute the success rate and average progress of each task over its 50 episodes, then average the ten tasks equally. All results use the scoring rules of 3 October 2026.
Decision budget
One decision is one accepted reply from the model. Invalid actions and the final SUBMIT count as decisions; network retries don’t. An episode runs out of decisions if it reaches 200 without a SUBMIT press or a stop.