RoboQuest
Generalist physical agents that search, inspect and test
RoboQuest is a benchmark of ten mobile manipulation tasks. In each one, information the robot needs is hidden at the start, such as where the targets are stored, how a box opens or which parcel is heavier, and the robot has to act to find it out. Of six frontier models, Gemini 4 Argon and GPT-6 Astra do best and are tied, solving 23.8% and 23.2% of episodes.
Search Room Find the complete set of hidden targets Task details
Leaderboard
Six frontier models act as robot policies through the same interface, each on the same 500 episodes: 50 per task.
- Updated
- 500 episodes per model
- Up to 200 decisions
- 3 camera views
- Ends at SUBMIT
- Methods and scoring
Success rate
Episodes solved (%) · Higher is better
Task progress
Part of the goal reached (%) · Higher is better
Cost per episode
List-price cost of one episode (USD) · Lower is better
Key findings
Full analysis-
23–24%
Gemini 4 Argon and GPT-6 Astra tie for the lead
They solve 119 and 116 of their 500 episodes, too close to tell apart. Claude Opus 5.5, GPT-6.1 Sol and Claude Fable 5.1 succeed in 11–14% of episodes and Gemini 3.8 Flash in 2.0%. A π₀.₅ fine-tuned on 1,000 of our demonstrations succeeds in 2% of Puzzle Box episodes and in none of the other tasks.
-
72–81%
Execution skills work in isolation
When the oracle plan is given, GPT-6 Astra, Claude Opus 5.5 and GPT-6.1 Sol succeed in 72–81% of isolated tests of eight execution skills, such as opening a drawer, flipping a cube or placing an object on a target. In the full tasks, only 18–21% of their failures are execution failures.
-
43–46%
Missing evidence is the largest cause
In 43–46% of failures, the model never collected the evidence it needed: a hiding place never opened, a cube face never turned to a camera, a parcel never weighed. Wrong decisions about evidence it did collect add another 23–31%.
-
94→37%
Learning by trial and error is hard
In Puzzle Box, a chain of sliding bolts locks the lid. With a cover over the bolts, the robot has to find their order by trying them. GPT-6 Astra still solves 96% of these puzzles, while Claude Opus 5.5 falls from 94% to 37%.
Success and cost
Cost at list prices, tokens and wall-clock time per episode, against success.
Gemini 4 Argon and GPT-6 Astra tie for the lead, at the same cost per episode
Success rate (%) against cost per episode at list prices (USD).
Results by task
Success rate and task progress of every model on each task, over 50 episodes per task.
Success rate by task
Episodes solved out of 50 (%). Higher is better.
How RoboQuest compares
In most manipulation benchmarks, everything a task depends on is visible from the start. Every RoboQuest task hides some of it, so the robot has to search, inspect or test to find it.
| Benchmark features | Evidence seeking | ||||||
|---|---|---|---|---|---|---|---|
| Benchmark | Long horizon | Hidden information | Memory | Mobile | Search | Inspect | Test |
| RLBenchVisuomotor skills | Partly | Partly | Not targeted | Not targeted | Not targeted | Not targeted | Not targeted |
| ManiSkill3Scalable manipulation | Partly | Not targeted | Not targeted | Not targeted | Not targeted | Not targeted | Not targeted |
| RoboTwin 2.0Bimanual manipulation | Partly | Not targeted | Not targeted | Not targeted | Not targeted | Partly | Not targeted |
| LIBEROLifelong skill transfer | Explicit target | Not targeted | Not targeted | Not targeted | Not targeted | Not targeted | Not targeted |
| CALVINLong-horizon manipulation | Explicit target | Partly | Not targeted | Not targeted | Not targeted | Not targeted | Not targeted |
| BEHAVIOR-1KHousehold activities | Explicit target | Partly | Partly | Explicit target | Explicit target | Not targeted | Not targeted |
| RoboCasa365Manipulation in kitchens | Explicit target | Not targeted | Explicit target | Explicit target | Partly | Partly | Partly |
| RMBenchMemory-dependent manipulation | Explicit target | Explicit target | Explicit target | Not targeted | Not targeted | Not targeted | Explicit target |
| RoboQuestEvidence-seeking manipulation | Explicit target | Explicit target | Explicit target | Explicit target | Explicit target | Explicit target | Explicit target |
Explicit targetPartlyNot targetedHidden information: the task depends on information that is not visible in the first observation
Dataset
Hugging Face- Scripted oracles know the hidden information but explore as if they did not: they open compartments nearest-first, turn each cube face by face and weigh parcels before choosing.
- Each 20 Hz step stores three camera views (two scene cameras and a wrist camera), the 16-dimensional robot state and the 12-dimensional action.
- Every frame has two language labels written from the robot’s point of view, a stage and a subtask, such as “try pulling the knob on the front of the box”.
- All 5,000 episodes succeed and are verified, and each task’s 500 episodes cover its workload levels in equal numbers.
- No demonstration shares an instance with the evaluation set, and the evaluation kitchen styles never appear in training. Released in LeRobot v2.1 format.
Interaction hours per task
Hours of interaction at 20 Hz in each task’s 500 demonstrations.
Citation
If RoboQuest is useful in your work, please cite our paper.
Liu Renhang, Navonil Majumder, Tej Deep Pala and Soujanya Poria. RoboQuest: Generalist Physical Agents that Search, Inspect and Test. arXiv preprint arXiv:2610.10388, 2026.
@article{liu2026roboquest,
title = {{RoboQuest}: Generalist Physical Agents that Search, Inspect and Test},
author = {Liu, Renhang and Majumder, Navonil and Pala, Tej Deep and Poria, Soujanya},
journal = {arXiv preprint arXiv:2610.10388},
year = {2026},
url = {https://arxiv.org/abs/2610.10388}
}