RoboQuest Paper GitHub

Tasks

Every task states a goal but hides information needed to reach it, such as where the targets are stored, what is written under a vessel or which parcel is heavier. The robot has to act to find it out.

The tasks test nine capabilities in four groups.

  • Evidence acquisitionDirected search, active perception
  • Evidence useEvidence integration, memory
  • Interactive inferenceAffordance discovery, experimental identification, physical causal inference
  • Action organizationConsequential action, long-horizon planning

Targeted directlySupporting

Search Room

Find every hidden item in the set.

Hidden
Where the targets are
Revealed by
Opening storage places until the set is complete

A set of items is hidden around a kitchen, some in the open and some behind doors and in drawers, mixed in with similar items that don’t belong. The robot has to gather the whole set on the tray and press SUBMIT. It isn’t told where the items are, how many storage places there are, or which ones are empty. So it has to choose where to look, learn from the empty places, remember what it has already checked, and stop once the set is complete.

Variation

Instances change the number and kind of items, the room layout, the storage units and the look-alike items.

Main parameters 4, 6 or 8 compartments; 2 or 3 targets

Capabilities
  • Directed search
  • Active perception
  • Evidence integration
  • Memory
  • Long-horizon planning

Locked Storage

Follow a chain of keys to reach the target.

Hidden
The target, behind locks
Revealed by
Finding the tokens that unlock each compartment

Each compartment opens only when its matching token sits on its reader. Some tokens are out of view at the start and others are locked inside other compartments, so one token leads to the next. The robot has to bring out the target item, and every token it finds changes where it can search next. If it closes a compartment with a token still inside, that token is locked away for good, so the robot has to plan what to do with each token as well as where to look.

Variation

Instances change the number of locked compartments, how the chain links them, where the tokens hide, and the dead ends.

Main parameters Chains of 1, 2 or 3 locks

Capabilities
  • Directed search
  • Evidence integration
  • Memory
  • Physical causal inference
  • Consequential action
  • Long-horizon planning

Blackout Search

Search a dark room, one lit spot at a time.

Hidden
Targets in the dark
Revealed by
Carrying a lamp and searching what it lights up

The room is dark. At the start the robot can see only a lamp, marked with a small glow, and the SUBMIT button. It has to pick up the lamp, carry it around and search by its light, since the wrist camera sees only what is lit. The robot has one gripper, so to take a target it must set the lamp down pointing at it, then work from memory in the dark.

Variation

Instances change the target, the room, where the lamp starts, whether there is a wall switch, and whether a timed light switches itself off.

Main parameters 4, 6 or 8 compartments; 2 or 3 targets

Capabilities
  • Directed search
  • Active perception
  • Evidence integration
  • Memory
  • Long-horizon planning

Inspect

The robot has to lift, turn or open objects to reveal properties it cannot see, such as a mark on a hidden face or a label underneath, and decide when it has seen enough.

Painted Cubes

Look at every face before deciding.

Hidden
Marks on faces not yet seen
Revealed by
Turning each cube to see every face

Each cube has marks painted on some of its faces, and some of those faces point down at the table or away from the cameras. The robot has to collect every cube that fits a rule, such as having exactly one painted face. A partial view can rule a cube out but can’t confirm it, so the robot has to decide how closely to look at each one.

Variation

Instances change the number of cubes, the rule, how the cubes start out, and whether the goal asks for one matching cube or all of them.

Main parameters 4, 6 or 8 cubes; some may start out of view or under a cover

Capabilities
  • Active perception
  • Evidence integration
  • Memory
  • Consequential action
  • Long-horizon planning

Marked Mugs

Read the label underneath without losing what’s inside.

Hidden
Labels under the vessels
Revealed by
Lifting or tipping each vessel to read its underside

Several identical mugs or bowls each hold a few loose balls, and a label on the bottom of each says where it belongs. The robot has to set every one upright in its place with its own balls still inside. Reading a label means lifting or tipping the vessel, which spills the balls. They stay nearby, and the robot has to put them back in the right vessel before it presses SUBMIT.

Variation

Instances change the number of vessels, what they hold, which label goes where, and how they start out.

Main parameters 3, 4 or 5 vessels; some may start out of view or under a cover

Capabilities
  • Active perception
  • Evidence integration
  • Memory
  • Consequential action
  • Long-horizon planning

Unfamiliar Containers

Work out how each box opens.

Hidden
How each container opens
Revealed by
Trying different ways to open each box

Several closed boxes each open in a different way: a hinged lid, a sliding lid, a drawer, a latch. Some don’t open at all. The items the robot needs are inside some of them. It doesn’t know which box holds what or how any of them open, so it has to find out by trying.

Variation

Instances change the number of boxes, the mix of opening mechanisms, where the items are, and how many decoys there are.

Main parameters 2, 3 or 4 items; some boxes may start out of view

Capabilities
  • Directed search
  • Memory
  • Affordance discovery
  • Experimental identification
  • Long-horizon planning

Test

The robot has to run small experiments, trying something and watching what happens, to work out a weight, a stamp pattern, an unlocking order or a slight tilt.

Stamp Composition

Learn what each stamp prints, then copy a pattern.

Hidden
Patterns and rotations of the stamps
Revealed by
Making test prints on a scrap sheet

Several stamps look alike from the outside, but each prints a different hidden pattern and starts at an unknown rotation. A reference card shows the design to make. The robot can make test prints on a scrap sheet, but anything printed on the final sheet is permanent and must match the reference exactly. So it has to test the stamps, read the prints, choose which stamps to use and at what angle, and then print the final design.

Variation

Instances change the number of stamps, their patterns, the target design, the starting rotations and the size of the test sheet.

Main parameters 2, 3 or 4 stamps; some may start out of view or under a cover

Capabilities
  • Evidence integration
  • Memory
  • Experimental identification
  • Consequential action
  • Long-horizon planning

Odd Parcel

Use a balance to find the parcel that’s different.

Hidden
Which parcel is odd
Revealed by
Comparing groups of parcels on a balance

The parcels all look the same, but one or two weigh differently. The only tool is a two-pan balance, and each pan holds several parcels. The robot has to put the odd parcel on the tray by itself. Weighing groups of parcels finds it in a few weighings; weighing them one at a time works too, but takes longer.

Variation

Instances change the number of parcels, whether the odd one is known to be heavier or could be lighter, how much each pan holds, and how the parcels look.

Main parameters 4, 6 or 8 parcels; some may start out of view or under a cover

Capabilities
  • Evidence integration
  • Memory
  • Experimental identification
  • Long-horizon planning

Puzzle Box

Find the order that unlocks the box.

Hidden
The order in which bolts block each other
Revealed by
Trying parts and watching which ones move

A box is held shut by a chain of parts, each blocking another until it is moved. Every part has a large handle and is easy to see, but which part blocks which is not. The robot has to take out the item inside and put it on the tray. It can only learn the order by trying parts and watching what moves, and some wrong moves jam another part until they are undone.

Variation

Instances change the number of parts, how they link together, decoy parts and trap moves.

Main parameters Chains of 4, 5 or 6 parts; the locks may be covered

Capabilities
  • Memory
  • Affordance discovery
  • Experimental identification
  • Physical causal inference
  • Consequential action
  • Long-horizon planning

Wobbly Stand

Use a ball to find a slight tilt, then fix it.

Hidden
Which legs are short, and by how much
Revealed by
Placing a ball and watching where it rolls

A stand has an uneven leg, so its top tilts by too little to see. A ball placed on top rolls toward the low side and gives the tilt away. Shims of different thicknesses lie nearby. The robot has to make the top level, so the ball stays where it is put, and leave everything released.

Variation

Instances change how many legs are short, how much the stand tilts, which shims are available, and whether it tilts in one direction or two.

Main parameters 1 or 2 short legs; sometimes several shims to choose from

Capabilities
  • Evidence integration
  • Memory
  • Experimental identification
  • Physical causal inference
  • Consequential action
  • Long-horizon planning