BURGERBenchmarking and Understanding Robotic Generalization via Embodied dRawing

BURGER Benchmarking and Understanding Robotic Generalization
via Embodied dRawing

1 Tsinghua University 2 Shanghai Qizhi Institute

* Equal contribution † Corresponding authors

PaperarXivDataCode

Announcement link coming soon

Drawing as a robotic task

How effectively can VLA models learn diverse behaviors as training data scales? Answering this requires large, varied robot datasets, which remain costly to collect on physical robots. We propose to study this question through drawing.

A robot starts from a blank canvas and follows a language instruction, yet each instruction allows many valid drawings and stroke orders. This makes the data naturally multimodal, while compositions of familiar objects let us test spatial understanding and generalization to new combinations.

Text-to-image models supply the sketches that our pipeline converts into robot demonstrations, while learned visual features let us evaluate the resulting drawings. Together, these make drawing an ideal setting for studying how vision-language-action models learn as demonstrations scale—and where their execution breaks down.

“Draw a hamburger.”
Simulation · 3×Real robot · 3×
Drag to compare. Simulation: generated demonstration. Real robot: learned policy.

Compare BURGER and LIBERO trajectories

The same drawing instruction admits different shapes, stroke orders, and execution routes. These four comparisons show the resulting diversity in BURGER alongside manipulation demonstrations from LIBERO.

Top: LIBERO demonstrations. Middle: BURGER demonstrations. Bottom: learned policy rollouts. Each panel accumulates 42 trajectories.

Across 40 sampled instructions per dataset, BURGER averages 7.13 effective trajectory modes, compared with 1.09 in LIBERO. The four examples above show how this variation appears in demonstrations and learned behavior.

What happens when VLAs deal with multimodal data?

A circle can be drawn clockwise or counterclockwise. We train on 100 demonstrations with the same shape and radius, using both directions in roughly equal proportions (56% clockwise, 44% counterclockwise). Position and starting point also vary. Can a VLA learn these alternatives and execute one consistently?

3×
OpenVLAStalls in local loops
3×
OFT · chunk 1Often misses contact
3×
OFT · chunk 8Draws, then overdraws
3×
π0-FASTFavors one direction
3×
π0.5Completes a single circle

Learned policies in simulation, shown at 3× speed. One example per setting; the measurements use 20 rollouts for each setting.

Execution changes with action chunking. OpenVLA-OFT makes contact in 6/20 rollouts at chunk length 1 and 20/20 at length 8.

Does the policy preserve both directions? π0-FAST draws clockwise in 95% of rollouts, compared with 56% of demonstrations. It also continues beyond a single revolution.

The same task exposes differences in contact, progress, termination, and coverage of the demonstrated behaviors. Inspecting the drawing process lets us distinguish these failure patterns.

All circle rollouts and measurements

Twenty circle rollouts per model setting, overlaid and colored by drawing direction.
100 training demonstrations; 20 rollouts per model setting.
SourceContactClockwiseStart entropyMean sweep
Demonstrations100/10056%0.97359.8°
OpenVLA20/2040%0.82874.6°
OFT · chunk 16/20100%0.35129.3°
OFT · chunk 820/2055%0.58586.1°
π0-FAST20/2095%0.63563.2°
π0.520/2055%0.84355.2°

Clockwise percentages use rollouts that make contact. Higher start entropy indicates more uniform starting points; a sweep near 360° indicates a single revolution.

Learning more than one drawing behavior

On more varied tasks, π0.5 also learns to produce different drawings for the same instruction. The helmet examples below compare training demonstrations with learned policy executions.

Helmet training demonstrations in the top two rows and different learned pi zero point five drawing sequences in the bottom row.
Training demonstrations above; learned policy executions below. Figure from the paper.

Language following and compositional generalization

The initial blank canvas does not reveal which object to draw. The instruction specifies the object and, for a composition, its spatial relation to another object. As the drawing develops, the policy uses the changing canvas as visual feedback.

We also ask whether a policy can combine familiar objects in pairs absent from training. The examples below show π0 executing a seen pair and a held-out pair.

Pair seen in trainingπ0 · simulation
Draw a leaf left of a moon.
Pair held out from trainingπ0 · simulation
Draw an apple left of a blueberry.

Selected learned-policy rollouts in simulation. Both objects in the held-out pair are familiar from training.

The evaluated policies can produce new combinations of familiar objects, while object categories absent from training remain difficult. Composition can also reduce drawing quality, especially when the instruction set includes vertical relations.

How learning changes with data scale

The generation pipeline lets us vary the number of object classes from 10 to 132 and demonstrations per class from 10 to 100. We use this sweep to study how three OpenPI-family models respond to additional data: the flow-matching models π₀ and π₀.₅, and the autoregressive π₀-FAST.

π0
Pi zero drawing score across numbers of training classes and demonstrations per class.
π0.5
Pi zero point five drawing score across numbers of training classes and demonstrations per class.
π0-FAST
Pi zero FAST drawing score across numbers of training classes and demonstrations per class.

Drawing similarity across numbers of training classes and demonstrations per class. Higher scores indicate greater similarity to the target object class in the learned sketch embedding.

π0 and π0.5 improve more consistently as data scale increases. π0-FAST shows more variable trends. The models differ in how effectively they use additional drawing demonstrations.

The demonstration generator provides a way to vary task diversity and examples per task within one robot setup, supporting further studies of how training data shapes learned behavior.

How drawing quality is measured

We train a projection head on frozen DINOv3 features to score drawings against their target class. In a study of 10 single-object classes, 66 participants ranked five drawings per class. The metric agrees more closely with their judgments than CLIPScore.

Agreement with human rankings
MetricSpearman ρKendall τbPairwise
CLIPScore0.5920.4330.803
BURGER metric0.8340.6550.875

Drawing with visual feedback on a real robot

On a Franka Panda, fine-tuned π₀.₅ policies complete partial drawings and continue after a human edit. The experiments below use different training datasets, indicated with each result. These are selected qualitative examples.

Real robot · 4.9×
Add the missing cup handleTraining: 100 simulated demonstrations (10 tasks) + 30 real demonstrations (3 tasks).
Real robot · 4.9×
Continue after a human draws the rear wheelTraining: 100 real demonstrations across 10 tasks. The robot is paused during the human edit.

In the bicycle experiment, the robot pauses while a person adds the rear wheel, then resumes from the updated canvas. The learned policy continues with the remaining structure.

Real-only training

π0.5 · 100 real demonstrations across 10 tasks.

Real robot · 6.6×
Birthday cake
Real robot · 6.5×
Guitar
Real robot · 6.6×
Train
Real robot · 6.6×
Butterfly

Same instruction, different stroke orders

“Draw a bicycle.” · 100 real demonstrations across 10 tasks.

Real robot · 6.6×
Start with the left wheel
Real robot · 6.6×
Start with the right wheel

Drawing after the camera moves

“Draw an apple.” · 30 real demonstrations across 3 tasks.

Real robot · 6.6×
Default front view
Real robot · 6.6×
Oblique view

From simulated data to real-robot drawing

We mix generated simulation demonstrations with a small real-robot dataset to fine-tune π0.5. These four objects appear in simulation training but are absent from the real-training set.

Simulated demonstrations
14,751across 132 tasks
Real demonstrations
100across 10 tasks
3× recording
Moon
3× recording
Banana
3× recording
Owl
3× recording
Cat

Dataset and demonstration generation

BURGER is designed to support studies of language-conditioned robot data. The dataset provides a starting point, and the generation pipeline lets researchers extend the task set and construct new training demonstrations.

The benchmark contains 28,200 simulated demonstrations: 132 single-object tasks and 150 compositional tasks, with 100 demonstrations per task. The compositional set contains 100 horizontal and 50 vertical tasks.

Simulated demonstrations
28,200
Single-object tasks
132
Composition tasks
150
Demonstrations per task
100

Diversity in drawing demonstrations

Consider “draw a helmet.” Both its shape and its stroke sequence can vary. A drawing dataset contains variation within each instruction as well as across categories.

3×
Demonstration 01 · Helmet AOutline first · clockwise
3×
Demonstration 02 · Helmet AVisor first · counterclockwise
3×
Demonstration 03 · Helmet BOutline first · clockwise
3×
Demonstration 04 · Helmet BLining first · counterclockwise

Each pair draws the same helmet with a different stroke order, direction, and starting point. Generated in simulation · 3× speed.

Single-object demonstrations

32 category previews · 132 categories in the full dataset

Scroll to browse generated demonstrations, replayed in simulation.

Generated spatial-composition demonstrations

Explore 18 tasks with three recorded demonstrations each. The inside / contains examples illustrate additional generated tasks beyond the paper’s horizontal and vertical composition set.

54 demonstrations

Demonstration

Left / right

6×
Leaf left of moonDemonstration 01
6×
Moon left of leafDemonstration 01
6×
Snowman left of leafDemonstration 01
6×
Leaf left of snowmanDemonstration 01
6×
Mug left of pizzaDemonstration 01
6×
Purse left of starDemonstration 01

Above / below

6×
Camera above baseballDemonstration 01
6×
Umbrella above butterflyDemonstration 01
6×
Guitar above owlDemonstration 01
6×
Bicycle below bookDemonstration 01
6×
Bucket below cloudDemonstration 01
6×
Ice cream below flowerDemonstration 01

Inside / contains

6×
Book inside backpackDemonstration 01
6×
Cat inside tentDemonstration 01
6×
Rabbit inside houseDemonstration 01
6×
House contains bedDemonstration 01
6×
Purse contains keyDemonstration 01
6×
Train contains suitcaseDemonstration 01
Expand any clip to explore its drawing.

Generated demonstrations recorded in simulation, shown at 6× speed with a final hold. These recordings show the generation pipeline’s output.

Open video ↗

Generate demonstrations for your own tasks

Start with a language prompt. The pipeline turns generated sketches into ordered strokes, then records the robot executing them—providing a scalable source of robot demonstrations for new objects and spatial compositions.

Draw a bicycle.
Generated line sketch of a bicycle
01 Generate a sketch

Use a text-to-image model, or supply a sketch.

02 Trace and order strokes

Extract the lines and choose a sequence for drawing them.

8.3×
03 Execute and record

Follow the strokes while recording observations and actions.

One pipeline, three simulation backends

Hamburger drawing executed in ManiSkill / SAPIEN
ManiSkill / SAPIEN
Hamburger drawing executed in MuJoCo / LIBERO
MuJoCo / LIBERO
Hamburger drawing executed in Isaac Sim
Isaac Sim

Each demonstration contains RGB observations, robot state, actions, and its language instruction.

Export datasets in LIBERO or LeRobot format.

Project overview

Narrated overview · 2:52

Scope and limitations

BURGER studies line drawing with a rigid tool. It simplifies the contact physics of painting and manipulation. Generated sketches and stroke-order heuristics shape the data distribution.

The evaluated policies can compose known objects, but struggle to draw unseen object categories. Drawing performance does not establish general manipulation ability.

Cite BURGER

@misc{xu2026burger,
  title = {{BURGER}: Benchmarking and Understanding Robotic Generalization via Embodied dRawing},
  author = {Xu, Xiangyu and Peng, Bangshuai and Liu, Ziyi and Zhao, Mengjie and Lu, Chenhao and Fan, Zhengyang and Xu, Huazhe and Hu, Kaizhe},
  year = {2026},
  eprint = {2610.40532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40532}
}