Intent Retargeting: Phone Videos to a Robot in LIBERO

I filmed my own hands moving a bowl on an iPhone, then used those clips to drive a Franka Panda in the LIBERO simulator. The robot reproduces the bowl's motion measured in the video and takes one detail from my technique: where on the rim I gripped.

RoboticsAI/MLComputer Vision
# overview

Problem to result

Who had the problem, what I built, the hardest part, and what it achieved.

01 · The problem

Robots usually learn manipulation from robot demonstrations, which are slow to collect. Simply replaying a person's hand motion on a robot fails (0 of 80 tasks in my tests), because a hand and a gripper move objects differently.

02 · What I built

A pipeline that turns my phone videos of moving a bowl into robot demonstrations, keeping where on the rim I gripped, and trains a robot policy from them.

03 · The hardest part, and how I solved it

Getting consistent demonstrations. Without my grip strategy the planner mixed rim pinches, drags and pushes, and a policy trained on that mix averaged them and failed more often. Seeding the planner with the grip angle measured from my video made it choose the rim pinch in 279 of 300 episodes.

04 · The results

96.2% success on held-out tasks with my grip strategy vs 73.6% without; 0.7 ms per policy query; 77.1% from camera images alone with SmolVLA; hand replay 0 of 80.

# graphs

Performance Graphs

Visualisations of model performance and results across experiments.

Scenario 1 · from results/ logs

Success on held-out tasks0%20%40%60%80%100%0123Random start + goal: 1 = random goal, 2 = right-clip goal, 3 = left-clip goalMLP · demos with my grip [left-clip goal · 149/150 · 95% CI 96.3–99.9%]SmolVLA from cameras [right-clip goal · 89/150 · 95% CI 51.3–66.9%]MLP · demos without [left-clip goal · 102/150 · 95% CI 60.2–74.9%]Replay of my hand motion [right-clip goal · 0/25 · 95% CI 0.0–13.3%]

Intent Retargeting — Results across all runs

# simulation

Live Simulation Output

Simulated console execution.

Outputs
Results summary
$_▋
# videos

Project Walkthroughs

Click any video to play.

My clip vs the robot · scenario 1
Click to enlarge
GIF

My clip vs the robot · scenario 1

My left-hand clip (left) and the Panda in LIBERO reproducing its effect on the bowl, seen from the front and from a top camera aimed like my phone.

My clip vs the robot · scenario 2
Click to enlarge
GIF

My clip vs the robot · scenario 2

My hand-held clip (left) and the Panda putting the bowl on the plate in LIBERO's official libero_goal task 8 scene, front and top views.

0:09

Scenario 2 · bowl and plate tracking

My hand-held clip with camera motion removed: the bowl (green) is measured relative to the plate (magenta) in every frame as I place it and take it off again.

# challenges

Problems & Fixes

What went wrong while building it, and how each problem was solved.

Setup and environment01 / 21

MediaPipe's pinned packages could break LeRobot

How it was fixed

Made two separate environments: .venv for LeRobot, LIBERO and SmolVLA, and .venv-extract for MediaPipe and OpenCV. See scripts/setup.sh