Intent Retargeting: Phone Videos to a Robot in LIBERO
I filmed my own hands moving a bowl on an iPhone, then used those clips to drive a Franka Panda in the LIBERO simulator. The robot reproduces the bowl's motion measured in the video and takes one detail from my technique: where on the rim I gripped.
Problem to result
Who had the problem, what I built, the hardest part, and what it achieved.
Robots usually learn manipulation from robot demonstrations, which are slow to collect. Simply replaying a person's hand motion on a robot fails (0 of 80 tasks in my tests), because a hand and a gripper move objects differently.
A pipeline that turns my phone videos of moving a bowl into robot demonstrations, keeping where on the rim I gripped, and trains a robot policy from them.
Getting consistent demonstrations. Without my grip strategy the planner mixed rim pinches, drags and pushes, and a policy trained on that mix averaged them and failed more often. Seeding the planner with the grip angle measured from my video made it choose the rim pinch in 279 of 300 episodes.
96.2% success on held-out tasks with my grip strategy vs 73.6% without; 0.7 ms per policy query; 77.1% from camera images alone with SmolVLA; hand replay 0 of 80.
Performance Graphs
Visualisations of model performance and results across experiments.
Scenario 1 · from results/ logs
Intent Retargeting — Results across all runs
Live Simulation Output
Simulated console execution.
Project Walkthroughs
Click any video to play.

My clip vs the robot · scenario 1
My left-hand clip (left) and the Panda in LIBERO reproducing its effect on the bowl, seen from the front and from a top camera aimed like my phone.

My clip vs the robot · scenario 2
My hand-held clip (left) and the Panda putting the bowl on the plate in LIBERO's official libero_goal task 8 scene, front and top views.
Scenario 2 · bowl and plate tracking
My hand-held clip with camera motion removed: the bowl (green) is measured relative to the plate (magenta) in every frame as I place it and take it off again.
Problems & Fixes
What went wrong while building it, and how each problem was solved.
MediaPipe's pinned packages could break LeRobot
Made two separate environments: .venv for LeRobot, LIBERO and SmolVLA, and .venv-extract for MediaPipe and OpenCV. See scripts/setup.sh
Source Code
GitHub repositories for this project.