What Changed in the Room?

Evidence-based 3D change detection from two phone videos of the same room: it reports objects added, removed or moved, in 3D and in plain language, and keeps viewpoint differences, occlusion and unseen areas from being reported as changes.

Computer VisionRoboticsAI/ML
# overview

Problem to result

Who had the problem, what I built, the hardest part, and what it achieved.

01 · The problem

Spotting what changed in a room from two phone videos is easy to get wrong: different viewpoints, occlusion and areas only one video saw all look like changes, and transparent objects are invisible to depth sensing.

02 · What I built

A tool that compares two walkthrough videos of the same room and reports what was added, removed or moved, in 3D and in plain language.

03 · The hardest part, and how I solved it

Transparent objects. Depth sees straight through a glass, so a removed glass left no trace. Two approaches failed; the one that worked compares matched before and after views by appearance and only accepts a change that scores well below nearby unchanged places seen through the same views.

04 · The results

All 8 recorded test pairs correct, including the removed drinking glass; a 2 m chair move measured at 1.93 m in a synthetic test; false changes on high-angle pairs cut from up to 7 to 0; 8 of 8 synthetic tests pass.

05 · Limitations
  • Thresholds (0.72 similarity, 0.10 below controls, 0.05 label margin) were tuned on the same 8 recorded pairs reported here; there is no separate held-out set.
  • No real recording of a moved object: the 1.93 m move comes from a synthetic test, so move measurement is unproven on real footage.
  • Ground truth for the recorded pairs was established by eye from matched views, not by an independent measurement.
# results

Results: table and top-down maps

The outcome for every recorded test pair, then one top-down map per pair.

Results by test pair

TestLighting / viewpointTrue changeSystem outputUnchecked (before / after)
test4cool whiteclear plastic jug addedjug added7% / 9%
test7cool whitedrinking glass removeddrinking glass removed11% / 4%
test6cool whitenone (control)no changes12% / 6%
test3cool white, high anglenoneno changes41% / 13%
test5cool white, high anglenoneno changes59% / 11%
test8cool white, high anglenoneno changes32% / 13%
test1warm, very dark, high angleno visible changeno changes43% / 10%
test9warm, high angleno visible changeno changes32% / 8%
  • 8 of 8 outputs match the ground truth. There is no test2 recording, so the pairs are test1 and test3 to test9.
  • Verification (DINOv2 similarity of matched crops): jug 0.47 vs controls 0.84; glass 0.71 vs controls 0.87. A change must score below 0.72 and at least 0.10 below its controls; static places score 0.76 to 0.89.
  • Registration tolerance (2.5 × measured ICP error, clamped to 1–5 cm): 1.1–1.5 cm on well-aligned pairs, 3.4–4.0 cm on high-angle pairs.
  • Unchecked: share of surfaces one video saw that the other didn't see clearly. Changes there are not reported.
Top-down map for test7: before and after camera paths, unchanged geometry, surfaces seen by only one video, and drinking glass removed
Top-down map for test4: before and after camera paths, unchanged geometry, surfaces seen by only one video, and jug added
Top-down map for test6: before and after camera paths, unchanged geometry, surfaces seen by only one video, and no change reported
Top-down map for test3: before and after camera paths, unchanged geometry, surfaces seen by only one video, and no change reported
Top-down map for test5: before and after camera paths, unchanged geometry, surfaces seen by only one video, and no change reported
Top-down map for test8: before and after camera paths, unchanged geometry, surfaces seen by only one video, and no change reported
Top-down map for test1: before and after camera paths, unchanged geometry, surfaces seen by only one video, and no change reported
Top-down map for test9: before and after camera paths, unchanged geometry, surfaces seen by only one video, and no change reported

test7 · top-down map · drinking glass removed

cool white light. Blue and orange: before and after camera paths; grey: unchanged; yellow: seen by only one video and left unchecked (11% of the before video, 4% of the after video); red: removed / old position, green: added / new position.

# outputs

Example output files

The real results summary, report.md and changes.json files the system wrote for the recorded test pairs.

Outputs
Results summary
WHAT CHANGED IN THE ROOM — 8 recorded test pairs (thresholds tuned on these pairs)
──────────────────────────────────────────────────────────────────────────────
test   light / viewpoint             ground truth              output
test4  cool white                    clear plastic jug added   added jug
test7  cool white                    drinking glass removed    removed drinking glass
test6  cool white                    no change (control)       no changes
test3  cool white, high angle        no change                 no changes
test5  cool white, high angle        no change                 no changes
test8  cool white, high angle        no change                 no changes
test1  warm, very dark, high angle   no visible change         no changes
test9  warm, high angle              no visible change         no changes

All 8 outputs match the ground truth (judged by eye from matched views).
Three high-angle pairs gave false alarms before the control-place verification step.

VERIFICATION (DINOv2 similarity of matched crops vs control places)
──────────────────────────────────────────────────────────────────────────────
jug   (test4, added)     0.47   controls 0.84
glass (test7, removed)   0.71   controls 0.87
static places, cool-white tests   0.76 to 0.89   (a change must score < 0.72 and >= 0.10 below controls)

SYNTHETIC TESTS (tests/test_changes.py)
──────────────────────────────────────────────────────────────────────────────
8/8 pass: hidden and unseen objects 100% unknown, static box 0% changed,
moved chair measured at 1.93 m (true 2 m) with no false rotation.
# videos

Project Walkthroughs

Click any video to play.

0:13

test7 · drinking glass removed

Reference (left) and test (right): the glass next to the black mug is outlined before and marked missing after. Found by the appearance channel, because the depth model sees through glass.

0:13

test4 · clear jug added

The new jug is outlined in the test video and a dashed box marks where it will appear in the reference. Its carved 3D box is 16 × 11 × 32 cm.

0:13

test6 · no change (control)

Same desk, nothing changed: the system reports no changes. Yellow tint marks surfaces only one video saw, which are left unchecked rather than reported.

# challenges

Problems & Fixes

What went wrong while building it, and how each problem was solved.

Setup and environment01 / 42

No Python, ffmpeg or COLMAP on the machine

Cause: Fresh environment

How it was fixed

Installed Python 3.11 and ffmpeg with winget; project venv; PyTorch 2.11 with CUDA 12.8

# repositories

Source Code

GitHub repositories for this project.