A Good Robot Needs Bad Footage

Robotics demos are usually edited to remove the embarrassing parts. Training data should do the opposite.

That was my main takeaway from Allen Institute for AI's MolmoBot work published on 11 March. The headline is about a new robotics system. The deeper lesson is about data honesty. If you want a model to survive the real world, you need to feed it more of the real world, including the clumsy, partially obscured, badly framed, slightly-failed moments that video teams normally trim away.

frame = shaky_camera()
goal = pick_up_the_thing()
outcome = almost_but_not_quite()
label = keep_this_example

Why failure footage matters

Robot learning has the same temptation as most machine learning fields: optimise for clean data because clean data is easier to reason about. The trouble is that clean data quietly teaches the wrong lesson. It tells the model that the world arrives in neat scenes, with objects fully visible, lighting stable, and hands behaving in predictable ways. Then the system meets an actual home, warehouse, or lab bench and discovers that the world did not read the benchmark spec.

Messy footage carries the operational information that polished data throws away. How does the scene look when the target object is partly hidden? What if the camera drifts? What if the grasp begins correctly and then slips? What if the human operator reaches in awkwardly and blocks half the frame? Those are not edge cases in robotics. They are the day job.

The engineering lesson

What I like about this line of work is that it shifts the romance away from the policy network and back toward the corpus. Robotics people know this already, but AI marketing often forgets it: the hero is usually the data pipeline. Collection quality. Coverage. Labelling discipline. The decision to keep examples that make the model look stupid because those are exactly the examples that teach it how not to be stupid next week.

I suspect something similar is about to happen across agent systems more broadly. Browser agents need traces from ugly websites. Coding agents need repos with flaky tests and baffling conventions. Office agents need messy documents rather than idealised toy forms. MolmoBot is a robotics story, but the data lesson generalises very well.

There is a pleasant irony here. We keep searching for smarter models, and the answer often comes back as 'show the model worse footage'. Not because the model enjoys suffering, but because robustness is usually learned from disturbance rather than perfection.

A good robot does not need a prettier dataset. It needs a truer one. That is harder work, less photogenic, and almost certainly more valuable.