NTU Lab and ACE Robotics Build Robot Training Worlds From Photos
A team at Nanyang Technological University and the Singapore robotics firm ACE Robotics has released Mundo, a world model that turns a handful of photographs of a real place into an editable three dimensional scene that robots can be trained inside. Take one to twelve pictures of a room, and Mundo builds a version of it with physics, then renders what a robot sees as it moves and manipulates objects in that copy.
The work targets a bottleneck that every robotics lab knows. Training a robot in the real world is slow, because every attempt costs time, floor space and a supervisor, while training it in a simulator is fast but brittle, because simulated rooms rarely look or behave like the one the robot will actually work in.
But the pitch here is specificity. Mundo is not trying to generate a generic kitchen. It rebuilds a particular hotel corridor or laundry room from photographs of that place, which means a policy trained in the copy has a chance of transferring to the original.
The pipeline runs in stages. A three dimensional foundation model recovers camera poses and dense geometry from the images, and a detector proposes a bounding box for each major object. Because detectors miss objects and misjudge their size, a vision language model agent then edits a scene program, renders it back from each input viewpoint, compares the render against the photograph and decides what to correct, while a numerical pass fits position, size and orientation.
And the scene it produces is more than a picture. Each object carries a name, pose, size, mass and friction, scripts apply actions and forces, and MuJoCo, the standard physics engine in robotics research, computes motion, contact and collisions. A video diffusion model then renders the result in the look of the original photographs, so the output reads as footage of the real place.
And the rendering stack is borrowed rather than built from scratch. The team fine tuned a MiniMax reference to video model on roughly 40,000 paired videos at 640 by 480, generating 124 frame clips and reusing the last 34 frames of each to keep long sequences continuous.
The demonstrations cover the tasks labs actually care about. Robots navigate a reconstructed hotel floor, open a washing machine in a laundry room, place an apple in a bowl and stack blocks, with head and wrist camera views and recorded trajectories. For large spaces the system produces camera moves lasting tens of seconds or minutes, choosing the most relevant photographs for each segment and overlapping them so the joins hold.
The limits are stated plainly. Reconstruction takes 10 to 20 minutes, rendering runs at about 1.5 frames per second on eight A800 GPUs at 20 inference steps, small objects can appear to float, and the scene has no long term memory, so a space that changes over weeks will drift from its copy.
But that last limit is the one to watch for anyone thinking about deployment. A factory floor is rearranged, a warehouse is repacked, and a model that cannot remember those changes will need rebuilding rather than updating.
Mundo converts one to twelve photos of a real space into an editable 3D scene with physics
A vision language model agent corrects object placement by comparing renders against the photos
MuJoCo handles motion, contact and collisions, and a video model renders the result
Demos include hotel floor navigation, opening a washing machine and stacking blocks
Code and weights are due by the end of next month, per the project repository
Source: Pandaily
