We provide Astra with GVHMR, a tool that reconstructs body pose and movement from video. Yet, since GVHMR does not consider 3D scene constraints, its direct results do not necessarily have correct contact with objects or the floor. A hand may penetrate a surface, a seated body may not line up with a chair, or the feet may float above the floor. To fix this, Astra first places the person roughly in the scene, and then refines the contact.
Once GVHMR recovers the motion, the camera and depth estimates are used to place the person in the 3D scene. Specifically, Astra adjusts the starting point and the overall orientation of the GVHMR trajectory, so that the person's depth in the GVHMR result matches the Pi3X depth at the person's pixels as closely as possible. After this adjustment, the interaction between the human motion and the scene is roughly correct, but human–object and human–floor contact still needs further optimization.
Contact comes next. Astra works out where the person should touch objects and where the feet should meet the floor, turns these into contact targets, and calls contact and ground-support optimization scripts. Like the previous step, this only adjusts the person's trajectory and keeps GVHMR's body pose unchanged.
The examples below show human motion reconstructed from video and placed in the room.
Getting contact right is hard for both Astra and human motion generation models. Astra can directly generate human motion sequence, yet it often looks unnatural and jumps between frames. Kimodo, a model that generates human and robot motion from text and constraints such as paths and joint positions, produces natural motion, but its contact with scene objects can still be off.
We combine them in one framework. Astra plans a rough path of human motion to travel in the room and specifies where and when each body part should touch an object, along with how the object moves during contact, so the targets move with it. Kimodo turns the rough path and joint targets into motion.
If the result looks wrong in the scene, Astra revises the path or contact targets, and then either reruns Kimodo or refines the contact with optimization scripts that correct the trajectory.
The examples below show generated G1 robot actions in the room.
Switch on the bedside lamp.
Move the bedside lounge table.
The examples below show results without Astra's involvement. Example 1 is pure human motion reconstruction by GVHMR; the person's contact with the sofa is inaccurate. Example 2 is generated automatically by Kimodo from the input prompt and contact inputs; the robot's contact with the table is inaccurate.