Physically Consistent 3D Multi-Object Scene Reconstruction

Limitations in Prior Work

Recently, research in 3D reconstruction shifts from achieving consistency in mere appearance and geometry to attaining a physically consistent model of a scene or an object. Past work on physics-aware 3D reconstruction falls into two primarily different categories, scene-level and object-level, and exhibit ample limitations. Efforts for scene-level reconstruction such as PhyRecon and 3D-RE-GEN only strive to achieve physically plausible states of the objects in the scene under an equilibrium by optimizing their geometry, and therefore, do not embody any notion of weight, elasticity and plasticity of objects therein. On the other hand, prior work for object-level reconstruction such as Pixie, PhysicsDreamer, OmniPhysGS, and NeRF2Physics are more grounded in physics and allow for simulation. However, they were primarily intended for a single object and not readily extendable to multiple objects.

In object-level physics-aware 3D reconstruction, the methodology of achieving physical parameters of objects is tightly coupled with the implementation of the simulator. The need for good physical parameters of objects in a scene can be less emphasized if the settings of the simulator can be configured specifically for that particular scene. In other words, we can have an arbitrarily specific configuration file dedicated for each scene to achieve desired simulation. This approach is obviously not desirable since physically reasonable simulation should depend more on good physical parameters and less on the configuration of the simulator. Prior work involve varying level of dependence on the simulator to achieve good simulation, and it’s important to work on a simulator that is easily generalizable for objects of varying types and quantities to reduce such dependence.