We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. Motivated by the bird’s-eye-view (BEV) representation commonly used in vision and robotics, we propose conditional neural groundplans, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations. Our method is trained self-supervised from unlabeled multi-view observations using differentiable rendering, and learns to complete geometry and appearance of occluded regions. In addition, we show that we can leverage multi-view videos at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The ability to separately reconstruct movable objects enables a variety of downstream tasks using simple heuristics, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. This highlights the value of neural groundplans as a backbone for efficient 3D scene understanding models.
Given a single image as input, our method represents the scene as a static and dynamic groundplan. This representation is then used to render novel views by compositing the contributions from the two groundplans using a neural renderer. Both static and dynamic groundplan can be rendered individually to present the static and dynamic (movable) parts of the scene respectively. In the video below, we show the composite rendering of the static and dynamic groundplans, as well as the static and dynamic groundplans rendered individually from circular camera trajectory.
Since our method computes independent groundplans for the static and dynamic (movable) components from the input image, the densities expressed by the dynamic groundplan enable segmentation in the bird's eye-view, 2D instance-level segmentation, and 3D bounding box prediction in an unsupervised setup.
The reliable localization of the objects using the dynamic groundplan provides individual object-centric representation for all movable objects in the input image. These object-centric representations can be individually manipulated to edit the scene, enabling object deletion, insertion, and rearrangement.