Improving 3D Scene Reconstruction: from Per-Scene to Generalizable

Abstract:  
In recent years, the 3D vision community has witnessed a significant shift towards Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) as primary focal points. These innovations have fundamentally transformed how we represent 3D space and learn these representations through deep learning techniques with 2D image supervision. Existing research in NeRF and 3DGS can be broadly categorized into per-scene and generalizable solutions. Per-scene approaches concentrate on optimizing 3D representations using numerous multi-view images, while generalizable solutions aim to learn from datasets containing diverse scenes, enabling models to generalize to new scenes with sparse views without necessitating retraining. This talk will present a series of advancements from our research group along this trajectory, such as ObjectSDF and ObjectSDF++ for enhancing surface reconstruction quality and efficiency, as well as MatchNeRF for improving generalizability in diverse scenarios.