Chen-Hsuan Lin - Learning 3D Registration and Reconstruction from the Visual World
Published at : October 22, 2021
Sep 21st 2021 at MIT CSAIL
Abstract: Humans learn to develop strong senses for 3D geometry by looking around in the visual world. Through pure visual perception, not only can we recover a mental 3D representation of what we are looking at, but meanwhile we can also recognize the location we are looking at the scene from. In this talk, I will discuss the problems of learning geometric alignment and dense 3D reconstruction, and the general importance of factorizing geometric information from visual data. I will discuss learning 3D shape priors from static RGB images from single-view supervision, as well as the problem of joint 3D registration and reconstruction: given a video sequence, how one can exploit pretrained 3D shape priors to register and refine 3D shape reconstruction, as well as a generic rendering prior from Neural Radiance Fields (NeRF) for learning neural 3D scene representations from noisy/unknown camera poses. Baking in suitable geometric priors allows learning models to effectively recover both the dense 3D scene structures and the corresponding camera poses using image synthesis as the proxy objective, and we believe this is an essential ingredient towards scalable learning of future spatial AI systems.
Bio: Chen-Hsuan Lin is a research scientist at NVIDIA Research. He received his Ph.D. in Robotics from Carnegie Mellon University, where he was advised by Prof. Simon Lucey. His research interests are computer vision and machine learning, with a focus on 3D reconstruction, neural rendering, and learning interpretable 3D representations from image/video data. Chen-Hsuan is a recipient of the NVIDIA Graduate Fellowship (2019) and has collaborated with Adobe Research and Facebook AI Research (FAIR) through research internships. He received his M.S. in Robotics from CMU and B.S. in Electrical Engineering from National Taiwan University. For more information, please visit https://chenhsuanlin.bitbucket.io/.
Abstract: Humans learn to develop strong senses for 3D geometry by looking around in the visual world. Through pure visual perception, not only can we recover a mental 3D representation of what we are looking at, but meanwhile we can also recognize the location we are looking at the scene from. In this talk, I will discuss the problems of learning geometric alignment and dense 3D reconstruction, and the general importance of factorizing geometric information from visual data. I will discuss learning 3D shape priors from static RGB images from single-view supervision, as well as the problem of joint 3D registration and reconstruction: given a video sequence, how one can exploit pretrained 3D shape priors to register and refine 3D shape reconstruction, as well as a generic rendering prior from Neural Radiance Fields (NeRF) for learning neural 3D scene representations from noisy/unknown camera poses. Baking in suitable geometric priors allows learning models to effectively recover both the dense 3D scene structures and the corresponding camera poses using image synthesis as the proxy objective, and we believe this is an essential ingredient towards scalable learning of future spatial AI systems.
Bio: Chen-Hsuan Lin is a research scientist at NVIDIA Research. He received his Ph.D. in Robotics from Carnegie Mellon University, where he was advised by Prof. Simon Lucey. His research interests are computer vision and machine learning, with a focus on 3D reconstruction, neural rendering, and learning interpretable 3D representations from image/video data. Chen-Hsuan is a recipient of the NVIDIA Graduate Fellowship (2019) and has collaborated with Adobe Research and Facebook AI Research (FAIR) through research internships. He received his M.S. in Robotics from CMU and B.S. in Electrical Engineering from National Taiwan University. For more information, please visit https://chenhsuanlin.bitbucket.io/.
Chen-HsuanLearningRegistration