“Living” 3D World Models Leveraging Crowd Sourced Data

Jan-Michael Frahm
Tuesday, March 3, 2020 - 1:25pm to 2:25pm
Lind 305

Crowd sourced imagery (images and video) is the richest data source available for 3D reconstruction of the world. The tremendous amounts of available imagery provided by photo/video sharing web sites, not only covers the world’s appearance, but also reflects the temporal evolution of the world, and its dynamic parts. It has long been a goal of computer vision to obtain life like virtual models from such rich imagery. The major current research challenges are the scale of the data, e.g. the Yahoo 100 million-image dataset (only presents a small fraction of what is needed to model our world), the diversity of data modalities (e.g. crowdsourced photos or satellite images), the robustness, the completeness of the registration, and the lack of data for dynamic elements. Specifically, we are currently facing significant challenges to process Internet scale crowd sourced imagery within a reasonable time frame given limited compute resources. This is particularly true as we move toward automatically creating content for virtual and augmented reality. The talk discusses the UNC group’s work on highly efficient image registration for the reconstruction of static 3D models from world-scale photo collections on a single PC in the span of six days, as well as the group’s related work on image-based search to address the scalability. It will also discuss the efforts to overcome the challenges achieving registration completeness and robustness. Additionally, the group’s work towards overcoming the lack of observations for the reconstruction of scene dynamics will be presented. This includes for example, reconstructing people and fountains, using crowd-sourced Flickr imagery and videos to achieve the goal of bringing the 3D models to life will be presented.

Jan-Michael Frahm is a research scientist manager at Facebook and a full professor at the University of North Carolina at Chapel Hill where he heads the 3D computer vision group. He received his Dr.-Ing. in computer vision in 2005 from the Christian-Albrechts University of Kiel, Germany. His dissertation, “Camera Self-Calibration with Known Camera Orientation” received the prize for the best Ph.D. dissertation of the year in CAU’s College of Engineering. His Diploma in Computer Science is from the University of Lübeck. His research interests include a variety of topics on the intersection of computer vision, computer graphics, AR & VR, and robotics. He has over 100 peer-reviewed publications, is a program chair f