Yasutaka Furukawa Brian Curless Steven M Seitz University of Washington Seattle USA Richard Szeliski Microsoft Research Redmond USA Reconstruction amp Visualization of Architectural Scenes ID: 316381
Download Presentation The PPT/PDF document "Reconstructing Building Interiors from I..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Reconstructing Building Interiors from Images
Yasutaka Furukawa Brian
Curless
Steven M. Seitz
University of Washington, Seattle, USA
Richard
Szeliski
Microsoft Research, Redmond, USASlide2
Reconstruction & Visualizationof Architectural Scenes
Manual (semi-automatic)
Google Earth & Virtual Earth
Façade [Debevec et al., 1996]CityEngine [Müller et al., 2006, 2007]AutomaticGround-level images [Cornelis et al., 2008, Pollefeys et al., 2008]Aerial images [Zebedin et al., 2008]
Google Earth
Virtual Earth
Zebedin
et al.
Müller
et al.Slide3
Reconstruction & Visualization
of Architectural Scenes
Manual (semi-automatic)
Google Earth & Virtual EarthFaçade [Debevec et al., 1996]CityEngine [Müller
et al., 2006, 2007]AutomaticGround-level images
[Cornelis et al., 2008, Pollefeys et al., 2008]
Aerial images [Zebedin et al., 2008]
Google Earth
Virtual Earth
Zebedin
et al.
Müller
et al.Slide4
Reconstruction & Visualization
of Architectural Scenes
Manual (semi-automatic)
Google Earth & Virtual EarthFaçade [Debevec et al., 1996]CityEngine [Müller et al., 2006, 2007]AutomaticGround-level images
[Cornelis et al., 2008, Pollefeys
et al., 2008]Aerial images
[Zebedin et al., 2008]
Google Earth
Virtual Earth
Zebedin
et al.
Müller
et al.Slide5
Reconstruction & Visualizationof Architectural Scenes
Google Earth
Virtual Earth
Zebedin
et al.
Müller
et al.
Little attention paid to indoor scenesSlide6
Our Goal
Fully automatic system for indoors/outdoors
Reconstructs a simple 3D model from images
Provides real-time interactive visualizationSlide7
What are the challenges?Slide8
Challenges - Reconstruction
Multi-view stereo (MVS) typically produces a dense model
We want the model to be
Simple for real-time interactive visualization of a large scene (e.g., a whole house)Accurate for high-quality image-based renderingSlide9
Challenges - Reconstruction
Multi-view stereo (MVS) typically produces a dense model
We want the model to be
Simple for real-time interactive visualization of a large scene (e.g., a whole house)Accurate for high-quality image-based renderingSimple mode is effective for compelling visualizationSlide10
Challenges – Indoor Reconstruction
Texture-poor surfacesSlide11
Challenges – Indoor Reconstruction
Texture-poor surfaces
Complicated visibilitySlide12
Challenges – Indoor Reconstruction
Texture-poor surfaces
Complicated visibility
Prevalence of thin structures
(doors, walls, tables)Slide13
Outline
System pipeline (system contribution)
Algorithmic details
(technical contribution)Experimental resultsConclusion and future workSlide14
System pipeline
Images
ImagesSlide15
System pipeline
Structure-from-Motion
Images
Bundler by Noah SnavelyStructure from Motion for unordered image collections
http://phototour.cs.washington.edu/bundler/Slide16
System pipeline
Images
SFMSlide17
System pipeline
Images
SFM
PMVS by Yasutaka Furukawa and Jean Ponce
Patch-based Multi-View Stereo Softwarehttp://grail.cs.washington.edu/software/pmvs/
Multi-view StereoSlide18
System pipeline
Images
SFM
MVSSlide19
System pipeline
Images
SFM
MVS
Manhattan-world Stereo
[Furukawa et al., CVPR 2009]Slide20
System pipeline
Images
SFM
MVS
Manhattan-world Stereo
[Furukawa et al., CVPR 2009]Slide21
System pipeline
Images
SFM
MVS
Manhattan-world Stereo
[Furukawa et al., CVPR 2009]Slide22
System pipeline
Images
SFM
MVS
Manhattan-world Stereo
[Furukawa et al., CVPR 2009]Slide23
System pipeline
Images
SFM
MVS
Manhattan-world Stereo
[Furukawa et al., CVPR 2009]Slide24
System pipeline
Images
SFM
MVS
Manhattan-world Stereo
[Furukawa et al., CVPR 2009]Slide25
System pipeline
Images
SFM
MVS
MWSSlide26
System pipeline
Images
SFM
MVS
MWS
Axis-aligned depth map merging
(our contribution)Slide27
System pipeline
Images
SFM
MVS
MWS
Merging
Rendering: simple view-dependent texture mappingSlide28
Outline
System pipeline (system contribution)
Algorithmic details
(technical contribution) Experimental resultsConclusion and future workSlide29
Axis-aligned Depth-map Merging
Basic framework is similar to volumetric MRF
[
Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]Slide30
Axis-aligned Depth-map Merging
Basic framework is similar to volumetric MRF
[
Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]Slide31
Axis-aligned Depth-map Merging
Basic framework is similar to volumetric MRF
[
Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]Slide32
Axis-aligned Depth-map Merging
Basic framework is similar to volumetric MRF
[
Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]Slide33
Axis-aligned Depth-map Merging
Basic framework is similar to volumetric MRF
[
Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]Slide34
Axis-aligned Depth-map Merging
Basic framework is similar to volumetric MRF
[
Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]Slide35
Key Feature 1 - Penalty termsSlide36
Key Feature 1 - Penalty terms
Binary penalty
Binary
encodes smoothness & dataSlide37
Key Feature 1 - Penalty terms
Binary penalty
Binary
encodes smoothness & dataUnary is often constant (inflation)Slide38
Key Feature 1 - Penalty terms
Binary penalty
Binary
encodes smoothness & dataUnary is often constant (inflation)
Weak regularization at interesting places
Focus on a dense modelSlide39
Key Feature 1 - Penalty terms
Binary penalty
Binary
encodes smoothness & dataUnary is often constant (inflation)
Weak regularization at interesting places
Focus on a dense modelWe want a simple modelSlide40
Key Feature 1 - Penalty terms
Binary penalty
Binary
encodes smoothness & dataUnary is often constant (inflation)Slide41
Key Feature 1 - Penalty terms
Binary penalty
Binary
encodes smoothness &
dataUnary is often constant (inflation)Slide42
Key Feature 1 - Penalty terms
Unary
encodes data
Binary penalty
Binary
encodes smoothness & data
Unary is often constant (inflation)Slide43
Key Feature 1 - Penalty terms
Binary
is smoothness
Unary encodes dataBinary penalty
Binary
encodes smoothness &
dataUnary is often constant (inflation)Slide44
Key Feature 1 - Penalty terms
Regularization becomes weak
Dense 3D model
Regularization is data-independent
Simpler 3D model
Binary penaltySlide45
Axis-aligned Depth-map Merging
Align
voxel
grid withthe dominant axesSlide46
Axis-aligned Depth-map Merging
Align
voxel
grid withthe dominant axesData term (unary)Slide47
Axis-aligned Depth-map Merging
Align
voxel
grid withthe dominant axesData term (unary)Slide48
Axis-aligned Depth-map Merging
Align
voxel
grid withthe dominant axesData term (unary)Slide49
Axis-aligned Depth-map Merging
Align
voxel
grid withthe dominant axesData term (unary)Smoothness (binary)Slide50
Axis-aligned Depth-map Merging
Align
voxel
grid withthe dominant axesData term (unary)Smoothness (binary)Slide51
Axis-aligned Depth-map Merging
Align
voxel
grid withthe dominant axesData term (unary)Smoothness (binary)Graph-cutsSlide52
Key Feature 2 – Regularization
For large scenes, data info are not completeSlide53
Key Feature 2 – Regularization
For large scenes, data info are not complete
Typical volumetric MRFs bias to general minimal surface
[Boykov and Kolmogorov, 2003]We bias to piece-wise planar axis-aligned for architectural scenesSlide54
Key Feature 2 – RegularizationSlide55
Key Feature 2 – RegularizationSlide56
Key Feature 2 – RegularizationSlide57
Key Feature 2 – RegularizationSlide58
Key Feature 2 – RegularizationSlide59
Key Feature 2 – Regularization
Same energy (ambiguous)Slide60
Key Feature 2 – Regularization
Same energy (ambiguous)
Data penalty: 0Slide61
Key Feature 2 – Regularization
Same energy (ambiguous)
Data penalty: 0
Smoothness penalty:
Data penalty: 0
Smoothness penalty: 24
Data penalty: 0Slide62
Key Feature 2 – Regularization
shrinkageSlide63
Key Feature 2 – Regularization
Axis-aligned neighborhood + Potts model
Ambiguous
Break ties with the minimum-volume solution
Piece-wise planar axis-aligned modelSlide64
Key Feature 3 – Sub-voxel accuracySlide65
Key Feature 3 – Sub-voxel accuracySlide66
Key Feature 3 – Sub-voxel accuracySlide67
Key Feature 3 – Sub-voxel accuracySlide68
Summary of Depth-map Merging
For a
simple
and axis-aligned modelExplicit regularization in binaryAxis-aligned neighborhood system & minimum-volume solutionFor an accurate modelSub-voxel refinementSlide69
Outline
System pipeline (system contribution)
Algorithmic details (technical contribution)
Experimental resultsConclusion and future workSlide70
Kitchen
- 22 images
1364 triangles
hall - 97 images3344 triangles
house
- 148 images
8196 triangles
gallery
- 492 images
8302 trianglesSlide71Slide72
DemoSlide73
Running Time
Kitchen
(22 images)
Hall (97 images)House (148 images)gallery (492 images)SFM137692
716MVS
38158147
130MWS
39.6281.3843.6
5677.4Merging
0.40.43.6
22.4
Running time of 4 steps [min]Slide74
Conclusion & Future Work
Conclusion
Fully aut
omated 3D reconstruction/visualization system for architectural scenesNovel depth-map merging to produce piece-wise planar axis-aligned model with sub-voxel accuracyFuture workRelax Manhattan-world assumptionLarger scenes (e.g., a whole building)Slide75
For More Details
Please refer to the paper and
our project website
http://grail.cs.washington.edu/projects/interior/3D viewer and dataset availableSlide76
For More Details
Come to a demo session this afternoon
(1:15pm)Slide77
For More Details
Come to a demo session this afternoon
(1:15pm)
Open-source SFM and MVS softwareBundlerNoah Snavelyhttp://phototour.cs.washington.edu/bundlerPMVS Version2Yasutaka Furukawa and Jean Poncehttp://grail.cs.washington.edu/software/pmvsSlide78
Acknowledgements
Sameer
Agarwal and Noah Snavely for support on SFM and discussionFunding sourcesNational Science Foundation grant IIS-811878SPAWARThe Office of Naval ResearchThe University of Washington Animation Research LabsDatasetsChristian Laforte and Feeling Software for KitchenEric Carson and Henry Art Gallery for gallerySlide79
Any Questions?
Images
SFM
MVS
MWS
Merging