SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks PowerPoint Presentation, PPT

SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks PowerPoint Presentation, PPT - DocSlides

Paper by John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger Dyson Robotics Lab, Imperial College London. Presentation by Chris Conte. Hey robot, go fetch me a Twix from the snack bar. ID: 720259

By luanne-stotts
Watch All docs

Likes : 4
Views : 5

Download this presentation

Click below link (As may be) to get this presentation.

SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks

Download Note - The PPT/PDF document "SemanticFusion: Dense 3D Semantic Mappin..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentations text content in SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks

SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks
Paper by John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger Dyson Robotics Lab, Imperial College London
Presentation by Chris Conte
Slide2
Hey robot, go fetch me a Twix from the snack bar
In order to understand most commands, computers need to know two aspects of their surrounding:
Where things are
What things areMotivation for divide:
Practical way to break up task
Humans do this!
Slide3
Where am I? Where is everything?
Problem often referred to as Simultaneous Localization and Mapping
Various algorithms to convert 2d video into a dense map of all of the surfaces and their positions within a scene, including the position of the cameraChallenges include loop closure, non-static objects, and distance (perspective is hardly changed with objects far away)
Specifically, ElasticFusion is used.
Slide4
ElasticFusion in Action
Creates a set of what are called Surface elements, which are characterized by a position (in 3D), normal, color, weight, radius, initialization timestamp t
0 and last updated timestamp t.Note: these are not complete objects. They are meant to represent local surface area around a point, while minimizing visual holes.
Slide5
What?
A semantic map is one based on concepts: the scene contains a chair and a desk
Previous work has been concerned with identifying specific objects from within the surface elements.
This paper uses a Convolutional Neural Network
Slide6
Why Use a CNN?
Very effective because of high degree of generalizability. Instead of identifying a make/model of chair, it can be trained to identify chairs, of all kinds.
It learns it’s own features weights, saving time and ideally, finding the optimal configurationRequires a lot of computing power, however.
Requires a lot of labelled training data, one thing that this paper did not have in abundance (<1000 images)
Slide7
Paper investigates two CNNs
First is from Eigen et al.Pre trained segmentation, augmented for depth detection on NYU dataset (complete)
Includes three separate scales in which to detect objects. Includes an (unused) depth approximation.
Slide8
Second is from Noh et. al.
The group also trained a CNN that took in a depth value provided by the localization and mapping approach for each individual pixel alongside an R, G and B value.The RGB weights were initialized from the original papers’ weights. Depth was initialized to the average of those weights, and then trained on the NYUv2 dataset 10k iterations on 2 days.
RGB-D segmentation alone showed an average class accuracy of 43.6%
Slide9
Merge the what and the
where
We can combine these two processes in a way that results in a semantic understanding of objects in a 3D space, allowing us to solve complex problems involving motion and identification. We also expect to see some improvements for segmentation: 3D spacial elements can more easily be identified than a 2D curve in single mergeHow to do this?
Slide10
Recursive Naive Bayes
The probability of element X belonging to an object A in frames q through k is the product across all frames i of the probability of the shape X being an object A in frame i
Yields a probability distribution P(Oi = oi) for all class labels, oi in O, the set of all classes.Naive in that it assumes independence between adjacent object elements.
Slide11
Advantage to recognize dependence
If we see grass below a patch of white, it is more likely a sheep or a cow. How?
Slide12
Conditional Random Fields
Probabilistic approach to addressing dependence between connected elements. Create a graph, each node being a surface element, and Gaussian edge potentials. For each regularization, we calculate the energy of assigning each class to each surface element.
Slide13
Energy Function
The pairwise smoothness term
unary data term
Slide14
Improvement was seen both with State of The Art CNNs and pre-trained CNNs.
Eigen improved class average accuracy from 59.9% to 63.2% (+3.3%) RGBD-CNN is (+2.3%)This confirms initial hypothesis that contour information, can increase accuracy of segmentation. Success!
Slide15
Conditional Random Fields regularization results were inconclusive
Added a massive amount of computational complexity.
Only contributed to a marginal gain of 0.5%Partially explained by the lack of parameter optimization.
Slide16
Semantic Segmentation does not need to be updated every frame
In practice, the CNN does not need to run very often to see improvement over the current single-frame literature. (As little as once every 5 seconds)
Slide17
Experimental Conclusions
Semantic segmentation only needs to happen at a rate of around 1HzApproach surpassed any semantic segmentation system published before.Some videos showed clearer improvements from the combination than othersRun time requires some compromises
Slide18
Videos with close objects and variation in perspective resulted in the best classification
The eventual classifications were dependent on the camera being able to understand 3D space first. We can see that some classifications were harmed compared to their 2D counterparts.
Possible extension: use CNN predictions to improve ElasticFusion?
Input
Ground Truth
Single Frame
SemanticFusion
Slide19
Computational power is a limiting factor
SLAM system requires 29.3ms, or 34HzCRF 0.05Hz or 20 seconds per frame on a CPUCNN and Bayesian updates are at 10Hz on an optimized GPUNew work in compressed CNNs could bring the time required down, however we are far from real time execution on a computer less than $1000
Slide20
Sources
gitHub: https://bitbucket.org/dysonroboticslab/semanticfusion/src
T. Whelan, S. Leutenegger, R. F. Salas-Moreno, B. Glocker, and A. J. Davison, “ElasticFusion: Dense SLAM without a pose graph,” in Proceedings of Robotics: Science and Systems (RSS), 2015. R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. J. Kelly, and A. J. Davison, “SLAM++: Simultaneous Localisation and Mapping at the Level of Objects,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013. [Online]. Available: http://dx.doi.org/10.1109/CVPR.2013.178
D. Eigen and R. Fergus, “Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Archi- tecture,” in Proceedings of the International Conference on Computer Vision (ICCV), 2015. John McCormac, Ankur Handa, Andrew J. Davison and Stefan Leutenegger, “SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks” in Robotics and Automation (ICRA), 2017 IEEE International Conference on, 2017.
Slide21
https://www.youtube.com/watch?v=cGuoyNY54kU
Video of the system in action!