Multicriterion representation for scene understanding Moos Hueting Aron Monszpart Nicolas Mellado University College London httpswwwimagestoreuclacukhomepcache10002aablackad80ejpg ID: 327400
Download Presentation The PPT/PDF document "MCGraph" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
MCGraph: Multi-criterion representation for scene understandingMoos Hueting∗ Aron Monszpart∗ Nicolas MelladoUniversity College London
https://www.imagestore.ucl.ac.uk/home/pcache/10002/aa/black_ad80e.jpgSlide2
MotivationIndoor scene analysisAcquisition easierLess constrained scenes, growing complexityTarget more complex: object counting segmentation, labellingSlide3
MotivationIndoor scene analysisAcquisition easierLess constrained scenes, growing complexityTarget more complex: object counting segmentation, labellingTypical processing:Vision (SLAM)3D2D images, temporal information, depth maps, pointclouds, 3D models
pointclouds, shapes, 3D meshes
localization, mapping, reconstruction, model recognition and fitting
local geometric or multi-scale features,
abstraction by primitives, shapes from collections,
inference of categorical knowledge,
functional information (interaction)Slide4
MotivationFew types of information at a timeComplex scene understanding different information domains at the same timeStacked, disjoint abstraction layersJoint representation with mutual refinementConcurrent information processing > iterativeSlide5
MCGraphMCGraph – a unified Multi-Criterion data representationStructure and meaning modelled separately,connected fullyFor understanding and processing of large-scale 3D scenesA standardized structure to format the data created by our community
Prior knowledge
Discovered knowledge
Abstraction graphSlide6
Related work - viewpoints2D3DArrangements of …features [Felzenszwalb10] , …recurring parts in shape collections [Zheng14] , …Slide7
Related work - viewpoints2D3DArrangements of …features [Felzenszwalb10] , …recurring parts in shape collections [Zheng14] , …Abstraction by …super-pixels [Zitnick04], …regions [Gould09, Kumar10] , …features[Hoiem08] , …planes [Gallup07] , …primitives [Schnabel07] , …cuboids [Fidler12,Shao14] , …Slide8
Related work - viewpoints2D3DArrangements of …features [Felzenszwalb10] , …recurring parts in shape collections [Zheng14] , …Abstraction by …super-pixels [Zitnick04], …regions [Gould09, Kumar10] , …features[Hoiem08] , …planes [Gallup07] , …primitives [Schnabel07] , …cuboids [Fidler12,Shao14] , …
Graph bases representation of …
RGB segmentation
[
FelzenszwalbHuttenlocher04], [Rother04], …
RGBD segmentation
[Zheng13], …
inter-shape analysis [Mitra14], …
joint shape segmentation [Huang11], ...
shape collections [Fish*14], …Slide9
Related work - criteriaFew criteriaAppearance co-occurrence, relative spatial layout [Hedau10]Primitive abstraction, physics [Gupta10]Semantic labelling [Huang13]Slide10
Related work - criteriaFew criteriaAppearance co-occurrence, relative spatial layout [Hedau10]Primitive abstraction, physics [Gupta10]Semantic labelling [Huang13]Multi-criteria 2DAppearance, shape, context [Shotton09]Verbal descriptions of actors, actions + object properties, relations [Zitnick13]Material, function, spatial envelope [Patterson14]Slide11
Related work - criteriaFew criteriaAppearance co-occurrence, relative spatial layout [Hedau10]Primitive abstraction, physics [Gupta10]Semantic labelling [Huang13]Multi-criteria 2DAppearance, shape, context [Shotton09]Verbal descriptions of actors, actions + object properties, relations [Zitnick13]Material, function, spatial envelope [Patterson14]Multi-criteria 3DRegularities of shape collections, function [Laga13]Intra and inter-object symmetries, physics, function [Mitra10]Slide12
Related work – formalisationInformation fusionMulti-Criteria Decision Analysis [Doumpos13]Hypergraphs [Zhang14]Low-level processingHmida et al. [2013]Separates knowledge from abstractionProcessing and representation coupled[Hmida et al. 2013]Slide13
Classic modelPrior knowledge: Graph nodes (object types)Graph edges (relationship types)Discovered knowledge:Graph layoutCoupled representation of a priori and discoveredSlide14
Proposed modelPrior knowledge, discovered knowledge and abstraction separateInspired by graph databasesLabelling = edge between object and labelSupports multi-criteria processingSlide15
1. Abstraction graphRepresents objects and relationsAbstractions may be connected to segments of dataSegments can be overlappingObject hierarchiesSlide16
2. Knowledge graphEncodes prior knowledge -> “knowledge units”HierarchicalMulti-criterion by separate sub-graphsIs defined a priori - portableCan have internal edgesSlide17
3. Relation setRepresents discovered knowledge, “labellings”Abstractions labels, or relation nodes labelsEdge can store parameters to represent an instantiation (i.e. primitive size)Slide18
ExampleSegmentation knowledge sub-graphSuper-pixelsRegionsSlide19
ExampleBounding boxBounding boxSegmentation knowledge sub-graphSlide20
ExampleAbstraction knowledge sub-graphSlide21
ExampleRelations knowledge sub-graphSlide22
ExampleLegs same size, and parallelSlide23
Common knowledge sub-graphsPrimitive proxiesI.e. primitive – polyhedron – cuboid, pyramidHierarchy induces high inference powerSemanticHierarchical object labelsClassification, function, etc.RelationshipsCoaxial, co-planar, equiangular [Li11]Covers, supports, occluded by, belong together [Gupta10]Can be searched by sub-graph matching [Schnabel08]Slide24
Common knowledge sub-graphsPrimitive proxiesI.e. primitive – polyhedron – cuboid, pyramidHierarchy induces high inference powerSemanticHierarchical object labelsClassification, function, etc.RelationshipsCoaxial, co-planar, equiangular [Li11]Covers, supports, occluded by, belong together [Gupta10]Can be searched by sub-graph matching [Schnabel08]Large freedom, high representational powerUse cases: Primitive abstraction, RGBD annotationSee paperSlide25
Extension 1. Scene collections for assisted livingRobot assistant needs to be able to reach the subject at all timesPro-active discovery of dynamic environmentNeeds to identify danger sourcesVision, scene-collections, intra-domain inferenceMCGraph: “Objects that easily tip over, and incur danger, when in contact with water”
Movable
Electric, movable, unstable
Emits waterSlide26
Extension 2. - Multi-criteria multi-scaleMulti-scale analysis of scene understanding [Mitra14]HKS [Sun09], GLS [Mellado12]Multi-scale similarity queries [Hou12]Only spatial domain, controlled environmentsOpen-world problemMulti-criteria multi-scale look-ups
Scan
?
Scale
Time
Location
ClassificationSlide27
Extension 3. Prior knowledge for object registrationGeometric priors, priors on part-relations, functionIf you discover, what you are scanning, you can use the extra information to enhance qualitylooks like engine
generic engine model prior
will have axes and wheels
looks like car
generic car model prior
function
l
ookup working CAD model
will have shiny material
d
ecrease RGB weight in registration
Co-occurrenceSlide28
ConclusionLimitationsPortability -> standardizationDiversity -> efficiencyData onlineFlexible and extendable representationA data structure to standardize storage of annotated dataCan start a discussion, debate and movement how to harness the powers of multi-criteria problem representations, focused on 3D scene understandingSlide29
Thank you!http://geometry.cs.ucl.ac.uk/projects/2014/mcgraph/Slide30Slide31Slide32Slide33Slide34