S M Ali Eslami Nicolas Heess John Winn CVPR 2012 Providence Rhode Island A Strong Model of Object Shape What do we mean by a model of shape A probabilistic distribution Defined on binary ID: 142571
Download Presentation The PPT/PDF document "The Shape Boltzmann Machine" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
The Shape Boltzmann Machine
S. M. Ali EslamiNicolas HeessJohn Winn
CVPR 2012Providence, Rhode Island
A Strong Model of Object ShapeSlide2
What do we mean by a model of shape?
A probabilistic distribution:Defined on binary
imagesOf objects not patchesTrained using limited training data
2Slide3
Weizmann horse dataset
3Sample training images
327 imagesSlide4
What can one do with an ideal shape model?
4Segmentation (due to probabilistic nature)Slide5
What can one do with an ideal shape model?
5Image completion (due to generative nature)Slide6
What can one do with an ideal shape model?
6Computer graphics (due to generative nature)Slide7
What is a strong model of shape?
W
e define a
strong
model of object shape as one which meets two requirements:
7
Realism
Generates samples
that look realistic
Generalization
Can generate samples that
differ from training images
Training images
Real distribution
Learned distributionSlide8
Existing shape models
8A comparison
Realism
GeneralizationGlobally
LocallyMean
✓
Factor Analysis
✓
✓
Fragments
✓
✓
Grid MRFs/CRFs
✓
✓
High-order
potentials
~
✓
✓
Database
✓
✓
ShapeBM
✓
✓
✓Slide9
Existing shape models
9Most commonly used architectures
MRFMean
sample from the model
sample from the modelSlide10
Shallow and Deep architectures
10Modeling high-order and long-range interactions
MRF
RBM
DBMSlide11
Deep Boltzmann Machines
ProbabilisticGenerativePowerfulTypically trained with many examples.
We only have datasets with few training examples.11
DBMSlide12
From the DBM to the ShapeBM
12Restricted connectivity and sharing of weights
DBM
ShapeBMLimited training data, therefore reduce the number of parameters:
Restrict connectivity,Tie parameters,Restrict capacity.Slide13
Shape Boltzmann Machine
13Architecture in 2D
Top hidden units capture object poseGiven the top units,
middle hidden units capture local (part) variabilityOverlap helps prevent discontinuities at patch boundariesSlide14
ShapeBM inference
14Block-Gibbs MCMC
image
reconstructionsample 1sample n
Fast: ~500 samples per secondSlide15
ShapeBM learning
Maximize with respect to Pre-training
Greedy, layer-by-layer, bottom-up,‘Persistent CD’ MCMC approximation to the gradients.Joint trainingVariational + persistent chain approximations to the gradients,Separates learning of local and global shape properties.
15Stochastic gradient descent
~2-6 hours on the small datasets that we considerSlide16
ResultsSlide17
Weizmann horses – 327 images
– 2000+100 hidden units
Sampled shapes
17
Evaluating the Realism criterion
Weizmann horses – 327
images
Data
FA
Incorrect generalization
RBM
Failure to learn variability
ShapeBM
Natural shapes
Variety of poses
Sharply defined details
Correct number of legs (!)Slide18
Weizmann horses – 327 images
– 2000+100 hidden units
Sampled shapes
18
Evaluating the Realism criterion
Weizmann horses – 327
images
This is great, but has it just overfit?Slide19
Sampled shapes
19Evaluating the Generalization criterion
Weizmann horses – 327 images – 2000+100 hidden units
Sample from the ShapeBMClosest image in training dataset
Difference between the two imagesSlide20
Interactive GUI
20Evaluating Realism and Generalization
Weizmann horses – 327 images – 2000+100 hidden unitsSlide21
Further results
21Sampling and completion
Caltech motorbikes – 798 images – 1200+50 hidden unitsTraining
imagesShapeBM samplesSamplegeneralization
ShapecompletionSlide22
Imputation scores
Collect 25 unseen horse silhouettes,
Divide each into 9 segments,Estimate the conditional log probability of a segment under the model given the rest of the image,Average over images and segments.
22Quantitative comparisonWeizmann horses – 327 images – 2000+100 hidden units
Mean
RBM
FA
ShapeBM
Score
-50.72
-47.00
-40.82
-28.85Slide23
Multiple object categories
Train jointly on 4 categories without knowledge of class:
23Simultaneous detection and completionCaltech-101 objects – 531 images – 2000+400 hidden units
Shape
completion
Sampled
shapesSlide24
What does h2 do?
Weizmann horsesPose information
24
Multiple categories
Class label information
Number of training images
AccuracySlide25
Summary
Shape models are essential in applications such as segmentation, detection, in-painting and graphics.The ShapeBM characterizes a strong model of shape:Samples are
realistic,Samples generalize from training data.The ShapeBM learns distributions that are qualitatively and quantitatively better than other models for this task.25Slide26
Questions
MATLAB GUI available athttp://
arkitus.com/Ali/Slide27
Questions
"The Shape Boltzmann Machine: a Strong Model of Object Shape"S. M. Ali Eslami, Nicolas Heess and John Winn (2012)
Computer Vision and Pattern Recognition (CVPR), Providence, USAMATLAB GUI available athttp://arkitus.com/Ali/Slide28
Shape completion
28Evaluating Realism and Generalization
Weizmann horses – 327 images – 2000+100 hidden unitsSlide29
Constrained shape completion
29Evaluating Realism and Generalization
Weizmann horses – 327 images – 2000+100 hidden units
ShapeBM
NNSlide30
Further results
30Constrained completion
Caltech motorbikes – 798 images – 1200+50 hidden units
ShapeBM
NN