From Colored Fields to Thin Junction Trees Yucheng Low Arthur Gretton Carlos Guestrin Joseph Gonzalez Gibbs Sampling Geman amp Geman 1984 Sequentially for each variable in the model ID: 409125
Download Presentation The PPT/PDF document "Parallel Gibbs Sampling" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Parallel Gibbs SamplingFrom Colored Fields to Thin Junction Trees
Yucheng
Low
Arthur
Gretton
Carlos
Guestrin
Joseph GonzalezSlide2
Gibbs Sampling [
Geman
&
Geman
, 1984]
Sequentially
for each variable in the model
Select
variable
Construct conditional given
adjacent assignments
Flip coin and update
assignment to
variable
2
Initial AssignmentSlide3
From the original paper on Gibbs Sampling:
“…the MRF can be divided into collections of [variables] with each collection assigned to an independently
running asynchronous processor.”
Converges to the
wrong
distribution!
-- Stuart and Donald
Geman
, 1984.
3Slide4
The problem with Synchronous Gibbs
Adjacent variables cannot be sampled
simultaneously.
Strong Positive
Correlationt=0
Parallel
Execution
t=2
t
=3
Strong Positive
Correlation
t
=1
Sequential
Execution
Strong
Negative
Correlation
4
Heads:
Tails:Slide5
Time
Chromatic Sampler
Compute a k-coloring of the graphical model
Sample all variables with same color in parallel
Sequential Consistency:
5Slide6
Properties of the Chromatic Sampler
Converges to the correct distributionQuantifiable
acceleration in mixingTime to updateall variables once
# Variables
# Colors# Processors
6Slide7
t
=2
t
=3
t
=4
t
=1
Properties of the
Synchronous
Gibbs Sampler
on
2-colorable
models
We can derive two
valid
chains:
Strong Positive
Correlation
t
=0
Invalid
Sequence
t
=0
t
=1
t
=2
t
=3
t
=4
t
=5
7Slide8
t
=2
t
=3
t
=4
t
=1
We can derive two
valid
chains:
Strong Positive
Correlation
t
=0
Invalid
Sequence
Chain 1
Chain 2
8
Properties of the
Synchronous
Gibbs Sampler
on
2-colorable
models
Converges to the
Correct DistributionSlide9
Theoretical Contributions on 2-colorable models
Stationary distribution of Synchronous Gibbs
Corollary: Synchronous Gibbs sampler is correct for single variable marginals.9
Variables in
Color 1
Variables in
Color 2Slide10
Models With Strong DependenciesSingle variable
Gibbs updates tend to mix slowly:
Ideally we would like to draw joint samples.Blocking10
Strong
Dependencies
X
1
X
2Slide11
An asynchronous Gibbs Sampler that adaptively
addresses strong dependencies.
Splash Gibbs Sampler11Slide12
Splash Gibbs SamplerStep 1:
Grow multiple Splashes in parallel:
12
Conditionally
IndependentSlide13
Splash Gibbs SamplerStep 2:
Calibrate the trees in parallel
13Slide14
Splash Gibbs SamplerStep 3:
Sample trees in parallel
14Slide15
Adaptively Prioritized SplashesAdapt the
shape of the Splash to span strongly coupled variables:
Converges to the correct distributionRequires vanishing adaptation15
Noisy Image
BFS Splashes
Adaptive SplashesSlide16
Experimental Results
Markov logic network with strong dependencies 10K Variables 28K Factors
The Splash sampler outperforms the Chromatic sampler on models with strong dependencies 16
Likelihood
Final Sample
Better
Splash
Chromatic
“Mixing”
Better
Splash
Chromatic
Speedup in Sample Generation
Better
Splash
ChromaticSlide17
Conclusions
Chromatic Gibbs sampler for models with weak dependenciesConverges to the correct distributionQuantifiable improvement in mixing
Theoretical analysis of the Synchronous Gibbs sampler on 2-colorable modelsProved marginal convergence on 2-colorable modelsSplash Gibbs sampler for models with strong dependenciesAdaptive asynchronous tree constructionExperimental evaluation demonstrates an improvement in mixing17