Sampling and Volume Computation in High Dimension - PowerPoint Presentation

Slide1

Sampling and Volume Computation in High Dimension

Santosh

Vempala

, Georgia TechSlide2

A really old problem

Given set K in n-dimensional space, estimate its volume.

E.g.,

Pyramids, wine barrels, …

Polytopes

Intersection of a polytope with ellipsoid(s)

Section of the

semidefinite

cone

Fundamental,

computational

: how much?Slide3

Computational model

Well-guaranteed Membership oracle:

Compact set

K is given by

a membership oracle: answers YES/NO to “

a point Numbers r, R s.t. Well-guaranteed Function oracleAn oracle that returns for any A point with Numbers r, R s.t. and

 Slide4

High-dimensional problems

Input:

A set of points S in n-dimensional space

or a distribution in

A function f that maps points to real values (could be the indicator of a set) Slide5

High-dimensional problems

What is the

complexity

of computational problems

as the dimension grows

?Dimension = number of variablesTypically, size of input is a function of the dimension.Slide6

Sampling and Integration

Numerous applications in diverse areas: statistics, networking, biology, computer vision, privacy, operations research etc.

This tutorial: mathematical and algorithmic foundations of sampling and integration.Slide7

Problem 1: Optimization

Input: function f:

specified

by an oracle,

point x, error parameter . Output: point y such that  Slide8

Problem 2: Integration

Input: function f:

specified

by an oracle,

point

x, error parameter . Output: number A such that: Slide9

Problem 3: Sampling

Input: function f:

specified

by an oracle,

point x, error parameter . Output: A point y from a distribution within distance of distribution with density proportional to f. Slide10

Problem 4: Rounding

Input: function f:

specified

by an oracle,

point

x, error parameter . Output: An affine transformation that approximately “rounds” f, between two balls. Slide11

Problem 5: Learning

Input:

i.i.d

. points

with labels

from an unknown distribution, error parameter .Output: A rule to correctly label 1- of the input distribution. Slide12

High-dimensional problems

Integration (volume

)

Optimization

Learning

Rounding SamplingAll intractable in general, even to approximate.Slide13

Structure

Q. What

structure

makes

high-dimensional problems computationally tractable? (i.e., solvable with polynomial complexity)Convexity and its extensions appear to be the frontier of polynomial-time solvability.Slide14

Tutorial outline

Part 1. Intro to high dimension and convexity

Volume distribution,

logconcavity

Ellipsoids

Lower boundsPart 2. AlgorithmsRoundingVolume/IntegrationOptimizationSamplingPart 3. ProbabilityMarkov chainsConductanceMixing of ball walkMixing of hit-and-runPart 4. GeometryIsoperimetryConcentrationLocalization and applicationsOpen problemsSlide15

An o..old p

roblem

Given a measurable, compact set K in n-dimensional space and

, find a number A such that:

K is given by a well-guaranteed membershi

p oracle. Slide16

Volume: first attempt

Divide and conquer:

Difficulty: number of parts grows exponentially in n.Slide17

Volume distribution

Volume(unit cube) = 1

Volume(unit ball)

drops exponentially with n.

For any central hyperplane, most of the mass of the unit ball is within distance . Section volume decays as at distance from the origin. Slide18

Volume distribution

Volume(unit cube) = 1

Volume(unit ball)

drops exponentially with n.

For any central hyperplane, most of the mass of the unit ball is within distance .Most of the volume is near the boundary: So, “Everything interesting for a convex body happens near its boundary” --- Imre Bárány. Slide19

Volume distribution

A,B

sets

in

, their

Minkowski sum is:For a convex body, the hyperplane section at contains .What is the volume distribution? Slide20

Brunn-Minkowski inequality

Thm

.

A, B compact sets in

Suffices to prove by taking the sets to be

 Slide21

Brunn-Minkowski inequality

Thm. A, B: compact sets in

Proof. First take A, B to be cuboids, i.e.,

=

.

 Slide22

Brunn-Minkowski inequality

Thm. A, B: compact sets in

T

ake A,B to be finite unions of disjoint cuboids:

A =

and B = . Induction.There exists an axis-parallel hyperplane s.t. at least one complete cuboid of A is on either side. Partition into Translate B s.t.

(induction)

Approximate to arbitrary accuracy by a finite union of cuboids.

 Slide23

Logconcave functions

i.e., f is nonnegative and its logarithm is concave.

 Slide24

Logconcave functions

Examples:

Indicator functions of convex sets are

logconcave

Gaussian density function,

exponential functionLevel sets, are convex.Many other useful geometric properties Slide25

Properties of logconcave functions

For two

logconcave

functions f and g

Their sum might not be

logconcaveBut their product h(x) = f(x)g(x) is logconcave And so is their minimum h(x) = min f(x), g(x).Slide26

Prekopa-Leindler inequality

Prekopa-Leindler

:

t

hen

Functional version of [B-M], equivalent to it.  Slide27

Properties of logconcave functions

Convolution

is

logconcave

E.g., any marginal:This follows from Prekopa-Leindler. Slide28

Volume: second attempt: Ellipsoids

Ellipsoid #1: John

e

llipsoid of a convex body K:

E = maximum volume ellipsoid contained in K

.Thm. For any convex body K, the John ellipsoid satisfies For any centrally-symmetric K, .  Slide29

Approximate John ellipsoid

Variant of the Ellipsoid algorithm:

suppose current center is

and axes are

Check if

. If so, output current ellipsoid.If not, then intersect E with halfspace H not containing and continue algorithm (replace E with min volume ellipsoid containing ).Thm. Algorithm outputs E satisfying  Slide30

Volume via (approximate) John ellipsoid

Using the Ellipsoid algorithm, in

polytime

Then

Polytime

, exponential approximation Slide31

Ellipsoid #2: Inertial ellipsoid

For a convex body K,

matrix of inertia:

Inertial ellipsoid:

Thm (KLS95). Also a factor sandwiching, but a different ellipsoid. Shown earlier up to constants by Milman-Pajor. Slide32

Isotropic position

For any distribution with bounded second moments, there is an affine transformation to make it isotropic.

Applying this to a convex body K:

Thus K “looks like a ball” up to second moments.

How close is it really to a ball?

K lies between two balls with radii within a factor of n. Slide33

Volume via ellipsoidal approximation

The Inertial ellipsoid can be approximated to within any constant factor (we’ll see how)

Therefore:

Polytime

algorithm, approximation Can we do better? Slide34

Ellipsoid #3: Milman Ellipsoid

For two compact sets

= #translates of

needed to cover

Thm

(Milman). For any convex body K, there is an ellipsoid E s.t., Many important consequences in convex geometry.Thm (Dadush-V.). Deterministc complexity of computing a Milman ellipsoid is time, approximation.Can we do better?! Slide35

Complexity of Volume Estimation

Thm

[E86, BF87]. For any deterministic algorithm that uses at most

membership

calls to the oracle for a convex body K and computes two numbers A and B such that

, there is some convex body for which the ratio B/A is at leastwhere c is an absolute constant.Thm [DF88]. Computing the volume of an explicit polytope is #P-hard, even for a totally unimodular matrix A and rational b. Slide36

Deterministic Volume Estimation

Thm

[

BF].

For deterministic algorithms:

# oracle calls approximation factor Thm [Dadush-V.13]. Matching approximation factor of in time  Slide37

Lower bound for volume estimation

[

Elekes

]

Membership oracle answers “YES” for points in unit ball, “No” for points outside.

After m queries, volume of K is between volume of convex hull and volume of unit ball. Lemma. Need exponentially many queries! Slide38

Lower bound for volume estimation

Lemma.

Proof. Let

ball of radius

around

Claim 1. Claim 2. .

is acute, i.e.,

and

are on the same side of orthogonal hyperplane through y. Hence,

.

 Slide39

Randomized Volume/Integration

[DFK89].

Polytime

randomized

algorithm that estimates volume to within relative error with probability at least in time poly(n, ). [Applegate-K91]. Polytime randomized algorithm to estimate integral of any (Lipshitz) logconcave function. Slide40

Volume: third attempt: Sampling

Pick random samples from ball/cube containing K.

Compute fraction

c

of sample in K.

Output c.vol(outer ball).Need too many samples!Slide41

Volume via Sampling [DFK89]

Let

Estimate

each ratio with random samples

.

(Markov Chain Monte-Carlo method)

 Slide42

Volume via Sampling

Claim.

 Slide43

Variance of estimate [DF91]

using k samples in each phase.

So

samples in each phase suffice.

Total number of samples

But, how to sample?

 Slide44

Sampling

Generate a point

uniform a compact set S

with density proportional to a function f

.We will discuss this in detail shortly!Slide45

Sampling

Input: function f:

specified

by an oracle,

point x, error parameter . Output: A point y from a distribution within distance of distribution with density proportional to f. Slide46

Algorithmic applications

Given a

blackbox

for sampling

logconcave

densities, we get efficient algorithms for: RoundingConvex OptimizationVolume Computation/Integrationsome Learning problemsSlide47

Rounding via Sampling

Sample

m random points from K;

Compute

sample mean

and sample covariance matrix .Output B(K-z) is nearly isotropic.Thm. C().n random points suffice to get . [Adamczak et al; improving on Bourgain, Rudelson] I.e., for any unit vector v,  Slide48

Complexity of Sampling

Thm

.

[KLS97] For a convex body, the

ball walk with an M-warm start reaches an (independent) nearly random point in poly(n,

R, M) steps.Thm. [LV03]. Same holds for arbitary logconcave density functions. Complexity is .Isotropic transformation makes R=O(); M can be kept at O(1).KLS’97 volume algorithm:  Slide49

Progress on Volume Computation

Power

New aspects

Dyer-Frieze-Kannan 89 23 everythingLovász-Simonovits 90 16 localization Applegate-K 90 10 logconcave integrationL 90 10 ball walkDF 91 8 error analysisLS 93 7 multiple improvementsKLS 97 5 speedy walk, isotropyLV 03,04 4 annealing, wt. isoper.LV 06 4 integration, local analysisCousins-V. 13, 14 3 Gaussian coolingSlide50

Simulated Annealing

[LV03, Kalai-V.04]

To estimate

consider a sequence

with

being easy, e.g., constant function over ball. Then, Each ratio can be estimated by sampling:Sample X with density proportional to Compute

Then,

.

.

 Slide51

Annealing [LV06]

Define:

phases

The final estimate could be

, so each ratio could be

or higher. How can we estimate it with a few samples?!

 Slide52

Annealing [LV03, 06]

Lemma

.

Although expectation of Y can be large (exponential even), we need only a few samples to estimate it!Lo-Ve algorithm:

 Slide53

Variance of ratio estimator

Let

Then,

.

(would be at most 1, if F was

logconcave

…)

 Slide54

Variance of ratio estimator

Lemma. For any logconcave f and a >0, the function

is also

logconcave

.

So is logconcave and Therefore:

.

f

or

.

 Slide55

Optimization via Annealing

For

Sample with density prop. to

.

Output X with

[Abernathy-

Hazan

15

]:

Interior point with entropic barrier

[

Bubek-Eldan

]

is equivalent to simulated annealing with exponential functions.

 Slide56

Annealing

Integration:

Sample from

.

Estimate Output

Optimization

:

Sample

from

.

Output X with max f(X).

 Slide57

Progress on Volume Computation

Power

New aspects

Dyer-Frieze-Kannan 89 23 everythingLovász-Simonovits 90 16 localization Applegate-K 90 10 logconcave integrationL 90 10 ball walkDF 91 8 error analysisLS 93 7 multiple improvementsKLS 97 5 speedy walk, isotropyLV 03,04 4 annealing, wt. isoper.LV 06 4 integration, local analysisCousins-V. 13, 14 3 Gaussian coolingSlide58

A MATLAB program

[

Cousins-V.13

]

Matlab

implementation of sampling and integration“volume computation matlab exchange”http://www.cc.gatech.edu/~bcousins/volume.htmlPlease download from MATLAB Exchange if you’d like to try it.Slide59

Rotated cubesSlide60

How to Sample?

Ball walk:

At

x,

-pick random y from -if y is in K, go to yHit-and-Run: At x, -pick a random chord L through x -go to a random point y on L