Lower bounds against convex - PowerPoint Presentation

Slide1

Lower bounds against convex relaxationsvia statistical query complexity

Based on:V. F., Will Perkins, Santosh Vempala. On the Complexity of Random Satisfiability Problems with Planted Solutions. STOC 2015V. F., Cristobal Guzman, Santosh Vempala. Statistical Query Algorithms for Stochastic Convex Optimization. SODA 2017V. F. A General Characterization of the Statistical Query Complexity. arXiv 2016

Vitaly Feldman IBM Research – AlmadenSlide2

The planBoolean constraint satisfaction problems

Convex relaxationsComparison with lower bounds against LP/SDP hierarchies(Known) sign-rank lower bounds via SQ complexitySlide3

MAX-CSPs

-SATGiven:

, where clause is OR of

(possibly negated) variables Is satisfiable?

MAX--CSPFind

for

-

ary

predicates

-SAT refutation

If is satisfiable output YES.If is random output NO with prob : -clauses are chosen randomly and uniformly from of all -clauses Unsatisfiable w.h.p. for

Best poly-time algorithm

uses clauses [Goerdt,Krivelevich 01; Coja-Oglan,Goerdt,Lanka 07; Allen,O’Donnell,Witmer 15]Conjectured to be hard [Feige 02]

 Slide4

Convex relaxation for MAX-CSPs

Objective-wise mapping:Denote

Which

allow such mappings?

Clause

over

Convex

function

over a convex body  Refutation gap :

:

(

) 

 Slide5

Outline

Optimization of in the stochastic setting has low SQ complexity 

Lower bound on statistical query complexity of stochastic -SAT refutation

Convexrelaxation

-

-

Convex optimization algorithmsYES: the support of is satisfiable Slide6

Lower bound example I

-Lipschitz convex optimization needs

-Lipschitz

convex optimization needs

For

and any convex

all convex funcs s.t. Then  

For

and any convex

all convex funcs s.t.

Then

 Slide7

Lower bound example II

General convex optimization needs

For

and any convex

all convex

funcs

over

with range

Then  Slide8

Lower bounds from algorithms

-Lipschitz convex optimization needs

convex optimization

needs

General convex

optimization

needs

 Random walks Center of gravityProjected gradient descentEntropic mirror descent(multiplicative weights) Slide9

Statistical queries [Kearns ‘93]

over

 Slide10

Statistical queries [Kearns ‘93]

is tolerance of the query;

SQ algorithm

oracle

Problem

:

If exists

a SQ algorithm that

solves

using

queries to

 Slide11

Statistical queries [Kearns ‘93]

is tolerance of the query;

SQ algorithm

oracle

Applications:

Noise-tolerant learning

[Kearns

93; …]

Private data analysis

[

Dinur,Nissim

03;

Blum,Dwork,McSherry,Nissim

05;

DMN,Smith

06

;

]

Distributed/low communication/memory ML

[

Ben-

David,Dichterman

98; Chu et al., 06;

Balcan,Blum,Fine,Mansour

12;

Steinhardt,G

.

Valiant,Wager

15;

F

. 16

]

Evolvability

[L. Valiant 06;

F

. 08; …]

Adaptive data analysis

[

Dwork

,

F

.,

Hardt,Pitassi,Reingold,Roth

14; …]

 Slide12

Outline

Optimization of in the stochastic setting has low SQ complexity 

Lower bound on SQ complexity of stochastic -SAT refutation

Convexrelaxation

-

-

Convex optimization algorithmsSlide13

Stochastic convex optimization (SCO)

Convex body Class of convex functions over

What is the SQ complexity of

?

:

Unknown

distribution

over

-minimize

over :Find s.t. Standard: Given i.i.d. samples SQ: Given  

 Slide14

SQ algorithms for

SCO

Reduction from an optimization oracle to SQ oracle

Direct analysis of an existing SCO algorithm

New/modified algorithm

Hard work

ReductionsSlide15

Zero-order/value oracle

𝜂-approximate value oracle for 𝑓 over Given

returns

,

If for all

then for any over , can simulate [P.Valiant ‘11] 

 Slide16

Corollaries

Known results for arbitrary and Ellipsoid-based:

queries to

[Nemirovski,Yudin 77;

Grotschel,Lovasz,Schrijver

88]

Random walks:

queries to

[Belloni,Liang,Narayanan,Rakhlin 15; F.,Perkins,Vempala 15] Corollary: For = {all convex funcs over with range } In high dimension weaker than full access/gradient oracle [Nemirovski,Yudin

‘77; Singer,Vondrak

‘15; Li,

Risteski ‘16]Slide17

First-order/gradient oracles

If then equivalent to

To implement

need to estimate

within

in

Assuming that

!

Global approximate gradient oracle of

over Given returns , s.t. for all  

 Slide18

Mean vector estimation

Easy case: Coordinate-wise estimation: for every , ask query

Let be the answer of

. Then

What about

?

Coordinate-wise estimation requires

In contrast,

samples suffice

Mean estimation in :Given distribution over Find s.t. , where

 Slide19

Kashin’s representation [Lyubarskii

, Vershynin 10]Use coordinate-wise mean-estimation in Kashin’s representation: For every , ask query

to

.

Vectors

provide

Kashin’s

representation with level

if : tight frame: low dynamic range: and  Corollary: Mean estimation in

can be solved using

queries to

 Thm [LV 10]: There exists Kashin’s representation of level for and can be constructed efficiently

 Slide20

Other norms

normsWhat about the general case?Always in

Mostly openDifferent from sample complexity for some hard to compute norms Slide21

Example corollaries

-Lipschitz SCO:

For any convex

all convex funcs s.t.

-Lipschitz SCO

:

For any

convex all convex funcs s.t.

 Slide22

Outline

Optimization of in the stochastic setting has low SQ complexity 

Lower bound on SQ complexity of stochastic -SAT refutationFrom SQ dimension to SQ complexityLower bound on SQ dimension of

-SAT

 Convexrelaxation

-

-

Convex optimization algorithmsSlide23

Stochastic -SAT refutation

 If s.t. the support of is satisfiable, output YESIf

, output NO with prob

 Slide24

SQ dimension

One-vs-many decision problems:Let be a set distributions over and be a reference distribution

over

:

for an input distribution

decide if

Fixed-distribution PAC learning

[

Blum,Furst,Jackson,Kearns,Mansour,Rudich 95; …]General statistical problemsLower bounds [F.,Grigorescu,Reyzin,Vempala,Xiao 13; FPV 15]Characterization [F. 16] Slide25

If

then any algorithm that solves

given access to

requires

>

queries

SQ dimension

of

[

F

. 16]

 Slide26

o

f-SAT refutation 

is a degree-

(multilinear) polynomial of with constant term

Concentration properties of low-degree polynomials over

:

for all

,

Hard family of distributions:

uniform

over all

-clauses in which satisfies an odd number of literals

Thm:

-

-

 Slide27

Outline

Optimization of in the stochastic setting has low SQ complexity 

Lower bound on SQ complexity of stochastic -SAT refutation

Convexrelaxation

-

-

Convex optimization algorithmsSlide28

Comparison with known approaches

Same:Objective-wise relaxation to functions over a fixed Incomparable/complementary: Known SQ based

Linear functions

Convex functions:

is

a

polytope

with bounded number of facets

is any convex body.

is boundedAssumes mapping s.t. and gapAssumes an gap in optimization outcomes“Variance”/“Overfitting”“Bias”/“Model misspecification”“Variance”/“Overfitting”“Bias”/“Model misspecification”Sherali-Adams,SOS/Laserre hierarchies: [Grigoriev 01; Shoenebeck 08; Charikar,Makarychev,Makarychev 09; O’Donnell,Witmer 14]LP extended formulations:

[Chan,Lee,Raghavendra,Steurer 13;

Kothari,Meka,Raghavendra

16][Barak,Moitra ’16]Slide29

Sign-rank lower bounds via SQ complexity

Dimension complexity:Let be a set of -valued functions over is the lowest

such that exists a mapping such that:

exists

, such that

,

Define

Then

For a matrix

,

Corollary:

Proved

by Forster

[2001]

Halfspaces over

can

be PAC learned

in

[

Blum,Frieze,Kannan,Vempala

96]

Learning of

(parity functions)

not

in

[Kearns 93; BFJKMR 95]

 Slide30

Conclusions

Convex relaxations fail for XOR constraint optimizationSQ complexity lower bounds bridge between algorithms and structural lower boundsExtensionsOther MAX--CSPsStronger -wise reductions [

F., Ghazi ‘17]Many open problems