Estimation amp Lifted Metrics J Saketha Nath IITH Joint Work with Pratik Jawanpuria Microsoft INDIA Piyushi Manupriya IITH Optimal Transport ID: 917466
Download Presentation The PPT/PDF document "Kernel Regularized Optimal Transport :" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Kernel Regularized Optimal Transport :Estimation & Lifted Metrics
J. Saketha Nath (IITH)
Joint Work with Pratik
Jawanpuria
(Microsoft, INDIA),
Piyushi
Manupriya
(IITH)
Slide2Optimal Transport
`
s
ource
t
arget
Optimal Transport
`
s
ource
t
arget
2
3
Optimal
Transport Plan
Optimal Transport
`
Cost function
is given
Optimal Transport
`
s
ource
t
arget
Optimal
Transport Plan
Optimal Transport
`
s
ource
t
arget
Optimal
Transport Plan
Optimal Transport
s
ource
target
Optimal
Transport Plan
Adapted from Gabriel
Peyré
and Marco
Cuturi
.
Computational Optimal Transport.
Slide8Application: Color Transfer
https://optimaltransport.github.io/slides-peyre/Applications.pdf
Slide9Application: Color Transfer
https://optimaltransport.github.io/slides-peyre/Applications.pdf
Slide10Application: Color Transfer
https://optimaltransport.github.io/slides-peyre/Applications.pdf
Application: Color Transfer
https://optimaltransport.github.io/slides-peyre/Applications.pdf
Slide12ML Application: Domain Adaptation
[
Courty
et.al., PAMI’15]
Slide13ML Application: Domain Adaptation
[
Courty
et.al., PAMI’15]
Slide14ML Application: Domain Adaptation
Optimal Transport
[
Courty
et.al.
, PAMI’15]
Slide15ML Application: Domain Adaptation
Optimal Transport
[
Courty
et.al.
, PAMI’15]
Slide16ML Application: Domain Adaptation
Courty
et.al., PAMI’15Courty et.al., NIPS’17Redko et.al., ECML’17Gautheron et.al., ECML’18Yan et.al., IJCAI’18Redko et.al., AISTATS’19Lee et.al., CVPR’19Kerdoncuff et.al., IJCAI’20Dhouib et.al., ICML’20Taherkhani et.al., ECCV’20Xu et.al., CVPR’20Cheng et.al., NeuroComp’20Montesuma et.al., CVPR’21Nguyen et.al., UAI’21Luo et.al., CVPR’21
Slide17-
Wasserstein Metric
needs to be a (ground) metric
-
Wasserstein Metric
needs to be a (ground) metric
Application: Generative Models
Unsupervised Wasserstein loss
Unsupervised Wasserstein loss
Arjovsky
et.al.
, ICML’17
I Gulrajani et.al., NIPS’17Petzka et al., ICLR’18
Genevay et.al., AISTATS’18Sanjabi et.al., NeurIPS’18
Tolstikhin et al., ICLR’18Dukler et.al., ICML’19Chen et.al., ICML’19Bunne et.al., ICML’19
Wu et.al., CVPR’19Liu et.al., ICCV’19Balaji et.al., NeurIPS’20
Xu et.al., NeurIPS’20Shen et.al., NeurIPS’20Fatras et.al., AISTATS’20Patrini et al.
, UAI’20Rout et.al., ICLR’22
Slide22Wasserstein Interpolation
, Barycenter problem
Adapted from Gabriel
Peyré
and Marco
Cuturi. Computational Optimal Transport.
Slide23Application: Shape interpolationSolomon et.al., Graphics’15
Slide24-
Wasserstein Metric = Kantorovich Metric
needs to be a (ground) metric
IPM metrics
needs to be a (ground) metric
MMD Metric
needs to be a characteristic kernel
Wasserstein vs MMD
Non-
Hilbertian
Lifts ground metricLocal minima issue less severeEstimation very challengingNumerical optimization
Hilbertian
Lifts similarity
Local minima issue more severe
Estimation is
easyClosed form
Wasserstein vs MMD
Non-
Hilbertian
Lifts ground metricLocal minima issue less severeEstimation very challengingNumerical optimization
Hilbertian
Lifts similarity
Local minima issue more severe
Estimation is
easyClosed form
Wasserstein vs MMD
Non-
Hilbertian
Lifts ground metricLocal minima issue less severeEstimation very challengingNumerical optimization
Hilbertian
Lifts similarity
Local minima issue more severe
Estimation is
easy
Closed form
Bottou
et.al., 2018
Slide30Wasserstein vs MMD
Non-
Hilbertian
Lifts ground metricLocal minima issue less severeEstimation very challengingNumerical optimization
Hilbertian
Lifts similarity
Local minima issue more severe
Estimation is
easyClosed form
Bottou
et.al., 2018
Slide31MMD regularized Optimal Transport
MMD regularized Optimal Transport
MMD regularized Optimal Transport
MMD regularized Optimal Transport
KL regularized Optimal Transport
Gaussian Hellinger-Kantorovich Metrics,
Liero
et.al.,
Inventiones
mathematicae
, 2018
needs to be Euclidean metric
MMD regularized Optimal Transport
MMD regularized OT: Statistical Estimation
Theorem:
Let
be a universal, normalized kernel and cost be continuous. Then,
.
[Nath &
Jawanpuria
, NeurIPS’20]
Slide38MMD regularized OT: Statistical Estimation
Theorem:
Let
be a universal, normalized kernel and cost be continuous. Then,
.
Smoothness
:
[Hutter&Rigollet’20]
Low-rank
:
[Weed&Rigollet’19]
Entropic Regularization
:
[
Genevay
et.al.’19]
MMD regularized OT: Statistical Estimation
Theorem:
Let
be a universal, normalized kernel and cost be continuous. Then,
.
A kernel embedding based viewpoint
More general regularization framework
Representer theorem
Dimension-free recovery of optimal plan/map
Out-of-sample estimation
[Nath & Jawanpuria, NeurIPS’20]
Slide40IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath &
Jawanpuria
, arXiv’22]
Slide41IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]Theorem: For any ground metric
, and any IPM,
is a norm-induced lifted metric over measures.
IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]Theorem: For any ground metric
, and any IPM,
is a norm-induced lifted metric over measures.
IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]Theorem: For any ground metric
, and any IPM,
is a norm-induced lifted metric over measures.
s.t.
IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]Theorem: For any ground metric
, and any IPM,
is a norm-induced lifted metric over measures.
s.t.
IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]Theorem: For any ground metric
, and any IPM,
is a norm-induced lifted metric over measures.
IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]
IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]
IPM regularized OT: Lifted Metrics
[
Manupriya
, Nath & Jawanpuria, arXiv’22]
Theorem:
Let
be a bounded metric
, and the witness functions,
, in the IPM’s generating set satisfy the uniform bound
. Then, for any
,
is a lifted metric over measures.
Simulations: Estimation
[Nath &
Jawanpuria
, NeurIPS’20]
Slide50Simulations: Metrics
MMD
Wasserstein
Bottou et.al., 2018
Slide51Simulations: Metrics
MMD
Wasserstein
Proposed
[Manupriya, Nath & Jawanpuria, arXiv’22]
Slide52Simulations: Downstream Application
[
Manupriya
, Nath & Jawanpuria, arXiv’22]
Slide53Summary
IPMs received hardly any attention compared to
divergences
MMD+OT alleviates the curse of dimensionality in estimationNew IPM+OT metrics show promise in initial ML applicationsFuture Work: Topological and Computational aspects
Slide54THANK YOU