Fast edge-directed - PowerPoint Presentation

Slide1

Fast edge-directed single-image super-resolution

Mushfiqur Rouf1 Dikpal Reddy2 Kari Pulli2 Rabab K Ward11University of British Columbia 2Light co

1Slide2

2x2

Single image super-resolutionSlide3

OverviewSlide4

Inverse problem formulation

4

Data fitting

Priors

Sparse gradient

Smooth contourSlide5

Inverse problem formulation

5Latent image

Sampling function

Observed image

Downsampling

ConvolutionSlide6

Fast edge-directed SISR

CombinesSparse gradient

“Smooth contour”

Small overhead

Prior more general purpose than SISR

SISR methods

Filtering / forward methods

Bilinear,

bicubic

, …

Anisotropic methods

Deep learning methods

Inverse / reconstruction methods

Total variation (TV) optimization

Markov random fields

Bilateral filtering + optimization

6Slide7

Image formation model

7Latent image

Anti-aliasing filter

Observed image

Downsampling

Convolution

Priors

Sparse gradient

Smooth contour

=

Forward model

[http

://

www.imaging-resource.com/PRODS/pentax-k3/pentax-k3SELECTIVE_LPF.HTM]Slide8

Inverse problem formulation

8Latent image

Anti-aliasing filter

Observed image

Downsampling

Convolution

Priors

Sparse gradient

Smooth contour

Only sharpens image across edges.

Edges could be jaggy.

Also smooths edge structure along the edge contours.

No jaggy edge.

Complementary

At what cost?

Is it worth the cost?

Is the method useful?

Data fittingSlide9

Inverse problem formulation

9Latent image

Anti-aliasing filter

Observed image

Downsampling

Convolution

Smooth contour

Sparse gradient

Total variation

Gradient

TV-only SR

Why not bilateral filter here?

Data fittingSlide10

Inverse problem formulation

10Latent image

Anti-aliasing filter

Observed image

Downsampling

Convolution

Total variation

Gradient

anisotropic interpolation

Downsampling

Smooth contour

L

ittle computation overhead

Improved reconstruction

TV-only SR

Proposed

Sparse gradient

Data fittingSlide11

Smooth contour prior – at what cost?

11

Edge directed interpolation

[Li and Orchard 2001]

Sparse gradient prior only

(TV optimization)

Improved reconstruction

Our improvement

Smooth contour

L

ittle computation overhead

L

ittle computation overhead

Improved reconstructionSlide12

Smooth contour prior

12

Star chart

Ground truth

Bicubic

TV optimization

EDI

Ours (sparse gradient and smooth contour)

LR inputSlide13

Inverse problem formulation

13Latent image

Anti-aliasing filter

Observed image

Downsampling

Convolution

Total variation

Gradient

anisotropic interpolation

Downsampling

Improved reconstruction

Sparse gradient

Data fitting

Smooth contour

L

ittle computation overheadSlide14

Choice of anisotropic interpolation

Goals:Local calculationsDirect estimation of interpolation weightsFast implementationWe choose: [Li and Orchard 2001]Newer versions of the method overkill

14Slide15

Intro to EDI [Li and Orchard 2001]

Edge directed interpolationDetects local “edge orientation”Sliding-window linear least-squares regressionsInterpolates along the edge15

[Li and Orchard 2001]Slide16

Intro to EDI [Li and Orchard 2001]

Two step process for upsampling16Slide17

Intro to EDI [Li and Orchard 2001]

Sliding window process17

?Slide18

Intro to EDI [Li and Orchard 2001]

DownsidesEdge misestimation artifactsFixed 2x2 upsamplingUpsampled image not sharpPerforms very well where the edge estimates are accurate

18

4x4

[http

://

chiranjivi.tripod.com/EDITut.html]Slide19

Our improvements to EDI

Wrapped up in a prior and used a data fitting term and a complementary priorRemoves misestimation artifactsRegularized regression

Speedup

 iterative application possible

19Slide20

Smooth contour prior

Combatting EDI artifactsTurn EDI into a prior – “Smooth contour prior”Add a data fitting term Add a complementary prior: sparse gradient20Slide21

EDI Speedup

Original EDI too slow to use iterativelyWe propose a speedup:Dynamic programming: Remove costly overlapping recalculations21Slide22

Inverse problem formulation

22Latent image

Anti-aliasing filter

Observed image

Downsampling

Convolution

Total variation

Gradient

Edge directed

upsampling

Downsampling

Smooth contour

L

ittle computation overhead

Improved reconstruction

Sparse gradient

Data fittingSlide23

Primal dual optimization

ConvexMixture of L1 and L2 priors --> Primal dual method23Slide24

Primal dual optimization

24Slide25

Primal dual optimization

25

Primal dual formSlide26

Primal dual optimization

[Chambolle and Pock 2010]26

Standard TV optimization

Our priorSlide27

Results - dyadic

27

Ground truth

2x2Slide28

Results - dyadic

28

[He-Siu 2011]

PSNR: 32.25

SSIM

:

0.9858

2x2Slide29

Results - dyadic

29

[Kwon et al. 2014]

PSNR

:

34.97

SSIM

: 0.9961

2x2Slide30

Results - dyadic

30

Our method

PSNR: 35.13

SSIM

: 0.9928

2x2Slide31

Results - nondyadic

31

Ground truth

3x3Slide32

Results - nondyadic

32

[Yang 2010]

PSNR: 23.28

SSIM

:

0.9041

3x3Slide33

Results - nondyadic

33

[

Kwon et al. 2014

]

PSNR

:

24.17

SSIM

:

0.9218

3x3Slide34

Results - nondyadic

34

PSNR

:

23.93

SSIM

:

0.9122

Our method

3x3Slide35

Comparisons with deep learning

35

Ground truth

4x4Slide36

Comparisons with deep learning

36

Our method

PSNR

:

32.95

SSIM

:

0.9442

4x4Slide37

Comparisons with deep learning

37

[Dong

et al. 2014

]

PSNR

: 33.12

SSIM

: 0.9504

4x4Slide38

Comparisons with deep learning

38

[

Timofte

et

al. 2014

]

PSNR

: 33.28

SSIM

: 0.9513

4x4Slide39

Conclusions

Proposed a novel natural image priorFast. Complementary to sparse gradient priorAny anisotropic upsampling method can be usedPotentially deep learning methods? (Future work!)Application in SISRSimilar image reconstruction problems (future work)39Slide40

Fast edge-directed single-image super-resolution

Mushfiqur Rouf1 Dikpal Reddy2 Kari Pulli2 Rabab K Ward11University of British Columbia 2Light co

40

Thanks!