Image and Video Upscaling from Local Self Examples - PowerPoint Presentation

Slide1

Image and Video Upscaling from Local Self Examples

Gilad Freedman

Raanan Fattal

Hebrew University of JerusalemSlide2

Background and

o

verview

Algorithm

description

L

ocal

self similarity

Non-dyadic filter bank

Filter

design

ResultsSlide3

Single image upscaling

1.

l

arge

2. realistic

3. faithful4. fastSlide4

Previous workparametric image modelexample based

Freeman et al. 2002

generic looking edges

Sun et al. 2008

Shan et al 2008

Fattal 2007

Glasner

et al. 2009

noisy

resultSlide5

New approach: locale example based

c

orner step edge

l

ine step edge

non

s

mooth shading

local self

s

imilarity

small upscaling ratios

1/2

4/5

new non-dyadic filter bank

Increase exemplar quality and size

maintain search locality

novel components:

l

ocal self similarity

n

on-dyadic filter bankSlide6

Background and overview

Algorithm description

L

ocal

self similarity

Non-dyadic filter bank

Filter

design

ResultsSlide7

Local self-examples upscaling

l

ow

p

ass

original

i

mage

h

igh

p

ass

i

nterpolated image

f

requency

contentSlide8

l

ow pass

h

igh

p

ass

i

nterpolated

i

mage

For each patch:

Search a

local

area for best example

Take corresponding patch

Add to interpolated image

frequency

content

Local

self-examples upscalingSlide9

Local self-examples upscaling

l

ow

p

ass

high pass

i

nterpolated

i

mage

Repeat for all patches, to fill the high frequencies

f

requency

contentSlide10

Overview

Algorithm description

Local self similarity

Non-dyadic filter bank

Filter

design

ResultsSlide11

Local self similaritycropped downscaled

 Slide12

Local self similarityPatches in original image can matched locally with ones in downscaled versionSlide13

Local examples are enoughfull imageimage

d

atabase

query

db

i

mage

l

ocal

4.0 2.9 3.55

1.6 1.05 1.05

2.7 2.05 2.05

3.3 2.96 3.06

6.5 5.61 5.61

best

m

atchesSlide14

Visual assessment – external, exact NN, localLarge externalexample database

Searching the

entire image

Searching local

regions in image

e

xternal database

g

lobal search

local

s

earchSlide15

Comparison of example search methodsSlide16

Background and overview

Algorithm description

L

ocal

self similarity

Non-dyadic

filter bank

Filter

design

ResultsSlide17

Need for non-dyadic scalings

large ratios

mixed ratios

small ratiosSlide18

Dyadic filters

1:2

full frequency content

h

igher half

l

ower half

dyadic filter bankSlide19

Non-dyadic filter bank

1:2

4:5

f

ull frequency content

h

igher part

l

ower part

n

on-dyadic filter bankSlide20

Non-dyadic filters: downscaling

1. convolve with 2 filters

2. subsample each by 3

d

yadic case:

e

xample for the 2:3 ratio:

1. convolve with one filter

2. subsample by 2Slide21

Non-dyadic filters: upscaling

1. zero upsample by 2

2

. convolve with

2 filters

3. sum

d

yadic case:

e

xample for the 2:3 ratio:

1. zero upsample by

1

2. convolve with 1 filterSlide22

Use of the filters in upscaling

Upscaling using inverse scaling filters

Smoothing by downscaling and upscalingSlide23

Background and overview

Algorithm description

L

ocal

self similarity

Non-dyadic filter bank

Filter design

ResultsSlide24

When interpolating, smooth areas come from

input

Uniformly

spaced

grids should remain

uniform

1. Uniform stretch

0

255

brightness

grid coordinatesSlide25

2. Consistency

upsample

d

ownsample

The interpolated image, if downscaled should be equal to the input.

Formally,

Previous methods achieve consistency by solving large linear systems to

achieve this propertySlide26

3. PSF modeling

L

arge image - small camera point spread function

Small image - large camera point spread function

Difference between

point spread functionsSlide27

4. Low frequency span

frequency

When upsampling don’t add new frequencies

Upsampling

filter should be low-pass

o

riginal

interpolatedSlide28

5. Singularities preservation

blurred Image

interpolated image

s

imilar

amount of blurSlide29

Real time video upsampling on GPU

m

ain GPU memory

GPU cores

NTSC to full HD @ 24 fps

Search and filter-banks

are both

local

operationsSlide30

Background and overview

Algorithm description

L

ocal

self similarity

Non-dyadic filter bank

Filter

design

ResultsSlide31

Bicubic

x3

(zoomed in

)

Ours X3

(zoomed in)Slide32

Bicubic

x3

Ours X3Slide33

Genuine Fractals™ x4

Ours X4Slide34

Glasner

et al. 2009 x4

Ours X4Slide35
Slide36
Slide37

Thank you!Paper & additional results can be found at: www.cs.huji.ac.il/~giladfreedmn