L ecture 3 filtering and frequencies CS 590134 future 572 Spring 2016 Prof Alex Berg Credits to many other folks on individual slides Today filtering and frequency analysis of images ID: 435090
Download Presentation The PPT/PDF document "Computational Photography" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Computational PhotographyLecture 3 – filtering and frequenciesCS 590-134 (future 572) Spring 2016
Prof. Alex Berg(Credits to many other folks on individual slides)Slide2
Today
filtering and frequency analysis of images
.
TodaySlide3
Slides from Derek HoiemSlide4
Sampling and ReconstructionSlide5
© 2006 Steve Marschner •
5
Sampled representations
How to store and compute with continuous functions?Common scheme for representation: sampleswrite down the function’
s values at many points
[FvDFH fig.14.14b / Wolberg]Slide6
© 2006 Steve Marschner •
6
Reconstruction
Making samples back into a continuous functionfor output (need realizable method)for analysis or processing (need mathematical method)amounts to
“
guessing
”
what the function did in between
[FvDFH fig.14.14b / Wolberg]Slide7
1D Example: Audio
low
high
frequenciesSlide8
© 2006 Steve Marschner •
8
Sampling in digital audio
Recording: sound to analog to samples to discPlayback: disc to samples to analog to sound againhow can we be sure we are filling in the gaps correctly?Slide9
© 2006 Steve Marschner •
9
Sampling and Reconstruction
Simple example: a sign waveSlide10
© 2006 Steve Marschner •
10
Undersampling
What if we “missed” things between the samples?Simple example: undersampling a sine wave
unsurprising result: information is lostSlide11
© 2006 Steve Marschner •
11
Undersampling
What if we “missed” things between the samples?
Simple example:
undersampling
a sine wave
unsurprising result: information is lost
surprising result: indistinguishable from lower frequencySlide12
© 2006 Steve Marschner •
12
Undersampling
What if we “missed” things between the samples?
Simple example:
undersampling
a sine wave
unsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also, was always indistinguishable from higher frequencies
aliasing
:
signals
“
traveling in disguise
”
as other frequenciesSlide13
Aliasing in video
Slide by Steve SeitzSlide14
Aliasing in imagesSlide15
What’s happening?
Input signal:
x = 0:.05:5; imagesc(sin((2.^x).*x))
Plot as image:
Alias!
Not enough samplesSlide16
AntialiasingWhat can we do about aliasing?Sample more oftenJoin the Mega-Pixel craze of the photo industry
But this can’t go on forever
Make the signal less
“wiggly” Get rid of some high frequenciesWill loose informationBut it’s better than aliasingSlide17
© 2006 Steve Marschner •
17
Preventing aliasing
Introduce lowpass filters:remove high frequencies leaving only safe, low frequencieschoose lowest frequency in reconstruction (disambiguate)Slide18
© 2006 Steve Marschner •
18
Linear filtering: a key idea
Transformations on signals; e.g.:bass/treble controls on stereoblurring/sharpening operations in image editingsmoothing/noise reduction in tracking
Key properties
linearity: filter(
f + g
) = filter(
f
) + filter(
g
)
shift invariance: behavior invariant to shifting the input
delaying an audio signal
sliding an image around
Can be modeled mathematically by
convolutionSlide19
© 2006 Steve Marschner •
19
Moving Average
basic idea: define a new function by averaging over a sliding windowa simple example to start off: smoothingSlide20
© 2006 Steve Marschner •
20
Weighted Moving Average
Can add weights to our moving averageWeights […, 0, 1, 1, 1, 1, 1, 0, …] / 5 Slide21
© 2006 Steve Marschner •
21
Weighted Moving Average
bell curve (gaussian-like) weights […, 1, 4, 6, 4, 1, …]Slide22
© 2006 Steve Marschner •
22
Moving Average In 2D
What are the weights H?
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
000000000
Slide by Steve SeitzSlide23
© 2006 Steve Marschner •
23
Cross-correlation filtering
Let’s write this down as an equation. Assume the averaging window is (2k+1)x(2k+1):
We can generalize this idea by allowing different weights for different neighboring pixels:
This is called a
cross-correlation
operation and
written:
H is called the
“
filter,
”
“
kernel,
”
or
“
mask.
”
Slide by Steve SeitzSlide24
Gaussian filteringA Gaussian kernel gives less weight to pixels further from the center of the window
This kernel is an approximation of a Gaussian function:
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
000000
012124212
1
Slide by Steve SeitzSlide25
Mean vs. Gaussian filtering
Slide by Steve SeitzSlide26
Convolutioncross-correlation:
A convolution operation is a cross-correlation where the filter is flipped both horizontally and vertically before being applied to the image:
It is written:
Suppose H is a Gaussian or mean kernel. How does convolution differ from cross-correlation?
Slide by Steve SeitzSlide27
© 2006 Steve Marschner •
27
Convolution is nice!
Notation:Convolution is a multiplication-like operationcommutativeassociative
distributes over addition
scalars factor out
identity: unit impulse
e
= […, 0, 0, 1, 0, 0, …]
Conceptually no distinction between filter and signal
Usefulness of associativity
often apply several filters one after another: (((
a
*
b
1
) *
b
2
) *
b
3
)
this is equivalent to applying one filter: a * (
b1 * b2 *
b
3
)Slide28
Tricks with convolutions
=Slide29
Slides from Alexei Eros and Steve SeitzSlide30
Salvador Dali,
“
Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln
”, 1976Salvador Dali, “Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln
”
, 1976
Salvador Dali
“
Gala Contemplating the Mediterranean Sea,
which at 30 meters becomes the portrait
of Abraham Lincoln
”
, 1976Slide31Slide32Slide33
A nice set of basis
This change of basis has a special name…
Teases away fast vs. slow changes in the image.Slide34
Jean Baptiste Joseph Fourier (1768-1830)had crazy idea (1807): Any univariate
function can be rewritten as a weighted sum of sines and cosines of different frequencies. Don’
t believe it?
Neither did Lagrange, Laplace, Poisson and other big wigsNot translated into English until 1878! But it’s (mostly) true!called Fourier Seriesthere are some subtle restrictions
...the manner in which the author arrives at these equations is not exempt of difficulties and...his analysis to integrate them still leaves something to be desired on the score of generality and even
rigour
.
Laplace
Lagrange
LegendreSlide35
A sum of sinesOur building block: Add enough of them to get any signal f(x) you want!
How many degrees of freedom?What does each control?Which one encodes the coarse vs. fine structure of the signal?Slide36
Fourier TransformWe want to understand the frequency w of our signal. So, let’s reparametrize the signal by w instead of x
:
f(x)
F(
w
)
Fourier
Transform
F(
w
)
f(x)
Inverse Fourier
Transform
For every
w
from 0 to inf,
F(
w
)
holds the amplitude
A
and phase
f
of the corresponding sine
How can
F
hold both? Complex number trick!
We can always go back:Slide37
Time and Frequencyexample : g(t)
= sin(2pf
t
) + (1/3)sin(2p(3f) t)Slide38
Time and Frequencyexample : g(t)
= sin(2pf
t
) + (1/3)sin(2p(3f) t)
=
+Slide39
Frequency Spectraexample : g(t)
= sin(2pf
t
) + (1/3)sin(2p(3f) t)
=
+Slide40
Frequency SpectraUsually, frequency is more interesting than the phaseSlide41
=
+
=
Frequency SpectraSlide42
=
+
=
Frequency SpectraSlide43
=
+
=
Frequency SpectraSlide44
=
+
=
Frequency SpectraSlide45
=
+
=
Frequency SpectraSlide46
=
Frequency SpectraSlide47
Frequency SpectraSlide48
FT: Just a change of basis
.
.
.
*
=
M *
f(x)
=
F(
w
)Slide49
IFT: Just a change of basis
.
.
.
*
=
M
-1
*
F(
w
)
=
f(x)Slide50
Finally: Scary MathSlide51
Finally: Scary Math…not really scary:is hiding our old friend:So it’s just our signal
f(x) times sine at frequency w
phase can be encoded
by sin/cos pairSlide52
Extension to 2D
in Matlab, check out: imagesc(log(abs(fftshift(fft2(im)))));Slide53
Fourier analysis in images
Intensity Image
Fourier Image
http://sharp.bu.edu/~slehar/fourier/fourier.html#filteringSlide54
Signals can be composed
+
=
http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering
More: http://www.cs.unm.edu/~brayer/vision/fourier.htmlSlide55
Man-made SceneSlide56
Can change spectrum, then reconstructSlide57
Low and High Pass filteringSlide58
The Convolution TheoremThe greatest thing since sliced (banana) bread!The Fourier transform of the convolution of two functions is the product of their Fourier transforms
The inverse Fourier transform of the product of two Fourier transforms is the convolution of the two inverse Fourier transforms
Convolution
in spatial domain is equivalent to multiplication in frequency domain!Slide59
2D convolution theorem example
*
f
(
x
,
y
)
h
(
x
,
y
)
g
(
x
,
y
)
|F
(s
x
,s
y
)|
|H
(s
x
,s
y
)|
|G
(s
x
,s
y
)|Slide60
Why does the Gaussian give a nice smooth image, but the square filter give edgy artifacts?
Gaussian
Box filter
FilteringSlide61
GaussianSlide62
Box FilterSlide63
Fourier Transform pairsSlide64
Low-pass, Band-pass, High-pass filters
low-pass:
High-pass / band-pass:Slide65
Edges in imagesSlide66
What does blurring take away?
originalSlide67
What does blurring take away?
smoothed (5x5 Gaussian)Slide68
High-Pass filter
smoothed – originalSlide69
Band-pass filteringLaplacian Pyramid (subband images)Created from Gaussian pyramid by subtraction
Gaussian Pyramid (low-pass images)Slide70
Laplacian PyramidHow can we reconstruct (collapse) this pyramid into the original image?
Need this!
Original
imageSlide71
Why Laplacian?
Laplacian of Gaussian
Gaussian
delta functionSlide72
Project 2: Hybrid Images
http://www.cs.illinois.edu/class/fa10/cs498dwh/projects/hybrid/ComputationalPhotography_ProjectHybrid.html
Gaussian Filter!
Laplacian Filter!
Project Instructions:
A. Oliva, A. Torralba, P.G. Schyns,
“
Hybrid Images,
”
SIGGRAPH 2006
Gaussian
unit impulse
Laplacian of GaussianSlide73
Early processing in humans filters for various orientations and scales of frequencyPerceptual cues in the mid frequencies dominate perceptionWhen we see an image from far away, we are effectively subsampling it
Early Visual Processing: Multi-scale edge and blob filters
Clues from Human PerceptionSlide74
Frequency Domain and Perception
Campbell-Robson contrast sensitivity curveSlide75
Da Vinci and Peripheral VisionSlide76
Leonardo playing with peripheral visionSlide77
Unsharp Masking
-
=
=
+
aSlide78
Freq. Perception Depends on Color
R
G
BSlide79
Lossy Image Compression (JPEG)
Block-based Discrete Cosine Transform (DCT)Slide80
Using DCT in JPEG The first coefficient B(0,0) is the DC component, the average intensityThe top-left coeffs represent low frequencies, the bottom right – high frequenciesSlide81
Image compression using DCTQuantize More coarsely for high frequencies (which also tend to have smaller values)Many quantized high frequency values will be zero
EncodeCan decode with inverse dct
Quantization table
Filter responses
Quantized valuesSlide82
JPEG Compression SummarySubsample color by factor of 2People have bad resolution for colorSplit into blocks (8x8, typically), subtract 128For each block
Compute DCT coefficients forCoarsely quantize
Many high frequency components will become zero
Encode (e.g., with Huffman coding)
http://en.wikipedia.org/wiki/YCbCr
http://en.wikipedia.org/wiki/JPEGSlide83
Block size in JPEG Block sizesmall blockfaster
correlation exists between neighboring pixelslarge blockbetter compression in smooth regions
It
’s 8x8 in standard JPEGSlide84
JPEG compression comparison
89k
12kSlide85
Image gradient
The gradient of an image:
The gradient points in the direction of most rapid change in intensity
The gradient direction is given by:
how does this relate to the direction of the edge?
The
edge strength
is given by the gradient magnitudeSlide86
Effects of noiseConsider a single row or column of the imagePlotting intensity as a function of position gives a signal
Where is the edge?
How to compute a derivative?Slide87
Where is the edge? Solution: smooth first
Look for peaks in Slide88
Derivative theorem of convolution
This saves us one operation:Slide89
Laplacian of GaussianConsider
Laplacian of Gaussian
operator
Where is the edge?
Zero-crossings of bottom graphSlide90
2D edge detection filters
is the
Laplacian
operator:
Laplacian of Gaussian
Gaussian
derivative of GaussianSlide91
Try this in MATLABg = fspecial('gaussian',15,2);imagesc(g); colormap
(gray);surfl(g)gclown = conv2(
clown,g,'same
');imagesc(conv2(clown,[-1 1],'same'));imagesc(conv2(gclown,[-1 1],'same'));dx = conv2(g,[-1 1],'same');imagesc(conv2(clown,dx,'same'));lg = fspecial('log',15,2);
lclown
= conv2(clown,
lg
,'same');
imagesc
(
lclown
)
imagesc
(clown + .2*
lclown
)
Find your own image,
clown
doesn’t seem to be a default image anymore
Slide92
For next class
Read over the slides and the textbook, Chapter 3.