2017 Yong Jae Lee UC Davis Announcements PS0 out today due 414 Friday at 1159 pm Carefully read course website Signup for piazza 2 Plan for today Image formation Image noise Linear filters ID: 708821
Download Presentation The PPT/PDF document "Linear Filters April 6 th" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, noncommercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Linear Filters
April 6th, 2017
Yong Jae Lee
UC DavisSlide2
AnnouncementsPS0 out today; due 4/14 Friday at 11:59 pmCarefully read course website
Signup for piazza2Slide3
Plan for todayImage formationImage noiseLinear filters
Examples: smoothing filtersConvolution / correlation3Slide4
Image Formation
4
Slide credit:
Derek
HoiemSlide5
Digital cameraA digital camera replaces film with a sensor array
Each cell in the array is lightsensitive diode that converts photons to electronshttp://electronics.howstuffworks.com/digitalcamera.htm
5
Slide credit:
Steve SeitzSlide6
Digital imagesSample
the 2D space on a regular gridQuantize each sample (round to nearest integer)Image thus represented as a matrix of integer values.
2D
1D
6
Slide credit:
Kristen
Grauman
, Adapted from Steve SeitzSlide7
Digital images
7
Slide credit:
Derek
Hoiem
CMOS sensorSlide8
Digital color images
8
Slide credit:
Kristen
GraumanSlide9
Color Sensing: Bayer Grid
Estimate RGB at each cell from neighboring values
http://en.wikipedia.org/wiki/Bayer_filter
Slide by Steve SeitzSlide10
R
G
B
Color images, RGB color space
Digital color images
10
Slide credit:
Kristen
GraumanSlide11
Images in Matlab
Images represented as a matrixSuppose we have an NxM RGB image called “
im
”
im
(1,1,1) = topleft pixel value in Rchannel
im
(y, x, b) = y pixels down, x pixels to right in the
b
th
channel
im
(N, M, 3) = bottomright pixel in Bchannel
imread
(filename) returns a uint8 image (values 0 to 255)
Convert to double format (values 0 to 1) with im2double
0.92
0.93
0.94
0.97
0.62
0.37
0.85
0.97
0.93
0.92
0.99
0.95
0.89
0.82
0.89
0.56
0.31
0.75
0.92
0.81
0.95
0.91
0.89
0.72
0.51
0.55
0.51
0.42
0.57
0.41
0.49
0.91
0.92
0.96
0.95
0.88
0.94
0.56
0.46
0.91
0.87
0.90
0.97
0.95
0.71
0.81
0.81
0.87
0.57
0.37
0.80
0.88
0.89
0.790.850.490.620.600.580.500.600.580.500.610.450.330.860.840.740.580.510.390.730.920.910.490.740.960.670.540.850.480.370.880.900.940.820.930.690.490.560.660.430.420.770.730.710.900.990.790.730.900.670.330.610.690.790.730.930.970.910.940.890.490.410.780.780.770.890.990.93
0.920.930.940.970.620.370.850.970.930.920.990.950.890.820.890.560.310.750.920.810.950.910.890.720.510.550.510.420.570.410.490.910.920.960.950.880.940.560.460.910.870.900.970.950.710.810.810.870.570.370.800.880.890.790.850.490.620.600.580.500.600.580.500.610.450.330.860.840.740.580.510.390.730.920.910.490.740.960.670.540.850.480.370.880.900.940.820.930.690.490.560.660.430.420.770.730.710.900.990.790.730.900.670.330.610.690.790.730.930.970.910.940.890.490.410.780.780.770.890.990.93
0.920.930.940.970.620.370.850.970.930.920.990.950.890.820.890.560.310.750.920.810.950.910.890.720.510.550.510.420.570.410.490.910.920.960.950.880.940.560.460.910.870.900.970.950.710.810.810.870.570.370.800.880.890.790.850.490.620.600.580.500.600.580.500.610.450.330.860.840.740.580.510.390.730.920.910.490.740.960.670.540.850.480.370.880.900.940.820.930.690.490.560.660.430.420.770.730.710.900.990.790.730.900.670.330.610.690.790.730.930.970.910.940.890.490.410.780.780.770.890.990.93
R
G
B
row
column
11
Slide credit:
Derek
HoiemSlide12
Image filteringCompute a function of the local neighborhood at each pixel in the image
Function specified by a “filter” or mask saying how to combine values from neighborsUses of filtering:Enhance an image (denoise, resize, increase contrast,
etc
)
Extract information (texture, edges, interest points,
etc
)
Detect patterns (template matching)
12
Slide credit:
Kristen
Grauman
, Adapted from Derek
HoiemSlide13
Motivation: noise reduction
Even multiple images of the
same static scene
will not be identical.
13
Slide a
dapted from Kristen
GraumanSlide14
Common types of noise
Salt and pepper noise: random occurrences of black and white pixelsImpulse noise: random occurrences of white pixels
Gaussian noise
: variations in intensity drawn from a Gaussian normal distribution
14
Slide credit:
Steve SeitzSlide15
Gaussian noise
>> noise =
randn(size(im
)).*sigma;
>> output =
im
+ noise;
What is impact of the sigma?
Slide credit:
Kristen
Grauman
Figure from Martial Hebert
15Slide16
Motivation: noise reduction
Even multiple images of the same static scene will not be identical.
How could we reduce the noise, i.e., give an estimate of the true intensities?
What if there’s only one image?
16
Slide credit:
Kristen
GraumanSlide17
First attempt at a solutionLet’s replace each pixel with an average of all the values in its neighborhood
Assumptions: Expect pixels to be like their neighborsExpect noise processes to be independent from pixel to pixel
17
Slide credit:
Kristen
GraumanSlide18
First attempt at a solutionLet’s replace each pixel with an average of all the values in its neighborhood
Moving average in 1D:
18
Slide credit:
S.
MarschnerSlide19
Weighted Moving AverageCan add weights to our moving averageWeights [1, 1, 1, 1, 1] / 5
19
Slide credit:
S.
MarschnerSlide20
Weighted Moving AverageNonuniform weights [1, 4, 6, 4, 1] / 16
20
Slide credit:
S.
MarschnerSlide21
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000
0000000000
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
21
Slide credit:
Steve SeitzSlide22
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000000
00000000
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000000
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
22
Slide credit:
Steve SeitzSlide23
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000000
00000000
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000000
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
23
Slide credit:
Steve SeitzSlide24
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000000
0000000010
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000000
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
24
Slide credit:
Steve SeitzSlide25
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000000
0000000010
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000000
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
25
Slide credit:
Steve SeitzSlide26
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000000
000000001020
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000000
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
26
Slide credit:
Steve SeitzSlide27
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000000
000000001020
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000000
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
27
Slide credit:
Steve SeitzSlide28
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
00000000000000900000000000
00000000102030
000000000000000000000009090909090000009090909090000009090909090000009009090900000090909090900000000
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
28
Slide credit:
Steve SeitzSlide29
Moving Average In 2D
0
10
20
30
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90909000000000000000900
000000000000000029Slide credit: Steve SeitzSlide30
Moving Average In 2D
0
10
20
30
30
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
9090909000000000000000900
000000000000000030Slide credit: Steve SeitzSlide31
Moving Average In 2D
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0000000000000090000000000
0000000001020303030201002040606060402003060909090603003050808090603003050808090603002030505060402010203030303020101010
100000031Slide credit: Steve SeitzSlide32
Correlation filtering
Say the averaging window size is 2k+1 x 2k+1:
Loop over all pixels in neighborhood around image pixel
f[
i,j
]
Attribute uniform weight to each pixel
Now generalize to allow
different weights
depending on neighboring pixel’s relative position:
Nonuniform weights
32
Slide
adapted from Kristen
GraumanSlide33
Correlation filtering
Filtering an image: replace each pixel with a linear combination of its neighbors.The filter “kernel” or “mask” h[
u,v
] is the prescription for the weights in the linear combination.
This is called
crosscorrelation
, denoted
33
Slide
adapted from Kristen
GraumanSlide34
Averaging filter
What values belong in the kernel
h
for the moving average example?
0
10
20
30
30
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
“box filter”
?
34
Slide
adapted from Kristen
GraumanSlide35
Smoothing by averaging
depicts box filter:
white = high value, black = low value
original
filtered
What if the filter size was 5 x 5 instead of 3 x 3?
35
Slide credit:
Kristen
GraumanSlide36
Boundary issuesWhat about near the edge?the filter window falls off the edge of the image
need to extrapolatemethods:clip filter (black)wrap aroundcopy edgereflect across edge
36
Slide credit:
S.
MarschnerSlide37
Boundary issuesWhat about near the edge?the filter window falls off the edge of the image
need to extrapolatemethods (MATLAB):clip filter (black): imfilter(f, g, 0)wrap around: imfilter(f, g, ‘circular’)copy edge: imfilter(f, g, ‘replicate’)reflect across edge: imfilter(f, g, ‘symmetric’)
37
Slide credit:
S.
MarschnerSlide38
Boundary issuesWhat is the size of the output?MATLAB: output size / “shape” options
shape = ‘full’: output size is sum of sizes of f and gshape = ‘same’: output size is same as f
shape
= ‘valid’: output size is difference of sizes of
f
and
g
f
g
g
g
g
full
f
g
g
g
g
same
f
g
g
g
g
valid
38
Slide credit:
Svetlana
LazebnikSlide39
What if we want nearest neighboring pixels to have the most influence on the output?
Removes highfrequency components from the image (“lowpass filter”).
Gaussian filter
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
90
90
90
90
0
0
0
0
0
90
0
90
90
90
0
0
0
0
0
90
90
90
90
90000000000000
009000000000000000000121242121This kernel is an approximation of a 2d Gaussian function:39Slide credit: Steve SeitzSlide40
Smoothing with a Gaussian
40
Slide credit:
Kristen
GraumanSlide41
Smoothing with a boxfilter
41Slide42
Gaussian filters
What parameters matter here?Size of kernel or maskNote, Gaussian function has infinite support, but discrete filters use finite kernels
σ
= 5 with 10 x 10 kernel
σ
= 5 with 30 x 30 kernel
42
Slide credit:
Kristen
GraumanSlide43
Gaussian filters
What parameters matter here?Variance of Gaussian: determines extent of smoothing
σ
= 2 with 30 x 30 kernel
σ
= 5 with 30 x 30 kernel
43
Slide credit:
Kristen
GraumanSlide44
Matlab
>> hsize = 30
;
>> sigma = 5;
>>
h
=
fspecial(‘gaussian
’,
hsize
, sigma);
>>
mesh(h
);
>>
imagesc(h
);
>>
outim
=
imfilter(im
,
h
); % correlation
>>
imshow(outim
);
outim
44
Slide credit:
Kristen
GraumanSlide45
Smoothing with a Gaussian
for sigma=1:3:10 h =
fspecial('gaussian
‘,
hsize
, sigma);
out =
imfilter(im
,
h
);
imshow(out
);
pause;
end
…
Parameter
σ
is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing.
45
Slide credit:
Kristen
GraumanSlide46
Properties of smoothing filtersSmoothingValues positive
Sum to 1 constant regions same as inputAmount of smoothing proportional to mask size
Remove “highfrequency” components; “lowpass” filter
46
Slide credit:
Kristen
GraumanSlide47
Filtering an impulse signal
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
a
b
c
d
e
f
g
h
i
What is the result of filtering the impulse signal (image)
f
with the arbitrary kernel
h
?
?
47
Slide
adapted from Kristen
GraumanSlide48
f
h
h
Convolution
Convolution:
Flip the filter in both dimensions (bottom to top, right to left)
Then apply crosscorrelation
Notation for convolution operator
48
Slide
adapted from Kristen
GraumanSlide49
Convolution vs. correlation
Convolution
Crosscorrelation
For a Gaussian or box filter, how will the outputs differ?
If the input is an impulse signal, how will the outputs differ?
49
Slide
adapted from Kristen
GraumanSlide50
Predict the outputs using correlation filtering
0
0
0
0
1
0
0
0
0
*
= ?
0
0
0
1
0
0
0
0
0
*
= ?
1
1
1
1
1
1
1
1
1
0
0
0
0
2
0
0
0
0

*
= ?
50
Slide credit:
Kristen
GraumanSlide51
Practice with linear filters
0
0
0
0
1
0
0
0
0
Original
?
51
Slide credit:
David LoweSlide52
Practice with linear filters
0
0
0
0
1
0
0
0
0
Original
Filtered
(no change)
52
Slide credit:
David LoweSlide53
Practice with linear filters
0
0
0
1
0
0
0
0
0
Original
?
53
Slide credit:
David LoweSlide54
Practice with linear filters
0
0
0
1
0
0
0
0
0
Original
Shifted left
by 1 pixel with correlation
54
Slide credit:
David LoweSlide55
Practice with linear filters
Original
?
1
1
1
1
1
1
1
1
1
55
Slide credit:
David LoweSlide56
Practice with linear filters
Original
1
1
1
1
1
1
1
1
1
Blur (with a
box filter)
56
Slide credit:
David LoweSlide57
Practice with linear filters
Original
1
1
1
1
1
1
1
1
1
0
0
0
0
2
0
0
0
0

?
57
Slide credit:
David LoweSlide58
Practice with linear filters
Original
1
1
1
1
1
1
1
1
1
0
0
0
0
2
0
0
0
0

Sharpening filter:
accentuates differences with local average
58
Slide credit:
David LoweSlide59
Filtering examples: sharpening
59
Slide credit:
Kristen
GraumanSlide60
Properties of convolutionShift invariant:
Operator behaves the same everywhere, i.e. the value of the output depends on the pattern in the image neighborhood, not the position of the neighborhood.Superposition: h * (f1 + f2) = (h * f1) + (h
* f2)
60
Slide credit:
Kristen
GraumanSlide61
Properties of convolutionCommutative:
f * g = g * fAssociative(f * g) * h = f * (g * h)Distributes over addition f * (g + h) = (f * g) + (f * h)
Scalars factor out
kf * g = f * kg = k(f * g)
Identity:
unit impulse e = […, 0, 0, 1, 0, 0, …]. f * e = f
61
Slide credit:
Kristen
GraumanSlide62
SeparabilityIn some cases, filter is separable, and we can factor into two steps:
Convolve all rowsConvolve all columns62
Slide credit:
Kristen
GraumanSlide63
SeparabilityIn some cases, filter is separable, and we can factor into two steps: e.g.,
What is the computational complexity advantage for a separable filter of size k x k, in terms of number of operations per output pixel?
f
* (
g
*
h
) = (
f
*
g
) *
h
g
h
f
63
Slide credit:
Kristen
GraumanSlide64
Effect of smoothing filters
Additive Gaussian noise
Salt and pepper noise
64
Slide credit:
Kristen
GraumanSlide65
Median filter
No new pixel values introduced
Removes spikes: good for impulse, salt & pepper noise
Nonlinear filter
65
Slide credit:
Kristen
GraumanSlide66
Median filter
Salt and pepper noise
Median filtered
Plots of a
col
of the image
Matlab
: output
im
= medfilt2(im, [
h
w
]);
66
Slide
credit: Martial HebertSlide67
Median filterMedian filter is edge preserving
67
Slide credit:
Kristen
GraumanSlide68
Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006
Filtering application: Hybrid Images
68
Slide credit:
Kristen
GraumanSlide69
Application: Hybrid Images
Gaussian Filter
Laplacian Filter
Gaussian
unit impulse
Laplacian of Gaussian
69
Slide credit:
Kristen
Grauman
A.
Oliva
, A.
Torralba
, P.G.
Schyns
,
“Hybrid Images,”
SIGGRAPH 2006Slide70
Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006
70Slide71
Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006
71Slide72
SummaryImage formationImage “noise”Linear filters and convolution useful for
Enhancing images (smoothing, removing noise)Box filterGaussian filterImpact of scale / width of smoothing filterDetecting features (next time)Separable filters more efficient
Median filter: a nonlinear filter, edgepreserving
72Slide73
Coming upNext Friday (4/14):
PS0 is due via Canvas, 11:59 PMTuesday:Filtering part 2: filtering for features73Slide74
Questions?See you Tuesday!