Environmental Data Analysis with - PowerPoint Presentation

Slide1

Environmental Data Analysis with MatLab2nd Edition

Lecture 19:

Smoothing, Correlation and Spectra

Slide2

Lecture 01 Using MatLabLecture 02 Looking At Data

Lecture 03 Probability and Measurement Error

Lecture 04 Multivariate DistributionsLecture 05 Linear ModelsLecture 06 The Principle of Least SquaresLecture 07 Prior InformationLecture 08 Solving Generalized Least Squares Problems Lecture 09 Fourier SeriesLecture 10 Complex Fourier SeriesLecture 11 Lessons Learned from the Fourier Transform Lecture 12 Power SpectraLecture 13 Filter Theory Lecture 14 Applications of Filters Lecture 15 Factor Analysis Lecture 16 Orthogonal functions Lecture 17 Covariance and AutocorrelationLecture 18 Cross-correlationLecture 19 Smoothing, Correlation and SpectraLecture 20 Coherence; Tapering and Spectral Analysis Lecture 21 Interpolation Lecture 22 Linear Approximations and Non Linear Least Squares Lecture 23 Adaptable Approximations with Neural NetworksLecture 24 Hypothesis testing Lecture 25 Hypothesis Testing continued; F-TestsLecture 26 Confidence Limits of Spectra, Bootstraps

SYLLABUS

Slide3

Goals of the lectureexamine interrelationships betweensmoothing, correlation and power spectral density

Slide4

review

Slide5

AutocorrelationandCross-correlation

Slide6

Autocorrelation

Measure of correlation in time series

at different lagstu(t)

Slide7

Autocorrelation

Measure of correlation in time series

at different lagsttlag, t

a(t)

0

lag, multiply and sum area

no lag

Slide8

Autocorrelation

Measure of correlation in time series

at different lagsttlag, t

a(t)

0

lag, multiply and sum area

small lag

Slide9

Autocorrelation

Measure of correlation in time series

at different lagsttlag, t

a(t)

0

lag, multiply and sum area

large lag

Slide10

Autocorrelation

Measure of correlation in time series

at different lagsttlag, t

a(t)

0

lag, multiply and sum area

Slide11

Autocorrelation

Measure of correlation in time series

at different lagsttlag, t

a(t)

0

lag, multiply and sum area

a(t)=u(t)

⋆

u(t)

Slide12

crooss

-correlation

Measure of correlation between two time seriesat different lagst

t

u(t)

v(t)

Slide13

crooss

-correlation

Measure of correlation between two time seriesat different lagst

t

u(t)

v(t)

c(t)=u(t)

⋆

v(t)

Slide14

important relationshipsc(t) = u(t)⋆v(t) = u(-t)*v(t)c(

ω

) = u*(ω) v(ω) a(ω)= |u(ω)|2

Slide15

rough time series

frequency,

ω0

t

u(t)

lag, t

a(t)

0

sharp autocorrelation

wide spectrum

|u(

ω

)|

2

Slide16

smooth time series

frequency,

ω0tu(t)lag, ta(t)0wide autocorrelationnarrow spectrum|u(ω)|2

Slide17

lag, t

a(t)

0rough timeserieslag, ta(t)0

t

v(t)

t

v(t)

lag, t

a(t)

0

Slide18

Part 1Smoothing a Time Series

Slide19

smoothing as filtering(example of 3-point smoothing)

Slide20

non-causal

smoothing as filtering

(example of 3-point smoothing)

Slide21

fix-up

allow for a delay

Slide22

fix-upallow for a delay

d

smoothed and delayed = s * dobscausal filter, s

Slide23

triangular smoothing filters

s

i

s

i

index,

i

index,

i

3 points

21 points

Slide24

smoothing if

Neuse River Hydrograph

Slide25

questionhow does smoothing effect thethe autocorrelation of

d

Slide26

answerthe autocorrelation of s

acts as a smoothing filter on

the autocorrelation of d

Slide27

effect of smoothing on autocorrelation

Slide28

effect of smoothing on autocorrelation

autocorrelation of smoothed time series

Slide29

effect of smoothing on autocorrelation

autocorrelation of smoothed time series

everything written as convolution

Slide30

effect of smoothing on autocorrelation

autocorrelation of smoothed time series

everything written as convolutionregrouped

Slide31

effect of smoothing on autocorrelation

autocorrelation of smoothing filter

autocorrelation of time seriesconvolved with*

Slide32

answerthe autocorrelation of s

acts as a smoothing filter on

the autocorrelation of d

Slide33

Part 2What Makes a Good Smoothing Filter?

Slide34

then by the convolution theorem

d

smoothed(t) = s(t) * dobs(t)

Slide35

then by the convolution theorem

d

smoothed(t) = s(t) * dobs(t)so what’s this look like?

Slide36

example of auniform or “boxcar” smoothing filter

s(t)

time, tT

0

1/T

Slide37

take Fourier Transform

where

sinc(x) = sin(πx) / (πx)

Slide38

A) T=3

B)

T=21

Slide39

B) T=21

falls off with frequency (good)

Slide40

B) T=21

bumpy side lobes (bad)

Slide41

a box car filter does not suppress high frequencies evenlythe challenge

find a filter

that suppresses high frequencies evenly

Slide42

Normal FunctionFourier Transform of a Normal Functionis aNormal Function(which has no side lobes)

Slide43

A) L=3

B)

T=21B) T=3

Slide44

but a Normal Functionis non-causal(unless you truncate it, in which case it is not exactly a Normal Function)

Slide45

Normal FunctionBox

Car

Trianglesimplicitysidelobes

Slide46

Part 3Designing a Filter that Suppresses Specific Frequencies

Slide47

General form of the IIR Filter, f

Slide48

z-transform of the IIR filter

Slide49

General form of the IIR Filter

z-transform

Slide50

General form of the IIR Filter

z-transform

ratio of polynomials

Slide51

z-transform

v(z)

as a product of its factorsu(z) as a product of its factorsroots of u(z)

roots of

v(z)

z-transform of the IIR filter

ratio of polynomials

Slide52

so designing a filteris equivalent tospecifying the roots

of the two

polynomailsu(z) and v(z)

Slide53

at this point we need to explore the relationship between theFourier Transformand the

z-transform

Slide54

Answerthe Fourier Transformis thez-transformevaluated at a specific set of

z’s

Slide55

Relationship between Fourier Transform and Z-transform

since

Slide56

Relationship between Fourier Transform and Z-transform

since

Fourier Transform

Slide57

Relationship between Fourier Transform and Z-transform

since

Fourier Transformdiscrete times and frequencies

Slide58

Relationship between Fourier Transform and Z-transform

since

Fourier Transformdiscrete times and frequenciesz-transform

Slide59

Relationship between Fourier Transform and Z-transform

since

Fourier Transformdiscrete times and frequenciesz-transform

specific choice of

z’

s

Slide60

in wordsthe Fourier Transformis thez-transformevaluated at a specific set of

z’s

Slide61

there are N specific z’s

z

kor with θ

Slide62

real zq

imag

zunit circle, |z|2=1they plot as equally-spaced points around a “unit circle” in the complex z-plane

zero frequency

Nyquist

frequency

Slide63

Back to the IIR Filter

roots of

u(z)roots of v(z)

Slide64

Back to the IIR Filter(z-

z

ju) is zero at z=zjuproduces a low amplitude region near z=zjucalled a “zero”

Slide65

Back to the IIR Filter1/(z-

z

kv) is infinite at z=zkuproduces a high amplitude region near z=zkvcalled a “pole”

Slide66

so build a filter by placing the poles and zeros atstrategic points in the complex z-plane

Slide67

Rules zeros suppress frequencies

poles amplify frequencies

all poles must be outside the unit circle(so vinv converges)all poles, zeros must be in complex conjugate pairs(so filter is real)

Slide68

A)

B)

Slide69

A)

B)

zero near zero frequency suppresses low frequencies“high pass filter”zero near the Nyquist frequency suppresses high frequencies“low pass filter”

Slide70

A)

B)

Slide71

A)

B)

poles near ± a given frequency amplify that frequency“band pass filter”

poles and zeros near

± a given

frequency attenuate that frequency

“notch filter”

Slide72

something usefula tunable band pass filter

frequency,

f-fny+fny0|f(ω)|

2

f

1

f

2

-f

2

-f

1

Slide73

Chebychev

band-pass filter: 4 zeros, 4 poles

Slide74

Chebychev

band-pass filter: 4 zeros, 4 poles

2 zeros2 zerospole

pole

pole

pole

Slide75

Slide76

not quite as boxy as one might hope …

Slide77

Slide78

In MatLab

Slide79

Ground velocity at Palisades NY

Slide80

Ground velocity at Palisades NYLow pass filter

Slide81

Ground velocity at Palisades NYhigh pass filter