Landstations After Homogenization ATTILAH Ralf Lindau Dipdoc Seminar 23102017 Break and Noise Variance Homogenization To homogenize we consider the difference time series between two neighboring stations ID: 656536
Download Presentation The PPT/PDF document "Adjustment of Temperature Trends In" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Adjustment of Temperature Trends In
Landstations
After Homogenization
(ATTILAH)
Ralf LindauSlide2
Dipdoc
Seminar – 23.10.2017
Break and Noise Variance
HomogenizationTo homogenize we consider the difference time series between two neighboring stations.The dominating natural variance is cancelled out, because it is very similar at both stations. The relative break variance is increased and we have a realistic chance to detect the breaks.General proceedingRandom combinations of test breaks are inserted. That one explaining the maximum variance is considered to show the true breaks. Technical applicationDynamic Programming with Stop criterion
Noise
Var
Break
VarSlide3
Dipdoc
Seminar – 23.10.2017
Trend bias
Trend biasIf the positive and negative jumps do not cancel out each other, they introduce a trend bias. Underestimation of trend biasIt is impossible to isolate the full break signal from the noise. Thus, only a certain part of it can be corrected. A small fraction remains (which has to be corrected after homogenization).Slide4
Underestimation of jump height
Dipdoc Seminar – 23.10.2017The two fat horizontal lines indicate the true jump height Errors occur when the noise randomly (and erroneously) increases the data above the middle line.
Then, a part of Segment 2 is (erroneously) exchanged to Segment 1
Correct detectionx1 and x2 are determined as segment averages. x1 is nearly correct, but x2 is to high.Incorrect detectionx1’ and x2’ are determined as segment averages. x2’ is nearly correct, but x1’ is to low.In both cases the jump height is underestimated. Slide5
Dipdoc
Seminar – 23.10.2017
Obviously, this systematic underestimation depends on the interaction between noise and break variance.
To quantify this effect, the statistical properties of both break and noise variance has to be known.Nomenclaturek: Number of test breaks (here: 3)n: Number of true breaks (here: 7)m: Total length (here: 100)l: Test segment length (here: 14, 4, etc.)Slide6
Statistical Characteristic of the Noise Variance
Beta distributedSlide7
Dipdoc
Seminar – 23.10.2017
Example for noise variance
We insert k = 3 random test breaks and check the variance they are able to explain.Since we have pure noise, the test segments’ means are very close to zero.However, there is a small random variation: This is the explained variance.Slide8
Statistic for Noise Variance
Dipdoc Seminar – 23.10.2017We insert k = 3 test breaks at random positions into a random noise time series and calculate the explained variance.This procedure is repeated
1000 by 1000 times (1000 time series and 3000 test break positions). Relative explained variance:
Slide9
Probability density:
with Beta function
For noise the shape parameters are:
when
k
denotes the number of test breaks and
m
the total length
Beta distribution
Dipdoc
Seminar
–
23.10.2017 Slide10
The mean of a Beta distribution is given by:
Mean explained variance:
Maximum explained variance:
Behavior of Noiseoptimum
mean (random)
Dipdoc
Seminar
–
23.10.2017
/ (m-1)
RememberSlide11
Statistical Characteristic of the Break
Variance1. Heuristic approach2. Empirical approach 3. Theoretical approachSlide12
For true breaks, constant periods exist. Tested segment averages are the (weighted) means of such (few) constant periods.
This is quite the same situation as for random scatter, only that less independent data is underlying.Obviously, the number of breaks n plays the same role as the time series length m did before for noise.Thus, the first approximation is: First Approach
Dipdoc
Seminar – 23.10.2017 Slide13
Second
Approach (1/2)However, this would lead to
This is obviously only true, when all real breaks are actually matched by the test breaks, (which is not the case for random trials).
Consider k=3, n=7 and count the number of platforms in each test segment. Altogether, there are 11 “independents”, in general n+k+1.Thus, the second approximation is: Dipdoc Seminar – 23.10.2017 4 2 4 1
RememberSlide14
Second
Approach (2/2)This would lead to
This is rather reasonable, because for
n = k the situation is approximately:Each test segment contains one true break, thus two independents, which are then averaged. This leads to a reduction of the variance by a factor of 2. However, so far we did not take into account that the HSPs have different lengths.The effective number of true breaks must be smaller than the nominal. Dipdoc Seminar – 23.10.2017 4 2 4 1Slide15
E
ffective observation numberIf we generate i = 1…N random time series of length j = 1…m with each element being:
only a fraction of (m-1)/m can be found within the time series (because a fraction of 1/m is “lost” due to the variance of the time series means.
How large is this effect if a step function with n breaks is considered? Dipdoc Seminar – 23.10.2017 Slide16
Sketch of derivation (1/2)
Dipdoc Seminar – 23.10.2017
The mean of each time series is:
The “lost” variance is:
Which can be reduced to the sum over mean squared lengths:
Which is equal to the weighted sum over l
2
Slide17
Sketch of derivation (2/2)
Dipdoc Seminar – 23.10.2017 The sum of a product of two Binomial coefficients is solvable by the Vandermonde’s identity:Which leads to a solution for the l2 sum:
Inserted into the original expression, we obtain
for the “lost” external variance: The remaining internal variance is then:
Slide18
Third approach
Dipdoc Seminar – 23.10.2017 The relative unexplained variance of a test segment:with i: number of breaks within a test segment and n: number of breaks within the entire time seriesi = l/m n:
m = l(k+1):
with n* = n/2 +1 Similar to the second approach, but n counts only half.
Remember
Remember:Slide19
Statistical Characteristic of the Break
Variance1. Heuristic approach2. Empirical approach 3. Theoretical approachSlide20
Empirical
Var(k,n)Dipdoc Seminar – 23.10.2017
Empirical test with 1000 random segmentations
(fixed k) of 1000 time series (fixed n).Calculate the mean relative explained variance v from these 1,000,000 permutations. Repeat this procedure for all combinations of k = 1, …, 20 and n = 1, …, 20.20 functions v(k) for the different n.Slide21
Intelligent Fitting (1/3)
v/(1-v) is proportional to k.The slope is a function of n. (Numbers and lines do not cross). The slope is certainly not proportional, but rather reciprocal to n. (
slp(1) large, slp(20) small).
Thus, better to plot 1/slp(n). Dipdoc Seminar – 23.10.2017 Slide22
We expect horizontal lines, if the reciprocal slope is really independent from k.
This is largely confirmed. Averages over k gives than a value for each n. These seems to be rather linear in n. Thus, plot these averages as a function of n.Intelligent Fitting (2/3)Dipdoc Seminar – 23.10.2017
Slide23
with
a = 0.629 and b = 1.855 Intelligent Fitting (3/3)Dipdoc Seminar – 23.10.2017
0.629n + 1.855
solve for v:
a
nd insert
an+b
:
Slide24
Application of findings
Dipdoc Seminar – 23.10.2017
Summarizing the stepwise fitting:
The direct fit in the v/k space yields: The best
heuristic approach was:
Slide25
Conclusion
The explained noise variance is Beta distributed with:The explained break variance is Beta distributed with:
Dipdoc
Seminar – 23.10.2017