Trend

Section 3: Trend Analysis
Analysis of Climate and Weather Data | Trend Analysis | HS 2013 | christoph.frei [at] meteoswiss.ch
Outline
! ! ! ! ! !
General Concept Parametric and Non-Parametric Trend Tests Accounting for Serial Correlation Trend Analysis for Count Data Additional Remarks Material:
!! !! !!
Yue et al. 2002, Zhang & Zwiers 2004 Frei & Schr 2001
General Concept
!
Data series = trend + stochastic variations
x (t ), t = t1,.., tn , the data series
x (t ) = f (t ) + ! (t )
!
f (t ) : a deterministic function in time
! (t ) : stochastic variations, a random variable
Null hypothesis and alternative: !! H0: f(t) is constant (not depending on time) !! HA: f(t) is not constant Test:
!! !! !!
Construct test statistic, distinguishing betw H0 and HA, in presence of !(t) . Derive Null-Distribution !
General Concept
!
Assumptions
!! !! !!
Every trend analysis makes assumptions about !(t). Whether or not the data satisfies these, needs verification. If assumptions are violated, test results can be misleading.
Parametric & Non-Parametric Tests
Linear Regression
!
Concept: Linear trend + normal white noise
x (t ) = ! ! t + ! + " (t )
!!
parametric
! (t ) ! N ( 0, " ), " (t ) = "

1 2 ( xi ! x ) # n !1 i
Ordinary least squares regression

= r sx ! xt st "t = x !! "
2 sx =
dito st
rxt = cor ( x, t )
Null hypothesis and alternative: !! H0: " = 0 (i.e. data not linearly dependent on time), HA: " ! 0 Test statistic (linear regression T-Test):
! T= ! "
2 " res = " 2 t ! t "( i ) 2 ! i
" !
standard error of slope
! res
residual std. deviation
Trend Error and Sample Size

!
The uncertainty (standard error) of a trend estimate decreases with sample size (# of obs.). How quickly?
Trend Error and Sample Size

!
The uncertainty (standard error) of a trend estimate decreases with sample size (# of obs.). How quickly?
2 ! res = ! = 2 " ( ti ! t ) 2 " i 2 ! res 2 $ i 1' 2 L # "& ! ) % N 2( i
L: Length of time period N: Number of obs. in L!
linear in N!
!!
Trend uncertainty decreases with square root of sample size. Similar like uncertainty in the mean!
Linear Regression
! !
Null distribution !! T is distributed like students T with n-2 degrees of freedom Assumptions
!! !! !!
residuals are independent (i.e. no serial correlation) residuals(!) are normally distributed residual variance is constant
Verify assumptions
!! !! !! !! !!
Normality of residuals in histogram and Q-Q plot Equal variance by Tukey-Anscombe plot (residuals vs. fitted) Independence by auto-correlation function Data transformations may be used to ensure normality of residuals and constant variance T-Test is sensitive to outliers
R: lm, coef, resid, summary!

Analysis of Climate and Weather Data | Trend Analysis | HS 2013 | christoph.frei [at] meteoswiss.ch 9
Linear Regression
!
Example
" = 0.1m3"s-1 / a! ! T = 2.38! ! qT(0.025)= 1.99! qT(0.975)= +1.99! 68 degrees of freedom! ! p-value: 0.02! !
!! Trend is statistically significant, i.e. evidence for change

Linear Regression
Residual Analysis: Maximum winter-time runoff, Birs-Moutier 1931-2000
11
Theil-Sen Trend Estimate

!
A robust estimator for a linear trend (not a test!)
xi i = 1,.., n regular time series
" x k ! xi % !TS = median $ ' i< k # k !i &

Median of slopes between all data pairs
Theil 1950, Sen 1968

Mann-Kendall Trend Test

!
Test for a monotonic trend
!! !! !!
non-parametric
a special case of Kendall s tau test
not relying on an estimate of the trend itself (test only) usually combined with Theil-Sen trend estimate based on relative ranking of data (not on data itself)
Null hypothesis and alternative !! H0: Time series values are independent, identically distributed !! HA: There is a monotonic (not necessarily linear) trend Test statistic
S = " sign ( xk ! xi )
i< k
# 1 x>0 % with sign( x ) = $ 0 x = 0 % !1 x < 0 &

Mann 1945, Kendall 1975

!
Null distribution
!!
For large samples (about n>8) S is normally distributed with
E ( S ) = 0, var ( S ) !
!! !!
n " ( n # 1) " ( 2 n + 5) 18
If the residuals are mutually independent (e.g. no serial corr.) In practice use standardized test statistic Z and compare to Normal distribution
" 1/2 S ! 1 var S ( ) ( ) $ $ Z =# 0 1/2 $ S + 1 var S ( ) ( ) $ %
S>0 S=0 S<0
Remark
!!
Need correction of var(S) if there are some ties

see also Yue et al. 2002

!
Example
" = 0.087m3"s-1 / a! ! S = 387! Z = 1.96! ! qN(0.025)= 1.96! qN(0.975)= +1.96! ! p-value: 0.05! !
!! Trend is smaller and less significant than with linear regression

Accounting for Serial Correlation
16
Serial Correlation in Trend Tests

!
Serial correlation increases variance of trend estimates
true (stipulated) trend
r1=0.2!
!!
r1=0.4!
r1=0.6!
r1=0.8!
Random number experiment:

! 1000 synthetic time series composed of linear trend and AR(1) ! Trend estimated by Theil-Sen ! Sample size: 100, stipulated trend: 1.2*" change over the full series, !
r1: stipulated lag-1 autocorrelation.

Yue et al. 2002

! !
Serial correlation affects the Null-distribution of a trend test Experiment:
!! !! !!
False rejections of Mann-Kendall trend test for trendless AR(1) time series
Random AR(1) series, NO trend Mann-Kendall Test, #=5% Count rejections
False rejections (Type I errors) under H0 are much more frequent than nominal rate. Test is more liberal than anticipated.
significance level 5% 4 times the false rejections than nominal
Kulkarni & von Storch 1995; Yue et al. 2002

!
In the presence of serial correlation, trends can be estimated less accurately.
!!
The larger the correlation, the larger the uncertainty
! ! !
The Null-distribution becomes broader the larger the serial correlation. A trend test (without account of serial correlation) rejects NullHypothesis too often. There is a tendency for detecting a trend even when there is none.
19
Accounting for Serial Correlation

!
Pre-whitening:
!! !!
Idea: wash-out serial correlation from the data before testing Pre-whitening:
xi! = ( xi " r1 # xi"1 ) (1 " r1 )

r1: lag-1 autocorrelation
!!
If xi is a combination of a trend and AR(1), then the pre-whitened series xi has the same trend as xi , has serially uncorrelated residuals (from the trend), has a larger residual variance than the underlying AR(1).
the self-acting cleaning agent
von Storch 1995; Wang & Swail 2001, Yue et al. 2002, Zhang & Zwiers 2004
Proof
!
See e.g. Wang and Swail 2001 (Appendix A)
xt = ! + " t + qt
qt = r1 ! qt"1 + #t
AR(1)
xt ! r1 " xt!1 = ! ! ! " r1 + " t ! r1" (t ! 1) + qt ! r1qt!1 # "# $ !# # "## $ !

r1! +! t (1!r1 )
!t
= ! (1 ! r1 ) + r1 ! ! + t ! ! (1 ! r1 ) + !t
xt! = ! ! + " " t +
!t (1 # r1 )
!! = ! + "
var(!t )
r1 (1 # r1 )
var(!t ) (1 " r12 )
-> increase in variance
var(resid ( xt! )) =
(1 " r1 )
var(qt ) =
var(resid ( xt! )) = var(qt )
(1 + r1 ) > var(q ) t (1 " r1 )
21
Example
Daily Mean Temperature in Davos
r1 = 0.75
by factor 4 !
22
Question
!
Pre-whitening increases variance of time series by:
(1 + r1) (1 " r1)
Also when " =0, i.e. when no trend. Does this ring a bell ?
23
Question
!
Pre-whitening increases variance of time series by:
(1 + r1) (1 " r1)
Also when " =0, i.e. when no trend. Does this ring a bell ?
$1 + r1 ' s2 x xi from AR(1) " var( x ) = & ) n %1 # r1 (
Effective sample size Uncertainty of means.

24
Side-Effect of Pre-Whitening
!
Pre-whitening washes-out part of the trend
stipulated trend stipulated r1
r1 estimated from AR(1) with trend: !! r1 is mostly overestimated

in time series with trend.
Trend estimated after pre-whitening: !! True trend is underestimated in pre-whitened time series.
Simulated series with n=100, linear trend, AR(1), Theil-Sen trend estimator.
Yue et al. 2002
Recursive Estimation
!
Recursive pre-whitening & detrending
T-series
recursive loop
Initial Estim: r1 from acf, trend from pw. data
Detrending: New r1 from residuals
Prewhitening: New trend from pre-wht originals
no
new=old ?
yes
!
Test Result
Mann-Kendall: Test in pre-wht originals
Wang & Swail 2001, Zhang & Zwiers 2004

Example
!
River Rhine: Mean annual runoff (Mainz)
original Theil-Sen Trend Mann-Kendall Test p-val Lag-1 autocorrelation 2.9 m3 s-1 a-1 0.10 0.23
rec. pre-whitened 3.3 m3 s-1 a-1 0.17 0.20

27

!
False rejections with pre-whitening + Mann-Kendall
False rejections of MannKendall trend test with pre-whitening
significance level 5%
!!
False rejections close to nominal rate (a) also with AR(1)

Kulkarni & von Storch 1995; Yue et al. 2002
Application
!
Stream flow trends at undisturbed rivers in CH
Mean seasonal stream flow Trend in maximum winter stream flow
Fraction of rivers showing statistically significant (#=0.1) trends. Using the MannKendall trend test with pre-whitening for serial correlation. Black bars for increasing trends, empty bars for decreasing trends.
Maximum winter stream flow trend for individual river catchments. Upward triangle for significant upward trends, dots for non-significant trends.
quantiles of daily stream flow
Birsan et al. 2005

Trend Analysis for Count Data
30
Trend Analysis for Count Data

!
Event counts
!! !!
# threshold exceedances, # of heat days (Tmax > 30C)
# of winter days with rain > 12.5 mm (3 events on average per winter)
Linear regression ?
!! !! !! !!
Discrete and bounded data Possibly neg. predictions Skewed, residuals not normal Variable variance
Mann-Kendall ?
!!
Many ties!
31
Logistic Regression
!
Based on binomial distribution !! e(t): # events at t, a realization of binomial random variable E(t)!
parametric
E (t ) ~ B ( m, ! (t ))
!! !!
t = 0 .. T
m: max. possible counts (e.g. 90 winter days) $(t): event probability is time dependent
Transformed linear time dependence
! ( " ( t )) = # ! t + $
! (.) : the link function

x ( ] 0,1[ ) ! ( x ) ( ]!*, +*[
" x % ! ( x ) = logit ( x ) = log $ ' #1 ! x &

!!
A stochastic model for event counts with parameters #, ".

McCullagh & Nelder 1989, see also Frei & Schr 2001
Estimation and Test

! !
Logistic regression is special case of Generalized Linear Models Estimation (Maximum Likelihood Principal) !! estimates for #, ", (those values where log-likelihood is max)
!! !!
event probability as function of time (inverse of logit function) odds ratio $ (magnitude of trend)
!=
! (T ) (1 " ! (T ))
! (0) = exp ( ! # T ) (1 " ! (0))
!"
! (T ) ! (0)
if ! << 1
Testing (Maximum Likelihood Principle)

!! !!
Curvature of Likelihood function gives approx. confidence interval for ". Reject Null-Hypothesis ("=0) if CI of " does not contain 0.
R: glm!
Assumptions
!
Implied assumptions
!! !! !!
Counts are strictly binomially distributed Probability is the same with each toss in a group (each day of the year) No auto-correlation
Variable probabilities and auto-correlation

lead to larger than nominal binomial variance (overdispersion), inflate the dispersion of the Null-Distribution, can be corrected in testing, taking account of over-dispersion in the Nulldistribution.
McCullagh & Nelder 1989

Example
!
Heavy precipitation trends in CH
Odds ratio $ : 2.88! p-value: 710-5 Overdispersion: 1.6
! m $(t)!
Data variance is larger than nominal Binomial variance by a factor of 1.6!
Frei & Schr 2001

Application
!
Heavy precipitation trends in CH
Trend in the # of intense precipitation events in winter (events above the long-term 90% quantile) between 1901-2000. blue=increase, red=decrease, filled symbols: p-value < 0.05
Evolution of intense precipitation events. Northern Switzerland. Median of events at stations in domain.
Schmidli & Frei 2005

Additional Remarks
37
Fishing Expeditions
!
Want to be scientifically successful ? Then you should:
!! !!
Conduct many hypothesis tests, Publish the significant results only.
Of 100 independent tests, 5% (or whatever #) will be statistically significant BY CHANCE!
38
The Mexican Hat

!
Is the Mexican Hat natural (H0) or man made (HA)?
For development and testing of a hypothesis you need TWO independent datasets.
Subjective selection of data can trick the result of a hypothesis test.
The Mexican Hat desert of Utah/Arizona
von Storch 1995


Trend

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Trend

Hochgeladen von

Copyright:

Verfügbare Formate

Section 3: Trend Analysis

f (t ) : a deterministic function in time

! (t ) : stochastic variations, a random variable

Section 3: Trend Analysis

Parametric & Non-Parametric Tests

! (t ) ! N ( 0, " ), " (t ) = "

Ordinary least squares regression

standard error of slope

residual std. deviation

Trend Error and Sample Size

Trend Error and Sample Size

L: Length of time period N: Number of obs. in L!

R: lm, coef, resid, summary!

!! Trend is statistically significant, i.e. evidence for change

Theil-Sen Trend Estimate

" x k ! xi % !TS = median $ ' i< k # k !i &

Theil 1950, Sen 1968

Mann-Kendall Trend Test

a special case of Kendall s tau test

# 1 x>0 % with sign( x ) = $ 0 x = 0 % !1 x < 0 &

Mann-Kendall Trend Test

For large samples (about n>8) S is normally distributed with

" 1/2 S ! 1 var S ( ) ( ) $ $ Z =# 0 1/2 $ S + 1 var S ( ) ( ) $ %

S>0 S=0 S<0

Need correction of var(S) if there are some ties

Mann-Kendall Trend Test

!! Trend is smaller and less significant than with linear regression

Section 3: Trend Analysis

Accounting for Serial Correlation

Serial Correlation in Trend Tests

Random number experiment:

r1: stipulated lag-1 autocorrelation.

Serial Correlation in Trend Tests

Random AR(1) series, NO trend Mann-Kendall Test, #=5% Count rejections

Serial Correlation in Trend Tests

The larger the correlation, the larger the uncertainty

Accounting for Serial Correlation

xi! = ( xi " r1 # xi"1 ) (1 " r1 )

the self-acting cleaning agent

xt ! r1 " xt!1 = ! ! ! " r1 + " t ! r1" (t ! 1) + qt ! r1qt!1 # "# $ !# # "## $ !

var(resid ( xt! )) = var(qt )

(1 + r1 ) > var(q ) t (1 " r1 )

(1 + r1) (1 " r1)

(1 + r1) (1 " r1)

$1 + r1 ' s2 x xi from AR(1) " var( x ) = & ) n %1 # r1 (

Effective sample size Uncertainty of means.

r1 estimated from AR(1) with trend: !! r1 is mostly overestimated

Initial Estim: r1 from acf, trend from pw. data

Detrending: New r1 from residuals

Prewhitening: New trend from pre-wht originals

Mann-Kendall: Test in pre-wht originals

Wang & Swail 2001, Zhang & Zwiers 2004

rec. pre-whitened 3.3 m3 s-1 a-1 0.17 0.20

Serial Correlation in Trend Tests

False rejections close to nominal rate (a) also with AR(1)

quantiles of daily stream flow

Birsan et al. 2005

Section 3: Trend Analysis

Trend Analysis for Count Data

Trend Analysis for Count Data

# threshold exceedances, # of heat days (Tmax > 30C)

Transformed linear time dependence

! (.) : the link function

" x % ! ( x ) = logit ( x ) = log $ ' #1 ! x &