STATS 200 (Stanford University, Summer 2015)

Lecture 10:

Tests Based on Asymptotic Properties

A hypothesis test (or more precisely, a collection of hypothesis tests with the same test
statistic) is only useful if we can determine a way to set its significance level, at least approximately. This task requires knowledge of the approximate distribution of the test statistic,
which may be easier to find in the asymptotic limit. If we choose a test statistic for which
the asymptotic distribution is known, then we can use it to construct a hypothesis test.

10.1

Wald Tests

Recall that under certain regularity conditions, the asymptotic distribution of the maximum
likelihood estimator is normal. More specifically, if n is the MLE of , then

n(n ) D N [0,

1
],
I1 ()

where I1 () denotes the Fisher information per observation. Equivalently,

In () (n ) D N (0, 1),
where In () = n I1 () denotes the Fisher information in the entire sample of size n. Note
that In () depends on the unknown parameter . However, we also know that under suitable
regularity conditions, we also have

In (n ) (n ) D N (0, 1),
Jn (n ) D N (0, 1),
where Jn = `Xn (n ) (sometimes called the observed information).
Definition of Wald Tests
The asymptotic results above suggest two similar hypothesis testing methods that are useful
for large n. One test of H0 = 0 versus H1 0 with approximate size is to reject H0
if and only if

In (n ) n 0 c,
(10.1.1)
where c is the number such that P (Z c) = for a standard normal random variable Z.
Another test of the same hypotheses with approximate size is to reject H0 if and only if

Jn n 0 c,
(10.1.2)
where c is the same as above.
Note: Recall that if Z N (0, 1), then W = Z 2 21 . Thus, it is equivalent to state
these tests as rejecting H0 if and only if the squares of their respective test statistics are
at least k, where k is the number such that P (W k) = for a 21 random variable W .

The two tests with rejection regions defined by (10.1.1) and (10.1.2) are called Wald tests.

Lecture 10: Tests Based on Asymptotic Properties

Example 10.1.1: Let X1 , . . . , Xn iid Exp(), where > 0, and consider testing H0 = 2
versus H1 2. Earlier in the course, we showed that the maximum likelihood estimator
of is
n
1
n =
= ( X n) .

n
i=1 Xi
Next, note that
`Xn () =

n
2
n
(n
log

Xi ) = 2
2

i=1

n ) = n/
2n .
n ) = n/
2n . Similarly, Jn = ` (
so In () = E [`X ()] = n/2 , and hence In (
Xn
Then the Wald test statistic defined by both (10.1.1) and (10.1.2) is

n
2

n 2 = n 1 = n 1 2X n .
In (n ) n 2 = Jn n 2 =
2n
n

Thus, the Wald test of these hypotheses is to reject H0 if and only if this test statistic is at
least c, where c is some critical value. (To obtain size = 0.05, we would take c 1.96.)
The two versions of the Wald test sometimes coincide, as in Example 10.1.1, but this is not
always the case.
Example 10.1.2: Let X1 . . . , Xn be a iid random variables with the Cauchy distribution
centered at , which has pdf f (x) = 1 [1 + (x )2 ]1 , where R is unknown. Then the
maximum likelihood estimator n of cannot be computed in closed form and must instead
be found numerically (unless n 2). Now note that
`X1 () =

2(X1 )
2[1 (X1 )2 ]

2
]}
{
=
log

log[1
+
(X

)
[
]
=
1
2 .
2
1 + (X1 )2
[1 + (X1 )2 ]

It can be shown (after some calculus) that E [`X1 ()] = 1/2, so the Fisher information
is In () = n/2. On the other hand, Jn must be calculated by evaluating `Xn () at the
numerically calculated MLE, and the result will not (in general) be equal to n/2. Hence, the
two formulations of the Wald test would not coincide in this case.

When people simply refer to the Wald test when the two formulations do not coincide,
it is not necessarily clear which test they mean. There exist technical theoretical reasons
why (10.1.2) may be slightly better, but these reasons are beyond the scope of this course.
Reference: Efron, B., and Hinkley, D. (1978). Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information. Biometrika,
65 457482.

Despite these theoretical advantages, textbooks tend to prefer (10.1.1). However, statistical
software usually uses (10.1.2) since it is more straightforward to calculate numerically.
Note: You may wonder why we have not considered rejecting H0 if and only if

In (0 ) n 0 c.
We could indeed construct and use such a hypothesis test, and some people might
also refer to this type of test as a Wald test. However, we will see later in the course
that this version of the Wald test would make the associated Wald confidence interval
considerably more awkward to calculate.

Lecture 10: Tests Based on Asymptotic Properties

10.2

Score Test

Recall that under certain regularity conditions, the asymptotic distribution of the score
function `X () itself is normal. More specifically,
1
1
n[ `Xn () 0] = `Xn () D N [0, I1 ()].
n
n
Then it follows that

1
n I1 ()

`Xn () =

1
In ()

`Xn () D N (0, 1),

i.e., the quantity [In ()]1/2 `Xn () has an approximate N (0, 1) distribution if n is large.
Definition of Score Test
The asymptotic result above suggests a hypothesis testing method that is useful for large n.
Specifically, a test of H0 = 0 versus H1 0 with approximate size is to reject H0 if
and only if

1
In (0 )

`Xn (0 ) c,

(10.2.1)

where c is the number such that P (Z c) = for a standard normal random variable Z.
Note: Recall that if Z N (0, 1), then W = Z 2 21 . Thus, it is equivalent to state this
test as rejecting H0 if and only if the square of the test statistic above is at least k,
where k is the number such that P (W k) = for a 21 random variable W .

A test with rejection region defined by (10.2.1) is called a score test.

Example 10.2.1: Let X1 , . . . , Xn iid Exp(), where > 0, and consider testing H0 = 2
versus H1 2. The score function is
`Xn () =

n n
1
(n log Xi ) = Xi = n( X n ),

i=1

i=1

where X n = n1 ni=1 Xi . Next, note from Example 10.1.1 that In () = n/2 . Then the score
test statistic is

1
1
1
`Xn (2) =

n( X n ) = n 1 2X n ,
2
In (2)
n/4
and the score test rejects H0 if and only if this test statistic is at least as large as some
critical value c. (To obtain size = 0.05, we would take c 1.96.) Note that in this
particular example, the score test coincides with the Wald tests from Example 10.1.1.

Unlike a Wald test, a score test does not involve the maximum likelihood estimator of the
parameter of interest. Thus, even when no closed-form solution for the MLE exists, it may
still be possible to express a score test in closed form.

Lecture 10: Tests Based on Asymptotic Properties

Example 10.2.2: Let X1 , . . . , Xn be iid continuous random variables with pdf

( + x)2
f (x) =

if x 0,
if x < 0,

where > 0 is unknown. The score function is

`Xn () =

n n
2
[n log 2 log( + Xi )] =
.

i=1 + Xi
i=1

Note that no closed-form expression for the maximum likelihood estimator exists since solving
the equation
2
n n
=
i=1 + Xi
for symbolically is impossible in general (unless n 4). However, consider a score test of
H0 = 0 versus H1 0 for some specified 0 > 0. Observe that
`Xn () =

n
n
2

`Xn () = 2 +
,
2

i=1 ( + Xi )

and it can be shown (after some calculus) that

In () = E [`Xn ()] =

n 2n
n
2 = 2.
2

3
3

Then the score test statistic is

n

n
2
1 n 20
`Xn (0 ) =

= 3n 1
=
n i=1 0 + Xi
n/(302 ) 0 i=1 0 + Xi
In (0 )

3 n Xi 0

,
n i=1 Xi + 0

and the score test rejects H0 if and only if this test statistic is at least as large as some
critical value c. (To obtain size = 0.05, we would take c 1.96.) Thus, we can express a
score test for this example in closed form despite the fact that no closed-form solution exists
for the maximum likelihood estimator.

10.3

Asymptotics of Likelihood Ratio Tests

If we have already chosen to resort to asymptotic results to determine the distribution of

a test statistic, it is logical to ask whether there exist such results for the likelihood ratio
statistic itself. The following theorem provides precisely such a result.
Theorem 10.3.1. Let (Xn ) be the likelihood ratio statistic for testing H0 = 0 versus
H1 0 based on the sample Xn . Then under the regularity conditions of Section 7.4 of
Lecture 7,
2 log (Xn ) D 21 if = 0 ,
where 21 denotes a chi-squared random variable with one degree of freedom.

Lecture 10: Tests Based on Asymptotic Properties

Proof. Let n denote the MLE of . A Taylor expansion of `Xn (0 ) around `Xn (n ) yields
2
1
`Xn (0 ) = `Xn (n ) + `Xn (n ) (0 n ) + `Xn (n ) (0 n ) +
2
2
1
= `Xn (n ) + `Xn (n ) (0 n ) +
2

since `Xn (n ) = 0. (Also note that the purpose of the regularity conditions is to allow us to
ignore the higher-order terms.) Now observe that
2 log (Xn ) = 2 log[

2
LXn (0 )
] = 2[`Xn (0 ) `Xn (n )] `Xn (n ) (0 n )
LXn (n )

by the Taylor expansion. Then

2
2
2
2 log (Xn ) `Xn (n ) (0 n ) = Jn (n 0 ) = [ Jn (n 0 )] .
Now observe that if the true value of is 0 , then the sequence of random variables in square
brackets converges in distribution to a N (0, 1) random variable. Since the square of a N (0, 1)
random variable is a 21 random variable, the result then simply follows from the continuous
mapping theorem for convergence in distribution.
If n is large, Theorem 10.3.1 shows how to find a critical value that yields a likelihood ratio
test with approximate size . Specifically, a test of H0 = 0 versus H1 0 with
approximate size is to reject H0 if and only if
2 log (Xn ) C
where C is the number such that P (W C) = for a 21 random variable W , or equivalently
the number such that P (Z C 1/2 ) = for a N (0, 1) random variable Z.
Note: This test is equivalent to taking the original critical value c for the likelihood
ratio test as defined in Lecture 9 to be c = exp(C/2).

Example 10.3.2: Let X1 , . . . , Xn iid Exp(), where > 0, and consider testing H0 = 2
versus H1 2. From Example 9.0.1 of Lecture 9, the likelihood ratio statistic is
n

(Xn ) = [2X n exp(1 2X n )] .

Note that
2 log (Xn ) = 2n[1 + log(2X n ) 2X n ].
To obtain a likelihood ratio test with approximate size , we should reject H0 if and only if
this test statistic is at least as large as some critical value C. (To obtain size = 0.05, we
would take C 1/2 1.96, and hence C 3.84.)

Lecture 10: Tests Based on Asymptotic Properties

10.4

Summary and Comparison of Asymptotic Testing Methods

The Wald tests, score test, and likelihood ratio test provide different ways to construct
hypothesis tests of H0 = 0 versus H1 0 with approximate size for large n.
The Wald tests are based on the difference between 0 and the MLE n .
The score test is based on the difference between slope of the log-likelihood at 0 and
the slope of the log-likelihood at n (the latter of which is simply zero).
The likelihood ratio test is based on the difference between the log-likelihood at 0 and
the log-likelihood at n .
Also note that when these tests are computed numerically, they require calculations involving
the log-likelihood at different points.
The Wald test (when based on Jn ) only involves the behavior of the log-likelihood at
and around its global maximum n .
The score test only involves the behavior of the log-likelihood at and around 0 .
The likelihood ratio test involves the behavior of the log-likelihood at both 0 and n .
Reparametrization
These tests also exhibit different behavior in regard to reparametrization. Suppose = g()
and 0 = g(0 ) for some one-to-one function g.
The Wald tests of H0 = 0 versus H1 0 will differ from the Wald tests of
H0 = 0 versus H1 0 . Hence, we could (for the same data) reject that = 0
but fail to reject that = 0 , or vice versa.
The score and likelihood ratio tests are invariant to reparametrization, i.e., it is irrelevant for these tests whether or is treated as the parameter.
Quality of Approximation in Finite Samples
Finally, remember that the results developed in this lecture are approximate results that hold
when the sample size is large. As a general rule of thumb, the quality of the approximation
tends to be better for the score test and likelihood ratio test than for the Wald tests.
Note: You may wonder, then, why Wald tests are so widely used in practice. One
reason is that each hypothesis testing method gives rise to an associated method for
constructing confidence intervals. The general form of the confidence intervals arising
from the Wald test is extremely straightforward, which is not true for the score and
likelihood ratio tests. We will see all of these ideas later in the course.

Also be aware that people sometimes perform Wald tests or score tests by finding the critical
value based on a tn1 distribution instead of a N (0, 1) distribution. This can sometimes
improve the quality of the approximation for finite n. Similarly, the likelihood ratio test can
be performed by finding the critical value based on the distribution of the square of a tn1
random variable (which is called the F1, n1 distribution) instead of a 21 distribution.