Power-Divergence Goodness-of-Fit Statistics:

Small Sample Behavior in One Way

Multinomials and Applications to Multinomial
Processing Tree (MPT) Models
Vicente Núñez-Antón and Miguel A. Garcı́a-Pérez
Universidad del Paı́s Vasco (Bilbao, Spain) and Universidad Complutense (Madrid, Spain)

• Provided there is an efficient method to determine π̂ = f (θ̂) ∈ Π0

Summary Introduction and Methodology that is most consistent with O, the power divergence statistic
k λ
(  )
λ 2 X Oi
• Evaluate small-sample behavior of power-divergence goodness-of- Pk 2I (O : ê) = Oi −1 (2)
• Let O = (O1, O2, . . . , Ok ) with k > 1, i=1 Oi = n and Oi ≥ 0 λ(λ + 1) êi
fit statistics with composite hypotheses with multinomial models i=1
(for all 1 ≤ i ≤ k) be the empirical distribution of n observations
of up to five cells and up to three parameters into k classes with λ ∈ R, êi = nπ̂i = nfi(θ̂), and s < k − 1 is asymptotically
• Assess performance by comparing asymptotic test sizes with exact k Pk
• Let π = (π1, π2, . . . , πk ) ∈ (0, 1) , with i=1 πi = 1 be a discrete distributed as a χ2 r.v. on k − s − 1 d.f. (Cressie and Read, 1984)
sizes obtained by enumeration in the near right tail (i.e.,1 − α ∈ distribution describing the probability of an observation’s falling
(0.90, 0.95]), and in the far right tail (i.e., 1 − α ∈ (0.95, 0.99]) • Asymptotic result may not be accurate in small-sample cases =⇒
into each class. Then, the probability of O under π is for simple null hypotheses =⇒ inaccurate test sizes when at least
• Address all combinations of power-divergence estimates of indices one expectation is small (Garcı́a-Pérez and Núñez-Antón, 2001)
k Oi
ν and power-divergence statistics of indices λ, such that ν, λ ∈ Y π i =⇒ some expectations are likely to be small with composite hy-
P (O; π) = n! (1)
{−1/2, 0, 1/3, 1/2, 2/3, 1, 3/2} Oi ! potheses
i=1
• Obtain combinations of ν and λ that provide a sufficiently accu- • Need to study accuracy of this asymptotic approximation with
rate asymptotic approximation (i.e., average exact size is no larger • Many goodness-of-fit problems =⇒ parametric models =⇒ π ∈
composite hypotheses
than ±10% of the nominal asymptotic test size), provided that the Π0 with Π0 = {π ∈ (0, 1)k : π = f (θ)}, where θ =
smallest expectation in the composite hypotheses exceeds five: (θ1, . . . , θs) ∈ Rs (s ≥ 1), f (θ) = (f1(θ), . . . , fk (θ)) ∈ (0, 1)k , • We evaluate the small-sample accuracy of the asymptotic approx-
Pk imation for a broad set of conditions involving a range of one-way
– In the near right tail, when ν = 0 and λ = 1/2 i=1 fi(θ) = 1
multinomial models with up to three parameters (i.e., 1 ≤ s ≤ 3)
• Testing the fit of the model Π0 to the data O (H0 : π ∈ Π0) and up to five cells (i.e., 2 < k ≤ 5), as a function of the power-
– In the far right tail, when ν = 0 and λ = 1/3
requires estimating θ divergence index λ

1 2 3

• Include one-parameter models (s = 1) with k = 3, 4, and 5 cells,

a two-parameter model (s = 2) with k = 4 cells, and a three Models included in the study Main Results: One Parameter Models
parameter model (s = 3) with k = 5 cells (shown on page 5)
(a) Model 3.1, ν = 1, λ = 1 (b) Model 3.5, ν = 1, λ = 1
k=3 k=4 k=5

Average relative error

1.0 1.0 1.0 0.6 0.6

• For each k, one-parameter models =⇒ expansion of [θ + (1 − 0.8

Model 3.1
1 0.8
Model 4.1
0.8
Model 5.1
0.4 0.4
Probability

0.6 0.6 0.6

0.2 0.2
4 2 5

θ)]m, 1 ≤ m ≤ k − 1 and the πi’s are polynomials in θ ranging

2
0.4 0.4 0.4
3 0 0
2
3 1 4 0.2 0.2
0.2 0.2 0.2
3 1
0.4 0.4
0.0 0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.6 0.6

from first degree [models 3.1, 4.1 and 5.1] up to (k − 1)-th degree
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
1.0 1.0 1.0
Model 3.2 Model 4.2 Model 5.2 Parameter estimate Parameter estimate
0.8 0.8 0.8 5 1
Probability

0.6 3 1 0.6 4 1 0.6

[models 3.2, 3.5, 4.3, 4.5, 5.4 and 5.5] 0.4

0.2
2
0.4

0.2
2
3
0.4

0.2 4
3
2
• Asymptotic distribution becomes poorer an approximation (i.e.,
0.0 0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

ARE increases) as the value of the estimated parameter θ̂ setting

Parameter θ

• Sample sizes n = 5k, 10k, 20k, and 40k

1.0 1.0
Model 4.3 Model 5.3
0.8 0.8
5 1
Probability

0.6 4 1 0.6

0.4 0.4 4

the composite hypothesis moves towards the boundaries of the

• Study covers power-divergence statistics of indices λ = 0.2 3 2 0.2 2
3

parameter space (complete probability degeneration)

0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Parameter θ
1.0

−1/2, 0, 1/3, 1/2, 2/3, 1, 3/2 =⇒ parameter estimates were ob- 0.8
Model 5.4
Probability

5 1
0.6

tained by minimizing power-divergence measures of indices ν =

0.4

0.2
4
3
2 • Smallest ARE occurs at the value of θ̂ where the smallest proba-
0.0

−1/2, 0, 1/3, 1/2, 2/3, 1, 3/2 =⇒ include all cases of matched and 1.0

0.8
Model 3.5
1.0

0.8
Model 4.5
0.0

1.0

0.8
0.2 0.4

Model 5.5
0.6 0.8 1.0

bility in the composite hypothesis is larger

1
Probability

1 1
0.6 2 0.6 0.6

unmatched statistics and estimate indices (λ = ν, Read and • Symmetric and asymmetric models present, accordingly, symmet-
3 4
0.4 0.4 0.4 3
2 2
0.2 0.2 0.2
3 4

Cressie, 1988 and λ 6= ν, Garcı́a-Pérez, 1994)

5

ric or asymmetric ARE patterns

0.0 0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Parameter θ Parameter θ Parameter θ

(b) Optimal unmatched case

• Discrepancy criteria? =⇒ FE(x; λ; ν): exact distribution func- One parameter, k = 3 π; [θ = (1 − θ)]
Average relative error

0.6 0.6 0.6

Model 3.1 Model 4.1 Model 5.1
0.4 0.4 0.4
3.1 {(θ, 2/3θ, 1/3θ)} 0.2
ν = 0, λ = 1/2
0.2
ν = 0, λ = 1/2
0.2
ν = 0, λ = 1/2

2
tion of the power-divergence statistic, FA(x; d): asymptotic
0 0 0
3.2 {(θ2 , 2θθ, θ )} 0.2 0.2 0.2
2 2 2 0.4 0.4 0.4
3.5 {(θ2 + 1/3θ , 2θθ + 1/3θ , 1/3θ )} 0.6
0.0 0.2 0.4 0.6 0.8 1.0
0.6
0.0 0.2 0.4 0.6 0.8 1.0
0.6
0.0 0.2 0.4 0.6 0.8 1.0

chi-squared distribution function on d d.f, Fdiff (x; λ; ν; d) = One parameter, k = 4 π; [θ = (1 − θ)]

Average relative error

0.6 0.6 0.6

Model 3.2 Model 4.2 Model 5.2
0.4 ν = 0, λ = 1/2 0.4 ν = 0, λ = 1/2 0.4 ν = 0, λ = 1/2
4.1 {(1/3θ, 2/3θ, 1/3θ, 2/3θ)} 0.2 0.2 0.2

FE(x; λ; ν) − FA(x; d), Rnear = {x : FA(x; d) ∈ (0.90, 0.95]} 4.2 {(θ2 , 2/3θθ, 4/3θθ, θ )}
2 3
2 0
0.2
0.4
0
0.2
0.4
0
0.2
0.4

4.3 {(θ3 , 3θ2 θ, 3θθ , θ )} 0.6 0.6 0.6

and Rfar = {x : FA(x; d) ∈ (0.95, 0.99]}. Then, use the Average

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
3 2 3 3 Parameter estimate
Average relative error

4.5 {(θ3 + 1/4θ3 , 3θ2 θ + 1/4θ , 3θθ + 1/4θ , 1/4θ )} 0.6

0.4
Model 4.3
ν = 0, λ = 1/2
0.6
0.4
Model 5.3
ν = 0, λ = 1/2

One parameter, k = 5 π; [θ = (1 − θ)] 0.2 0.2

Relative Error (ARE): 5.1 {(1/6θ, 1/3θ, 1/2θ, 1/3θ, 2/3θ)}

2
0
0.2
0.4
0
0.2
0.4

5.2 {(θ2 , 1/3θθ, 2/3θθ, θθ, θ )} 0.6

0.0 0.2 0.4 0.6 0.8 1.0
0.6
0.0 0.2 0.4 0.6 0.8 1.0

2 3 Parameter estimate
Average relative error

0.6
Z  5.3 {(θ3 , 2θ2 θ, θ2 θ, 3θθ , θ )} Model 5.4

|Fdiff (x; λ; ν; d)|

0.4 ν = 0, λ = 1/2
2 3 4 0.2
5.4 {(θ4 , 4θ3 θ, 6θ2 θ , 4θθ , θ )}
dx /kRik1, i ∈ {near, far} (3)
0
4 4 2 4 3 4 4 0.2
5.5 {(θ4 + 1/5θ , 4θ3 θ + 1/5θ , 6θ2 θ + 1/5θ , 4θθ + 1/5θ , θ )} 0.4

Ri 1 − FA(x; d)
0.6
Two parameters, k = 4 π; [θi = (1 − θi )] 0.0 0.2 0.4 0.6 0.8 1.0
Average relative error

0.6 0.6 0.6

4.2p {(θ1 θ2 , θ1 θ2 , θ1 θ2 , θ1 θ2 )} 0.4
Model 3.5
ν = 0, λ = 1/2 0.4
Model 4.5
ν = 0, λ = 1/2 0.4
Model 5.5
ν = 0, λ = 1/2
0.2 0.2 0.2
Three parameters, k = 5 π; [θi = (1 − θi )] 0 0 0
0.2 0.2 0.2
5.3p {(1/2θ1 θ2 , 1/2θ1 θ2 , θ1 θ2 , θ1 θ3 , θ1 θ3 )} 0.4 0.4 0.4
0.6 0.6 0.6
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Parameter estimate Parameter estimate Parameter estimate

4 5
6

• Differences for optimal matched (λ = ν) and unmatched (λ 6= ν) • Resultant ranges for the optimal pairing of λ and ν (i.e., the one • Minimum admissible value for the smallest expected frequency?
cases were minimal that produced the largest aggregated breadth at the smaller sample =⇒ optimal pairing together with the condition that the smallest
• Optimal unmatched pair never provides worse patterns that the size for the range of the value of θ̂ for which ARE ≤ 0.10 for the and/or second smallest expectation must satisfy to guarantee ARE
optimal matched pair two regions) =⇒ λ = 1/2 and ν = 0 ≤ 0.10 in either the near or far right tails
• Accuracy of the asymptotic approximation is mostly determined • Plots for the null hypotheses resulting in ARE ≤ 0.1 and in ARE
> 0.1, using the value of the smallest and second smallest expec- Main Results: Multi-Parameter Models
by the statistic index λ and not so much by the index ν used
in parameter estimation =⇒ practical recommendation =⇒ use tations as coordinates near tail far tail
ν = 0, λ = 1/2 ν = 1, λ = 1 ν = 0, λ = 1/2 ν = 1, λ = 1

MLE (ν = 0) and look for optimal λ =⇒ λ = 1/2 for all models 1

0.8
0.6
1
0.8
0.6
Model Tail ν λ Condition n = 20
Second parameter

λ = ν = –1/2 λ=ν=0 λ = ν = 1/3 0.4 0.4

Average relative error

0.6 0.6 0.6 0.2 0.2

0.4 0.4 0.4
0.2 0.2 0.2
3.1 near 0 1/2 E2 ≥ 5 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
1 1
0 0 0 far 0 1/2 E2 ≥ 8 0.8 0.8
0.2 0.2 0.2
3.2 near 0 0 E1 ≥ 8 0.6 0.6
n = 160
0.4 0.4 0.4
0.6 0.6 0.6 far 0 0 E1 ≥ 8 0.4
0.2
0.4
0.2
5k 10k 20k 40k 5k 10k 20k 40k 5k 10k 20k 40k
3.5 near 0 1/2 E1 ≥ 2 (not near equiprobability) 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
λ = ν = 1/2 λ = ν = 2/3 λ=ν=1 λ = ν = 3/2
far 0 1/2 E1 ≥ 4 (not near equiprobability)
Average relative error

0.6 0.6 0.6 0.6 First parameter First parameter

0.4 0.4 0.4 0.4
4.1 near 0 [1/2,2/3] E1 ≥ 2
0.2 0.2 0.2 0.2
0 0 0 0 far 0 1/3 E1 ≥ 2
0.2
0.4
0.2
0.4
0.2
0.4
0.2
0.4
4.2 near
far
0
0
2/3
1/2
E1 ≥ 2
E1 ≥ 3
• Plots that yield surfaces defined on the plane where all possible
pairs of parameter values setting up the composite hypothesis lie
0.6 0.6 0.6 0.6
5k 10k 20k 40k 5k 10k 20k 40k 5k 10k 20k 40k 5k 10k 20k 40k
Sample size Sample size Sample size Sample size
4.3 near [1/3,2/3] 1/2 E1 ≥ 2
far [0,2/3] 1/3 E1 ≥ 3
4.5 near [0,2/3] 1/3 E1 ≥ 3 (not near equiprobability)
• ARE increases with decreasing sample size far [0,2/3] 1/3 E1 ≥ 3 and E2 ≥ 12 (not near equiprobability) • Plots were created as gray level images instead of perspective ren-
5.1 near 0 1/2 E1 ≥ 1
far 0 1/3 E1 ≥ 1 ditions of the corresponding surface
• Indices λ = ν ∈ [1/3, 2/3] yield the best patterns at any sample 5.2 near
far
0
0
1/3
1/3
E1 ≥ 1
E1 ≥ 2
size across the entire set of models 5.3 near [1/3,2/3] 1/2 E1 ≥ 2 • In these plots, white represents ARE ≤ 0.1 and increasing gray
far [0,2/3] 1/3 E1 ≥ 2
5.4 near [0,2/3] 1/3 E1 ≥ 1 levels represent values of ARE in successive intervals of width 0.1
• For sample sizes n ≥ 20k, ARE’s lie below 0.1 in both the near far [0,2/3] 1/3 E1 ≥ 2
5.5 near [0,2/3] 1/3 E1 ≥ 2 or E1 ≥ 1 and E2 ≥ 4 (not near equiprobability) up to the interval (0.9,1], which is represented in black
and far right tails =⇒ difference between actual and nominal sig- 4.2p
far
near
0
[0,1]
1/3
[1/3,2/3]
E1 ≥ 4 or E1 ≥ 2 and E2 ≥ 4 (not near equiprobability)
E1 ≥ 5
nificance is less than 10% of the nominal level far [0,2/3] [1/3,1/2] E1 ≥ 5 • Optimal pairing of λ and ν (i.e., the one producing the largest
5.3p near 0 [1/3,3/2] E2 ≥ 5
far 0 [1/3,1/2] E2 ≥ 5 area of white in the plots) =⇒ ν = 0 and λ = 1/2
• ARE(λ = ν = 1/3) < ARE(λ = ν = 1/2) < ARE(λ = ν = 2/3)
• Same conclusion for three-parameter model where two-dimensional
• λ = ν = 1 (Pearson’s X 2 with minimum chi-squared estimates)
slices were analyzed instead
is better than λ = ν = 0 (likelihood ratio statistic G2 with MLE)
7 8 9