Classification Accuracy Assessment

Dr. Sao Hone Pha

Director and Head
Remote Sensing and GIS Research Center
Yangon Technological University

Date: 15th, October 2018

Certificate Training Course on Basic Concept of Remote Sensing & GIS (1.11.2018) to (30.11.2018)
 2

Accuracy Assessment
Accuracy
The degree (often expressed as a percentage) of correspondence between observation and reality.

We usually judge accuracy against existing maps, large scale aerial photos, or field checks
(information used as reference, considered “truth”).

 Comparison to two sources of information

 Remote Sensing derived classification map
 Reference Test information
 The relationship between the two sets of information's is expressed as a matrix known as
 ERROR MATRIX / CONFUSION MATRIX / CONTINGENCY TABLE
 3

Classification Error Matrix

Error matrices compare, on a category-by-category basis, the relationship between known
reference data (ground truth) and the corresponding results of an automated classification.

Square matrix with the number of rows and columns equal to the number of categories
whose classification accuracy is being assessed.

Error matrix stems from classifying the sampled training set pixels and listing the known cover
types for training (columns) versus the pixel actually classified into each land cover category
by the classifier (rows).
 4

Several Characteristics about Classification Performance

Major diagonal of the matrix  The training set pixels that are classified into the proper
and cover categories

All nondiagonal elements  Errors of Omission of the matrix (or)Errors of Commission

 5

Errors
Omission Error (related to “producer accuracy”)

 Error of exclusion
 Pixels are not assigned to its appropriate class, or
Pixels are omitted from the actual class they belong
 Correspond to non diagonal column element

Commission Error (related to “user accuracy”)

 error of inclusion
 pixel is assigned to a class to which it does not belong,
(or) pixels were improperly included in that category
 represented by non diagonal row element
 6

ERROR of Omission Vs. Commission

(From a Class1 Perspective)

Commission error: pixels in Class2 Omission error: pixels in Class1

erroneously assigned to Class 1 erroneously assigned to Class 2

Class 1 Class 2
Number of
pixels

0 Digital Number 255

 7

ERROR of Omission Vs. Commission

(From a Class2 Perspective)

Omission error: pixels in Commission error: pixels in

Class2 erroneously assigned to Class1 erroneously assigned to
Class 1 Class 2

Class 1 Class 2
Number of
pixels

0 Digital Number 255

 8

Overall Accuracy =
The total number of correctly classified pixels
The total number of reference pixels

Accuracy of individual category =

The number of correctly classified pixel in each category
The total number of pixels in the corresponding row or column
 9

Producer’s accuracy = (1-Omission error)

Omission error of a class = 100 – (% Accuracy of a class)

Producer Accuracy  indicates how well training set pixels of the given cover types are classified
The number of correctly classified pixels in each category (major diagonal)
The number of training set pixels used for that category (column total)

User Accuracy  indicates the probability that a pixel classified into a given category actually represented
that category on the ground
The number of correctly classified pixels in each category (major diagonal)
The total number of pixels that were classified in that category (row total)

User’s Accuracy = (1-Commission Error)

Commission error of a class = 100 – (% Accuracy of a class)
 10

So, from this assessment we have three measures of accuracy which address subtly different
issues:

Overall accuracy :

takes no account of source of error (errors of omission or commission)

User accuracy :
measures the proportion of each TM class which is correct.

Producer accuracy

measures the proportion of the land base which is correctly classified.

 11
∧
Kappa Coefficient ( 𝒌𝒌 or “KHAT”)

K statistics is a measure of the difference between the actual agreement between the
reference data and an automated classifier and the chance agreement between the reference
data and a random classifier.

∧
Conceptually 𝑘𝑘 can be defined as

Observed accuracy - chance agreement

k =
ˆ
1 - chance agreement
 12
∧
Kappa Coefficient ( 𝒌𝒌 or “KHAT”)

Observed accuracy - chance agreement

k =
ˆ
1 - chance agreement

The statistic serves as an indicator of the extent to which the percentage correct values of an
error matrix are due to the “true” agreement versus “chance” agreement

A true agreement (observed accuracy) approaches 1 and chance agreement approaches 0

then Kappa approaches 1

A kappa of 0 suggest that the classification is very poor and the given classifier is no better
than a random assignment of pixels
 13

∑x
i =1
ii

Observed Accuracy =
N
r

1 ∑ (x i+ ⋅ x+ i )
Change Agreement = i =1
N N

Where
r = number of rows in the error matrix
xii = number of observations in row i and column i (on the major diagonal)
xi+ = total of observations in row i (shown as marginal total to right of the matrix)
x+i = total of observations in column i (shown as marginal total at bottom of the matrix)
N = total number of observations included in matrix
 14

The KHAT statistic is computed as

Observed accuracy - chance agreement

k =
ˆ
1 - chance agreement
r r

∑x ii
1 ∑ (x i+ ⋅ x+ i )
i =1
- i =1

N N N
k̂ = r

1 ∑ (x i+ ⋅ x+ i )
1 - i =1
N N
r r
N ∑ xii − ∑ ( xi + ⋅ x+ i )
kˆ = i =1
r
i =1

N − ∑ ( xi + ⋅ x+ i )
2

i =1
 15

• Note : Overall Accuracy is biased on the correctly classified pixels (the major
diagonal) and excludes the omission and commission errors

• Kappa in addition to the correctly classified pixels, includes the off diagonal elements
as a product of Row total and column total
 16
CLASSIFIED IMAGE REFERENCE DATA

From airphotos, field data, etc.

1 Generate a random sample of points

2 For each point, compare the map class with the “true” class

3 Construct and analyze an error matrix...

 17

Error Matrix Resulting from Classifying Random ly Sam pled Test

Reference Data

Classification Data Water Sand Forest Urban Corn Hay Row total

Water 226 0 0 12 0 1 239

Sand 0 216 0 92 1 0 309

Forest 3 0 360 228 3 5 599

Urban 2 108 2 397 8 4 521

Corn 1 4 48 132 190 78 453

Hay 1 0 19 84 36 219 359

Column total 233 328 429 945 238 307 2480

Producer’s accuracy User’s accuracy

W = 226/233 = 97% U = 397/945 = 42% W =226/239= 94% U = 397/521 = 76%

S = 216/328 = 66% C= 190/238= 80% S = 216/309 = 70% C = 190/453 = 42%

F = 360/429 = 84% H = 219/307 =71% F = 360/599 =60% H = 219/359 = 61%

Overall accuracy = (226+ 216+ 360 + 397+ 190+219)/2480= 65%

 18

Error Matrix Resulting from Classifying Random ly Sam pled Test

r r
N ∑ xii − ∑ ( xi + ⋅ x+ i )
kˆ = i =1
r
i =1

N 2 − ∑ ( xi + ⋅ x+ i )
i =1

∑x
i =1
ii = 226 + 216 + 360 + 397 + 190 + 219 = 1608

∑ (x
i =1
i+ ⋅ x+ i ) = (239 × 233) + (309 × 328) + (599 × 429)

+ (521× 945) + (453 × 238) + (359 × 307) =

1,124, 382

(2480 × 1608) − 1,124, 382

kˆ = 0.57
(2480) 2 − 1,124, 382
 19

How accurate should the classified map be?

General rule of thumb is 85% accuracy

What’s a good Kappa?

General range
K < 0.4: poor
0.4 < K < 0.75: good
K > 0.75: excellent