Framework for network-level pavement condition assessment using

remote sensing data mining

Stefanos S. Politis,a,* Zhanmin Zhang,a Sareh Kouchaki,b Carlos H. Caldasa
a
University of Texas at Austin, Department of Civil, Architectural and Environmental Engineering, 301 E. Dean
Keeton St., Austin, TX, United States of America, 78712
b
Wood Environment & Infrastructure Solutions, Inc., 16414 San Pedro Avenue, San Antonio, TX, United States of
America, 78232

Abstract. Pavement condition monitoring is fundamental for the efficient allocation of resources in transportation
asset management. However, data collection involves laborious and costly procedures. This study intends to
investigate the usage of remote sensing data for network-level pavement condition assessment offering a more cost-
effective alternative and a rapid infrastructure assessment tool which can be used in the aftermath of natural disasters.
Based on an extensive literature review, a data mining framework was established to train models that predict the
pavement condition of different road segments. The framework exploits the inherent information of multispectral
images by generating spectral related attributes. To identify pavement sampling areas, an automated procedure using
image segmentation replaces manual surface digitizing. Unlike previous research, different classification models to
approximate the mapping function from spectral information to pavement conditions. A preliminary case study was
conducted with data provided by the City of Dallas and multispectral images acquired from the Texas Natural
Resources Information System. The mean-shift segmentation algorithm was used to locate noise introducing areas on
the pavement surface. Four different classification models were trained using k-nearest neighbors, naïve Bayes,
support vector machines, and a multilayer perceptron. The developed models were employed to predict the road
surface condition class of a test set not included in the training procedure. The multilayer perceptron presented the
highest accuracy level of 71 percent, showing that the framework might have the potential for future implementation.

Keywords: classification, data mining, image processing, infrastructure management, pavement condition, remote
sensing.

*Stefanos Politis, E-mail: stfpolitis@utexas.edu

1 Introduction

Pavement data collection technologies have seen a widespread application during the past

decade 1. Traditional manual field inspections and video logging used to collect pavement

condition data are getting replaced by mobile automated systems. These systems include high-

speed laser, acoustic and infrared imaging sensors. Using measurements of surface distress,

roughness, deflection and skid resistance, pavements are assigned an index score that reflects their

1
overall condition. Transportation infrastructure management agencies have developed various

such indices and use them as a basis for resource allocation and maintenance decisions.

Maintenance divisions or external vendors conduct automated surveys using equipment

mounted to vans designed for collecting pavement and roadway characteristic data. Assessment of

the condition of pavements through existing inspection approaches can be expensive, laborious,

and time-consuming 2. The costs of automated pavement condition data collection and processing

vary. Full-featured collection and processing averages more than $30 per lane-km ($50 per lane-

mi) and may reach $125 per lane-km ($200 per lane-mi) 3. The cost can be even higher in urban,

high-traffic areas. By considering the vast network size that a typical transportation agency

manages, the total cost of pavement condition surveying and processing can reach several million

dollars per year. Furthermore, traditional pavement data collection involves disruptions of traffic

due to its stop-and-go, low-speed nature 4. Significant safety hazards arise due to that fact and there

is no capability to rapidly assess pavement condition after natural disasters. As an alternative,

developments in remote sensing and data science have shown potential in using high-resolution

images to assess pavement conditions inexpensively 5. Sensors mounted on different platforms,

such as unmanned aerial vehicles, airplanes and satellites can capture these images.

Data collection requirements for network and project-level decisions present major

differences 6. At the network level, agencies collect a large amount of pavement condition data,

which is transformed into composite indices or scores. This level of information is most

appropriate for decision-makers to prioritize pavement segments and to make multi-year

projections for the overall network condition. On the other hand, project-level data collection

involves detailed distress identification and severity assessments. Engineers can use these

assessments to select specific maintenance and rehabilitation treatments and estimate project costs.

2
 7
To assist in data collection and satisfy the corresponding requirements, Paterson and Scullion

proposed dividing data needs into different information quality levels. Each level correlates to the

degree of sophistication required for transportation asset management decision making. In this

context, they can aggregate very detailed data into progressively higher-level forms. Such forms

can be specific key performance measures or indicators that combine key factors from several

pieces of information. Bennett and Paterson 8 defined five such information quality levels (IQL).

These are ranging from project-level (IQL 1) to high-level system performance monitoring (IQL

5) data. The methods employed in this paper focus more on the planning and performance

evaluation information level (IQL 4) based on this classification. Consequently, this research

focuses on assessing the overall pavement condition and does not consider detecting individual

distresses.

After delimitating the scope of this paper, a review of the remote sensing applications for

pavement performance assessment is provided. The applications that this review entails, involve

pavement condition information in the planning and performance evaluation information level

(IQL 4) and utilize aerial and satellite images of high and very high spatial resolution. Although

common pavement distresses, such as cracking and rutting, cannot be directly detected in these

aerial and satellite images, there have been many interesting efforts that try to model the

relationship between some spectral attributes and higher-level condition indicators. For that

reason, both hyperspectral and multispectral data have been used. A summary of the sensors and

data types that has been used in the literature for pavement condition assessment can be found in

Table 1. It is noted that the spatial resolution mentioned in that table refers to pan-sharpened

images for sensors with panchromatic bands. For the ground spectrometers the resolution refers to

3
the diameter of the corresponding circular areas that are measured on ground based on the

measurement distance and the instrument’s field of view.

Table 1 Spectral data used in previous research for pavement condition assessment.
Authors Sensor name Platform Data type Spectral range Spatial resolution
Herold et al. 9 IKONOS Satellite Multispectral 0.45 - 0.85 μm ~ 0.8 m
AVIRIS Airborne Hyperspectral 0.35 - 2.50 μm ~ 3.6 m
ASD FR FieldSpec Ground Hyperspectral 0.35 - 2.50 μm ~ 0.4 m
Herold et al. 10 HyperSpectir Airborne Hyperspectral 0.45 - 2.45 μm ~ 0.5 m
ASD FR FieldSpec Ground Hyperspectral 0.35 - 2.40 μm ~ 0.4 m
Kavzoglu et al.11 IKONOS Satellite Multispectral 0.45 - 0.85 μm ~ 0.8 m
ASD FR FieldSpec Ground Hyperspectral 0.35 - 2.50 μm ~ 0.4 m
Pascucci et al.12 MIVIS Airborne Hyperspectral 0.43 - 12.70 μm ~ 3.0 m
Mei et al. 13,14 MIVIS Airborne Hyperspectral 0.43 - 12.70 μm ~ 3.0 m
ASD FieldSpec3 Ground Hyperspectral 0.35 - 2.50 μm ~ 0.1 m
Mei & Salvatori 15 IKONOS Satellite Multispectral 0.45 - 0.85 μm ~ 4.0 m
ASD FieldSpec3 Ground Hyperspectral 0.35 - 2.50 μm ~ 0.1 m
Andreou et al. 16 CASI 550 Airborne Hyperspectral 0.40 - 1.00 μm ~ 0.5 m
SVC GER-1500 Ground Hyperspectral 0.28 - 1.09 μm ~ 0.2 m
Mettas et al. 17 Landsat 7 ETM+ Satellite Multispectral 0.44 - 2.40 μm ~ 15.0 m
SVC HR-1024 Ground Hyperspectral 0.35 - 2.50 μm ~ 0.1 m
Mohammadi 18 HyMap Satellite Hyperspectral 0.45 - 2.49 μm ~ 4.0 m
Pan et al. 19 WorldView-2 Satellite Multispectral 0.45 - 0.80 μm ~ 0.5 m
Shahi et al. 20 WorldView-2 Satellite Multispectral 0.45 - 0.80 μm ~ 0.5 m

Many attempts have been made to utilize hyperspectral field spectroscopy in order to

characterize pavement conditions 9–11,13–17. The objective of these efforts is the development of a

relationship that is later extrapolated for usage in remote sensing images to provide assessments
9
with high area coverage. Herold et al. observed that for recently paved roads, the spectral

reflectance was generally lower across the range of the measured spectrum. On the contrary, for

old and deteriorated road surfaces, a general increase in reflectance was found. This difference

was higher in the infrared spectrum. The corresponding spectral signatures derived from the

imagery was presented and compared to the ground measurements. It was observed that there were

large differences that can be attributed to atmospheric and systematic noise, as well as spectral

mixture effects of surfaces other than pavements. The authors also identified several bands that

could be used to identify raveling and aging effects. A similar methodology was also followed by
4
 11 10
Kavzoglou et al. . In later work, Herold et al. integrated hyperspectral data with in situ

pavement condition surveys. This was the only effort found in the cited literature that utilized

pavement condition data collected according to established protocols. The research focused only

on the empirical relationship between the reflectance difference of two specific bands with the

pavement condition index and a structural index. The two bands were in the blue visible and near-

infrared spectrum.
12
Pascucci et al. focused on pavement surface reflectance in the long-wave infrared

spectrum. The authors tried to exploit the absorption property of limestone at a specific band of

that spectrum. The notion behind this research was that since limestone was the dominant

aggregate of asphalt concrete in the study area, it could be captured in the thermal infrared

spectrum. This threshold was used to discriminate between pavements in good condition and those

that had to be checked for maintenance. However, the threshold that the authors identified was

based on limited field observations. Later, Mei et al. 13–15 explored pavement aggregate exposure

using a quantitative metric in the form of an index. This metric was based on area calculation of

exposed aggregate coverage in natural color band images. The authors identified four asphalt

surface brightness clusters in commercial multispectral images. These clusters were assumed to be

associated with bitumen removal.

Andreou et al. 16 developed a spectral library for asphalt from field measurements, defining

five potential categories of asphalt condition and minimizing the dimension of the hyperspectral

space. These categories ranged from good condition to highly distressed pavements. Different

processing methods were used, and it was found that principal component analysis performed well

at distinguishing between asphalt conditions. Mohammadi 18 used hyperspectral data and asphalt

signatures assuming that the mean reflectance from the visible to the short-wave infrared spectrum

5
would be disproportional with the pavement condition. Three states of condition (good,

intermediate, and bad) were defined, and the classification results were compared to limited field
17,21,22
visits. Mettas et al. , developed separability indices for cracked and uncracked pavement.

The authors recognized that the blue visible and the short-wave infrared spectrum presented the

highest separability values. The indices were extended for an in-band analysis of a multispectral

satellite sensor.

There is a general shift in the literature from approaches that tried to find the highest

correlation between specific bands of the spectrum and a performance indicator. Recent works 19,20

have instead tried to move away from costly and generally unavailable hyperspectral datasets and

try to exploit as much information given overall by the spectral and spatial patterns of commercial

multispectral images. Shahi et al. 20 used object-based image analysis to differentiate between good

and poor condition classes using satellite images. Spatial, spectral, textural, and color-related

attributes were generated, and different feature selection techniques were used and compared.

These techniques included support vector machine, random forest, and chi-square algorithms that

were evaluated to select the most effective one in identifying the best set of attributes. Pan et al. 19

used also commercial satellite images and spectral mixture analysis to classify the age of the

pavements. Some asphalt pavement pixels were covered by other objects (e.g., vegetation,

vehicles, sidewalks), and spectral unmixing was performed to calculate the percentage of

pavement surface found in the mixed pixels. Based on field investigation and in situ measurements,

aged asphalt pavements were categorized into four stages: preliminarily aged, moderately aged,

heavily aged, and distressed.

A very important issue found in the review of the cited literature is that most of the existing

applications are not based on actual condition data but rather use proxy indicators such as

6
assessments of pavement age and brightness that are interrelated with aggregate exposure and

bitumen removal. Also, the vast majority of developed models are not adequately validated. The

validation is most usually based on limited subjective on-site evaluations conducted by the

researchers rather than using standardized pavement data collection protocols. The only exception

found was the work of Herold et al. 10 that utilized Roadware PCI data in developing a relationship

with a specific spectral attribute. However, even in that research the proposed relationship was not

applied in unseen data and there is no estimate concerning the generalization error of the model.

The relationship was based on a limited number of pavement condition data samples and exhibited

relatively low correlation. Therefore, applicability of remote sensing data in pavement condition

assessment remains obscure.

Furthermore, previous research in the field of pavement condition assessment using remote

sensing has not fully exploited developments in computer vision and data science. Most of the

studies focus only on limited image processing steps and consider specific prediction model

formulations. In this paper, we develop a comprehensive methodological framework to address

the aforementioned shortcomings. The main research challenge that we intend to tackle is

exploring whether remote sensing can be practically used to assess pavement condition. To address

this question different machine learning models are trained and tested against unseen data in order

to provide an overview of their generalization accuracy. The framework includes a multitude of

techniques to analyze image data. Also, instead of manually digitizing pavement sampling areas

upon the images, an automated method employing image segmentation is implemented. A

description of the methodological framework is provided in the next section.

3 Methodological Framework

This research focuses on developing a network-level road surface condition classification

7
framework that exploits publicly available multispectral images. The outputs of the framework

will provide high-level information to transportation asset managers. This will contribute to

reducing pavement data collection costs. To achieve this goal, we use different supervised data

mining techniques to classify road pavements’ condition based on the inherent multispectral ortho-

image information. A synthesis of best practices found in the literature is used in developing the

framework. The methodological framework is generic. That means it describes several basic steps

for analysis using readily available data, as shown in Fig. 1. It is divided into three main stages,

which include data selection, analysis, and data mining techniques that are embedded for extracting

useful information. A description of the processes included in the framework is provided.

Fig. 1 Framework for pavement condition assessment using remote sensing data.

3.1 Data Selection

The first stage of the framework deals initially with collecting and preparing pavement condition

data for analysis. The critical factors of this process are the selection of specific pavement types

8
and condition updates based on maintenance projects completed between pavement inspections

and imagery sensing. Pavements that are made of different materials (e.g. asphalt or cement

concrete) have different characteristics and thus separate models might have to be developed.

However, in this research we focus on asphalt pavements that usually comprise the majority of a

city’s street network. Except for pavement condition inspection data, transportation agencies

usually publish an inventory of geospatial data that corresponds to the centerlines of the road

network. This inventory is usually provided in vector format and is made available for public use.

In this research, it is assumed that readily available centerline information will provide a sufficient

pavement surface sample while advanced road centerline extraction techniques can be used to

increase location precision 23. In case the road segments’ location and geometry are not available,

roadway centerlines can be extracted using remote sensing data. This topic has been addressed

extensively by many researchers, but it is not part of the scope of this investigation.

Multispectral images capture data within specific wavelength ranges of the electromagnetic

spectrum. The wavelengths are separated using instruments that are particularly sensitive to these

or by using specific filters. These images may capture data from frequencies beyond the visible

range, i.e. infrared and ultra-violet. Thus, spectral imaging allows the extraction of additional

information the human eye cannot sense. Contrary to hyperspectral imaging where often hundreds

of contiguous spectral bands are captured, in multispectral images a small number typically 3 to

15 of spectral bands are included. Many organizations perform imaging acquisition programs on

a recurrent basis, which has made multispectral images easily accessible comparing to

hyperspectral data that are generally limited. For this reason, multispectral images are utilized in

the framework to extract spectral attributes that can be used for pavement condition classification.

9
3.2 Data Analysis

The first step for data analysis is data cleaning. This involves removing irrelevant attributes from

the pavement condition inspection tables, as well as the attribute table of the roadway linework

vector data. The removed attributes include street names and numbers, roadway type and other

information. After this process, each road segment is characterized only by the corresponding

pavement condition index attribute that is selected. Pavement condition data can be cross-checked

with maintenance and rehabilitation history for changes and inaccurate records. The pavement

condition data can be integrated with road centerline and project information data, using a linear

referencing tool or a unique key attribute. In this way, the inspection values of the pavement

condition can be assigned to the corresponding road segments.

After integrating the linework with the condition inspection data, the vector features that

contain the information must be re-projected to the coordinate system of the multispectral image.

This is necessary for the two datasets to align spatially (Fig. 2a) in support of geoprocessing

operations to take place later. By aligning the two datasets, buffer zones with variable width are

then created around the linework so that pavement sections can be sampled in the multispectral

images (Fig. 2b). In locations where two street segments intersect, a small circular buffer zone is

removed to avoid overlays in pavement sampling. The multispectral images need to be

radiometrically corrected to reduce errors in their intensity. The process improves the

interpretability and quality of remotely sensed data. Radiometric calibration and correction are of

extreme importance when comparing data sets over multiple periods.

An important aspect of the classification model development is the transformation of

intensity from digital numbers to apparent reflectance. This topic is of high importance when using

multiple images from different sensing periods or different sensors. Reflectance is a spectral unit

10
that provides a common measure between different sensing systems. The spectral signature of a

pavement surface is not transferable if the intensity values of the electromagnetic radiation are

recorded in digital numbers. Digital numbers are image specific. They depend on the gain and bias

of the sensor at each band, the viewing geometry, the location of the sun, as well as weather and

other factors. In certain applications, it is more useful to convert the digital number values to

reflectance. This conversion will enable direct application of classification models to other images

for rapid assessment of pavement conditions. In cases where the sensing specifications are not

known, the empirical line method along with pseudo-invariant surfaces can be proved useful tools

for reflectance transformation 24.

11
Fig. 2 Study area subregion: (a) centerline, (b) buffer zones, (c) segmentation and (d) extracted pavement sections.

An attribute generation procedure can be applied to extract not only spectral but also

textural and geometric information from the different bands. Many such metrics can be used, some

of which were employed in Shahi et al. 20. The different vector polygons are used to extract the

information. If the distribution of each attribute value cannot be used as an input, basic descriptive

12
statistics can be calculated for each polygon. These might include minimum and maximum values,

ranges, mean, median, mode, standard deviation, variance, percentiles, etc.

After forming the attribute table, a supervised classification scheme is used to identify and

extract the clusters that represent pavement surfaces. The objective is to remove all the other

sections covered by non-pavement surfaces automatically. There are two ways to deal with this

classification procedure. The first is to create a training set by manually assigning a binary class

that separates pavement and non-pavement covered polygons. Some classification algorithms that

can be used for this classification are described in Sec. 3.4. An alternative method would be using

existing libraries of the spectral signatures of different materials. For this research the first method

has been employed and object-based binary classification has been used to extract pavement

surfaces. In this process the clustering algorithm used for segmenting the image and creating the

corresponding objects has a crucial role in the final accuracy of the results. The objects that are

extracted are subsequently dissolved within each road segment (Fig. 2d). Following this procedure,

the main noise-introducing areas, such as trees, cars, and shadows, are excluded from the sampling.

By removing the noisy areas, the zonal attributes are subsequently recalculated. The new

calculation involves the collection of remaining polygons that are located within a buffer zone of

each road segment of the linework. A spectral unmixing technique can be employed for pixels

adjacent to noisy areas. Spectral unmixing is the procedure that decomposes the measured

spectrum of a mixed pixel into a collection of constituent spectra or endmembers. A set of

corresponding fractions indicates the proportion of each material present in the pixel 19.

An attribute selection or a more sophisticated dimensionality reduction process is needed

to guarantee the efficiency of the classification algorithms. There are many approaches to deal

with this issue 20. Attribute selection can be based in terms of correlation and information gain.

13
Dimensionality reduction and feature extraction can be performed by principal components

analysis 16, linear discriminant analysis, canonical components, non-negative matrix factorization,

or other techniques. Since the pavement condition classes might be dominated by imbalanced

datasets leading to overrated accuracy results, a resampling method can optionally eliminate this

negative effect. Examples of such methods are undersampling, oversampling, synthetic data

generation, and cost-sensitive learning.

3.3 Clustering

A clustering module is embedded to extract pavement surface areas from remote sensing data,

excluding surfaces that might introduce noise in the distributions of the pixel intensities. It can be

observed that noisy areas, such as those covered with vehicles, vegetation, and shadows, are

included in those resulting buffer zones. To deal with this problem, an image segmentation

procedure coupled with binary classification can be utilized. The intention of image segmentation

is to separate areas with different spectral characteristics, such as markings, vegetation, building,

and vehicles, from pure pavement surfaces. Also, based on the quality of the road centerline

geospatial data and the remote sensing images, the roads might not always align properly. As a

result, the buffer zones might not capture all the pavement surfaces. Many clustering algorithms

can be explored for image segmentation starting from a simple distance-based k-means to a more

sophisticated mean-shift algorithm, shown in Fig. 2c. However, there might be areas in which the

segmentation algorithm might not discern different objects accurately. Consequently, errors in the

spectral information of some polygons might remain.

14
3.3.1 Mean-shift

For segmentation, the mean-shift clustering algorithm 25 can be selected to be used among

others. This algorithm does not require specifying the number of clusters in the data ex ante. On

the contrary, mean-shift builds upon the probability density function for a set of data. The

algorithm works by placing a kernel on each point in the data set. By considering a set of points in

an input space corresponding to all the pixels of a multi-band image, a window is assumed to be

centered on each point. This window has a specific radius and corresponds to the kernel. If p is an

initial estimate and pξ is an input sample point, the Gaussian kernel or radial basis function K,

which is typically used, can be expressed as:

K ( p − p ) = e
2
− p −p
, (1)

where γ is a parameter that sets the “spread” of the kernel. The above function determines the

weight of nearby points for the re-estimation of the mean. The weighted mean of the density in the

window determined by K is:

 K ( p − p ) p 
p B ( p )
g (p ) = , (2)
 K ( p − p )
p B ( p )

where B(p) is the neighborhood of p, i.e. a set of points for which K(pξ) is non-zero.

Mean-shift is a hill-climbing algorithm, which involves shifting this kernel to a higher

density region at each iteration until convergence. The difference g(p) – p is called mean-shift and

at each iteration, the algorithm sets p = g(p) until g(p) converges. At convergence, there is no

direction at which a shift can accommodate more points inside the kernel.

15
3.4 Classification

For both the noisy area detection and the pavement condition classification schemes, the samples

under study must be divided into three groups: training, validation, and test. The training dataset

is used to fit the parameters of the model. A validation dataset is useful for fine-tuning the

hyperparameters of the classifier. The test data group is used to provide an unbiased evaluation of

the final model fit on the data and provide an estimate of the generalization error. Whether a class

resampling technique is used or not, the three groups should have the same level of variability to

achieve more accurate results.

Many classifiers can be used to predict the condition class of a pavement segment. These

include artificial neural networks, Bayesian classifiers, instance-based learners, tree-based

classifiers, and ensemble techniques that enrich base classifiers with metaheuristic techniques. We

can compare the performance of these classifiers based on specific metrics. The computational

effort and the accuracy that each classifier achieves are some of the most important performance

factors. The accuracy of a classifier on the test set is the percentage of test set tuples that were

correctly classified. It can be calculated as:

Number of correct predictions

Accuracy = . (3)
Total number of predictions

The classifier with the best performance can be selected for pavement management system

application. In the following sections, four classifiers that are used in the preliminary case study

are discussed. The classifiers that are selected are simple enough to demonstrate the applicability

of the framework and are available in most data mining platforms and software.

3.4.1 k-Nearest Neighbors

The k-nearest neighbor algorithms use instance-based learning, where the function is only
16
approximated locally and are considered among the simplest of classification techniques 26. A k-

nearest neighbor classifier may use Euclidean distance D as a metric among others. It searches the

pattern space for the “k” training tuples closest to the unknown tuple. For example, for the first

data tuple to be classified based on the information of the other given instances, the following

distances must be calculated:

M
D ( t1 , t n ) =  (t − tnm ) n = 2,
2
1m ,N , (4)
m =1

where tn the vector tn1,…,tnM of the attribute values of the n-th data tuple, N the number of data

instances, and M the number of attributes.

All the tuples are sorted based on the minimum calculated distance from the tuple to be

classified. The number of nearest neighbors “k” is specified explicitly. If only one neighbor is

used, then the class C of the first tuple is:

C1 = arg min D ( t1 , t n ) . (5)

Cn , n2, , N 

Predictions from more than one neighbor can be based on a majority vote or weights according to

their distance from the test instance.

3.4.2 Naïve Bayes

A naïve Bayes classifier is based on the assumption of attribute independence 27. Let the n-th data

instance be represented as an M-dimensional vector tn, where M is the number of attributes after

the dimensionality reduction procedure. The classifier assigns to each instance probabilities that

its class Cn is equal to all existing classes ck. According to the Bayes theorem, the posterior

probability equals the product of the prior with the likelihood of an instance given the class, divided

by the probability of the evidence:

17
 P ( Cn = ck ) P ( t n Cn = ck )
P ( Cn = ck t n ) = k = 1, ,K n = 1, ,N . (6)
P (tn )

Since the probability of the evidence is fixed, the posterior probabilities are proportional to the

nominator while the probability acts as a scaling factor. Then, according to the assumption of the

naïve Bayes classifier, each attribute is assumed to be independent of each other. Consequently,

the class label, which has the maximum value, is selected as the class label of the test data:

M
Cn = arg max P ( Cn = ck )  P ( tnm Cn = ck ) n = 1, ,N . (7)
k1, , K  m =1

3.4.3 Multilayer Perceptron

The multilayer perceptron is a specific feed-forward artificial neural network. It comprises at least

three layers of nodes 2. In this study, the number of layers is equal to the number of attributes

selected after the dimensionality reduction process. In each node i, a sigmoid activation function

is used to map the weighted sum input xi to the output f (xi). A logistic function can be used as an

activation function given by:

1
f ( xi ) = . (8)
1 + e − xi

The training of the perceptrons is performed by changing the weights wij of the links

between the nodes i and j of two layers after each data tuple is processed. The change of weights

is related to the amount of the output error compared to the expected result yj. The most widely

used loss function is the squared error, which can be expressed at the n-th data point as:

E (n) =
1
 (
yj − f (xj ) . )
2
(9)
2 j

The training process is carried out through backpropagation of the activation function. Using

gradient descent, the change of the weight will then be:

18
 E ( n )
wij ( n ) = −r f ( xi ( n ) ) , (10)
x j ( n )

where r is the learning rate, which is selected so that the weights converge to a set of values

efficiently.

3.4.4 Support Vector Machines

Support vector machines use nonlinear mapping to transform the original attribute space into a

higher dimensional feature space 28. Within this higher dimensional space, this classifier searches

for the linear optimal separating hyperplane, which corresponds to the decision boundary

separating the tuples of one class from another. With an appropriate nonlinear mapping to a

sufficiently high dimension, data can always be separated by a hyperplane. The classifier finds this

hyperplane using support vectors. These are the essential training tuples that define the hyperplane

margins. For a binary classification problem, the classifier seeks to minimize the hinge loss:

1 n 
 n  max ( 0,1 − ci ( w  z i − b ) )  +  w , (11)
 i =1 

where zi is the i-th tuple in the higher dimensional feature space, ci is the class of that tuple, and w

is the normal vector of the hyperplane. The parameter λ denotes the trade-off between increasing

the margin size and ensuring that each tuple lies on the correct side of the hyperplane. Thus, for

sufficiently small values of λ, the second term in the loss function will become negligible. For

multiclass problems, the labels are drawn from a finite set of several elements by reducing the

problem into separate binary classification problems.

4 Preliminary Case Study

The applicability of the framework was explored with a case study utilizing road inspection data

19
provided by the Department of Public Works of the City of Dallas. We used a subset of 362 data

instances covering a wide street network paved with asphalt concrete. The pavement condition

data were collected using in situ measurements from a van-mounted sensor by an external vendor.

The derived pavement condition index for each data instance corresponds to average values of

street segments with a length of 30 m approximately. The pavement condition index numerically

denotes the average surface condition of each 30-m pavement segment with respect to the distress
29
level and size . An index value of 100 signifies pavements in excellent condition and no

distresses. On the other hand, a value of 0 indicates a pavement that has completely failed and is

closed to traffic. We discretized the pavement condition data into three bins corresponding to a

custom rating scale. The labels of these three classes along with their corresponding pavement

condition index values were poor (0-55), fair (55-70), and good (70-100). Multispectral ortho-

images with a 30-cm pixel resolution were acquired from the Texas Natural Resources Information

System. These were captured using a Leica ADS100 digital camera sensor system as part of the

Statewide Texas Orthoimagery Project under the Texas Orthoimagery Program. The flights were

conducted from October 28th to November 8th and the images depict leaf-on conditions. The image

acquisition period is very important as it might drastically help in segregating vegetated noisy

objects from asphalt surfaces. Each image tile consists of four bands while. These bands along

with their corresponding spectral range are: red (619-651 nm); green (525-585 nm); blue (435-495

nm); and near-infrared (808-892 nm). The imagery covers a study area of 2.25 km2.

A visualization of some spectral characteristics of three pavement sample areas detected in

a subset of one image is provided in Fig. 3. The three sampled areas are marked with green, pink,

and blue. The area sampled in green color depicts a pavement in a relatively poor condition. This

is indicated by its lower pavement condition index, as there is an evident amount of cracking and

20
rutting. On the other hand, the pavement area in pink is sampled from a section with fair condition.

The blue depicts a pavement area covered with parked vehicles. The corresponding distributions

of the pixel intensities included in these areas are shown in the scatter diagram on the right. Each

grey dot represents the intensity of each pixel in the image and the corresponding colored dots

refer to the sampled areas on the left image. The intensity of the near-infrared band of the

multispectral image is on the vertical axis. In the horizontal axis, the corresponding intensity of

the green band is shown. The scatter plot indicates that the pavement in the worse condition shows

a higher variation of the intensity values in both channels. Finally, pavement covered with vehicles

introduces a significant amount of noise in the distribution of pixel intensities, as the variance of

these areas gets significantly higher. Thus, it is vital to exclude from sampling areas that induce

noise in the pavement segments’ spectral distributions.

Fig. 3 Sampled pavement areas: (a) location and (b) scatter plot of pixel intensities for these areas.

21
 The framework for data pre-processing, image segmentation, and pavement condition

classification was implemented through open source geographic information systems and machine
30–35
learning software . After running the mean-shift segmentation algorithm a supervised

procedure was followed in order to label the generated objects. One label was given to objects

corresponding to pavement surfaces and a second one to non-pavement noisy areas. This procedure

led to a binary classification problem that achieved ~98% accuracy within the established buffer

zone. Thus, in forming the final pavement sampling areas most probably only a very small

percentage of noisy objects was included in the analysis. In this initial stage, only spectral related

attributes were generated, and a simple feature selection procedure based on information gain was

employed. The mean and the standard deviation from the near-infrared band as well as attributes

from other bands provided significant information gain.

Over-optimistic estimates due to overfitting might occur if the accuracy of the classifier is

calculated using the same data that were used for training. The classifier’s accuracy should be

measured using a test set comprising class-labeled tuples not used to train the model. Thus, the

original dataset was divided: 70 percent for training and 30 percent for testing. In other words, out

of the 362 pavements segments in the dataset, 253 were used to derive the model. The rest 109

segments were kept out of the training procedure to estimate the accuracy. This test sample that is

practically “unseen” by the classification algorithms is later used to estimate the accuracy of each

classification model. Thus, the accuracy results that we report and the segments that have been

misclassified do not correspond to the training error but rather the generalization error of the

models. This is the common practice of verifying reliability in data mining applications.

To demonstrate the effect of the clustering module, two runs were conducted for each

classifier. The first run involved pavement sampling without applying the clustering module. Thus,

22
zonal statistics calculations included noisy area intensities in this case. In the second run, the

sampling was conducted only in clusters that represent pavement surfaces after having excluded

the noisy areas. Figure 4 shows that there was a significant increase in the accuracy of each

classifier by using the clustering procedure and ruling out non-pavement covered areas.

Fig. 4 Accuracy level achieved for each pavement condition classification model.

The accuracy of the naïve Bayes, multilayer perceptron, support vector machines, and k-

nearest neighbors was 62.15, 70.71, 69.38 and 56.07 percent respectively. Without following the

clustering and noise filtering process, the accuracy achieved was much lower for all the algorithms.

The corresponding results were 38.67, 40.33, 39.50 and 37.29 percent accordingly. The multilayer

perceptron achieved the highest accuracy. Artificial neural networks are universal approximators

and it seems that they adjust better to the intricate nature of the problem.

A downside of the multilayer perceptron is that its training time is slightly higher than the

rest of the algorithms. The theoretical time complexity of the specific implementations of the

classifiers along with the actual training and testing times is shown in Table 1. Time complexity

is expressed in terms of the limiting behavior of the algorithms where M is the number of features

and N the number of instances and K the number of classes. The complexity of testing time for

23
support vector machines depends on the number of the support vectors nsv. For the multilayer

perceptron the corresponding time complexity depends also on the network architecture. This

includes the number of hidden layers h, the number of perceptrons npr of each layer and the number

of epochs t. It is assumed that all hidden layers have an equal number of nodes.

Table 2 Theoretical complexity and actual training and testing times.

Algorithm Training Testing
Complexity Time Complexity Time
Naïve Bayes O(NM) 0.03 s O(KM) 0.01 s
Multilayer Perceptron O(KNM t nprh) 0.70 s O(K nprh) 0.01 s
k-Nearest Neighbors O(1) 0.00 s O(NM) 0.03 s
Support Vector Machines O(N2M+N3) 0.14 s O(M nsv) 0.02 s

Using the trained multilayer perceptron, we predict the pavement condition classes of the

362 segments. Figure 5 presents the model results in contrast with the actual inspection data. The

three classes poor, fair, good are marked with red, yellow, and green. The misclassified predictions

are highlighted with a cyan box (Fig. 5b).

Fig. 5 Pavement condition classes: (a) ground truth and (b) predicted by the multilayer perceptron model.

24
6 Discussion

This study investigated remote sensing data usage for network-level pavement condition

assessment. A detailed data mining framework was established, utilizing the inherent spectral

characteristics of pavement surfaces. These are used to train models that will be capable of

predicting the pavement condition of different road segments. In contrast with previous literature

we explore the applicability of such models by testing them in unseen pavement condition data. A

preliminary case study was conducted with data provided by the City of Dallas and remote sensing

images acquired from the Texas Natural Resources Information System. Contrary to previous

research that utilizes hyperspectral and commercial image products, the methodology that we

develop can be applied with images that are publicly available. The case study examined four

classification algorithms. Out of those classifiers, a multilayer perceptron achieved 71 percent

accuracy in classifying pavements in three different condition states. However, due to the

algorithms complexity it might require higher training time depending on the network architecture.

This issue was of little importance in the case study since training times of all four classifiers were

less than one second. However, significant efficiency issues might arise for larger datasets. Further

analysis based on the trade-off between accuracy and efficiency should be conducted in that case.

Research has shown that measurement variation and errors can influence pavement
36
inspection data depending on the data source . For this reason, the accuracy achieved is

satisfactory compared to a ~33.3 percent equal probability of a segment belonging to one of the

three classes at random. This is an indicator of an underlying connection between image inherent

spectral information and pavement condition. The results also show a significant increase on the

accuracy when the clustering module is employed. Thus, the framework might have the potential

for future implementation if further research is conducted on the different constituent steps. For

25
that effort it is also important to develop models trained to classify the conditions of rigid

pavements. The usage of larger training datasets will contribute to better approximations of the

classification models to the underlying mapping functions of remote sensing data to pavement

conditions.

7 Conclusion

Based on the results generated from the classification models, it seems that the methodological

framework can be potentially used for high-level information checks. More specifically, the

predicted classes can be compared with existing pavement condition datasets. The comparison will

help to identify potential errors in data collection. The framework is based on synthesizing

knowledge from the literature and is directed towards guiding future research on this topic.

Applying more sophisticated data processing techniques and classification algorithms will provide

results with even higher accuracy.

Acknowledgments

The authors wish to express their gratitude to Mr. Cosmin Spiridon and Ms. Jennifer Nicewander,

Department of Public Works of the City of Dallas, for providing pavement condition inspection

data. We also extend our appreciation to the Texas Water Development Board for providing public

access to the Texas Natural Resources Information System. This includes the multispectral ortho-

imagery that was used in the case study.

Disclosures

The authors have no relevant financial interests in the manuscript and no other potential conflicts

of interest to disclose.

26
References

1. L. M. Pierce, G. McGovern, and K. A. Zimmerman, “Practical Guide for Quality

Management of Pavement Condition Data Collection,” 170 (2013).

2. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and

Techniques, 2nd ed., Morgan Kaufmann, San Francisco, CA (2005).

3. K. H. McGhee, “NCHRP Synthesis of Highway Practice 334: Automated Pavement

Distress Collection Techniques,” Transportation Research Board of the National

Academies, Washington, D.C. (2004) [doi:10.17226/23348].

4. G. Ullman, J. Ragsdale, and N. Chaudhary, “Recommendations for Highway

Construction, Maintenance, and Service Equipment Warning Lights and Pavement Data

Collection System Safety,” Texas Department of Transportation, Austin, TX, U.S.A.

(1998).

5. E. Schnebele et al., “Review of remote sensing methodologies for pavement management

and assessment,” Eur. Transp. Res. Rev. 7(2) (2015) [doi:10.1007/s12544-015-0156-6].

6. G. W. Flintsch and K. K. McGhee, “NCHRP Synthesis of Highway Practice 401: Quality

Management of Pavement Condition Data Collection,” Transportation Research Board of

the National Academies, Washington, D.C. (2009) [doi:10.17226/14325].

7. W. D. O. Paterson and T. Scullion, “Information Systems for Road Management: Draft

Guidelines on System Design and Data Issues,” The World Bank, Washington, D.C.

(1990).

8. C. R. Bennett and W. D. O. Paterson, A Guide to Calibration and Adaptation, The World

Bank, Washington, D.C. (2000).

9. M. Herold et al., “Spectrometry and Hyperspectral Remote Sensing of Urban Road

27
 Infrastructure,” Online J. Sp. Commun.(3), 1–29 (2003).

10. M. Herold et al., “Imaging Spectrometry and Asphalt Road Surveys,” Transp. Res. Part C

Emerg. Technol. 16(2), 153–166 (2008) [doi:10.1016/j.trc.2007.07.001].

11. T. Kavzoglu, Y. E. Sen, and M. Cetin, “Mapping urban road infrastructure using remotely

sensed images,” Int. J. Remote Sens. 30(7), 1759–1769 (2009)

[doi:10.1080/01431160802639582].

12. S. Pascucci et al., “Road Asphalt Pavements Analyzed by Airborne Thermal Remote

Sensing: Preliminary Results of the Venice Highway,” Sensors 8(2), 1278–1296 (2008)

[doi:10.3390/s8021278].

13. A. Mei, R. Salvatori, and A. Allegrini, “Analysis of Paved Areas with Field Data and

MIVIS Hyperspectral Images,” Ital. J. Remote Sens. 43(2), 147–159 (2011)

[doi:10.5721/ItJRS201143212].

14. A. Mei et al., “Integration of Field and Laboratory Spectral Data with Multi-Resolution

Remote Sensed Imagery for Asphalt Surface Differentiation,” Remote Sens. 6(4), 2765–

2781 (2014) [doi:10.3390/rs6042765].

15. A. Mei and R. Salvatori, “Urban Mapping Using Ikonos Imagery,” Int. J. Remote Sens.

Geosci. 2(3), 55–58 (2013).

16. C. Andreou, V. Karathanassi, and P. Kolokoussis, “Investigation of Hyperspectral Remote

Sensing for Mapping Asphalt Road Conditions,” Int. J. Remote Sens. 32(21), 6315–6333

(2011).

17. C. Mettas et al., “Risk Provision Using Field Spectroscopy to Identify Spectral Regions

for the Detection of Defects in Flexible Pavements,” Nat. Hazards 83(1), 83–96, Springer

Netherlands (2016) [doi:10.1007/s11069-016-2262-8].

28
18. M. Mohammadi, “Road Classification and Condition Determination Using Hyperpsectral

Imagery,” in International Archives of the Photogrammetry, Remote Sensing and Spatial

Information Sciences 39(B7), pp. 141–146, International Society for Photogrammetry and

Remote Sensing, Melbourne (2012) [doi:10.5194/isprsarchives-XXXIX-B7-141-2012].

19. Y. Pan et al., “Mapping Asphalt Pavement Aging and Condition Using Multiple

Endmember Spectral Mixture Analysis in Beijing, China,” J. Appl. Remote Sens. 11(1),

016003 (2017) [doi:10.1117/1.JRS.11.016003].

20. K. Shahi, H. Z. M. Shafri, and A. Hamedianfar, “Road Condition Assessment by OBIA

and Feature Selection Techniques Using Very High-Resolution WorldView-2 Imagery,”

Geocarto Int. 32(12), 1389–1406 (2017).

21. C. Mettas et al., “Monitoring asphalt pavement damages using remote sensing

techniques,” in Proceedings of SPIE, Third International Conference on Remote Sensing

and Geoinformation of the Environment 9535, Society of Photo-Optical Instrumentation

Engineers, Paphos, Cyprus (2015) [doi:10.1117/12.2195702].

22. C. Mettas et al., “Detection of Asphalt Pavement Cracks Using Remote Sensing

Techniques,” in Proceedings of SPIE, Remote Sensing Technologies and Applications in

Urban Environments 10008, T. Erbertseder, T. Esch, and N. Chrysoulakis, Eds., Society

of Photo-Optical Instrumentation Engineers, Edinburgh (2016) [doi:10.1117/12.2240682].

23. W. Wang et al., “A Review of Road Extraction From Remote Sensing Images,” J. Traffic

Transp. Eng. 3(3), 271–282, Elsevier Ltd (2016) [doi:10.1016/j.jtte.2016.05.005].

24. A. Mei et al., “The Use of Suitable Pseudo-invariant Targets for MIVIS Data Calibration

by the Empirical Line Method,” ISPRS J. Photogramm. Remote Sens. 114, 102–114,

Elsevier B.V. (2016) [doi:10.1016/j.isprsjprs.2016.01.016].

29
25. J. Michel, D. Youssefi, and M. Grizonnet, “Stable Mean-Shift Algorithm and its

Application to the Segmentation of Arbitrarily Large Remote Sensing Images,” IEEE

Trans. Geosci. Remote Sens. 53(2), 952–964, Institute of Electrical and Electronics

Engineers (2015) [doi:10.1109/TGRS.2014.2330857].

26. D. W. Aha, D. Kibler, and M. K. Albert, “Instance-Based Learning Algorithms,” in

Machine Learning 6(1), pp. 37–66, Kluwer Academic Publishers, Boston (1991)

[doi:10.1007/BF00153759].

27. G. H. John and P. Langley, “Estimating Continuous Distributions in Bayesian Classifiers,”

in Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pp.

338–345, Morgan Kaufmann, San Mateo, CA (1995).

28. C. C. Chang and C. J. Lin, “LIBSVM: A Library for Support Vector Machines,” ACM

Trans. Intell. Syst. Technol. 2(3), Association for Computing Machinery, New York, NY

(2011) [doi:10.1145/1961189.1961199].

29. ASTM, “D6433-18: Standard Practice for Roads and Parking Lots Pavement Condition

Index Surveys,” ASTM International, West Cornhohocken, PA, U.S.A. (2018)

[doi:10.1520/D6433-18].

30. QGIS Development Team, “QGIS Geographic Information System,” 2.18, Open Source

Geospatial Foundation (2009).

31. OTB Development Team, “Orfeo ToolBox - Open Source Processing of Remote Sensing

Images,” 6.6.0, CNES (2002).

32. GDAL/OGR Contributors, “GDAL/OGR Geospatial Data Abstraction Software Library,”

Open Source Geospatial Foundation (2018).

33. GRASS Development Team, “Geographic Resources Analysis Support System (GRASS

30
 GIS) Software,” 7.4, Open Source Geospatial Foundation (2018).

34. O. Conrad et al., “System for Automated Geoscientific Analyses (SAGA) v. 2.1.4,”

Geosci. Model Dev. 8(7), 1991–2007 (2015) [doi:10.5194/gmd-8-1991-2015].

35. E. Frank, M. A. Hall, and I. H. Witten, “The WEKA Workbench. Online Appendix,” in

Data Mining: Practical Machine Learning Tools and Techniques, 4th ed., Morgan

Kaufmann (2016).

36. P. A. Serigos et al., “Automated Distress Surveys: Analysis of Network Level Data (Phase

III) (FHWA Report 0-6663-3)” (2015).

Stefanos S. Politis is a graduate research assistant at the Center for

Transportation Research of the University of Texas at Austin. He received his

MS degree in surveying engineering from the National Technical University

of Athens in 2016. He is a PhD candidate at the University of Texas at Austin

and holds the John E. Kasch Endowed Graduate Fellowship. His current research interests include

remote sensing and machine learning applications in transportation asset management and

simulation of infrastructure networks. He has served as vice president at the Institute of

Transportation Engineers and Intelligent Transportation Society student chapters.

Zhanmin Zhang is a professor at the University of Texas at Austin. He has

been actively conducting research in the engineering and management of

infrastructure systems and the applications of advanced information systems

to infrastructure management for more than 20 years. Dr. Zhang earned his

31
PhD degree in civil engineering from the University of Texas at Austin in 1996. He is an author

or co-author of more than 150 technical papers, reports, and articles.

Sareh Kouchaki is a project engineer at Wood Environment and

Infrastructure Solutions. Her main activities are linked to the development of

statistical models for pavement performance. She received her Ph.D. degree

in Civil Engineering from the University of Texas at Austin in 2019 and her

MS degree in Road and Transportation Engineering from the Iran University of Science and

Technology in 2013. She also earned a MS degree in Statistics from the University of Texas at

Austin in 2017. She has published five peer-reviewed articles in top-tier transportation journals

Carlos H. Caldas is an associate professor at the University of Texas at Austin

Department of Civil, Architectural and Environmental Engineering. He has a

Ph.D. in Civil Engineering from the University of Illinois at Urbana-

Champaign and has more than eighteen years of experience in construction

engineering and project management. His research interests include information technologies and

sensors for construction engineering and project management, advanced data analysis techniques,

project performance assessment, productivity improvement, knowledge management, and

infrastructure systems construction.

32