Image Processing for Surface Quality Control

in Stainless Steel Production Lines

C. Spinola1, J.M. Caero-Nieto1,2, M.J. Martin-Vazquez1, J.M. Bonelo4, F. Garcia-Vacas1,
G. Moreno-Aranda1, S. Espejo3, G. Hylander1, J. Vizoso4
1

Department of Electronics, University of Mlaga, 29071 Mlaga, Spain, cspinola@ctima.uma.es

2
T.C.C. S.A., C/ Robinson Crusoe 5, 29006 Mlaga, Spain, jmcanero@ctima.uma.es
3
Department of Electronics and Electromagnetism, University of Sevilla, 41012 Sevilla, Spain, espejo@imse.cnm.es
4
Acerinox, S.A., Factora del Campo de Gibraltar, 11370 Los Barrios (Cdiz), Spain, josemaria.bonelo@acerinox.com
Abstract In this paper we present an image processing
algorithm to detect and measure the amount of residual
oxide remaining on the surface of stainless steel coils for
quality control in a production line. This algorithm
processes in real time the images taken by the acquisition
system which we have designed for this purpose and which
has been installed in the finish line of a stainless steel
production factory. We present here a more robust and
reliable algorithm than the initial one, which has been
adapted to deal with non-ideal conditions such as nonperfect homogeneous lighting, different surface finish,
water marks, etc., which usually occur in practice.
Keywords Image processing, quality control, residual oxide
detection, stainless steel production.

I. INTRODUCTION
Stainless steel is mainly produced as flat sheets then rolled
in coils several hundred meters long and around one and a half
meters wide. The desired thickness is obtained by extrusion in
the hot and cold rolling mill process [1]. After that it is
submitted to the annealing process at temperatures above 900
C so that the internal stress can be released and the crystalline
structure regenerated in order to recover its metallurgical and
mechanical properties. In all these processes the steel surface
oxidizes appearing layers of iron oxides (external layers) and
iron-chromium oxides (internal layers). They have to be
eliminated in a subsequent pickling process [2][4] by
submitting the coil surface to mechanical treatment with grain
blasting and wire brush and also to chemical treatment by
submerging it in hydrofluoric and sulphuric acid tanks which
eliminate the oxide layer and regenerate the passive layer,
Cr2O3, of stainless steel [5][7].
If cleaning is not completed due to an excess in line speed
or other anomalies, oxide stains can remain randomly attached
to the surface, taking the form of small, dark spots typically
sized from 50 to 200 microns and lodged in the valleys of the
roughness or other imperfections on the surface (Fig. 1).
When these stains are small (one tenth of a mm or less), they
cannot be seen by the naked eye of the quality control staff in
the line. Nevertheless in subsequent processes they are made

978-1-4244-6494-4/10/$26.00 2010 IEEE

evident, lowering the steel quality and producing important

economic losses. This type of marking is called residual
scale in the specialized argot.
The steel surface has to be inspected at the end of the line to
determine its quality. Before the existence of this automatic
inspection system, the oxide detection was scarcely done
because it was necessary to stop the line and to inspect the
surface with an ad hoc lens. The reflectance of the stainless
steel surface and its roughness made the inspection difficult
because the brightness could hide the residual scale and the
roughness shadow could be confused with oxide stains.
To increase the productivity and quality of the production,
we have designed and developed a system able to
automatically acquire clear images of the surface and inspect
them to detect residual scale in real time, while the sheet is
moving forward in the process line [8],[9].
Such a system has been installed in two factories of one of
the main producers of stainless steel in the world, namely in
the Spanish and South African factories.
In this paper we present the algorithm we have developed to
detect the residual scale on the stainless steel surface. This
algorithm has been improved thanks to the experience
acquired working with the system. It can fulfill the real time
requirements and can also adapt to the different steel grades
and surface finish. It is also prepared to deal with practical
situations that occur in production like water marks and nonhomogeneous lighting in the images [10], [11].

Fig. 1. An example of residual scale. The smallest divisions are mm.

II. IMAGE ACQUISITION SYSTEM

III. RESIDUAL SCALE DETECTION ALGORITHM

The stainless steel surface is shiny but also rough. The

stains to be detected are very small and the steel sheets are
moving along the line. That is why great magnification and
short exposure time are required. Therefore illumination is a
key factor; it has to be intense, homogeneous and diffuse to
avoid glitters (great reflectance) and shadows (surface
roughness). The image acquisition system has specifically
been designed for this application and to deal with the hard
industrial environment and the difficulty of the problem (Fig.
2).
It consists of a high resolution camera, stroboscopic lights
and diffusing screens, all of them inside a hermetic steel case
placed on a tray which can be moved transversally to the sheet
movement. The details concerning this system can be found in
references [8][12].
The user is not aware of each individual stain, but only of its
density and average size. As a result, we do not need to make
a full scan of the surface. The acquisition device moves over
the coil width as it advances along the line and takes several
images in a second. They are grey scale, 27x21 mm in size,
with an effective resolution of 1056x822 pixels, i.e. 1964
pixel/mm2. These values have been selected to guarantee that
a 50 micron stain has enough pixels to be surely detected.
The histograms of the acquired images have similar
features. They fit quite well to a normal distribution
probability curve. The shape and position of the Gaussian bell
change according to the features of the steel being processed,
like surface reflectivity and roughness, but it also depends on
the features of the acquired image like lighting homogeneity
and sharpness. These changes can be quantified using
statistical indexes like the average, the median, the standard
deviation, the kurtosis, etc., and they are used to set the
optimal illumination and focus levels.

The processing must satisfy the following general

requirements:
a) Low computational costs as the images have to be
processed in real time.
b) It must be robust and reliable enough to correctly
discriminate pixels that represent oxide from others that
do not.
c) Images that show defects in most of their area, like
welding, cracks, holes, etc., should be discarded, as they
could affect the correctness of the results. (Fig. 3).

Fig. 3. Example of discarded images: (a) welding and (b) hole.

The starting hypothesis is that if there is any oxide in an

image, it should appear darker than the clean surface even in
the darkest valleys of the roughness [8]. Therefore, the aim of
the algorithm is to obtain a threshold value to discriminate this
oxide. In a certain image, the darkest pixels may be oxide but
we should not mistake the shadows for scale. Furthermore, the
algorithm should be flexible enough to adapt to images with
different luminance and sharpness and it must also be efficient
in computational cost.
According to this hypothesis, in the histogram of an image
with residual oxide, Fig. 4 (a), the pixels that represent it
should be the darkest ones, and therefore they should be
counted in the lowest gray levels of the histogram, separated
from the main Gaussian bell, where most of the pixels that
represent the rest of the surface are counted.
The histogram zone marked in Fig. 4 (b) has been amplified
in Fig. 5, where it can be appreciated that the pixels with the
darkest level of gray, which are supposed to be oxide, are
separated by a valley from the rest of pixels in the image.

Fig. 2. Scale Detection System. Isometric view (a) and Installed System at
Factory (b).

Fig. 4. (a) Image with residual scale and (b) its histogram.

Fig. 5. Image histogram with scale zoom (filtered histogram in red).

To make easier the analysis, a FIR (Finite Impulse

Response) low band filter of order 6 is applied to the
histogram curve. The delay introduced by the filter has been
corrected in the Fig. 5. The filtered output y(k) is given in (1)
n

y (k )

xn  k 

(1)

k 0

where x is the input values, n is the filter order and bk the filter
coefficients.
The partial histogram in Fig. 5 shows a real but very
favorable case of bimodal histogram. It is possible to use
threshold techniques [13][16] to discriminate pixels
representing oxide from those ones that do not. This threshold
should be found in the valley that separates both groups of
gray levels.
To enhance in the histogram the effect of possible oxide and
to accelerate the processing, we implemented a first procedure
[11] that analyzed the histogram of a reduced region 50x50
pixel wide. It focused on the darkest pixel in the image and
applied the described procedure to look for a valid threshold
value for the whole image.
But this type of bimodal histogram is not always obtained,
even in the presence of oxide. In other cases unimodal and
multimodal histograms appear and it is difficult to detect
residual oxide in them because it is not easy to find a reliable
threshold. Some situations where this usually happens are
represented in (Fig. 6), such as:
x
Non-homogeneous lighting in the image, produced by
progressive degradation of bulbs, non-perfect position of
lamps, etc. Fig. 6 (a).
x
Water marks on the sheet surface due to defects in the
drying process after the cleaning tanks, Fig. 6 (b).
x
Different surface roughness or textures in the same
image, Fig. 6 (c).
x
Small defects such as scratches, marks, etc., which are
not big enough to discard the image, Fig. 6 (d).
In this and other cases, the general hypothesis that pixels
which represent oxide must be the darkest ones is not fulfilled
in the image.

Fig. 6. Detection Dificulties: (a) lighting heterogeneities, (b) marks, (c)

variable rugosity and (d) defects.

An approach to the solution involves dividing the image in a

set of sub-images or windows in which the main hypothesis is
fulfilled, implementing the oxide detection independently in
each one of them. We have worked with different window
sizes according to the surface roughness and residual oxide
stains and we have found that windows greater than 25x25
pixels have enough area and enough pixels to be able to
perform the histogram analysis. Consequently we divide the
original image in 20x20 sub-images, what is the same as
choosing a window size of 41x53 pixels (1,30x1,02 mm),
(Fig. 7).
Thus, when processing the pixels of an image to calculate
its histogram, they do not count in a global histogram but in a
histogram for each window.
The next step is to search for the threshold level in each
histogram. Instead of searching for it in the whole histogram,
we previously determine a reduced range of it where it is most
likely to be. This range must adapt to the different reflectance
and roughness along different images and windows in the
same image. The correct threshold level will depend on the
luminance of the image or window. We have selected the
median because it is a statistical index less sensitive than the
average to the occurrence of extreme values [15].
We have implemented an empirical method to find this
correlation. A population of hundred of images which are
representative of the different types of surfaces processed in
the line has been collected. They were also acquired in nonperfect conditions, like those ones mentioned before. They
were analyzed one by one to determine the empirical threshold
level that produced the best results, regarding the residual
oxide detection. The values obtained in this way were
represented against the luminance median of the images (Fig.
8).

Fig. 9. Threshold searching on selection zone.

Fig. 7. Image Division in 20 x 20 squares.

A linear correlation was applied to find the straight line that

best fits to the population of points. This line is used to predict
approximately the central value of the range in which there is
a maximum probability to find the threshold level. The spread
of points along the line give us an indication of the range
amplitude. To set it we have used 2V, which give us 8 gray
levels with a confidence of 95%. This function allows us to
calculate the dynamic range of any window in a normal
image.
The threshold level of each window is searched in the
dynamic range of its histogram in the same way it was
searched for in the darkest window of the global image [11]. It
will be the gray level corresponding to the deepest valley in
the range (Fig. 9). In this figure, the dashed lines mark the
calculated dynamic range and the continuous one the
threshold value found. If there was not any valley, the lowest
value of the dynamic range is taken.
We summarize below the oxide detection algorithm of each
global image. Previously the images that have big defects or
anomalies were discarded.

a)
b)
c)
d)
e)
f)
g)
h)
i)

Dividing the image in windows.

Obtaining the histogram of them all.
Applying the FIR filter to each histogram.
Determining the dynamic range of maximum likelihood.
Calculating the threshold level.
Binarizing the sub-image with the found threshold level.
Joining binarized sub-images to form the global image.
Grouping contiguous pixels to form each oxide stain.
Counting and grouping the stains according to their sizes
and updating the statistical values (number of stains per
surface units and sizes).
It is important to study the increase in computational cost
involved in decomposing an image in multiple windows in
order to assure that the requirement of real time processing is
fulfilled. TABLE I and TABLE II show a comparison
between the computational cost of dividing (D) or non
dividing (ND) the image in windows. They are expressed in
number of CPU cycles in TABLE I and milliseconds in
TABLE II, and have been obtained using information about
the performance of Intel Processor and the operations needed
to carry out each step.
TABLE I
ALGORITHM COMPUTATIONAL COST (CPU CYCLES*)
Procedure

ND

D-ND

Histogram Calculation
Minimal Searching
Window Equalization
Filtering
Threshold Searching
Binarizing
Blobbing
TOTAL CYCLES

6,944,256
0
0
5,734,400
121,600
2,604,096
20,832,768
36,237,120

6,944,256
3,906,656
37,500
197,880
2,432
2,604,096
20,832,768
34,525,588

0
-3,906,656
-37,500
5,536,520
119,168
0
0
1,711,532

Fig. 8. Empirical threshold as function of image median. Confidence

bounds for 95 %.

CPU: Intel Core 2 Duo, 32 bits instructions (from www.intel.com).

TABLE II
ALGORITHM COMPUTATIONAL COST (CPU TIME*)
Procedure

Histogram Calculation
Minimal Searching
Window Equalization
Filtering
Threshold Searching
Binarizing
Blobbing
TOTAL TIME (ms)
*

2.893
0
0
2.389
0.050
1.085
8.680
15.098

ND

D-ND

2.893
1.627
0.015
0.082
0.001
1.085
8.068
14.385

0
-1.627
-0.015
2.306
0.049
0
0
0.713

Time in milliseconds. CPU: Intel Core 2 Duo, 2.4 GHz.

It can be seen that the relative increment in time, measured

in number of cycles, is about 5%, although the requirements
of memory are slightly higher, but this is not an
inconvenience. We have verified that this increase is adequate
to the real time requirements and it is compensated quite well
by the enhancement achieved in the behavior of the algorithm.
IV.

EXPERIMENTAL RESULTS

Fig. 10 and Fig. 11 show the results of the image processing

with the previous algorithm and the new one based on the
division of the image in multiple windows. In Fig. 10 we can
see the one of the images taken by the acquisition system. In
part (c) we have amplified the selected area. The detection
results of the first algorithm are shown in Fig. 10 (d) and those
of the new algorithm in Fig. 10 (b). In all the cases the high
performance of the algorithm with multiple windows is
evident.

Fig. 10. (a) Image with superficial defects, (c) image zoom, processed
image with divided (b) and non divided (d) window algorithm.

Fig. 11. (b) Water marked image without scale, processed image with
divided (a) and non divided (b) window algorithm. (e) Water marked
image with scale, processed image with divided (d) and non divided (f)
window algorithm.

In normal conditions (Fig. 10) the stains are better outlined

when applying a local threshold. In the presence of anomalies
such as water marks (Fig. 11) applying a global threshold is
error prone and pixels that are not oxide appear to be (Fig.
11(c), (f)) or vice versa. In this case the performance of the
new algorithm is clearly higher.
V. CONCLUSIONS
In this article we have presented an algorithm to detect and
measure the rate of residual oxide on the surface of stainless
steel. It can be applied to real time quality control while the
steel sheets are being produced in the factory.
This algorithm is based on an initial work [10],[11] and
presents an important enhancement in behavior and robustness
in the case of abnormalities on the surface and images when
working in real condition, like scratches, water marks and
other small superficial defects or non-ideal lighting
conditions.
The introduction of a dynamic range of maximum
likelihood gives more stability and flexibility to the search for
the threshold value. This range is calculated like an empirical
function of the luminance median of a population of
representative images.
The small increment in computational cost is clearly
compensated by the advantages obtained and the new
algorithm has proved to be more adequate to deal with
practical situations in the stainless steel production factories.
In future works we could study if the dynamic range is more
precisely or effectively found using other techniques like
neural networks over the population of training images.

VI.

ACKOWLEDGEMENT

We would like to thank Acerinox management for

supporting this investigation and the Information Systems and
Quality Control departments for its help and essential
suggestions offered in this research.
REFERENCES
[1]

[2]

[3]

[4]

[5]
[6]

[7]
[8]

[9]

Jha, B.; Jha, P.; Singh, C. D. Process Technology for the Continuous
hot band annealing of 17% Cr ferritic stainless steel, in Journal of
Material Engineering and performance, 2002. Pp. 11.
Bailey, J.; Hirsch, P. Recrystallization Process In Some Polycrystalline
Metals. Proceedings Royal Society of London Series. A-Mathematical
and Physical Sciences, 267, 1962.
Cheng, S.; Kuan, S.; Tsai, W. Effect of Water Vapor on Annealing
Scale Formation on 316 SS, en: Corrosion Science. Vol. 48. 2006. Pp.
634649.
Waanders, F. B.; Vorster, S. W.; Engelbretch, A. Mssbauer and SEM
Characterization of the Scale on Type 304 Stainless Steel, en: Scripta
Materalia. Vol. 42. 2000. Pp. 9971,000.
Avner, Sydney H. Introduccin a la metalurgia fsica, 2nd ed.,
McGraw-Hill, Mxico, 1995.
Li, L.; Caenen, P.; Jiang, M. Electrolytic Pickling of the Oxide Layer
on Hot-rolled 304 Stainless Steel in Sodium Sulphate, in Corrosion
Science. vol. 50. 2008. Pp. 2,8242,830.
Davis, J. R. Stainless steels, 3rd ed. Ohio, U.S.A.: ASM International,
1999.
Martn, D. Arquitectura dinmica para inspeccin visual de superficies
en tiempo real. Tesis Doctoral, Departamento de Inteligencia Artificial,
UNED, 2007.
Spnola C. G.; Espejo Meana, S.; Morillas Castillo, S.; Caero Nieto, J.
M.; Garca-Vacas, F.; Guinea Daz, D.; Martn-Vzquez, M. J.
Dispositivo para la deteccin y clasificacin de xido residual en lneas
de produccin de laminados metlicos. Patent. Publication number:
WO 2008/119845, World Intellectual Patent Organization (W.I.P.O.),
Application number: PCT/ES2007/000768, Publication date: 09 October
2008.

[10] Spnola, C. G.; Bonelo, J. M.; Caero Nieto, J. M.; Espejo, S.; Morilla,
S.; Martn-Vzquez, M. J. Procesamiento de imgenes para la deteccin
de xidos residuales en lneas de produccin de laminados metlicos,
en: XV Seminario Anual de Automtica, Electrnica Industrial e
Instrumentacin (SAAEI08). Cartagena, Espaa: 9-11 de Septiembre
2008. ISBN: 978-84-96997-04-2.
[11] Spnola, C. G.; Bonelo, J. M.; Caero Nieto, J. M.; Espejo, S.; Morilla,
S.; Martn-Vzquez, M. J. Residual oxides detection and measurement
in stainless steel production lines, en: IEEE International Conference
on Computational Intelligence for Measurements Systems and
Applications (CIMSA 2009). Volume 1. Hong Kong, China: 11-13 de
Mayo 2009. Pp. 7679. ISBN: 978-1-4244-3820-4.
[12] Martn-Vzquez, M. J.; Spnola, C. G.; Caero Nieto, J. M.; Bonelo, J.
M.; Vizoso, J. Sistema de visin artificial para la deteccin de
soldaduras en tiempo real en lneas de proceso de acero inoxidable, en:
XVI Seminario Anual de Automtica, Electrnica Industrial e
Instrumentacin (SAAEI09). N 12. Legans, Espaa: 1-3 de Julio 2009.
Pp. 2,3452,356. ISBN: 978-84-692-2596-7.
[13] Davies, E. R.; Machine Vision: theory, algorithms, practicalities., 3rd
ed., Elsevier, Amsterdam, December 2004. Pp. 102129. ISBN: 978-012-206093-9.
[14] Pajares, G.; de la Cruz, J. M.; Visin por Computador, 1st ed., Editorial
Ra-ma, Madrid, 2001. Pp. 179-187. ISBN: 84-7897-472-5.
[15] Pratt, W. K.; Digital Image Processing, 3rd ed., Wiley Interscience,
New York, 2001.
[16] Mitra, S.K., Digital Signal Processing. A Computer Based Approach,
1st ed., McGraw-Hill, New York, 1998.