FPGA design for real time flaw detection on edges

using the LEDges technique

Ygo N. Batista

Cristiano C. de Araujo

Abel G. S. Filho

Federal Institute of Education, Science

and Technology of Pernambuco - IFPE
Pesqueira, Pernambuco, Brazil
Email: ygo@pesqueira.ifpe.edu.br

Federal University
of Pernambuco - UFPE
Recife, Pernambuco, Brazil
Email: cca2@cin.ufpe.br

Federal University
of Pernambuco - UFPE
Recife, Pernambuco, Brazil
Email: agsf@cin.ufpe.br

the system in section IV and present the FPGA implementation

in section V. A case study is described in VI. Results are
discussed in section VII. Finally, we conclude and present the
future works in section VIII.
II. I MAGE PROCESSING STEPS FOR A TYPICAL
AUTOMATED VISUAL INSPECTION SYSTEM

The image processing for AVI applications can be divided

into three distinct levels of processing. Figure 1 shows these
levels in a typical AVI process. The first part, low level processing, relates to image acquisition and quality enhancement.

I. I NTRODUCTION

978-1-4673-2608-7/12/$31.00 2012 IEEE

Representation
Segmentation
Preprocessing
Image Acquisition

Level Processing

Description

High

Recognition

Intermediate

Interpretation

Knowledge Base

Automated Visual Inspection (AVI) for dimensional inspection is one of the main machine vision applications. AVI
enables inspections in hazardous or hard-to-access environments and can improve productivity and quality in industrial
production lines [1]. For instance, the AVI system implemented by C. Fernandez [2] made orange classification 27
times faster and mistakes reduced by 65% when compared to
human inspection.
Despite the relevance of AVI for industrial automation, it
still faces some challenges. Many AVI systems are based on
computationally intensive algorithms and expensive hardware.
This limits the application of AVI to certain domains. Thus
there is a growing demand for simpler algorithms implemented
in efficient embedded systems.
In order to improve the inspection rate and the image
resolution, this paper presents an implementation in FPGA
of the LEDges technique. We applied it as an AVI system to a
real industrial problem: flaw detection on edges of toothpaste
tubes. We measured and compared some parameters of the
developed AVI system with other related works. Compared
to the ARM-based implementation of the LEDges [3], the
results were much better. But these comparisons are not trivial
when other techniques are used since every inspection has
its specific requirements and we cannot test these approaches
using an identical test set. However, these measures show that
the developed system has a very good inspeciton rate.
This paper is structured as follows. We explain the image
processing pipeline in section II while in section III we
describe the LEDges technique. We show the architecture of

Results

Low

AbstractThis work presents a FPGA design for real time flaw

detection on edges based on LEDges technique. The LEDges, on
one hand, significantly reduces the computational effort to perform the image segmentation, representation and description. On
the other hand reduces the use of costly architectural resources
such as processor and memory. Thus the FPGA design of the
LEDges allows the implementation of automated visual inspection
systems satisfying the increasing demand for performance. We
have developed, implemented and applied the FPGA design to a
real industrial problem, where defects were successfully detected
on edges of toothpaste tubes. We achieve lower response time
and lower use of computational resources than other solutions
which have same computational complexity.

Problem Domain
Fig. 1.

Image processing steps of a typical AVI system.

The intermediate-level processing involves the steps of (i)

segmentation, (ii) representation and (iii) description of the
image. Segmentation may be one of the most important and
complex activities throughout the image processing. In this,
regions of interest (ROI) are identified and then represented
in terms of its internal features (pixels that make up the region)
or external features (edges). The next step is to describe the
ROI into a digital set of numbers that can be processed by a
computer system. The high level processing is responsible for
image recognition, and finally, its interpretation indicating if
the object under inspection is or is not within the acceptable
standards for the industry.

Finally, the knowledge base contains accumulated knowledge about the inspection, provided by man and / or by
machine [4]. For instance, accumulated experiences of staff
in relation to an manual inspection to be automated, which
are already known results related to the problems and their
solutions.
III. LED GES
The LEDges [3] is a AVI technique for real time automated
visual dimensional inspections which is based on simplicity
of the algorithm and low complexity of hardware presented
in [5][10], but with the improvement of detecting all edges
of the object in the images acquired by the camera, not
just the outermost edges. Previously, this capability was only
possible in the works based on depth cameras, laser and projectors [11][14], which has a long processing time, resourceintensive computing, and delicate and complex image capture
devices. This makes it unfeasible their use in embedded and
real-time AVI systems with the following requirements: (i)
Simple classification of the object between good or flaw; (ii)
Fast response time without the use of high speed cameras;
(iii) A fuzzy background with many objects making the image
processing harder; (iv) A low space available to assemble the
new AVI system to preexisting production machinery. To the
best of our knowledge there are no solutions available that
meet all these requirements.
The LEDges is described in four steps as presented in
Figure 2. First the object is illuminated with a high power
structured light source and its image is captured by the camera.
The result of this step is an image displaying the lighted areas
of the objects in the foreground in higher intensities, while the
background and the shadows are displayed in lower intensities.
In the thresholding step the acquired image is easily segmented into two levels of intensity: (i) the background and
the shadows are indicated by regions with zero intensity, color
black, (ii) and light areas in the foreplane are indicated by the
maximum intensity, color white.
In the third step, signature generation, the thresholded image
is represented by its edges and described as a signature.
Each edge is represented numerically in terms of the distance
between the beginning of the image lines and relevant blackto-white and white-to-black transitions of the thresholded
image. This set of distances makes the image signature. Note
that, in general, the Hough transform, used in [15], [16] would
require greater use of hardware.
The last step is the comparison, which compares the generated signature with a standard good objects signature and

Scene

Input

Image
capture

Thresholding

generates as output a binary flag that indicates whether the

object is a good one or there is any defect on its edges.
One advantage of the technique is that it allows all steps (image capture, thresholding, signature generation and compare)
to be executed in a pipelined approach. As a consequence, the
memory usage is significantly reduced, since it is not necessary
to store the image for processing, reducing the implementing
cost and the total processing time.
As a restriction of the technique, as well as many AVI
systems based on light reflection, the LEDges depends on the
positioning of illumination and camera, as well as the object
color and texture of the material. Despite these restrictions,
this technique is well fitted to an industrial environment where
they can usually be met as it is a controlled environment.
IV. A RCHITECTURE D ESIGN
The definition of the architecture to be used for implementation of the LEDges is based on the computational resources
required to perform the image processing and organization
of these resources in order to meet the industry requirements
regarding the inspection.
The main objective is to properly meet the specified functionality and to achieve the speed and accuracy rate required.
A secondary objective has economic nature, is to reduce
production costs. Related with the user interface, the flexibility
to update the parameters of the inspection is another desirable
quality. The effort of the engineering team for development of
the AVI system and the timing of availability of the product on
the market also differ between architectures. Finally, there is
concern about energy efficiency. However, embedded systems
typically have lower power consumption than the acceptable
defined by the industry.
The procedure for architecture definition was the following:
(i) Preliminar analysis of the algorithm for the identification
of the necessary controllers, (ii) computer modeling of the
LEDges using the FREEMAT software (Fig. 3), (iii) optimization with the identification of performance bottlenecks in data
processing and verification of really necessary data widths,
(iv) preparation of a list with characteristics and constraints of
potential architectures for checking the technical adequacy of
the LEDges with the architectures. As a result of this analysis,
the specified functionality can be met adequately, which is the
main objective in the choice of architecture, with the FPGA
design.
The second priority is to achieve the necessary inspection
speed keeping the hit rate obtained in the modeling. In an

Signature
generation

Processing Steps
Fig. 2.

LEDges processing steps.

Comparison

Result

Output

Fig. 3. Computer modeling of the LEDges on FREEMAT software. The acquired and thresholded image are displayed, as well as the image description as
a signature.

architecture based on the low cost FPGA TERASIC kit DE270, the LEDges can be applied to process images with good
resolution (640 x 480 pixels), detecting dimensional errors of
0.5 mm for the mechanical assembly performed and with a
rate of 100 inspections per second. Finally, computer modeling
allows a fast implementation in FPGA with good flexibility
to update the parameters. Therefore, the choice of an FPGAbased architecture achieves the objectives for the design.
V. FPGA- BASED ARCHITECTURE
The available hardware is the DE2-70 development kit,
including the D5M digital camera and LTM LCD display,
all from the Taiwanese company TERASIC. The DE2-70 is
equipped with a low cost FPGA Cyclone II EP2C70F896C6.
In addition to the components already available in the kit, it
was also necessary to install a high-power LED, required to
perform the LEDges illumination [3].
The AVI system was segmented into six major functional
blocks: (i) image acquisition; (ii) thresholding; (iii) signature
generation; (iv) comparison; (v) LCD displaying. Each block
comprises one or more hardware modules implemented in
FPGA as presented in Figure 4. The figure also shows the
inputs and outputs of the modules.
VI. C ASE STUDY: FLAW DETECTION ON EDGES OF
TOOTHPASTE TUBES

We implemented and applied the LEDges as an AVI system

to a real industrial problem consisting on the flaw detection
on edges of toothpaste tubes in a production line. In this case
study we tested the effect (i) of lighting, (ii) of processing
only the R component in the RGB signal and (iii) of changes
in the camera exposure time. Two parameters of interest were

not adjusted, (i) image resolution and (ii) camera focus, once
it requires major changes in architecture or new mechanical
assemblies.
VII. R ESULTS
We observed some good features of the LEDges when
compared to the AVI systems presented in section III. These
comparisons are not trivial since every inspection has its
specific requirements and we cannot test these approaches
using an identical test set.
The Table 1 shows that the less complex systems only
detect the outermost edges of the object. Those which detect
the internal edges have a complex, or very specific hardware,
and slow image acquisition. Furthermore, none of them were
implemented on embedded system. In fact most of them were
implemented in large industrial computers. Only the LEDges
is able to get the inspection time of 10ms, detecting all edges,
although implemented in low-cost embedded hardware.
Regarding the architecture based on FPGA and digital
camera, the image was acquired faster and with superior
resolution than the microcontroller-based architecture [3]. The
new inspection rate of 10ms is highlighted. We further note
that the two architectures use low-cost processors. We used
only 3% of the total available resources in the low-cost
Cyclone II FPGA.
The size of the final signature was different between implementations. The FPGA presented four times more points
than the microprocessor-based architecture. The more points
you have, the inspection will be more accurate. But for the
simple case study, both AVI systems presented hit rate of
100%. Other features are shown in Table 2 in a comparison
between implementations based on microcontroller and FPGA.

Inspection apparatus
RGGB

RGB
1

101

201

301

401

501

601

26
51
76
101
126
151

GOOD

CAMERA
SETTING
VIA I2C

COMPARISON

SIGNATURE COMPOSITION

IDENTIFICATION OF
RELEVANT TRANSITIONS

THRESHOLDING

RGGB TO RGB
CONVERSOR
+ R FILTER

IMAGE
CAPTURE

CMOS CAMERA

BAD

D5M KIT

FPGA

DISPLAY SELECTOR
SDRAM1 MEMORY CONTROLLER

SDRAM1

SDRAM2 MEMORY CONTROLLER

SDRAM2

LCD SETTING
VIA SPI

LCD DATA
INTERFACE
DE2-70 KIT

Legend:
LCD DISPLAY

LTM KIT

Fig. 4.

Block diagram for FPGA-based architecture.

ACQUISITION
THRESHOLDING
SIGNATURE GENERATION
COMPARISON
LCD DISPLAYING
DATA FLOW PIPELINE

Table 1: Comparison between techniques

Embedded /
Able to
Inspection
detect all Use of computational
time
resources.
the edges

Technique

Related
works

Note

Extraction of foreground from image

pairs, with or without specific lighting

Sun et al. [8]

Very effective for

image segmentation

N/A
(1)

No

Illumination with infrared light to

identify the foreground

Wu, Boulanger
e Bischof [9]

High performance in
the segmentation

100 ms

No

Uses specific and

high cost cameras

66 ms

Yes

Detection of foreground images with

static background

Wang et al.
[11]
Casares et al.
[4]

Simple algorithm

N/A

No

Use of the laser lighting

Liao et al. [12]

3000 ms

Yes

Use of standard projectors

Wang et al.
[13]

500 ms

Yes

LEDges

Batista [3]

10 ms

Yes

Use of depth cameras

High precision;
complex lighting
High precision;
complex lighting
High
performance

No (2) /
Moderate
No /
Low
No /
Low
No (2) /
Moderate
No /
Moderate
No /
High
Yes /
Very low

N/A - Not avaiable

(1) Need to acquire two images
(2) Technique developed for use in embedded systems

Table 2: Comparison between LEDges implementations

Microcontroller-based
Architecture
Inspection rate;
Costs; efforts of the
Objectives / Parameters optimized
Resolution.
engineering team.
FPGA Cyclone II 2C70
LPC2148 - Core ARM7 TDMI
Main processor
3%
9%
Use of the main processor
CMOS Digital
CCD Analog NTSC
Camera technology
10 ms / inspection
16.66 ms / inspection
Inspection rate
640 x 480 pixels
Analog : 120 lines
Image resolution
300
72
Size of final signature
Yes (1)
No
Able to adjust resolution and inspection rate
Low
Very low
Main processor cost
Low
Very low
Camera cost
Moderate
Low
Effort to develop and update design
Moderade
High
Flexibility to update the parameters
100%
100%
Hit rate in inspections
(1) Adjustable from 2592 x 1944 pixels @ 70 ms/inspection to 640 x 480 pixels @ 10 ms/inspection
Parameters

FPGA-based Architecture

Table 3: Used FPGA resources

FPGA Module
Image capture
RGGB to RGB conversor + R filter
Camera setting via I2C
Thresholding
Identification of relevant transitions
Signature composition
Comparison
Total

Combinational
functions

Logic registers

Memory usage

55
146
229
67
128
162
203
990 / 68416
(1.5%)

48
94
134
32
93
135
100
636 / 68416
(0.9%)

0
19152
0
0
0
1000
12080
32232 / 1152000
(2.8%)

Table 3 presents the resourcers related to implementation

of the LEDges modules on the low-cost FPGA Cyclone II
EP2C70F896C6. The report was generated by the Quartus II.
As can be seen, the use of FPGA resources is very low and
most of the resources are used by the image acquisition.
VIII. C ONCLUSIONS AND F UTURE W ORK
Several works present techniques for inspections of edges
on objects placed in the foreground. However, these complex solutions showed long processing time, intensive use of
computational resources, delicate and complex image capture
devices. The technique implemented in this work, LEDges,
performs a robust image segmentation through a simple and
efficient thresholding, which enables high performance with
low use of computational resources.
In the case study, inspection of toothpaste tubes, the FPGA
implementation resulted in a very fast inspection when compared with those presented by competing techniques capable
of performing the same type of inspection. Futhermore, we
used only 3% of the total available resources in the low-cost
Cyclone II FPGA. Thus it is possible multiple implementations
of LEDges in a single FPGA, that can be useful for inspection
of objects from multiple images or several different angles.
The hit rate was 100% proving that it is possible to perform
image processing in simple embedded systems. But make
no mistake, the simplicity can only be achieved through hard
work (Clarice Lispector).
As future work, it will be necessary to develop a system for
functional verification, make more accurate measurements of
precision and accuracy, develop a supplement for FREEMAT
automating the code conversion to Verilog. Then the FPGAbased AVI system will be ready to be tested in new highpeformance applications.
ACKNOWLEDGMENT
The authors would like to thank Optanica Photonic Solutions for technical support and Rafael C. Neto for his aid with
translation. In addition, the authors thank FACEPE for partial
financial support in the design development.

R EFERENCES
[1] E. Malamas, E. Petrakis, M. Zervakis, L. Petit, and J. Legat, A survey
on industrial vision systems, applications and tools, Image and Vision
Computing, vol. 21, no. 2, pp. 171188, 2003.
[2] C. Fernandez, J. Suardiaz, C. Jimenez, P. Navarro, A. Toledo, and
A. Iborra, Automated visual inspection system for the classification
of preserved vegetables, in Industrial Electronics, 2002. ISIE 2002.
Proceedings of the 2002 IEEE International Symposium on, vol. 1.
IEEE, 2002, pp. 265269.
[3] Y. Batista, LEDges: A new automatic visual inspection technique for real
time flaw detection on edges. Masters dissertation. Federal University
of Pernambuco, Brazil, 2011.
[4] T. Brosnan and D. Sun, Improving quality inspection of food products
by computer visiona review, Journal of Food Engineering, vol. 61,
no. 1, pp. 316, 2004.
[5] M. Casares, S. Velipasalar, and A. Pinto, Light-weight salient foreground detection for embedded smart cameras, Computer Vision and
Image Understanding, vol. 114, no. 11, pp. 12231237, 2010.
[6] R. Bolle, J. Connell, N. Haas, R. Mohan, and G. Taubin, Object imaging
system, May 20 1997, uS Patent 5,631,976.
[7] J. Sun, Y. Li, S. Kang, and H. Shum, Flash matting, ACM Transactions
on Graphics (TOG), vol. 25, no. 3, pp. 772778, 2006.
[8] J. Sun, S. Kang, Z. Xu, X. Tang, and H. Shum, Flash cut: Foreground
extraction with flash and no-flash image pairs, in Computer Vision and
Pattern Recognition, 2007. CVPR07. IEEE Conference on. IEEE, 2008,
pp. 18.
[9] Q. Wu, P. Boulanger, and W. Bischof, Robust real-time bi-layer video
segmentation using infrared video, in Computer and Robot Vision,
2008. CRV08. Canadian Conference on. IEEE, 2008, pp. 8794.
[10] , Bi-layer video segmentation with foreground and background
infrared illumination, in Proceeding of the 16th ACM international
conference on Multimedia. ACM, 2008, pp. 10251026.
[11] L. Wang, C. Zhang, R. Yang, and C. Zhang, Tofcut: Towards robust
real-time foreground extraction using a time-of-flight camera. Citeseer,
2010.
[12] C. Liao, J. Su, Y. Lin, and Y. Tarng, Application of the structured
illumination method for automated visual inspection of the loudspeaker
cones, Journal of Materials Processing Technology, vol. 200, no. 1-3,
pp. 5970, 2008.
[13] Z. Wang, H. Du, S. Park, and H. Xie, Three-dimensional shape
measurement with a fast and accurate approach, Applied Optics, vol. 48,
no. 6, pp. 10521061, 2009.
[14] M. Chang and S. Park, Automated scanning of dental impressions,
Computer-Aided Design, vol. 41, no. 6, pp. 404411, 2009.
[15] S. Zhang and Z. Liu, A robust, real-time ellipse detector, Pattern
Recognition, vol. 38, no. 2, pp. 273287, 2005.
[16] C. Chien, Y. Cheng, and T. Lin, Robust ellipse detection based on
hierarchical image pyramid and hough transform, JOSA A, vol. 28,
no. 4, pp. 581589, 2011.
[17] F. Duan, Y. Wang, H. Liu, and Y. Li, A machine vision inspector for
beer bottle, Engineering Applications of Artificial Intelligence, vol. 20,
no. 7, pp. 10131021, 2007.