Mechatronics 10 (2000) 869880

Robotic sorting of paper items from a random pile

C. Jones*, G. Ross, J.R. Hewit, A.P. Slade
Department of Applied Physics, Electronics & Mechanical Engineering, University of Dundee, Dundee, DD1 4HN, UK Received 2 April 1999; accepted 20 June 1999

Abstract A random stack of documents is physically sorted into ordered stacks according to type and condition. This is done by rst locating the topmost item using a machine vision system with algorithms based upon the Hough Transform. The edges of the items are identied from the shadows created by illuminating the pile from dierent directions. Having located the position and orientation of the topmost item, a robot arm is programmed to pick the item from the pile using an end-eector with vacuum picking mechanisms. The robot arm oers the picked item ``on-the-y'' to devices which ascertain its physical tness and classication. Finally, the item is placed on the appropriate one of a number of piles of similar category. 7 2000 Elsevier Science Ltd. All rights reserved.

1. Introduction Bin-picking by robot is a generic task common to a wide range of industries. Examples include the picking of piston rods from metal bins in a car engine plant, the picking of apples from boxes of fruit, and the picking of socks from a random pile at the output of a knitting machine. As these three examples show, the bin-picking process can involve parts that are rigid and geometrically determinate, or parts that are exible and deformable with little geometrical denition. The former are easier to bin-pick than the latter for
* Corresponding author. 0957-4158/00/$ - see front matter 7 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 9 5 7 - 4 1 5 8 ( 9 9 ) 0 0 0 4 5 - 8

870

C. Jones et al. / Mechatronics 10 (2000) 869880

two major reasons their positions and orientations are easier to determine, and stay xed after determination, and they are easier to pick up by a robot endeector. In all cases, however, it is likely that the same basic processes will be used to eect the bin-picking procedure. Computer vision will be used to locate the parts to be picked and, in particular, to dierentiate between the part to be picked next and the other parts in the bin. A suitable end-eector, possibly of a special design, will be used to do the actual picking under the direction of the computer vision system. As well as these basic processes there may be other processes taking place during the bin-picking procedure. For example, it is sometimes possible to apply tests to the picked part during the picking process; apples may be inspected for blemishes or poor coloration; socks may be sized on the y; electronic components may be tested for continuity or for resistance etc. In the following a description is given of an industrial bin-picking procedure applied to the picking, classifying and sorting of paper items presented as a random pile. This application poses a number of dicult problems. These problems are discussed and ways of overcoming them are presented which will prove to be applicable in a wide range of applications.

2. Picking of paper items There are many examples of industrial and commercial operations where the picking of paper items must be done accurately, robustly and fast. Banks must sort piles of banknotes of dierent denominations into other piles each containing only a single denomination; the Post Oce must sort piles of letters and postcards prior to delivery; the counting of votes during an election is done by sorting the voting papers. Traditionally, all of these tasks are done by humans who use their visual sense to locate the appropriate piles of items and, in some cases, to detect the next item to be picked. The human operator also makes use of tactile sensing and coordination between the hand and the eye to eect an ecient pick. As mentioned above, inspection operations are also undertaken the voting paper is inspected to determine the voter's preference; the letter is inspected to locate the address. Where robots are proposed to replace the human it is essential that these sensory and co-ordinative capabilities are replicated as far as technology will allow. For paper items, sensory and co-ordinative functions are used during the following repeated sequence: . . . . nd the topmost item on the pile; determine its position and orientation; determine whether it is defective; control a suitable end-eector into place to pick the item;

C. Jones et al. / Mechatronics 10 (2000) 869880

871

. apply tests on-the-y to the item; . deposit the item in a suitable manner. These individual functions are described in detail below. 3. Machine vision aspects A random pile of media contained within a rectangular bin presents a binpicking scenario which is further complicated by the variable form and type of such media. These items of media may have any or all of the following: printed data on their surfaces; folded corners; torn edges or attachments such as staples or tape. The contents of the bin are viewed from above with a monochrome CCD camera and a typical pile as seen by the camera is shown in Fig. 1. Items within the bin are to be removed singly from the top of the pile, hence requiring the determination of the uppermost item of the pile. A wide range of currency and cheques together with the possible inclusion of `foreign' objects such as business cards, bus tickets etc. may be in the bin. This renders direct pattern recognition unfeasible and item location is derived from the physical boundary of items since the uppermost item will generally have its complete boundary visible. 3.1. Edge enhancement Previously, thin and planar mechanical parts have been located in a bin-picking situation [1]. These parts were located via edge tracking after grabbing a series of images illuminated from a range of positions. Our application uses a similar lighting arrangement, however, we create shadows at the physical edges of the media, but need to remove any printed data (noise) from the image [2]. By successively illuminating the bin from above each side we create the desired

Fig. 1. Typical pile of paper items.

872

C. Jones et al. / Mechatronics 10 (2000) 869880

shadows. A simple subtraction procedure would then, to a rst approximation, remove noise. In reality the intensity of each image varies across each image; theoretically this variation is non-linear, however in practice it is found to be approximately linear. We therefore modify the images by taking on-line measurements of the intensity at each side of the bin, and applying a correction factor to each image. The on-line measurements enable accurate estimation of the correction factor irrespective of the output of each source which may degrade over time. Fig. 2 shows histograms of intensity through the image (Fig. 1) where source A is located parallel with the upper edge of the image and source B is located

Fig. 2. (a) Source A energised. (b) Intensity corrected. (c) Source B energised. (d) Intensity corrected. (e) Eect of [(b)(d)+128].

C. Jones et al. / Mechatronics 10 (2000) 869880

873

parallel with the lower edge. The original variations of intensity are shown in Figs. 2(a) and (c) and the respective corrected intensities are shown in Figs. 2(b) and (d). Fig. 2(e) shows a histogram illustrating the result of a subtraction and a 128 grey level shift to indicate physical edges which step up or down when moving across the bin from top to bottom in Fig. 1. The histograms of Figs. 2(a)(e) indicate the procedure applied when light sources parallel to the top and bottom of the image in Fig. 1 are applied. Further identical processing takes place for sources applied at the left and right edges. The nal result after edge detection and binarisation is shown in Fig. 3. 3.2. Feature detection and location The items constituting the pile of media are essentially rectangular but no edge orientation information is derived from the enhancement process and therefore the use of a Generalised Hough transform [3] is not possible for object location. The fragmentation of physical edge features renders other methods such as boundary coding [4], for example, unusable. We locate individual features of the items via a Hough transformation and have developed a simple, accurate and robust procedure to locate these features, known as the COA algorithm [5]. The fragmentation of the edge data and its variable width together with the lack of edge orientation data produces a cluttered parameter space and non-ideal peak formations. The COA algorithm searches for regions indicating peaks in parameter space and is particularly eective when peaks are sharp [6] a phenomenon which exists in this case. Having located these regions, the algorithm takes a slice through the peak at a level below the summit of the peak and computes the centre of area of this region. The eect of this is to ignore errors from the distant parts of the distribution and also errors around the focal point of

Fig. 3. Binary edge-detected image.

874

C. Jones et al. / Mechatronics 10 (2000) 869880

the peak; this can be likened to the Least Median of Squares approach to mode location [7]. We obtain sub-pixel accuracy of parameter estimation with little computational eort and the method has proved to be robust. 3.3. Object location Final location of the uppermost object of the pile is achieved by an analysis of the detected features based on a simple search of these features and incorporates tolerances on the angular location of each feature. The nature of the items within the bin and the preliminary image processing stages are such that very few items indicated in the binary edge image have truly parallel and perpendicular features. The complete image processing and analysis procedure incorporates many tradeos, generally implemented via thresholding. Ultimately, the detection and location of the uppermost item is not guaranteed. However, within the object location procedure, safeguards are built in such that failure to locate the uppermost item is indicated. Under these circumstances the contents of the bin are disturbed mechanically and the procedure is repeated until successful identication and location is achieved. 4. Media picking and handling Individual items are to be picked from the bin and transported to sensing stations and storage trays. A commercial industrial robot (Staubli RX90) is utilised to provide six degrees of freedom. The automated handling of exible materials [8] is a dicult task which in general requires a specially designed end eector. The nature of the media to be handled in this application is such that it may range from crisp new banknotes or cheques, to very worn, torn banknotes and may be of a wide range of sizes. The end eector designed to pick and handle this wide range of media is shown in Fig. 4. 4.1. Picking Media can only be attached to the upper side and vacuum provides an ideal method of attachment. The inner section of the end-eector houses two suction cups which attach to the desired item. The movement of the robot in the zdirection, when picking, is controlled by a vacuum sensor located in the line. The variable form of the pile of media renders optical methods of detecting the height of the pile ineective. Once attached to an item the end-eector is lifted away and the inner section retracts. An ultrasonic transmitter is located within this extendible section for condition sensing. 4.2. Secure attachment for transport and sensing The physical condition of this media is unknown at the picking and initial

C. Jones et al. / Mechatronics 10 (2000) 869880

875

handling stages. Secure attachment is necessary so that accurate positioning over sensing stations can be achieved. It is also required because high dynamic and aerodynamic forces may exist when transporting the media at high speed. High attachment forces are applied to each item irrespective of damage, such as torn edges and porosity due to wear or damage, by application of vacuum to a porous pad. A high mass ow rate fan is used to supply the vacuum force which, by careful choice of the spacing and size of the holes in the pad, applies maximum attachment force and attens the items to assist sensing. 5. Sensing Having located the top item of the unordered stack of items and picked that item, the document must be at least recognised if automatic sorting and placement are to be carried out. It may also be necessary, depending on the document, to validate it and screen it for condition. For example, a pile of valuable documents, such as stamps or tickets, might contain forgeries or items too soiled to be acceptable in some way. The establishment of these properties would seem simple given that an imaging device is available within the system. This, however, is optimised for a very particular operation. To extend its uses to recognition etc. would require a separate illumination system, a dierent image set and further processing time. It is also unlikely that ecient validation and condition screening could be performed without using colour imaging. Recognition itself could prove time-consuming for the existing system due to registration being only as good as

Fig. 4. The end-eector.

876

C. Jones et al. / Mechatronics 10 (2000) 869880

the edge detection (22 mm laterally and 228 skew). Thus, OCR would be timeconsuming even for a small set of items. The technique the system described actually uses the one which utilises the robotic handling of the documents such that recognition, validation and condition can be established while the document is in transit between the bin and its nal destination, leaving the imaging system to establish the next top document ready for picking when the end-eector returns. 5.1. Recognition Recognition can be assisted by data on item size. The dimensions of items can be determined to within 2 mm by the edge detect system. If the application is to sort several dierent types of items into classes and reject all others as a separate class, then it is possible rst to reject all items whose dimensions are incorrect. However, even if each known class has unique dimensions this does not make its members unique in the universal document set. Some further form of dierentiation is necessary. In this case the colour and spatial pattern along a xed stripe are utilised to recognise known document types. Whilst in transit, the robot arm carries the item at a constant speed across a stationary colour sensor. This sensor has a wide bandwidth photodiode as the sensing element, with various LEDs positioned around the detector to provide known bandwidth illumination. The LEDs range from blue to near IR. The position of the LEDs and the aperture of the photodetector port are designed to exclude specular reections from the item. The separate LEDs pulse sequentially and the reections from the passing item provide a pulse train of colour information at a constant rate. Sample start/stop is automatically controlled and is dependent on the position of the end-eector and item dimensional information. The data is de-multiplexed into individual colour signatures which are compared using Bayesian classication (Eq. (1)) to a database from known items. Four stripes for each item are stored in the database to cover the four possible ways the item might be presented to the sensor. The closest t is determined. Those items within tolerance are deemed to be recognised. 5.2. Validation Depending on the tolerance of the recognition algorithm, most simple forgeries would be rejected at the recognition stage. This would include opportunist forgeries made by scanning originals and subsequently printing on normal printer paper using a desk top printer. The more valuable items such as currency, stamps, event tickets etc., are targeted by professional counterfeiters using more sophisticated printing techniques. To counteract these forgeries the system examines the signatures resulting from the colour dierences and by applying time dependent discriminant analysis to the signatures. These discriminant functions are generated by training the system to highlight the locations of dierences. This is shown in Fig. 5.

C. Jones et al. / Mechatronics 10 (2000) 869880

877

5.3. Condition-screening There are many aspects of an item covered by the term `condition-screening'. It involves the aesthetic appearance of the item, its age and the degree of wear, and the presence of folds, tears and attachments. All of these features can be determined in various ways. Here age and degree of wear are considered and is expressed subjectively by the limpness of the item. Appearance is included implicitly, via the recognition and validation algorithms, but only for the specic path examined. To determine limpness, ultrasonic transmittance is employed. A transmitter is located in the end-eector pad and at one point, during the passage of the item from the pile, the transmitter passes directly above a receiver in a known position. Processing of the limpness data must take into account the item recognition data. This is because limpness is subjective, for example, a limp business card may not be as limp as a new banknote. 5.4. Sorting Sorting is very application dependent. In this case rejects at any stage are sent to a specic reject bin for further mechanical or manual sorting. Items which are recognised and pass the condition screening process are sorted into ordered stacks within machine tolerance, or inserted into subsequent machines for binding or for redistribution. 6. Results and discussion The system was tested using ve known types of genuine paper document mixed

Fig. 5. Classication using colour.

878

C. Jones et al. / Mechatronics 10 (2000) 869880

randomly with unknown documents and colour copies of the known documents. The results for the various aspects of the system are given below. 6.1. Recognition Using the Bayesian function (Eq. (1)) the system correctly identied 100% of the ve types of item it was trained to recognise. The system also had a 0% false accept rate, i.e. all unknown documents were rejected. Lj x i mij 2
i 2 sij

class j 4 Lj minLj 6.2. Validation

Tests for validation were carried out on genuine and copied items, where the copies ranged from colour scans and those from desk top colour printers to those that were printed using multicolour lithographic printing. In every case the copy was rejected as can be seen in Fig. 6. 6.3. Screening Condition screening was also almost entirely successful. It was found that the detected signal of a new item was substantially dierent to that of a used but subjectively acceptable one. Both were dierent again to that of a used and unacceptable item. These results have been obtained for all types of item investigated, though the signal levels were dierent for each of the document types examined.

Fig. 6. Validation results.

C. Jones et al. / Mechatronics 10 (2000) 869880

879

Fig. 7. Vision system success rate.

6.4. Vision The machine vision sub-system is three-stage (edge enhanced and detection, feature parameter estimation, feature analysis) with each stage itself being multistage. For example, the COA algorithm rst located peaks in parameter space and then estimates feature location. Many thresholds are required to be set to achieve successful object detection and location and each stage is highly dependent on the ecacy of previous stages. Sampling, using a wide range of items, resulted in success of detection and location as indicated in Fig. 7. No complete failure to detect and locate was indicated, however, success was not necessarily achieved at the rst attempt. Three modes of failure to detect and locate may occur: 1. there may be insucient edge data for any particular edge feature to be detected; 2. there may be excessive noise in the image so that the search procedure registers occlusion; 3. the detected `edge' may be too wide so that the COA algorithm fails to locate any particular feature. In the results obtained, where failure occurred, they occurred with the following frequencies; Mode 1: 10; Mode 2: 4; Mode 3: 7. References
[1] Dessimoz JD, Kunt M, Zurcher JM. Recognition and handling of overlapping industrial parts. In: Proceeding of the Ninth International Symposium on Industrial Robots, Washington DC, 1979. p. 35766. [2] Jones C, Hewit JR. The extraction of physical edge features of deformed printed paper documents, 1998 (in preparation). [3] Ballard DH. Generalising the Hough transform to detect arbitrary shapes. Pattern Recognition 1981;13:11122. [4] Freeman H. On the encoding of arbitrary geometric congurations. IEEE Trans Electron Comput 1961;10:2608.

880

C. Jones et al. / Mechatronics 10 (2000) 869880

[5] Jones C, Hewit JR. The accurate and robust location of edge features of paper documents via the COA algorithm. Pattern Recognition Letters 1998 (submitted for publication). [6] Davies ER. Modelling peak shapes obtained by Hough transform. IEE Proceedings E 1992;139(1):912. [7] Davies ER. Machine vision: theory, algorithms, practicalities. 2nd ed. Academic Press, 1997. [8] Taylor P. Handling of exible materials in automation. International workshop on advanced robotics and intelligent machines, University of Salford, 1994.