Research of Shoeprint Image Stream Retrival Algorithm with Scale-Invariance Feature Transform

Zhongwei Li
College of Software Nankai University Tianjin, China lizhongwei@nankai.edu.cn

Chiahung Wei
Department of Information Management Ching Yun University Taiwan rogerwei@cyu.edu.tw

Yue Li
College of Software Nankai University Tianjin, China Liyue80@nankai.edu.cn

Tsaiyang Sun
Department of Information Management Ching Yun University Taiwan

AbstractShoeprints left at the crime scene provide valuable information to criminal investigation due to distinctive patterns in the sole. Those shoeprints are often incomplete and noise ones because incomplete and noise prints may be resulted from external destruction or incomplete contact between shoe sole and the ground surface. In this study a method including feature extraction of keypoints and similarity matching is proposed for recognition and retrieval of partial and noise shoeprint images. Experimental results show the full-size prints and toe prints perform best among all shoeprints. Furthermore, this system also demonstrates its robustness against noise because there is a very slight difference in comparison between original shoeprints and noise shoeprints. Keywords-Shoeprint Retrieval, Outsole Recognition, Scale-Invariance Feature Transform, Keypoint Detection, Forensic Science

I.

INTRODUCTION

Shoeprint refers to the mark made by outside surface of the sole of a shoe [1]. When a person takes a step, his/her shoe exerts pressure on the surface of the floor which leaves an impression in the floor [1]. Shoeprints are considered important forensic evidences in legal courts [2]. To perform forensic analysis, shoeprint images are normally sampled using photography, gel, or electrostatic lifting from crime scene [3]. Subsequently, at forensic shoeprint laboratory, this shoeprint image can be used as evident to build a crime case or made comparison against the database of shoeprints found at other crime scenes [4]. Before pattern recognition techniques were applied for shoeprint recognition, forensic shoeprint examiners used to manually annotate and classify shape and pattern primitives of shoeprints, e.g., circles, squares, zig zags, etc. In this way, identifying an example shoeprint from databases requires searchers to describe the print in a similar manner as it is depicted by shoeprint annotators [2, 5-7]. Since this is a laborious and time consuming task to compare a shoeprint with existing shoeprints in the database, computer-based assistance to the task may greatly improve the efficiency to crime investigation [8].

Some studies have recently been reported in references [3, 9-11]. Algarni & Hamiane [9] described shoeprints based on seven Hus moment invariants, which are invariant to rotation and changes in spatial resolution. Since Hus moment invariants belong to a region-based shape descriptor, it can only performed well when the investigated shoeprints are full-size images. This limitation devalues the practical application because many shoeprints left at crime scenes are often partial impressions rather than complete impression. A way to solve the partial shoeprint recognition problem is to make use of other local and invariant features abundant on the partial shoeprint. Some researchers have worked on the problem of matching a partial shoeprint to full or partial template shoeprint. In de Chazal et al [3], a reference shoeprint image is preprocessed and performed discrete Fourier transformation. Then, the transformed reference image is used to generate the power spectral density, which represents the levels of different spatial frequencies for the shoeprint pattern. The feature is invariant to translational and rotation. In their experiments, the correction rates at the first and fifth rank are 65% and 87% on full-prints, and 55% and 78% for partials. Nibouche, Bouridane, Gueham, & Laadjel [10] propose a solution to rotated partial shoeprint retrieval, based on the combined use of local features. Patil and Kulkarni [11] used Gabor feature map of shoeprint images to extract multiresolution features, which are invariant to intensity and rotation. It is very common to obtain incomplete shoeprints from crime scenes because partial prints may be resulted from external destruction or incomplete contact between shoe sole and the ground surface. The performance of a retrieval system for partial prints is of considerable interest and importance. Therefore, the main contributions of this paper are summarized below: a) A method including local invariant features and similarity matching is proposed and applied for recognition and retrieval of outsole patterns. b) Partial images of shoeprints are created and tested to verify the performance of the proposed system. c) Noise, including black and white stains, is added to verify the performance of the proposed system.

978-1-61284-774-0/11/$26.00 2011 IEEE

5488

II.

PROPOSED METHOD

A flow diagram of the shoeprint retrieval system proposed in this study is illustrated in Figure 1. The proposed method firstly constructs different scale spaces in order to detect local extrema in the underlying shoeprint images. Those local extrema are considered as useful keypoints in the image. Next, the features of those keypoints are extracted to represent their local patterns around keypoints. Then, the system computes the cross correlation between the query image and each shoeprint image in the database. On the basis of similarity measure, the database shoeprints that most likely belong to an identical pattern to the query shoeprint are ranked in order. A. Feature Extraction Scale-invariance feature transform (SIFT) [12] is one of the popular methods to detect keypoints of an object so that the same object in another image can be recognized with invariance to scale, rotation, translation and illumination. The basic idea behind the keypoint detection method assumes that most salient points exist in corners of objects and are seen as interest points. It is denoted that though salient points do not only exist in corners, those points in corners are considered more stable and useful than those in edges for object recognition and similarity matching applications.

scales. To find local maxima and minima, each pixel in the DoG image is compared to eight neighbors in the same scale and nine neighbors in the neighboring scales. The pixel is considered as a candidate keypoint only if it is local maxima or minima. However, some pixels may be mistakenly localized because those with low contrast are sensitive to noise [12]. To rule out those improper keypoints, we have to verify whether each point could reasonably be selected in another image of the same shoeprint pattern. A 3D quadratic function is fitted to the selected points in order to filter the point in low contrast [13]. D ( x ) , described in (3), is expanded at the select point x by Taylor expansion.

(3) where D and its derivatives are evaluated at the selected point and x = ( x, y )T is the offset from this point. Taking the derivative of this function with respect to x and setting it equal to zero, we can determine the extremum, x , to be
2 D 1 D (4) x 2 x The extremum can help us verify the keypoints in low contrast and reject those points. We substitute (4) into (3) which result in = x

D( x ) = D +

1 D 2 D x + xT x 2 x x 2

1 DT x (5) 2 x ) is less than 0.03. To A given point is rejected if D( x determine if an extrema point is along an edge, using the Hessian matrix ) = D + D( x
Figure 1. Flow chart of shoeprint recognition and retrieval using local features of keypoints.

Dxx H = Dxy

Dxy Dyy

(6)

B. Scale Space Construction The first step of the SIFT is to identify stable points from shoeprint pattern which is invariant to scale, rotation and translation. Assume x, y represent the coordinates of a pixel in image I. The scale space L( x, y , ) can be constructed by utilizing Gaussian filter G ( x, y , ) to smooth the base image I ( x, y ) as described in (1).

where D is the second partial derivative of the DoG image at a scale,

(1) L( x, y, ) = G ( x, y, ) I ( x, y ) where is defined as the width of the filter, and * is the convolution operation. Then, Difference-of-Gaussian (DoG) images of a shoeprint image I , as shown in Equation (2), can be created by subtracting each Gaussian image from the previous Gaussian image in scale.

DoG ( x, y, ) = L( x, y, k ) L ( x, y, ) (2) where k is a constant multiplicative factor used for varying the scale. C. Local Extrema Detection DoG images are used to detect potential keypoints through the findings of local maxima and minima across different

Figure 2. Scale space construction of shoeprint images.

The following inequality is used to find edge and corner points. If

5489

trace( H )2 (1 + 2 )2 ( ( r + 1) 2 ) (r + 1)2 = , = < 2 det(H ) r2 r 12

2

(7)

, where 1 < (1+ r)2 . As r = 10, the extrema point is considered to be a corner; otherwise, the point is rejected as an edge point. After rejecting points based on contrast, edge value, and stability, the remaining points are assigned to describe the pattern of the given shoeprints. D. Orientation Assignment For each selected keypoint, the dominant orientation of the gradient of the points within a window around the point is determined from the quantized histogram of orientations. The magnitude and orientation of the gradient of each point is found and its orientation is stored within one of 36 bins. For each point within a window of W around thee keypoint, the weighted gradient magnitude is added to the bin corresponding to that points orientation and the gradient magnitude is weighted by a Gaussian centered at the keypoint with standard deviation.

210mm 297mm piece of paper for each shoe. As seen in dataset no. 1 of Table 1, each shoe was used to stamp and create five shoeprints with different qualities. The five shoeprints, which belong to the same pattern category, were of variable quality with some prints clearly showing the full detailed of shoe mark while others only captured part of the shoe mark. Dataset no. 1 includes 86 separate pattern categories, each containing five full shoeprints from either right or left shoes. This results in 430 full-size shoeprints in total. This dataset of full shoeprint images was generated by digitizing the paper shoeprints in 256-level gray scale. 2) Original Partial Shoeprints Since partial shoeprints are more common at crime scenes than full shoeprints, the performance of a system in identifying partial images of shoeprints is of considerable interest. To generate partial shoeprint images, each full-size shoeprint images were utilized to generate partial images. Fig. 1 illustrates that a full-size shoeprint is utilized to generate four partial shoeprints, each containing approximately 50 percent of a full shoeprint. Therefore, four datasets of partial shoeprints were created as seen in dataset no. 2 to no. 5 of Table 1. 3) Noise Full-Size and Partial Shoeprints Prints of outsole pattern are often disfigured due to some factors, e.g., fine grain or stone crammed on the outsole, and blurring caused by foot slippage. These factors cause some stains (i.e., black spots) or erase some regions (i.e., white spots) on shoeprints, increasing the difficulty in image analysis. For this reason, it is necessary to examine the effectiveness of shoeprint retrieval using noise shoeprints. To generate noise shoeprint images, each image of the datasets from no.1 to no. 5 was taken to randomly add five black spots and white spots within the outsole. This results in another five datasets as shown from dataset no. 6 to no. 10 of Table 1.

(a)

(b)

Figure 3. (a) Keypoint detection on shoeprint using SIFT. Scale and direction of orientation are indicated by arrows (b) Salient keypoints matching between two shoeprint images.

E. Keypoint Matching As described in the previous section, the two feature descriptors magnitude and orientation of keypoints can be extracted from each shoeprint image. For similarity matching cross correlation for each feature is computed so as to determine the similarity between the query shoeprint and the database shoeprint images. Cross correlation c is a measurement of estimating the degree to which two vectors are correlated. Ai and Bi denote the i-th elements in vector A and B, respectively. Finally, the values of c for the two features are summed to represent the final score of the similarity. III. EXPERIMENTS In this study the experiments were conducted to investigate the following issues: 1) Performance evaluation of retrieving full-size shoeprints; 2) Performance evaluation of retrieving partial shoeprints, such as toe and heel; 3) Performance evaluation of retrieving noise full-size and partial shoeprints. A. Shoeprint Databases 1) Original Full-Size Shoeprints In this study shoeprints were obtained by inviting participants to tread on an inkpad and then stamp on a

Figure 4. Illustration of partial shoeprints generated from a full-size shoeprint.

B. Experimental Results and Discussions In this study the measurement of performance for the proposed system is cumulative match scores that are plotted on vertical axis of a graph [3]. Cumulative match score is estimated as the number of shoeprint images retrieved from the same shoeprint category to the total number of shoeprint images stored in the database. The horizontal axis of the graph in Figure 5 represents the percentage of database images reviewed for the whole database. Each point on the curve represents the probability of a match within the top n percent of all sorted images. A shoeprint retrieval system with perfect retrieval results in the curve that passes through the upper left corner. Therefore, the closer the curve is to the upper left

5490

corner, the higher the overall cumulative match scores of the evaluation. Figure 5(a) demonstrates that those curves represent different types of outsoles, whose overall performance are ranked as full-size, toe, heel, right-half, and left-half from highest to lowest. The curve of toe prints demonstrates the performance as high as that of full-size prints. It is also observed that those split in horizon, including toe and heel, perform better those split in vertical, including right-half, and left-half. Another experimental result shown in Figure 5(b) is to demonstrate whether noise will decrease the performance of the proposed method. Figure 5(b) shows that, overall, the existence of noise does not result in significant differences as compared with Figure 5(a). Figure 5(a) and (b) both show that the proposed system can retrieve toe prints and heel prints more effectively than left-half prints and right-half prints. The finding can be attributed the fact that the patterns in some left-half and right-half prints were disfigured so that the local features of those half-prints were changed, thereby affecting pattern analysis on shoeprints.
TABLE I. THE DATASETS OF SHOEPRINT IMAGES USED FOR PERFORMANCE EVALUATION. quality original original original original original stains & erasion stains & erasion stains & erasion stains & erasion stains & erasion categorie images in images totally s category 86 86 86 86 86 86 86 86 86 86 5 5 5 5 5 50 50 50 50 50 430 430 430 430 430 4300 4300 4300 4300 4300

broken patterns. Hence, we are currently investigating ways of describing patterns of right-half prints and left-half prints. In addition, many partial-shoeprints left from the same shoe contain only a small region overlapped, which would be a crucial issue to survey in the future.

(a)

(b)

Figure 5. Performance evaluation based on original shoeprints (a) without noise (b) with noise. ACKNOWLEDGMENT

This paper is supported by "the Fundamental Research Funds for the Central Universities". REFERENCES
[1] W. J. Bodziak, Footwear impression evidence: Detection, recovery, and examination, 2 ed. Boca Raton, FL: CRC Press, 2000. [2] S. Mikkonen, et al., "Use of footwear impressions in crime scene investigations assisted by computerised footwear collection system," Forensic Science International vol. 82, pp. 6779, 1996. [3] P. de Chazal, et al., "Automated Processing of Shoeprint Images Based on the Fourier Transform for Use in Forensic Science," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, pp. 341-350, 2005. [4] C. Huynh, et al., "Automatic classification of shoeprints for use in forensic science based on the fourier transform," presented at the IEEE International Conference on Image Processing, 2003. [5] W. Ashley, "What shoe was that? The use of computerised image database to assist in identification," Forensic Science International, vol. 82, pp. 7-20, 1996. [6] A. Girod, "Computer classification of the shoeprint of burglarsshoes," Forensic Science International, vol. 82, pp. 59-65, 1996. [7] S. Mikkonen and T. Astikainen, "Databased classification system for shoe sole patterns identification of partial footwear impression found at a scene of crime," Journal of Forensic Science vol. 39, pp. 1227-1236, 1994. [8] T. J. Hannigan, et al., "Survey of 1276 shoeprint impressions and development of an automatic shoeprint pattern matching facility," Science & Justice, vol. 46, pp. 79-89, 2006. [9] G. AlGarni and M. Hamiane, "A novel technique for automatic shoeprint image retrieval," Forensic Science International, vol. 181, pp. 10-14, 2008. [10] O. Nibouche, et al., "Rotation Invariant Matching of Partial Shoeprints," presented at the Proceedings of 13th International Machine Vision and Image Processing Conference, 2009. [11] P. M. Patil and J. V. Kulkarni, "Rotation and intensity invariant shoeprint matching using Gabor transform with application to forensic science," Pattern Recognition, vol. 42, pp. 1308-1317, 2009. [12] D. G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, vol. 60, pp. 91-110, 2004. [13] M. Brown and D. G. Lowe, " Invariant features from interest point groups.," in Proceedings of British Machine Vision Conference, 2002, pp. 656-665.

Dataset no.1 no.2 no.3 no.4 no.5 no.6 no.7 no.8 no.9 no.10

outsole full-size toe heel left-half right-half full-size toe heel left-half right-half

It is also found that those curves in Figure 5(a) and (b) demonstrate a very slight difference in comparison between original shoeprints and noise shoeprints. This suggests that keypoints scattered within shoeprints can be found by the proposed method and the number of detected keypoints is great enough for keypoint matching. Although those added noises result in some change in the description of some keypoint, most keypoints are not affected by those added noises. IV. CONCLUSIONS This work proposed local invariant features and keypoint matching for recognition and retrieval of shoeprint images. The performance of the proposed system was evaluated in terms of cumulative match score and percentage of database images reviewed. The proposed algorithm demonstrates its robustness of the proposed algorithm against noise. The experimental results also show that the performance of retrieving toe prints and heel prints is better than that of retrieving left-half prints and right-half prints. The difficulty in retrieving left-half prints and right-half prints can be attributed to the incomplete or

5491