Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm

Pi19404
April 5, 2013

Contents

Contents
Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm 3
0.1 0.2 0.3 0.4 0.5 0.6 Introduction . . . . . . . . . . . . . . . . . . . . . Introduction to Dynamic Time warping . . . DTW . . . . . . . . . . . . . . . . . . . . . . . . . . Fast DTW . . . . . . . . . . . . . . . . . . . . . . Data Pre Processing . . . . . . . . . . . . . . . Gesture Normalization . . . . . . . . . . . . . . . 0.6.1 Registering candidate . . . . . . . . . . 0.6.2 Rejecting invalid Gesture . . . . . . . . 0.6.3 Re-sampling points . . . . . . . . . . . . . 0.6.4 Scaling . . . . . . . . . . . . . . . . . . . . . 0.6.5 Translation . . . . . . . . . . . . . . . . . . 0.7 Rotational Invariance . . . . . . . . . . . . . . . . 0.8 Rejecting gestures . . . . . . . . . . . . . . . . . 0.9 Robustness to rotation . . . . . . . . . . . . . . 0.10 Computing the Similarity Score . . . . . . . . . 0.10.1 Loading the Template Gestures . . . 0.10.2 Android Gesture Capture Application 0.11 Code . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 4 5 6 6 6 7 8 9 10 11 11 12 12 13 14 15 15

2 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm
0.1 Introduction
In the present article a touch based gesture recognition system is implemented on android platform using Fast Dynamic time warping algorithm.

0.2 Introduction to Dynamic Time warping

Dynamic time warping algorithm aims at aligning two sequences . The element of sequences are typically represented as feature vectors and alignment of two sequences is determined by alignment of feature vectors . Consider a two feature vector sequences R = R1 ; : : : ; RM and T = T1 ; : : : ; TN which characterize two discrete time sequences R and T . The aim is to establish correpondance between two given sequences and output a measure indicating degree of correspondance. One method is euclidean distance measure,that aligns point i of one sequence with point i of another sequence.The euclidean distance between two sequences is distortin measure. The task would be to establish a suitable starting matching point in sequence 2 corresponding to given starting point in sequence 1. For a given starting point in sequence 1 ,we could check for

3 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm all the possible points in sequence 2 and choose the point that provides minimum distortion as optimal alignment. Euclidean distance is sensitive to distortions in time domain like time delay for example two sequences representing same signal at different frequencies. Dynamic time warping can provide a better performance than euclidean allignement method.

0.3 DTW
Dynamic time warping (DTW) is a technique that finds the optimal alignment between two ordered sequences if one sequence may be AIJwarpedAI non-linearly by stretching or shrinking . The duration of two sequences can be different however in case of present case of gesture recognition the gesture sequences are normalized and resampled to have a fixed number of points.The DTW is used to determine the similarity of the two sequences based on the cost of optimal alignment of two gesture sequences. A distortion measure is defined to calculate the distance between i th element of seq 1 and j th element of seq2 represented as d(i; j )
0 0

The task of DTW algorithm is to find a path from (1,1) to (M,N) such that sum of distortion measure over all elements is minimum. Like in euclidean distance measure every possible path between the two sequences can be evaluated. However in this case in is more complex than simple point matching as sequence could be compressed,expanded along with being delayed There are an exponential number of paths from one sequence to next through the warping matrix. DTW is computed by identifying an optimal alignment path between two sequences. The main aim is to establish a ordered correspondance between points in one sequences with the points in another sequences .

4 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm

The problem of alignment can be represented graphically in form on a 2D grid where axis of grid represent the elements of the sequence. To find the best match between these two sequences we can find a path through the grid which minimises the total distance between them. In the given example the first and second elements in sequences x and y match The third element and fourth of sequence x matches with second element of sequence y.This indicates the section of y pattern element appearing in a stretched form in the input. The fourth element of input also matches with the third element of second sequence. This indicates section of y pattern being compressed in the x sequence. Once path/correspondance between two sequences has been found the distortion/distance measure can be computed. The best path/correspondance is the once which provides the least distortion/distance measure. The brute force method would be to compute the distance measure for all the possible path and choose the minimum Since gesture are represented as sequence of points in 2D space,DTW can be used for sequnce allignmnent of candidate and template gesture and select the template that provides the minimum distortion measure.

0.4 Fast DTW

The DTW algorithm has quadratic time and space complexity and thus can be used in online processing for short sequences .However using and approximation of DTW linear space and time complexity can be achieved. Fast DTW uses a multilevel approach that recursively projects a solution from a coarse resolution and refines the projected solution.The FAST DTW algorithm described in the paper Stan and Philip (2007)

5 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm

0.5 Data Pre Processing

The data pre processing steps are the same ones in $1 gesture recognizer described in the earlier article only the method to compute the similarity between the candidate and template has been changed from using euclidean path distance to using FAST DTW algorithm. The pre-processing steps are included below.

0.6 Gesture Normalization

The template and the candidate points may contain different number of sampled points,they may differ in size and spatial location are not the same ie the template and candidate points would not line up. Hence the first step is to perform gesture normalization so that candidate and template points are transformed such that they can be compared this pre processing step is called as gesture normalization. The aim to transform the gesture so that they are invariant to translation and rotation

0.6.1 Registering candidate

The first step is to capture the candidate template.This step is called registering the candidate. As mentioned earlier the number of points captured would depend on the device spatial resolution. The gesture capture processing is defined to be in one of the three states dragged,released,start. The start state indicates that gesture has started and to clear any previous information stored. The dragged state indicates that unistroke is being performed without lifting the figure and the 2D co-ordinates of gesture are being captured.The released state

6 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm indicates that figure has been lifted and gesture capture process has been completed and to start with gesture recognition process. The class AndroidDollar defines android routines to capture the touch gesture performed by the user . The class Data Capture provide high level interface to capture the data and to initiate gesture recognition.AndroidDollar class contains a instance of DataCapture and whose methods are called based on the touch event detected by the user The java class DataVector is defined which captures the 2D coordinate information of drawn gesture. The DataCapture Class contains instance of DataVector. The class PUtils contains all the methods for gesture pre processing and recognition. The DataCapture class contains instance of PUtils class .

0.6.2 Rejecting invalid Gesture

A simple check is incorporated to check whether the gesture was intentional or not by specifying a path length criteria.If the path length of the gesture is less than a specified threshold no further processing is performed and no gesture recognized status will be displayed. The PathLength method defined in the PUtils class simply computes the sum of linear distances between all the adjacent points of the captured/template gesture.

1 2 3 4 5 6 7 8 9

public double PathLength ( Vector points ) { double length = 0; for ( int i = 1; i < points . size () ; i ++) { length += Distance (( Point ) points . elementAt ( i - 1) , ( Point ) points . elementAt ( i ) ) ; } return length ; }

7 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm

In the present implementation the path length threshold used in 100. The code is implemented by the method PathLength is the PUtils Class.

0.6.3 Re-sampling points

Once the gesture has been captured before the process of comparing the candidate gesture with template gesture some pre processing operations are performed .Resampling is one such operation. The re-sampling operations selects from the provided candidate/template gesture a fixed subset of points. This ensures than candidate and template have the same number of points enabling us to perform point based comparison. The method used for sampling the data points is uniform sampling.The path length is divided by the number of re-sampled points.This will be the interval length between the points. We start with the initial point and next point is selected such that distance between points is greater than of equal to interval length.Let points be labeled pt1 and pt2 A linear path is assume to exist between adjacent sample points. using simple trigonometric relationship of sin and cos we can estimate of location at distance of uniform path interval which lies between pt1 and pt2 . This new co-ordinate replaces pt2 in the candidate/template coordinate array and the same process is repeated till the last point of the co-ordinate array is reached.
1 2 3 4 5 6 7 8 9

double d = Distance ( pt1 , pt2 ) ; if (( D + d ) >= I ) { // computation of new co - ordinate using cos relationship double qx = pt1 . x + ( I / ( D + d ) ) * ( pt2 . x - pt1 . x ) ; // computation of new co - ordinate using sin relationship double qy = pt1 . y + ( I / ( D + d ) ) * ( pt2 . y - pt1 . y ) ; Point q = new Point ( qx , qy ) ; // adding the point in resampled array

8 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm
10 11 12 13 14 15 16 17 18 19 20

} else { // computing cumulative distance D=D+d; }

dstPts . addElement ( q ) ; // replacing the point in the source array srcPts . insertElementAt (q , i ) ; // resetting cumulative distance D = 0.0;

This is implemented by the method Re-sample in the PUtils Class. In the present implementation the number of re-sampled points used is 32.

0.6.4 Scaling
The next pre-processing step is to scale the co-ordinates such that third width and height remain within a fixed bounds.First the bounding width and height of current set of points are computed which is simply max(x) min(x) and max(y ) min(y ).The new width and height are denoted by W and H and all the points are scaled by factor W=max(x) min(x) and H=max(y ) min(y )
1 2 3 4 5 6 7 8 9

Rectangle B = BoundingBox ( points ) ; Vector newpoints = new Vector ( points . size () ) ; for ( int i = 0; i < points . size () ; i ++) { Point p = ( Point ) points . elementAt ( i ) ; double qx = p . x * ( size / B . Width ) ; double qy = p . y * ( size1 / B . Height ) ; newpoints . addElement ( new Point ( qx , qy ) ) ; }

This provides step provides invariance wrt scaling since all gestures are bounded to lie within rectangle of same size.This is implements by method ScaleToSquare1 in PUtils class. In the present implementation the scale is done so than bounding box is a square of dimension 250. The above method perform uniform scaling another method of scaling is non uniform scaling that maintains the aspect ratio. Compute the

9 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm ration between the width and height or viceversa of the bounding rectangle if ratio is close to 0 than 1 then perform non uniform scaling else perform uniform scaling. The ScaleDimTo method implements this in the PUtils Class.

0.6.5 Translation
The first step required is computation of mean/centroid of set of co-ordinate location.
1 2 3 4 5 6 7 8

Enumeration e = points . elements () ; while ( e . hasMoreElements () ) { Point p = ( Point ) e . nextElement () ; xsum += p . x ; ysum += p . y ; } return new Point ( xsum / points . size () , ysum / points . size () ) ;

This is implemented by the method Centroid in class PUtils. The next step is to translate all the points such that centroid lies at the origin of co-ordinate system. Translate all the points by (centroidx ; centroidy ).

1 2 3 4 5 6 7 8 9

Point c = Centroid ( points ) ; Vector newpoints = new Vector ( points . size () ) ; for ( int i = 0; i < points . size () ; i ++) { Point p = ( Point ) points . elementAt ( i ) ; double qx = p . x - c . x ; double qy = p . y - c . y ; newpoints . addElement ( new Point ( qx , qy ) ) ; }

This is implemented in the method TranslateToOrigin in the PUtils Class.

10 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm

0.7 Rotational Invariance

If rotational invariance is desired.The line joining the first point to the origin can be rotated so that it lies along 0 degree axis.This will ensure that all template are aligned . The RotateToZero method in PUtils class performs this action.

0.8 Rejecting gestures

After the pre processing step the candidate and template gesture are ready to be compared. However we can reject templates by incorporating crude comparison. compute the normalized vector 1 of gesture length. This between the starting point and the 8 is done by computing difference in x and y co-ordinate of the two points normalized by the distance between them.This vector is called StartUnitVector

1 2 3 4 5

Point i1 =( Point ) points . elementAt ( index ) ; Point i2 =( Point ) points . elementAt (0) ; Point v = new Point ( i1 . x - i2 .x , i1 . y - i2 . y ) ; double len = Math . sqrt ( v . x * v . x + v . y * v . y ) ; return new Point ( v . x / len , v . y / len ) ;

CalcStartUnitVector method of PUtils class computes this angle. To compare startunitvectors of template and candidate cosine of angle between the unit vectors is used as criteria The AngleBetweenUnitVectors method compute difference in angle
1 2

double n = ( v1 . x * v2 . x + v1 . y * v2 . y ) ; return Math . acos ( n ) ; // arc cosine of the vector dot product

In the present implementation the angle similarity threshold is set to 30 deg.

11 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm

0.9 Robustness to rotation

Although above pre-processing steps normalize the gesture wrt scale and translation,every time a gesture is drawn there may be subtle changes like small change in orientation.Hence some robustness to rotation is required to provide a good score. This is performed by rotating the candidate points about line joining the starting point and the centroid by  and + in small increments and computing score at each incremental angle. The best matching score amongst all the angles is selected.

0.10 Computing the Similarity Score

The final step is to compute the similarity score The similarity score is computed using the FAST DTW algorithm. As method earlier a scan of angle withing a specified range is performed. It is assumed that maximum score exits amongst the angle. Instead of performing a uniform search the Golden section search algorithm is used to search for global maximum from list of candidate by by successively narrowing the range of values inside which the maximum score is known to exist. The cost function or score is provided DistanceAtAngle function which computes the distance between the candidate rotated by specified angle and template.The score is compute by computing the euclidean distance between the 2D set of vectors. DistanceAtBestAngle method in PUtils Class compute the best score by applying golden section scan algorithm about range of specified angles. DistanceAtAngle method in PUtils Class performs transformation about the points such that all points are rotated by specified angle before comparison. The Fast DTW code is available at the site fastdtw:Dynamic

12 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm Time Warping (DTW) with a linear time and memory complexity.The code has been modified to construct the time series object from vector of points in TimeSeries class.The remaining code is the same.

1 2 3 4 5 6 7

DistanceFunction distFn = DistanceFunctionFactory . getDistFnByName ( " EuclideanDistance " ) ; Vector newpoints = RotateBy ( points , theta ) ; TimeSeries template = new TimeSeries () ; TimeSeries example = new TimeSeries () ; TimeWarpInfo info = FastDTW . getWarpInfoBetween ( template , example ,20 , distFn ) ; info = DTW . getWarpInfoBetween ( template , example , distFn ) ; double d = info . getDistance () ;

This code is present in the DistanceAtAngle function of PUtils Class. The Recognize method in PUtils class provides interface to gesture recognition routine.

0.10.1 Loading the Template Gestures

The template gesture were captured using desktop application and co-ordinates are stored in a csv files .A directory is create for each class of template and multiple files can be placed withing each directory.For each files a template object will be created. In the CSV files raw x and y co-ordinates of gesture trajectory of mouse pointer captured by desktop application are written side by side and this entire CSV string is loaded into a integer array.Hence while loading the template nth location is assigned to x co-ordinate while (n + 1)th location of integer array is assigned to y co-ordinate of 2D co-ordinate object representing Point Class. The templates are loaded once during initialization. Comparison is made with all the loaded templates and best score is selected amongst the templates. Below are examples of templates demonstrated in the present application.The template consist of following shapes made in clockwise and anti clockwise manner.The templates are labeled as (1,11) ,(2,12) ,(3,15) (9,13),(10,14) for clockwise and anticlockwise respec-

13 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm tively. An example of data contained in CSV files is shown below

75,154,81,150,85,146,90,142,96,139,101,135,105,130,110,128,114,124, 119,120,122,117,125,113,128,108,130,103,132,96,133,93,133,89,133,85, 133,81,133,75,133,72,133,69,133,65,131,62,128,60,125,59,122,57,119,57, 116,57,113,56,110,56,108,56,107,57,105,62,104,67,103,71,103,75,103,78,103, 81,103,86,103,90,103,93,103,97,104,101,105,104,107,107,109,111,111,116, 113,119,115,123,119,127,122,130,126,133,129,136,133,139,136, 142,140,147,143,150,147,153,150,155,152,157,155,158 This contains x and y co-ordinates stored in adjacent locations.

0.10.2 Android Gesture Capture Application

The android gesture capture exampled code is based on the TouchPaint example provided in the android samples directory under graphics sub-directory.Few modifications we made to gesture capture interface. The AndroidGesture capture utility will contain a instance of DataCapture which provides high level interface to gesture registering and recognizing methods.

14 | 16

Android :Single Stroke Gesture Recognition using Fast Dynamic Time Warping algorithm When the gesture is desired to be captured On method of DataCapture class is called with the first boolean parameter set to true and second,third parameters are x and y co-ordinates respectively. When the gesture is completed the On method of DataCapture Class is called with first boolean parameter set to false and second and third parameters are the final x and y co-ordinates respectively.

0.11 Code
The code can be found in code repository https://github.com/ pi19404/m19404/tree/master/Android/AndroidGesture1.2 or https://code. google.com/p/m19404/source/browse/Android/AndroidGesture1.2/. The header files is located in jni directory.The library files are not placed in the repository download them from appropriate packages on send a mail separately for download link.

15 | 16

Bibliography

Bibliography
[1] Lisa Anthony and Jacob O. Wobbrock.  A lightweight multistroke recognizer for user interface prototypes. In: Proceedings of Graphics Interface 2010. GI '10. Ottawa, Ontario, Canada: Canadian Information Processing Society, 2010, pp. 245 252.

1839214.1839258.
[2]

isbn: 978-1-56881-712-5. url: http://dl.acm.org/citation.cfm?id=

http://web.science.mq.edu.au/~cassidy/ comp449/html/ch11s02.html.
Dynamic Time Warping. at fastdtw:Dynamic Time Warping (DTW) with a linear time and memory complexity.

[3]

url: http://code.google.com/p/fastdtw/.

[4] [5]

Ricardo Gutierrez-Osuna.  Introduction to Speech Processing. In: CSE@TAMU. Salvador Stan and Chan Philip.  Toward accurate dynamic time warping in linear time and space. In: vol. 11. 5. Amsterdam, The Netherlands, The Netherlands: IOS Press, Oct. 2007, pp. 561580.

id=1367985.1367993.

url: http://dl.acm.org/citation.cfm?

16 | 16