2010 3rd International Congress on Image and Signal Processing (CISP2010)

3D Multi-touch Recognition Based Virtual Interaction

Changbo Wang, Jinqiu Dai
Software Engineering Institute, East China Normal University, Shanghai, 200062
AbstractMulti-touch interaction is a more free interaction means, however, recent related applications limit in the 2D area, such as interaction desktop, shop window. For the applications in virtual exhibition, this paper proposes a 3D multi-touch recognition method. The different gestures firstly are defined for gesture recognition. Then a character recognition algorithm based on directions is presented to achieve the input and erasing gesture of characters. Further, based on a quantitative touch feedback mechanism, the three-dimensional multi-touch interaction algorithm is presented. Finally, we implement the multi-touch interaction display in different conditions, including moving, rotating, zoom for three-dimensional scene. Keywords- Multitouch;3D Interaction; Gesture Recognize; Virtual Exhibition

Long Wang
Shanghai YGTH Imaging Design Co., Ltd Shanghai, 200092 design, including registration and the reusability of gestures. They also give the examples for noting, erasing, duplicating and cutting. Elias et al. [8] proposed a dictionary for multitouch points, which includes a large amount of entities, each responding to a gesture. User can also define his or her own language for each gesture. Moeller et al. [9] proposed a model of gesture recognition. Firstly it captures human operation video set by camera, then transforms the three-dimensional information into two-dimensional information and categorizes and recognizes gesture through learning platform. Currently multi-touch interaction applications mainly focus on the handset, notebooks, digital entertainment products and display desk, etc. However, these applications mainly limit to the small sized equipments, such as interaction desk, interaction window, and the sizes are also very regular. In recent years, the virtual exhibition industry has seen rapid growing, especially the shanghai 2010 expo has further improved the development of related technologies. But we noticed that traditional interaction technology can not satisfy the interaction demands for these isomerous equipments of virtual exhibition. At the same time, current multi-touch recognition interaction technologies have some limitations such as insuffient touch sensibility. So this paper studies further about the recognition, feedback and application technology of multi-touch recognition interaction. And extends the multitouch recognition technology to three dimensions spaces and realizes the three dimensional interaction operation and virtual display. II. MULTI-TOUCH RECOGNITION BASED ON PICTURE

I.

INTRODUCTION

Human-Computer Interaction is a technology studying how to achieve better interaction between human and computer. Unlike traditional single touch technology, multi-touch technology using touch screen or touch pad can receive input from multi-points of the screen spontaneously. Multi-touch technology emphasizes the human-centered interaction, which ensures the complex gestures responding on a computer, and the machine identification of human gestures, and then the feedback, resulting in more convenient and interactive experience. Multi-touch recognition technology first appeared at the University of Toronto. Nimish [1] built a multi-touch screen with finger pressure sensor glass. As a new mean of interaction between human and computer, it allows users to use two or more fingers to touch the screen and operate. Jefferson et al. [2] showed the public new touch screen supporting multi-operation in 2006. Apple and Microsoft Inc. released the iphone and computers respectively, which applied the multi-touch technology [3]. After then, there have been many applications of multi-touch technology. Multi-touch interaction technology includes touch reception, gesture recognition and feedbacks, etc. Currently the hardware platform for touch can be categorized into 4 types: Capacitive based [4]IR based FTIR based and Multi-camera platform [5]. It captures touch points through multi-touch hardware platform, then tracks touch points and records their movement to form and recognize gestures and makes feedbacks. Gesture recognition is the key in the multi-touch interaction. Recently interactive gesture recognition methods include: template matching, feature collection, neural network and Hidden Markov Model. Damaraju et al. [6] proposed two kinds of gestures, parameterized gesture and non-parametered gesture. Wu et al. [7] proposed a set of principles for gesture

A. The recognition process of touch objects Based on the multi-touch device platform, recognizing these touch points is the key. To meet different requirements of different objects, we adopt the recognition pipeline based on pictures. Firstly we collect the overall grayness feature and regional outline information according to the object picture, then collect feature points according to the combined features of appearance and pixels. And then we do automatically match based on the collected information, including selecting opposite samples reasonably and efficiently, and use the opposite samples to train the recognition machine. Finally the construct recognition machine is presented. B. Recognized gestures definition The biggest advantage of multi-touch recognition is that it has realized free interaction based on different gestures. The definition of gestures is the key. The user information is

978-1-4244-6516-3/10/$26.00 2010 IEEE

1478

collected to decide which gesture used in different application environment, and then categorize gesture into the combination of many gestures. Thus the user intent is transformed into a combination of a series of simple gestures with a certain link. Specifically, it is to form the corresponding expectation and feedback action by defining a certain amount of points and the combined relationship between points. First it needs to recognize how many points are there on the touch screen. Then as the touch points move, the position and the distance of positions will be updated. As for the movement trend of different points, we define different gesture templates for different operations in different application demands. We also can complete an interaction task by collaboration. For example, by clockwise rotation of both hands, we can realize the rotation of picture. And we can also realize the zooming of picture by moving one point while another unmoved.

sequence. Finally calculate the average distance between template point and target point to select the best gesture to match. To accelerate the recognition, we select eight directions as the directions for gesture. As shown in Figure 2, categorize the movement of gestures into eight directions. If one angle of the movement does not overlap on the eight directions, we can choose the nearest direction as its direction. Each change in directions can be demonstrated by numbers from one to eight. Based on the eight directions template, we can define characters and number databases. Each input character will be a sequence defined by these eight directions. We call it as Direction character sequence. Based on the majority habit of writing, these Direction character sequences from the composition of the standard directions templates form a template library. For example, the letter B is defined by a sequence of characters as: "260123401234", just as shown in Fig. 3. The same way is as for other characters and numbers.

Figure 2. Gesture-based character recognition processes

Figure 1. Recognized gestures definition (the starting positions of the fingers are shown in red, the blue trails show the path of directions of movement during the gesture).

Just as shown in Figure 1, we can develop 16 different gestures in our current learning and recognition system. For different points numbers, there are four types of gestures each. C. Character recognition based on gestures For the current multi-touch recognition interaction application, it is a common method to adopt multi-touch instead of mouse input. However, unlike keypad input, the input of characters is the bottleneck of multi-touch interaction. Currently there is few research works related to the input recognition based on multi-touch interaction. So we further studied the character recognition based on gesture. The recognition method based on template is often used in PC. However, due to the fact that templates are too fixed and small changes of gesture can affect recognition adequacy in multi-touch interaction. So that type of method can not be directly used in gesture interaction. The obvious feature of gestures in touch interaction is direction-oriented. So here we propose a multi-touch recognition technology based on directional gesture template. We firstly define many directional gestures as picture template database based on the features of human gesture at virtual display and then collect samples from target gesture and recognize target gestures as an orbit in time

When fingers move on touch screen, the direction of movement change at each time, the change of the direction will be stored into a string, which is its direction characters sequence. As shown in Figure 2, "B" character inputted by the touch operation records the sequence: "22661223440123". Then we can compare it with the character sequences in template library. In order to determine which character is matching the best, here we define an edit distance between two strings. Specifically, one string changing into the other one requires the minimum number of editing. Here the editing operations include: replacing a character to another character, inserting a character, deleting a character. It can easily get that the smaller edit distance is the better matching. Figure 2 is the recognition process of the character "B", so that we can easily input characters through touch interaction. Through the comparison of characters sequence, the computing process is so quick to meet with the real-time requirement of virtual interactive display. III. GESTURE-BASED 3D MULTI-TOUCH RECOGNITION
FEEDBACK

A. Quantitative touch feedback mechanism Gesture-based multi-touch recognition feedback often achieves an interactive task through collaborative action. Here the implementation of multi-touch movement is a time-varying process of behaviors, thus require real-time analysis and

1479

processing for the operation of multi-touch process, and inform the upper application system to real-time response. Specifically, we first define the contact interaction of the transition model, constantly in the motion process of recording the trajectories of two points, through the campaign model to deal with a variety of action identification. Current multi-touch touch feedback rules mostly limit to qualitative feedback [10]. A virtual display of the touch interaction often involves the need for quantitative movement distance scaling parameters and other information. Here we propose a quantitative feedback mechanism rules. During users move or rotate, the distance and direction of touch movement are recorded, and can translate them into quantitative control for virtual display. Suppose the distance l is the distance of the finger sliding on the touch screen, we can calculate the movement of virtual objects in 3D display scene:

layer independently associated so that objects associated with the touch points more accurately, to prevent the phenomenon of string points. Relative to the adding operation, there is also deleting operation. When the finger left the touch screen, you need to contact to respond to the destruction operation. So the three dimensional operations including move, rotate, scale, roaming and other operations, can be gained by multi-touch interaction. IV. IMPLEMENTATION AND RESULTS Based on the above method, we have established a multitouch interactive platform for the display. We further implemented a virtual multi-touch display system. For the 3D touch-based interaction, Figure 3 is the rotation of virtual objects in 3D space by the touches interaction of two fingers, where the red points in scene is the positions of touch points. It can be used in Virtual Museum of Science and related applications of virtual display.

L = ( Lx , L y ) = (l cos

W H , l sin ) w h

Where, LX and Ly are the distance that objects move along the direction x and y , id the angle between the direction of

w, h are the length and width of the touch screen, and W , H are the length and width of display
movement and X-axis. scene. At the same time, we also import the inertia into to make a smooth moving processing, when the user finished dragging object fingers on a live touch desktop. It can be calculated by that moment of the instantaneous velocity, acceleration and decrease slowly to 0, which can meet people's habits better. For the rotation operation, we let one point constant, and another touch point on the screen moves some distance Ld. Assuming that the distances before and after movement with another touch point are L1 and L2, respectively. According to the law of cosines, the rotation angle of virtual objects in screen can be obtained. Rotary gesture calculates the rotation angle of the two touch points connected with virtual objects, to get the actual rotation. The scaling gesture also can get the scaling multiple number by the moment the ratio calculated before and after objects. B. Interactive 3D multi-touch walkthrough Based on the above quantitative feedback rules, we can achieve three-dimensional multi-touch interactive display. On a three-dimensional scene, when fingers touch on a multitouch screen, the hardware system will immediately capture the location of the point and feedback to display system. Then the system get the contact coordinates, and calculated from the screen location of the nearest object point, and the ID of the point associated with related objects in order to determine the response of objects, such as a table within the room to rotate, zoom photos on the wall, roaming in one room, etc. As the three-dimensional objects in virtual scene are complex, the touch points associated with objects are complex than the twodimensional plane. Here we adopt a hierarchical 3D objects connected in various independent layers, and then contact each

Figure 3. 3D virtual rotation for multi-touch display effects (red circle is a touch point)

Figure 4 is the 3D roaming in a room based multi-touch interaction. In Fig.4 (a), the small circle is the position of touch points (the same for other figures). Fig.4(b) is for the two scaling the wall of pictures, you can also drag the wall image. Figure 5 is the character input based on touch identification. When entering the number "B", the Direction character sequence is shown in the blue box.

Rotation in the house

1480

[1].

[2].

[3].

[4].

[5]. (b) Scaling the wall of pictures Figure 4. 3D virtual navigation based on touch interaction [6].

[7].

[8]. [9].

[10]. Figure 5. Touch-based character input (blue box sequence of characters that is the direction of a sequence of characters)

WANG De-xin, ZHANG Mao-jun, XIONG Zhi-hui, Survey on multitouch research [J], Application Research of Computers, 2009, 26(7): 2404-2410. Jefferson Y. Han. Low-Cost multi-touch sensing through frustrated total internal reflection [A]. Proc of the 18th Annual ACM Symposium On User Interface Software and Technology[C]. New York. ACM Press, 2005:315. MA Yi-de, DAI Ruo-lan, LI-Lian. Automated image segmentation using pulse coupled neural networks and image's entropy [J]. JOURNAL OF CHINA INSTITUTE OF COMMUNICATIONS , 2002, 23: 46-50. LIU Yu, WANG Yu-mei, ZHANG Lin. An information-driven sensor selection algorithm for target estimation in sensor networks. Journal of Beijing University of Posts and Telecommunication, 2006,29: 62-66. WANG De-xinZHANG Mao-junXIONG Zhi-huiLIU Qing-bao. A Multi-touch System Using Four Cameras[J]. JOURNAL OF COMPUTER AIDED DESIGN&COMPUTER GRAPHICS, 2009, 21:1507-1511. Damaraju, S., Kerne, A. Multitouch Gesture Learning and Recognition System. , Extended Abstracts of IEEE Workshop on Tabletops and Interactive Surfaces, Amsterdam, Netherlands, 2008, 102-104. Wu, M.; Shen, C., Ryall, K., Forlines., C., Balakrishnan, R.Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces. Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems. Tabletop, Adelaide, South Australia., 2006, pp. 185192. Elias J., Westerman G., Wayne C., et al., Multi-touch gesture directory United States20070177803 [P]. 2007. Moeller, J., Kerne, A., Scanning FTIR: Unobtrusive Multi-Touch Sensing through Waveguide Transmissivity Imaging, Proceedings of the 4th International Conference on Tangible, Embedded and Embodied Interaction, Cambridge, MA, 2010, 25-27. REN Ya-xiang, Suvery of human-computer interaction development based on hand posture and gesture recognition[J].Computer Engineering and Design, 2006, 27: 1201-1205.

V.

SUMMARY AND FUTURE WORKS

This paper proposes a multi-touch recognition method for three-dimensional objects. Based on the touch-based gesture recognition template, template based interactive touch input character recognition, and touch feedback mechanism based on quantitative three-dimensional touch interaction, we successfully realize three-dimensional interactive multi-touch for large-size virtual display applications. Future work includes: further definition and implementation of more complex gestures, interactive means of long-range touch and interact based on bank card identification, and further improving the identification of three dimensional multi-touch gestures and collaborative feedback technology, the new touch-based interaction Virtual display roaming system.

ACKNOWLEDGEMENTS This paper was supported by Science and Technology Breakthrough Major Project of Shanghai Municipality of China under Grant No. 09511500100, and Shanghai Rising-Star Program (A type) under Grant No. 08QA1402500. REFERENCES

1481