Multimodal Speech-Gesture Interface for Handfree Painting on a Virtual

Paper Using Partial Recurrent Neural Networks as Gesture Recognizer

Andrea Corradini, Philip R. Cohen
Oregon Graduate Institute for Science and Technology
Center for Human-Computer Communication
20000 N.W. Walker Rd, 97006 Beaverton, OR
andrea@cse.ogi.edu

-
ABSTRACT We describe a pointing and speech alternative
to the current paint programs based on traditional devices like
mouse, pen or keyboard.
We used a simple magnetic field tracker-based pointing system
as input device for a painting system to provide a convenient
means for the user to specify paint locations on any virtual
paper. The virtual paper itself is determined by the operator
as a limited plane surface in the three dimensional space.
Drawing occurs with natural human pointing by using the
hand to define a line in space, and considering its possible
intersection point with this plane. The recognition of pointing
gestures occurs by means of a partial recurrent artificial
neural network. Gestures along with several vocal commands
are utilized to act on the current painting in conformity with a
predefined grammar.
Keywords
User-centered Interface, Painting Tool, Pointing Gesture, Speech
Recognition, Communication Agent, Multimodal System,
Augmented and Virtual reality, Partial Recurrent Artificial Neural
Network.
1. INTRODUCTION
The natural combination of a variety of modalities such as
speech, gesture, gaze, and facial expression makes human-
human communication easy, flexible and powerful.
Similarly, when interacting with computer systems, people
seem to prefer a combination of several modes to a single
one alone [12, 191. Despite the strong efforts and deep
investigations in the last decade, human-computer
interaction (HCI) is in its childhood and therefore its
ultimate goal, aiming at building natural perceptual user
interfaces, remains a challenging problem.
Two concurrent factors produce awkwardness. First, current
HCI systems make use of both rigid rules and syntax over
the individual modalities involved in the dialogue. Second,
speech and gesture recognition, gaze tracking, and other
channels are isolated because we do not understand how to
integrate them to maximize their joint benefit [20,2 1,25,30].
While the first issue is intrinsically dificult (everyone
claims to know what a gesture is, but nobody can tell you
precisely), progress is being made in combining different
modalities into a unified system. Such a multimodal system,
allowing interactions that more resemble everyday
communication, becomes more attractive to users.
1.1 Related Work
Like speech, gestures vary both from instance to instance
for a given human being and among individuals. Beside this
temporal variability, gestures vary even spatially making
them more difficult to deal with. For the recognition of
those single modalities, only few systems make use of
connectionist models [3,7,17,27] for they are not
considered well suited to completely address the problem of
time alignment and segmentation. However, some neural
architecture [ 10,14,29] has been put forward and
successfully exploited to partially solve problems involving
the generation, learning or recognition of sequence of
patterns.
Recently, several research groups have more thoroughly
addressed the issue of combining verbal and nonverbal
behavior. In this context, most of such multimodal systems
have been quite successful in combining speech and gesture
[4,6,26,28] but, to our knowledge, none exploits artificial
neural networks.
One of the first such systems is Put-That-There [4] which
uses speech recognition and allows for simple deictic
reference to visible entities. A text editor featuring a multi-
modal interface that allows users to manipulate text using a
combination of speech and pen-based gestures has been
presented in [28]. Quickset [6] along with a novel
integration strategy offers mutual compensation between
pen and voice modalities.
Among gestures, pointing is a compelling input modality
that has led to friendlier interfaces (such as the mouse-
enabled GUI) in the past. Unfortunately, few 3D systems
that integrate speech and deictic gesture have been built to
detect when a person is pointing without special hardware
support and to provide the necessary information to
determine the direction of pointing. Most of those systems
have been implemented by applying computer vision
techniques to observe and track finger and hand motion.
The hand gesture-based pointing interface detailed in [24]
was proposed to track the position of the fingertip in which
the user points map directly into the 2D cursor movement
on the screen. Fukumoto et a1 [ l l ] report a glove-free
camera based system providing pointing input for
applications requiring computer control from a distance
(such as a slide presentation aid). Further stereo-camera

0-7803-7278-6/02/$10.00 Q2002 IEEE 2293

techniques for detection of real time pointing gestures and
estimation of direction of pointing have been exploited in
[S, 8, 131. More recently, [26] describes a bimodal
speechlgesture interface, which is integrated in a 3D visual
environment for computing in molecular biology. The
interface lets researchers interact with 3D graphical objects
in a virtual environment using spoken words and simple
hand gesture.
In our system, we make use of the Flock of Birds (FOB)
[ 11, a six-degree-of-freedom tracker device based on
magnetic fields to estimate the pointing direction. In an
initialization phase, the user is required to set the target
coordinates in the 3D space that bound his painting region.
With natural human pointing behavior, the hand is used to
define a line in space, roughly passing through the base and
the tip of the index finger. This line does not usually lie in
the target plane, but may intersect it at some point.
We recognize pointing gestures by means of a hybrid
partial recurrent artificial neural network (RNN) consisting
of a Jordan network [ 141 and a static network with buffered
input to handle the temporal structure of the movement
underlying the gesture. Concurrently, several speech
commands can be issued asynchronously. They are
recognized using Dragon 4.0, a commercial speech engine.
Speech along with gestures is then used to put the system
into various modes to affect the appearance of the current
painting. Depending on the spoken command, we solve for
the intersection point and use it either to directly render ink
or to draw a graphical object (e.g. circle, rectangle, or line)
at this position in the plane. Since we implemented the
speech and tracking modules are on different machines, we
employed our agent architecture to allow the different
modules to exchange messages and information.
2. DEICTIC GESTURES
As for HCI, currently no comprehensive classification of
natural gestures exists that would help in establishing a
methodology for gesture understanding. However, there is a
general agreement in defining the class of deictic or
pointing gestures [9,15,18]. The term deictic is used in
reference to gestures or words that draw attention to a
physical point or area in course of a conversation.
Among natural human gestures, pointing gestures are the
easiest to identify and interpret. There are three body parts
which can conventionally be used to point: the hands, the
head, and the eyes. Here we are concerned only with
manual pointing.
In the western society, there are two distinct forms of
manual pointing which regularly CO-occur with deictic
words (like “this”, “that”, “those” etc.): the one-finger-
pointing to identify a single or a group of objects, a place or
a direction, and the flat-hand-pointing to describe paths or
spatial evolutions of roads, range of hills. Some researchers
[23] argue that pointing has iconic properties and represents
0-7803-7278-6/02/$10.00 02002 IEEE
a prelinguistic and visually perceivable event. In fact, in
face-to-face communication deictic speech never occurs
without an accompanying pointing gesture. In a shared
visual context, any verbal deictic expression like “there” is
unspecified without a parallel pointing gesture’. These
multiple modes may seem redundant until we consider a
pointing gesture as a complement to speech, which helps
form semantic units. We can easily realize that by speaking
with children. They compensate for their limited vocabulary
by pointing more, probably because they cannot convey so
much information about an objecdlocation by speaking as
they could by directing their interlocutor to perceive it with
his own eyes.
2.1 An Empirical Study
As described in the previous section, pointing is an
intentional behavior that aims at directing the listener’s
visual attention to either an object or a direction. It is
controlled both by the pointer’s eyes and by muscular sense
(proprioception).
In the real world, our pointing actions are not coupled with
“cursors”, yet our interlocutors can often discern the
intended referents, processing the pointing action and the
deictic language together.
We conducted an empirical experiment to investigate how
precise pointing is when no visual feedback is available.
We invited four subjects to point at a target spot on the wall
using a laser pointer. They did this task from six different
distances away from the wall (equally distributed from 0.5m
to 3m), ten times for each. Each time the subject attempted
to point at the target with the beam turned off. Once a
subject was convinced that he had directed a laser pointer
toward the bulls-eye, we turned on the laser and determined
the point the user was really aiming at. We determined the
distance between the bulls-eye the user was aiming for and
the actual area he indicated with the laser pointer. We then
computed overall error for each distance as the average
distance between desired and actual points on the wall over
all the trials for that distance from the wall.
The subjects were requested to perform this experiment
twice and in two different ways: in a “natural” way and in
an “improved” way. As natural way we asked the persons
involved in this experiment to naturally point at the target,
while in the improved way we specifically asked the person
to try to achieve the best result (some people put the laser
pointer right in front of the eye and closed the other, other
put it right in front of the nose, etc.). The outcome of the
experiment is shown in Figure 1.
1 This dos not happen in smlences that are used far referencing places, objects or w m t ~the interlocutors
have clear their mink because of the dialogue context. E.g.,in “Have you b m l o Italy? YS, I have
b m there twice” or “I watched Nuovo Cincma Paradiso at the TV yesterday. DiB‘t that film win an
Oscar in 1989?”, dcictic w o r k are UOI accompanied by pointing gstures. Neither they are in sentnrs
like e.g. “There shall come a time”, or “They all know that Lam is cute”, or “The house thot she built is
huge”, where they arc used as conjunctionor pronoun.
2294
As expected, we can see how with increasing distance the
error increases as well. In addition, when the user pointed at
the given spot from a distance of 1 meter the error
decreased from 9.08 to 3.89 centimeters from the “natural”
to the “improved” way.
Pointing Inaccuracy
0.5 I 1.5 2 2.5
distance from wall (m)
I+improved pointing +natural pointing
Figure 1: target pointing precision.
In light of this experiment, reference resolution of deictic
gestures without verbal language is an issue. In particular,
when small objects are placed close together, reference
resolution via deictic gesture can be impossible without the
help of spoken specification. In addition, direct mapping
between the 3D user input (the hand movement) and user
intention (pointing on the target plane) can be carefully
performed only with visual feedback (information on
current position). In the next session, we describe the
system that has been built according to these
considerations.
3. THE PAINTING SYSTEM
3.1 Estimating the pointing direction
For the whole system to work, the user is required to wear a
hand glove on whose top we put one FOB’s sensor. The
FOB is a six-degree-of-freedom tracker device based on
magnetic fields which we exploit to track the position and
orientation of the user’s hand with respect to the coordinate
system determined by the FOB’s transmitter The hand’s
position is given by the position vector reported by the
sensor at a frequency of approximately 50Hz. For the
orientation, we put the sensor almost at the back of the
index finger with its relative x-coordinate axis directed
toward the index fingertip. In this way, using the quatemion
values reported by the sensor, we can apply mathematical
transformations within quatemion algebra to determine the
unit vector X which unambiguously defines the direction of
the sensor and therefore that of pointing (Figure 2).
The point along with vector X is then used to determine
the equation of the imaginary line passing through and
having direction X.When the system is started for the first
time, the user has to choose the region he wants to paint in.
This is accomplished by letting the user choose three of the
0-7803-7278-6/02/$10.00 02002 IEEE
vertices of the future rectangular painting region. These
points are chosen by pointing at them. However, since this
procedure is to be done in the 3D space, the user has to aim
at each of the vertices from two different positions. The two
different vectors triangulate to select a point as vertex. In
3D space, two lines will generally not have an intersection.
In such cases, we will use the point of minimum distance
from both lines.
With natural human pointing behavior, the hand is used to
define a line in space, roughly passing through the base and
the tip of the index finger. Normally, this line does not lie in
the target plane but may intersect it at some point. It is this
point that we aim to recover.
For this reason, when the region selected in the 3D space is
3
neither a wall screen, nor a general surface on which the
input can be directly output (tablet, the computer’s monitor
I etc.), the system can be properly used only when the
magnetic sensor is aligned and used together with a light
pointer. However, in this situation we also implemented a
rendering module to draw the actual painting on the screen
regardless of the target plane chosen in the 3D space.
Figure 2: selecting a graphic tablet as target region for painting
enables directly visual feedback. The frame of reference of the sensor
is shown on the left. On the right an exemplarity painting is shown as
it appears on the tablet.
3.2 Motion Detection for Segmentation
In order for us to describe in detail the motion detector, we
first need to give some definitions.
We consider the FOB data stream static anytime the sensor
attached to the user’s hand remains stationary for at least
five consecutive FOB’S reports. In this case, we also refer
to the user as to be in the resting position. In a similar way,
we say the user is moving and we consider the data stream
dynamic whenever the incoming reports change in their
spatial location for at least five times in a row.
Static and dynamic data stream are defined in such a way
that they are mutually exclusive, but not exhaustive. In
other words, if one definition is satisfied, that implies that
the other is not. However, the converse situation is not the
case, since if one definition is not satisfied, this does not
imply that the other is. Such non-complementarity makes
the motion detector module robust against noisy data.
For real-time performance purposes, the FOB’s data are
currently downsampled to lOHz so that, both static and
2295
dynamic conditions need be hlfilled over a time range of addition, we fiunish the input layer with a time window to
approximately half a second. accommodate a part of the input.
The motion detector is in charge of reporting to the gesture
recognizer anytime transition from (into) dynamic into
(from) static occurs. If the transition is from static into
dynamic, the motion detector forwards the input stream to
the RNN for the classification to start. The RNN is queried
for the classification result if the following conditions hold
simultaneously:
a) the opposite transition from dynamic into static is
detected
b) the imaginary line described by the sensor in the
space intersects the target region (the virtual paper)
c) the time elapsed between the begin of the Figure 3: Example of R"for Gesture Recognition.
classification and the current time is less than a given The resulting network (Figure 3) is a hybrid combination of
threshold chosen according to the maximum duration a Jordan network [14] and a static network with buffered
of the gestures used for the €2" training input. The input layer is divided into two parts: the context
The motion detector provides the recognizer with explicit unit and the buffered input units, which currently contains 5
start and end points for classification. Therefore, the RNN time steps (i.e. half a second). The context unit holds a copy
needs not segmentation to identify these characteristics of of the output layer and from itself at the previous time step.
the movement. With the use of such a motion detector The recurrence's weight from the output layer is kept fixed
gestures need not to start with the hand in a given user- at 1. Indicating with p the strength of the self connection,
defined position, either. the updating rule at time instant t for the context unit C(t)
can be expressed in mathematical form as h c t i o n of the
3.3 Gesture Recognition by means of R" network's output O(t) as:
We use artificial neural networks to capture the spatially
nonlinear local dependencies and handle the temporal
structure of a gesture.
The most direct way to perform sequence recognition by
C(t)= C(t - 1)p + O(t - 1) = x:-*
r=O (L"i))
static artificial neural networks is to turn the largest It turns out that the context unit accumulates the past output
possible part of the temporal pattern sequence into an input values of the network in a manner depending on the choice
buffer on the input layer of a network. In such a network a of the parameter p i n the range [O,l]. Close to one, it
part of the input sequence is presented simultaneouslyto the
extends the memory further back into the past but causes
network by feeding the signal into the input buffer and next
loss of sensitivity to detail. This recurrence permits the
shifting it at various time intervals. The buffer must be
network to remember some aspects of the most recent past
chosen in advance and has to be large enough to both
giving the network a kind of short-term memory. At a given
contain the longest possible sequence and to maintain as
time, the network output depends as well on the current
much as possible context-dependent information but a large
input as on an aggregate of past values.
buffer means a large number of parameters and implicitly a
As input values for the RNN, we use the signs of the
large quantity of required training data for successhl
differences between the spatial components of the current
learning and generalization.
and previous sensor location. Thus, the input is three-
Partial recurrent artificial neural networks (RNNs) are a
dimensional, translation invariant, and its components can
compromise between the simplicity of these feedforward
assume only one value among { - 1,O, 1} .
nets and the complexity of recurrent models. RNNs are
During the training, the desired output is always associated
currently the most successhl architectures for sequence
with the center of the buffered input window. As target, we
recognition and reproduction by connectionist methods.
take the sample following that center within the training
They are feedforward models with an additional set of fixed
sequence. An additional output neuron is always presented
feedback connections to encode the information from the
the value 1 when the sequence is a deictic gesture, the value
most recent past. That recurrence does not complicate the
0 otherwise. Preclassified patterns are not necessary. The
training algorithm since its value is fixed and hence not
input vectors constituting the sequence are stepped through
trainable.
the context window time step by time step. The weight
We deploy a partial recurrent network in which the most
adjustment is carried out by backpropagation with least
recent output patterns, which intrinsically depend on the
mean square error function. Both hidden and output
input sequence, are fed back into the input layer. In

0-7803-7278-6/02/$10.00 02002 IEEE 2296

neurons compute the sigmoid activation function. The
additional output neuron is checked anytime a classification
result is required.
We tested the recognizer on four sequences, two of a person
performing pointing gestures toward a given virtual paper,
and two of the same person gesticulating during a
monologue without deictic gestures. Each sequence lasts 10
minutes and is sampled at 10Hz. One sequence for each
class was used for training and testing. The recognition rate
was up to 89% for pointing gestures, and up to 76% for
non-pointing gestures. While only a few sequences from the
deictic gesture data set were misrecognized (false negative),
much more movements from the non-pointing gesture data
set were misrecognized as pointing gesture (false positive).
This is not surprising since long lasting gestures, which
occur frequently during a monologue/conversation, are very
likely to contain segment patterns that are very similar to
deictic gestures. Due to the nature of the training data, the
performed test looks only at the boundary conditions (false
positivehegative). We plan to collect and transcribe data
from users during a conversational event where both deictic
and non-deictic gestures occur. Testing the system with this
more natural data will permit to assess more precisely the
performance of the recognizer.
3.4 The Speech Agent
We make use of Dragon 4.0, a Microsoft SAP1 4.0
compliant speech engine. This speech recognition engine
captures an audio stream and produces a list of text
interpretations (with association probabilities of correct
recognition) of that speech audio. These text interpretations
are limited by a grammar that is supplied to the speech
engine upon startup.
The following grammar specifies the possible self-
explanatory sentences:
1: <Sentence> = <answer> I <color> I <double> I <single>
2: <answer> = no / yes
3: <color> = green / red / blue /yellow / white /
magenta / cyan
4: <double> = draw on / draw off /zoom in / zoom out /
cursor on /cursor off/line begin /paste /
select end / select begin / line end / copy /
circle end / circle begin / rectangle end /
rectangle begin
5 : <single> > = exit / he& / undo / switch to foreground /
save /free buffer / switch to background /
send to background / cancel / restart 1
delete /load
Here, <single> and <double> refers to the sets of
commands which need to be issued without and with an
accompanying pointing gesture, respectively.
0-7803-7278-6/02/%10.00 c92002 IEEE
The user uses voice commands to put the system into
various modes that remain in effect until he changes them.
Speech commands can be entered at anytime and are
recognized in continuous mode.
3.5 The Fusion Agent
The Fusion Agent is a finite state automaton that is in
charge for two major functions, i.e., the rendering, and the
temporal fusion of speech and gesture information.
This rendering is implemented with OpenGL on a SGI
machine utilizing the Virtual Reality Peripheral Network
(VRF”) [2] driver for the FOB.
The fusion bases on a time-out variable. Once a pointing
gesture is recognized, a valid spoken command must be
entered within a given time (currently 4 seconds, as usually
speech follows gestures [22]) or another pointing gesture
must occur. Eventually, the Fusion Agent either takes the
action (such as changing drawing color, select the first point
of a line, etc.) associated with the speech command or
issues an acoustic warning signal.
The model nature of the state machine ensures consistent
command sequences (e.g., “line begin” can only be
followed by “undo”, “cancel” or “line end”). Depending on
the performed action, the system may undergo a state
change.
3.6 Agent Architecture
The modules implemented for tracking, pointing and
painting, and speech command recognition, need to
communicate with each other. Agents communicate by
passing Prolog-type
- . ASCII strings (Horn clauses) via
TCP/IP.
T
Audio
Figure 4: agent communication within the entire system.
The central agent is the facilitator. Agents can inform the
facilitator of their interest in messages which match
(logically unify) with a certain expression. Thereafter, when
the facilitator receives a matching message from some other
agent, it will pass it along to the interested agent. Since
ASCII strings and TCP/IP are common across various
2297
 platforms, agents can be used as software components that
can communicate across platforms.
In this case, the Speech Agent is running on a Windows
platform. The best off-the-shelf speech recognition engines
available to us (currently, Dragon) are on the Windows
platform. On the other hand, the Flock of Birds and the
VRPN server are set up for Unix. Therefore, it makes sense
to tie them together with the agent architecture (Figure 4).
Communication is straightforward. The Speech Agent
produces messages of the type parse-speech(Message)
which the facilitator forwards to the Fusion Agent. This
ladder, with some simple parsing, can then extract speech
recognition alternate interpretations and their associated
probabilities from the message strings. The command
associated with the highest probability value above an
experimental threshold (currently 0.85) is chosen.
4. Conclusions and Future Work
The presented system represents a real-time application of
drawing in space on a two-dimensional limited rectangular
surface. This is a first step toward a 3D multimodal speech
and gesture system for computer aided design and
cooperative tasks. A system might perhaps recognize from
the user’s input some 3D objects from an iconic library and
refine the user’s drawings accordingly. We anticipate
expanding the use of speech to operate with 3D objects.
Since the h i o n component is an agent, we are going to
’ make it a module in the entire QuickSet Adaptive Agent
Architecture [ 161, to further use it as a sort of virtual mouse
for the QuickSet [6] user interface. Possible alternative
applications for this system range from hand cursor control
by pointing to target selection in virtual environments.
5. ACKNOWLEDGMENTS
This research is supported by the Office of Naval Research, Grant
N00014-99-1-0377 and N00014-99-1-0380. Thanks to Rachel Coulston
for help editing and Richard M. Wesson for programming support.
6. REFERENCES
[ 11 httu://www.ascension-tech.com
[2] Taylor R.M., VRPN: A Device-Independent, Network-Transparent
VR Peripheral System, Proceedings of the ACM Symposium on
Virtual Reality Software and Technology, 2001.
[3] Boehm K., Broll W., Sokolewicz M., Dynamic Gesture Recognition
using Neural Networks; A Fundament for Advanced Interaction
Construction, SPIE Conf. Elect. Imaging Science & Tech., 1994.
[4] Bolt R.A. Put-That-There: voice and gesture at the graphics
interface. Computer Graphics, Vol. 14, No. 3, 1980,262-270.
[5] Cipolla R., Hadfield P.A., Hollinghurst, N.J., Uncalibrated Stereo
Vision with Pointing for a Man-Machine Interface, in Proc. of the
IAPR Workshop on Machine Vision Application, 163-166, 1994.
[6] Cohen P.R., et al., Quickset: Multimodal interactions for distributed
applications. Proc. of the 5th Int’l Multimedia Conf., 31-40, 1997.
[7] Corradini A., Gross H.-M., Camera-based Gesture Recognition for
Robot Control, Proceedings of the IEEE-INNS-ENNS International
Joint Conference on Neural Network, vol. IV, 133-138,2000,
0-7803-7278-6/02/$10.00 02002 IEEE
[8] Crowley J.L., Berard F., and Coutaz J., Finger Tracking as an Input
Device for Augmented Reality, in Proc. of the Int’l Workshop on
Automatic Face and Gesture Recognition, 195-200, 1995.
[9] Efron D., Gesture, Race and Culture, Mouton and Co., 1972.
[lo]Elman J.L., Finding Structure in Time, Cog. Sci., 14:179-211, 1990
[11] Fukumoto M., Mase K., and Suenaga Y., Realtime detection of
pointing actions for a glove-free interface, in Proceedings of IAPR
Workshop on Machine Vision Applications, 473-476, 1992.
[12]Hauptmann A.G., and McAvinney P., Gesture with speech for
graphics manipulation. International Journal of Man-Machine
Studies, Vol. 38,231-249, February 1993.
[13]Jojic N., et al., Detection and Estimation of Pointing Gestures in
Dense Disparity Maps, in Proceedings of International Conference
on Automatic Face and Gesture Recognition, 468474,2000,
[141Jordan M., Serial Order: A Parallel Distributed Processing Approach
Advances in Connectionist Theory, Lawrence Erlbaum, 1989.
[151Kendon A., The Biological Fundations of Gestures: Motor and
Semiotic Aspects, Lawrence Erlbaum Associates, 1986.
[16]Kumar S., Cohen P.R., Levesque, H.J., The Adaptive Agent
Architecture: Achieving Fault-Tolerance Using Persistent Broker
Teams, Proc. 4th Int’l Conf. Multi-Agent Systems, 159-166,2000.
[17]Lippmann R.P., Review of Neural Networks for Speech
Recognition, Neural Computation, 1:1--38, 1989.
[ 181McNeill D., Hand and Mind: what gestures reveal about thought, the
University of Chicago Press, 1992.
[191Oviatt S.L., Multimodal interfaces for dynamic interactive maps, in
Proceedings of Conference on Human Factors in Computing
Systems: CHI, 95-102, 1996.
[20] Oviatt S.L., Cohen, P.R., Multimodal interfaces that process what
comes naturally. Communication of the ACM, 43(3):45-53,2000,
[2 13Oviatt S.L., et al., Designing the user interface for multimodal
speech and gesture applications: State-of-the-art systems and
research directions, Human Comp. Interaction, 15(4):263-322,2000,
[22] Oviatt S., De Angeli A., Kuhn K., Integration and Synchronization
of Input Modes during Multimodal HCI, Proceedings of CHI ‘97,
415-422, 1997.
[23]Place U.T. The Role of the Hand in the Evolution of Language.
Psycoloquy, Vol. 11, No. 7,2000, httn://www.coasci.soton.ac.uk
[24]Quek F., Mysliwiec T.A., Zhao M., FingerMOuse: A Freehand
Computer Pointing Interface in Proc. of Int’l Conf. on Automatic
Face and Gesture Recognition, 372-377, 1995.
[25] Queck F., et al., Gesture and Speech Multimodal Conversational
Interaction, Tech. Rep., VISLab-01-01, University of Illinois, 2001.
[26] Sharma R., et al., SpeecWGesture Interface to Visual-computing
Environment, IEEE Comp. Graphics and Appl., 20(2):29-37,2000,
[27] Tank D.W., Hopfield J.J., Concentrating Information in Time:
Analog Neural Networks with Applications to Speech Recognition,
Proc. of the 1st Int’l Conf. on Neural Nets, Vol. IV, 455468, 1987.
[28] Vo M.T., Waibel A.A., Multimodal human-computer interface:
combination of gesture and speech recognition, InterCHI, 1993.
[29] Waibel A., et al., Phoneme Recognition Using Time-Delay Neural
Networks, IEEE Transactions on Acoustics, Speech, and Signal
Processing, 37( 12): 1888-1898, 1989.
[30] Wu L., Oviatt S., Cohen P.R., Multimodal Integration - A Statistical
View, IEEE Transactions on Multimedia, 1(4):334-34 1, 2000
2298