GSI DEMO: Multiuser Gesture / Speech Interaction over Digital Tables by Wrapping Single User Applications
Date
2006-05-18
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Most commercial software applications are designed for a single user
using a keyboard/mouse over an upright monitor. Our interest is exploiting
these systems so they work over a digital table. Mirroring what people do when
working over traditional tables, we want multiple people to interact with the
tabletop application and with each other via rich speech and hand gestures.
In previous papers, we illustrated multi-user gesture and speech interaction
on a digital table for geospatial applications Google Earth, Warcraft III
and The Sims. In this paper, we describe our underlying architecture: GSI
DEMO. First, GSI DEMO creates a run-time wrapper around existing single user
applications: it accepts and translates speech and gestures from multiple
people into a single stream of keyboard and mouse inputs recognized by the
application. Second, it lets people use multimodal demonstration instead of
programming to quickly map their own speech and gestures to these
keyboard/mouse inputs. For example, continuous gestures are trained by saying
Computer, when I do [one finger gesture], you do [mouse drag] . Similarly,
discrete speech commands can be trained by saying Computer, when I say
[layer bars], you do [keyboard and mouse macro] . The end result is that end
users can rapidly transform single user commercial applications into a
multi-user, multimodal digital tabletop system.
Description
Keywords
Computer Science