GSI DEMO: Multiuser Gesture / Speech Interaction over Digital Tables by Wrapping Single User Applications

Date
2006-05-18
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Most commercial software applications are designed for a single user using a keyboard/mouse over an upright monitor. Our interest is exploiting these systems so they work over a digital table. Mirroring what people do when working over traditional tables, we want multiple people to interact with the tabletop application and with each other via rich speech and hand gestures. In previous papers, we illustrated multi-user gesture and speech interaction on a digital table for geospatial applications Google Earth, Warcraft III and The Sims. In this paper, we describe our underlying architecture: GSI DEMO. First, GSI DEMO creates a run-time wrapper around existing single user applications: it accepts and translates speech and gestures from multiple people into a single stream of keyboard and mouse inputs recognized by the application. Second, it lets people use multimodal demonstration instead of programming to quickly map their own speech and gestures to these keyboard/mouse inputs. For example, continuous gestures are trained by saying Computer, when I do [one finger gesture], you do [mouse drag] . Similarly, discrete speech commands can be trained by saying Computer, when I say [layer bars], you do [keyboard and mouse macro] . The end result is that end users can rapidly transform single user commercial applications into a multi-user, multimodal digital tabletop system.
Description
Keywords
Computer Science
Citation