Raytheon BBN Technologies

Natural Communication with Avatars Through Speech and Gesture

Recent advances in speech and gesture recognition technologies (e.g., real-time speech recognition with large vocabulary, statistical grammars; low cost 3D time-of-flight cameras) coupled with intelligent agent/behavior modeling and speech synthesis technologies now make it possible to develop virtual role players (VRPs) that are able to understand relatively unconstrained speech and recognize human movement and gesture. These next-generation VRPs will be sufficiently sophisticated for use in team training and simulated urban environments as replacements for human role players.

To demonstrate the potential of these technologies, Raytheon BBN has integrated ongoing work on gesture, speech recognition, natural language understanding, and intelligent agent/behavior modeling technologies into a roadside security checkpoint training scenario running in VBS2™. This video illustrates the ability to control of one's own avatar using body movements and to communicate naturally with VRPs through speech and gesture.