Dialogue Systems Go Multimodal: The SmartKom Experience -- Facts and Figures About the SmartKom Project -- An Exemplary Interaction with SmartKom -- Multimodal Input Analysis -- The SmartKom Architecture: A Framework for Multimodal Dialogue Systems -- Modeling Domain Knowledge: Know-How and Know-What -- Speech Recognition -- Class-Based Language Model Adaptation -- The Dynamic Lexicon -- The Prosody Module -- The Sense of Vision: Gestures and Real Objects -- The Facial Expression Module -- Multiple Biometrics -- Natural Language Understanding -- The Gesture Interpretation Module -- Multimodal Dialogue Processing -- Modality Fusion -- Discourse Modeling -- Overlay: The Basic Operation for Discourse Processing -- In Context: Integrating Domain- and Situation-Specific Knowledge -- Intention Recognition -- Plan-Based Dialogue Management for Multiple Cooperating Applications -- Emotion Analysis and Emotion-Handling Subdialogues -- Problematic, Indirect, Affective, and Other Nonstandard Input Processing -- Multimodal Output Generation -- Realizing Complex User Wishes with a Function Planning Module -- Intelligent Integration of External Data and Services into SmartKom -- Multimodal Fission and Media Design -- Natural Language Generation with Fully Specified Templates -- Multimodal Speech Synthesis -- Scenarios and Applications -- Building Multimodal Dialogue Applications: System Integration in SmartKom -- SmartKom-English: From Robust Recognition to Felicitous Interaction -- SmartKom-Public -- SmartKom-Home: The Interface to Home Entertainment -- SmartKom-Mobile: Intelligent Interaction with a Mobile System -- SmartKom-Mobile Car: User Interaction with Mobile Services in a Car Environment -- Data Collection and Evaluation -- Wizard-of-Oz Recordings -- Annotation of Multimodal Data -- Multimodal Emogram, Data Collection and Presentation -- Empirical Studies for Intuitive Interaction -- Evaluation of Multimodal Dialogue Systems.

The result of four years of intensive research in a large multimodal dialogue project involving 12 partners from academia and industry, SmartKom is one of the most advanced multimodal dialogue systems worldwide and is a landmark project in the history of intelligent user interfaces. The system provides symmetric multimodality in a mixed-initiative dialogue system with an embodied conversational agent. The same software architecture and components are used in three fully operational application scenarios. The theoretical and practical foundations of SmartKom represent a new generation of multimodal dialogue systems that deal not only with simple modality integration and synchronization, but cover the full spectrum of multimodal dialogue. With contributions by leading scientists in the field, this book gives the first comprehensive overview of the results of this seminal project.