Multimodal Human Computer Interaction

"This book provides concepts, methodologies, and applications used to design and develop multimodal systems"--Provided by publisher.

Multimodal signal processing is an important research and development field that processes signals and combines information from a variety of modalities – speech, vision, language, text – which significantly enhance the understanding, modelling, and performance of human-computer interaction devices or systems enhancing human-human communication. The overarching theme of this book is the application of signal processing and statistical machine learning techniques to problems arising in this multi-disciplinary field. It describes the capabilities and limitations of current technologies, and discusses the technical challenges that must be overcome to develop efficient and user-friendly multimodal interactive systems. With contributions from the leading experts in the field, the present book should serve as a reference in multimodal signal processing for signal processing researchers, graduate students, R&D engineers, and computer engineers who are interested in this emerging field. Presents state-of-art methods for multimodal signal processing, analysis, and modeling Contains numerous examples of systems with different modalities combined Describes advanced applications in multimodal Human-Computer Interaction (HCI) as well as in computer-based analysis and modelling of multimodal human-human communication scenes.

The two-volume set LNCS 10271 and 10272 constitutes the refereed proceedings of the 19th International Conference on Human-Computer Interaction, HCII 2017, held in Vancouver, BC, Canada, in July 2017. The total of 1228 papers presented at the 15 colocated HCII 2017 conferences was carefully reviewed and selected from 4340 submissions. The papers address the latest research and development efforts and highlight the human aspects of design and use of computing systems. They cover the entire field of Human-Computer Interaction, addressing major advances in knowledge and effective use of computers in a variety of application areas. The papers included in this volume cover the following topics: HCI theory and education; HCI, innovation and technology acceptance; interaction design and evaluation methods; user interface development; methods, tools, and architectures; multimodal interaction; and emotions in HCI.

This book constitutes the strictly reviewed post-workshop documentation of the First International Conference on Cooperative Multimodal Communication held in Eindhoven, The Netherlands, in 1995. The volume presents an introductory survey and carefully re vised and updated full versions of three invited contributions and 14 papers selected for inclusion in the book after intensive reviewing. Among the issues addressed are intelligent multimedia retrieval, cooperative conversation, agent system communication, multimodal maps, multimodal plan presentation, multimodal user interfaces, multimodal dialog, and various systems for multimodal HCI.

This preface tells the story of how Multimodal Usability responds to a special challenge. Chapter 1 describes the goals and structure of this book. The idea of describing how to make multimodal computer systems usable arose in the European Network of Excellence SIMILAR – “Taskforce for cre- ing human-machine interfaces SIMILAR to human-human communication”, 2003– 2007, www. similar. cc. SIMILAR brought together people from multimodal signal processing and usability with the aim of creating enabling technologies for new kinds of multimodal systems and demonstrating results in research prototypes. Most of our colleagues in the network were, in fact, busy extracting features and guring out how to demonstrate progress in working interactive systems, while claiming not to have too much of a notion of usability in system development and evaluation. It was proposed that the authors support the usability of the many multimodal pro- types underway by researching and presenting a methodology for building usable multimodal systems. We accepted the challenge, rst and foremost, no doubt, because the formidable team spirit in SIMILAR could make people accept outrageous things. Second, h- ing worked for nearly two decades on making multimodal systems usable, we were curious – curious at the opportunity to try to understand what happens to traditional usability work, that is, work in human–computer interaction centred around tra- tional graphical user interfaces (GUIs), when systems become as multimodal and as advanced in other ways as those we build in research today.

Abstract: "Multimodal interaction is a way to make user interfaces natural and efficient with parallel and synergistic use of two or more input or output modalities. Two-handed interaction is a special case of multimodal interaction that makes use of both hands in a combined and coordinated manner. This dissertation gives a comprehensive survey on issues that are related to multimodal and two-handed interaction. Earlier work in human-computer interaction and related psychology is introduced within both these fields. The constructive part of this dissertation consists of designing and building a group of multimodal interaction techniques that were implemented in two research prototypes. The first prototype is an object-oriented drawing program that implements new tools that are controlled with two-handed input. The second prototype is a multimodal information kiosk that responds to both touch and speech input, and makes use of touch pressure sensing. In order to evaluate the success of constructive research, four empirical studies were conducted to compare the new interaction techniques to the conventional methods and to evaluate how the users react on them. The first of the studies compared a new direct manipulation tool to conventional menu and palette commands. The second evaluation was more informal and determined how an alternative way of drawing would be used by normal users. The third study was aimed at determining what is the best input device configuration to control the new tools. The last study evaluated different touch-based selection techniques in a multimodal touch and speech based information kiosk. The need for extensive interdisciplinary research is pointed out with a group of research questions that need to be answered to better understand multimodal human-computer interaction. The current knowledge only applies to a few special cases, and there is no unified modality theory of multimodal interaction that covers both input and output modalities."

The growing emphasis on multimodal interface design is fundamentally inspired by the aim to support natural, easy to learn and use, flexible, efficient, and powerfully expressive means of human-computer interaction. Most of the articles in this special issue present work in support of challenging applications such as algebra instruction, data summaries, and interaction with complex spatial displays. A collection of emerging research ideas on next-generation multimodal interfaces, it also addresses multimodal interface design for portable devices to be used in natural field settings. Additionally, it describes implemented systems that make computing accessible to the visually impaired.

Here is the third of a four-volume set that constitutes the refereed proceedings of the 12th International Conference on Human-Computer Interaction, HCII 2007, held in Beijing, China, in July 2007, jointly with eight other thematically similar conferences. It covers multimodality and conversational dialogue; adaptive, intelligent and emotional user interfaces; gesture and eye gaze recognition; and interactive TV and media.