Machine Learning Systems For Multimodal Affect Recognition

Markus Kächele offers a detailed view on the different steps in the affective computing pipeline, ranging from corpus design and recording over annotation and feature extraction to post-processing, classification of individual modalities and fusion in the context of ensemble classifiers. He focuses on multimodal recognition of discrete and continuous emotional and medical states. As such, specifically the peculiarities that arise during annotation and processing of continuous signals are highlighted. Furthermore, methods are presented that allow personalization of datasets and adaptation of classifiers to new situations and persons.

The very significant advances in computer vision and pattern recognition and their applications in the last few years reflect the strong and growing interest in the field as well as the many opportunities and challenges it offers. The second edition of this handbook represents both the latest progress and updated knowledge in this dynamic field. The applications and technological issues are particularly emphasized in this edition to reflect the wide applicability of the field in many practical problems. To keep the book in a single volume, it is not possible to retain all chapters of the first edition. However, the chapters of both editions are well written for permanent reference.

This book explores AI methodologies for the implementation of affective states in intelligent learning environments. Divided into four parts, Multimodal Affective Computing: Technologies and Applications in Learning Environments begins with an overview of Affective Computing and Intelligent Learning Environments, from their fundamentals and essential theoretical support up to their fusion and some successful practical applications. The basic concepts of Affective Computing, Machine Learning, and Pattern Recognition in Affective Computing, and Affective Learning Environments are presented in a comprehensive and easy-to-read manner. In the second part, a review on the emerging field of Sentiment Analysis for Learning Environments is introduced, including a systematic descriptive tour through topics such as building resources for sentiment detection, methods for data representation, designing and testing the classification models, and model integration into a learning system. The methodologies corresponding to Multimodal Recognition of Learning-Oriented Emotions are presented in the third part of the book, where topics such as building resources for emotion detection, methods for data representation, multimodal recognition systems, and multimodal emotion recognition in learning environments are presented. The fourth and last part of the book is devoted to a wide application field of the combination of methodologies, such as Automatic Personality Recognition, dealing with issues such as building resources for personality recognition, methods for data representation, personality recognition models, and multimodal personality recognition for affective computing. This book can be very useful not only for beginners who are interested in affective computing and intelligent learning environments, but also for advanced and experts in the practice and developments of the field. It complies an end-to-end treatment on these subjects, especially with educational applications, making it easy for researchers and students to get on track with fundamentals, established methodologies, conventional evaluation protocols, and the latest progress on these subjects.

This book constitutes the refereed post-workshop proceedings of the 5th IAPR TC9 Workshop on Pattern Recognition of Social Signals in Human-Computer-Interaction, MPRSS 2018, held in Beijing, China, in August 2018. The 10 revised papers presented in this book focus on pattern recognition, machine learning and information fusion methods with applications in social signal processing, including multimodal emotion recognition and pain intensity estimation, especially the question how to distinguish between human emotions from pain or stress induced by pain is discussed.

The Handbook of Multimodal-Multisensor Interfaces provides the first authoritative resource on what has become the dominant paradigm for new computer interfaces: user input involving new media (speech, multi-touch, hand and body gestures, facial expressions, writing) embedded in multimodal-multisensor interfaces that often include biosignals. This edited collection is written by international experts and pioneers in the field. It provides a textbook, reference, and technology roadmap for professionals working in this and related areas. This second volume of the handbook begins with multimodal signal processing, architectures, and machine learning. It includes recent deep learning approaches for processing multisensorial and multimodal user data and interaction, as well as context-sensitivity. A further highlight is processing of information about users' states and traits, an exciting emerging capability in next-generation user interfaces. These chapters discuss real-time multimodal analysis of emotion and social signals from various modalities, and perception of affective expression by users. Further chapters discuss multimodal processing of cognitive state using behavioral and physiological signals to detect cognitive load, domain expertise, deception, and depression. This collection of chapters provides walk-through examples of system design and processing, information on tools and practical resources for developing and evaluating new systems, and terminology and tutorial support for mastering this rapidly expanding field. In the final section of this volume, experts exchange views on the timely and controversial challenge topic of multimodal deep learning. The discussion focuses on how multimodal-multisensor interfaces are most likely to advance human performance during the next decade.

This latest volume in the series, Socio-Affective Computing, presents a set of novel approaches to analyze opinionated videos and to extract sentiments and emotions. Textual sentiment analysis framework as discussed in this book contains a novel way of doing sentiment analysis by merging linguistics with machine learning. Fusing textual information with audio and visual cues is found to be extremely useful which improves text, audio and visual based unimodal sentiment analyzer. This volume covers the three main topics of: textual preprocessing and sentiment analysis methods; frameworks to process audio and visual data; and methods of textual, audio and visual features fusion. The inclusion of key visualization and case studies will enable readers to understand better these approaches. Aimed at the Natural Language Processing, Affective Computing and Artificial Intelligence audiences, this comprehensive volume will appeal to a wide readership and will help readers to understand key details on multimodal sentiment analysis.

This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective. This book, emerging from the Special Issue of the Sensors journal on “Emotion and Stress Recognition Related Sensors and Machine Learning Technologies” emerges as a result of the crucial need for massive deployment of intelligent sociotechnical systems. Such technologies are being applied in assistive systems in different domains and parts of the world to address challenges that could not be addressed without the advances made in these technologies.

This book constitutes the thoroughly refereed post-workshop proceedings of the Fourth IAPR TC9 Workshop on Pattern Recognition of Social Signals in Human-Computer-Interaction, MPRSS 2016, held in Cancun, Mexico, in December 2016. The 13 revised papers presented focus on pattern recognition, machine learning and information fusion methods with applications in social signal processing, including multimodal emotion recognition, user identification, and recognition of human activities.