Multimodality

Gunther Kress, a pioneer in the field of multimodality and the co-author of the bestselling Reading Images, produces a comprehensive theoretical framework for the study of the topic providing sample analyses and suggestions for further reading.

The Graphical User Interface (GUI), as the most prevailing type of User Interface (UI) in today's interactive applications, restricts the interaction with a computer to the visual modality and is therefore not suited for some users (e.g., with limited literacy or typing skills), in some circumstances (e.g., while moving around, with their hands or eyes busy) or when the environment is constrained (e.g., the keyboard and the mouse are not available). In order to go beyond the GUI constraints, the Multimodal (MM) UIs apear as paradigm that provide users with great expressive power, naturalness and flexibility. In this thesis we argue that developing MM UIs combining graphical and vocal modalities is an activity that could benefit from the application of a methodology which is composed of: a set of models, a method manipulating these models and the tools implementing the method. Therefore, we define a design space-based method that is supported by model-to-model colored transformations in order to obtain MM UIs of information systems. The design space is composed of explicitly defined design options that clarify the development process in a structured way in order to require less design effort. The feasability of the methodology is demonstrated through three case studies with different levels of complexity and coverage. In addition, an empirical study is conducted with end-users in order to measure the relative usability level provided by different design decisions.

This accessible introduction to multimodality illuminates the potential of multimodal research for understanding the ways in which people communicate. Readers will become familiar with the key concepts and methods in various domains while learning how to engage critically with the notion of multimodality. The book challenges widely held assumptions about language and presents the practical steps involved in setting up a multimodal study, including: formulating research questions collecting research materials assessing and developing methods of transcription considering the ethical dimensions of multimodal research. A self-study guide is also included, designed as an optional stand-alone resource or as the basis for a short course. With a wide range of examples, clear practical support and a glossary of terms, Introducing Multimodality is an ideal reference for undergraduate and postgraduate students in multimodality, semiotics, applied linguistics and media and communication studies. Online materials, including colour images and more links to relevant resources, are available on the companion website at www.routledge.com/cw/jewitt and the Routledge Language and Communication Portal.

This textbook provides the first foundational introduction to the practice of analysing multimodality, covering the full breadth of media and situations in which multimodality needs to be a concern. Readers learn via use cases how to approach any multimodal situation and to derive their own specifically tailored sets of methods for conducting and evaluating analyses. Extensive references and critical discussion of existing approaches from many disciplines and in each of the multimodal domains addressed are provided. The authors adopt a problem-oriented perspective throughout, showing how an appropriate foundation for understanding multimodality as a phenomenon can be used to derive strong methodological guidance for analysis as well as supporting the adoption and combination of appropriate theoretical tools. Theoretical positions found in the literature are consequently always related back to the purposes of analysis rather than being promoted as valuable in their own right. By these means the book establishes the necessary theoretical foundations to engage productively with today’s increasingly complex combinations of multimodal artefacts and performances of all kinds.

This book meets the demands of scholars of Chinese linguistics as well as researchers on multimodality from a cross-linguistic and comparative perspective. It sheds new light on the traditional study of Chinese discourse and grammar. The volume brings together leading scholars working on the state-of-the-art research on this topic from all over the world, contributing to the understanding of the multimodal nature of human interaction at large.

The contributors in this collection question what kinds of relationships hold between narrative studies and the recently established field of multimodality, evaluate how we might develop an analytical vocabulary which recognizes that stories do not consist of words alone, and demonstrate the ways in which multimodality brings into fresh focus the embodied nature of narrative production and processing. Engaging with a spectrum of multimodal storytelling, from ‘low tech’ examples encompassing face-to-face stories, comic books, printed literature, through to opera, film adaptation and television documentary, stretching beyond to narratives that employ new media such as hypertext, performance art, and interactive museum guides, this volume examines the interplay of semiotic codes (visual, oral, aural, haptic, physiological) within each case under scrutiny, thereby exposing both points of commonality and difference in the range of multimodal narrative experiences.

In this wide-ranging collection, leading scholars, researchers, and emergent researchers from around the world come together and present examples of multimodal discourse analysis in practice. The book illustrates new theoretical, methodological and empirical research into new technologies such as the internet, software, CD-ROM, video, and older technologies such as film, newspapers, brands or billboards. Each chapter demonstrates how aspects of multimodal theory and method can be used to conduct research into these and other multimodal texts. Taken together, the book advances the field of multimodal discourse analysis and offers a range of current studies that address some of the most important questions facing researchers and students when investigating new or old technologies multimodally in linguistics, education, communication studies, sociology, anthropology and other fields.

Multimodality’s popularity as a semiotic approach has not resulted in a common voice yet. Its conceptual anchoring as well as its empirical applications often remain localized and disparate, and ideas of a theory of multimodality are heterogeneous and uncoordinated. For the field to move ahead, it must achieve a more mature status of reflection, mutual support, and interaction with regard to both past and future directions. The red thread across the disciplines reflected in this book is a common goal of capturing the mechanisms of synergetic knowledge construction and transmission using diverse forms of expressions, i.e., multimodality. The collection of chapters brought together in the book reflects both a diversity of disciplines and common interests and challenges, thereby establishing an excellent roadmap for the future. The contributions revisit and redefine theoretical concepts or empirical analyses, which are crucial to the study of multimodality from various perspectives, with a view towards evolving issues of multimodal analysis. With this, the book aims at repositioning the field as a well-grounded scientific discipline with significant implications for future communication research in many fields of study.