The Structure Of Multimodal Documents

This book develops a new framework for describing the structure of multimodal documents: how language, image, layout and other modes of communication work together to convey meaning. Building on recent research in multimodal analysis, functional linguistics and information design, the book examines the textual, visual, and spatial aspects of page-based multimodal documents and employs an analytical model to describe and interpret their structure using the concepts of semiotic modes, medium and genre. To demonstrate and test this approach, the study performs a systematic, longitudinal analysis of a corpus of multimodal documents within a single genre: an extensively annotated corpus of tourist brochures produced between 1967-2008. The book provides multimodal discourse analysts with methodological tools to draw empirically-based conclusions about multimodal documents, and will be a valuable resource for researchers planning to develop and study multimodal corpora.

The first systematic, corpus-based and theoretically rigorous approach to the description and analysis of multimodal documents. Drawing on academic research and the experience of designers and production teams, Bateman uses linguistically-based analysis to show how different modes of expression together make up a document with a recognisable genre.

This volume integrates multimodal theoretical frameworks with those from graphic communication and information design and applies this critical synthesis to the examination of the changes and relationships that occur when multimodal documents are distributed across various means and channels of consumption. Drawing on examples from popular newspapers and store catalogs, the book’s specific focus is on documents as sets, here defined as the collective of all the assorted forms of a document published across multiple mediums and modes. This approach affords a multi-layered analysis of multimodal documents more broadly, in addition to engaging in questions about the very definition of a document and the terminology we use in relation to documents, including genres, mediums, and modes. As both a critical examination of the theoretical frameworks employed in literature on documents and a way forward for new approaches to analyzing multimodal texts, this volume is key reading for students and scholars in multimodality, graphic communication, design, media studies, and information science.

Most dialogues are multimodal. When people talk, they use not only their voices, but also facial expressions and other gestures, and perhaps even touch. When computers communicate with people, they use pictures and perhaps sounds, together with textual language, and when people communicate with computers, they are likely to use mouse gestures almost as much as words. How are such multimodal dialogues constructed? This is the main question addressed in this selection of papers of the second Venaco Workshop, sponsored by the NATO Research Study Group RSG-10 on Automatic Speech Processing, and by the European Speech Communication Association (ESCA).

Multimodality’s popularity as a semiotic approach has not resulted in a common voice yet. Its conceptual anchoring as well as its empirical applications often remain localized and disparate, and ideas of a theory of multimodality are heterogeneous and uncoordinated. For the field to move ahead, it must achieve a more mature status of reflection, mutual support, and interaction with regard to both past and future directions. The red thread across the disciplines reflected in this book is a common goal of capturing the mechanisms of synergetic knowledge construction and transmission using diverse forms of expressions, i.e., multimodality. The collection of chapters brought together in the book reflects both a diversity of disciplines and common interests and challenges, thereby establishing an excellent roadmap for the future. The contributions revisit and redefine theoretical concepts or empirical analyses, which are crucial to the study of multimodality from various perspectives, with a view towards evolving issues of multimodal analysis. With this, the book aims at repositioning the field as a well-grounded scientific discipline with significant implications for future communication research in many fields of study.

"This book provides a well-rounded synopsis of the state-of-the-art in perceptual-based multimedia design"--Provided by publisher.

This volume advances the data-based study of multimodal artefacts and performances by showcasing methods and results from the latest endeavors in empirical multimodal research, representing a vibrant international and interdisciplinary research community. The collated chapters identify and seek to inspire novel, mixed-method approaches to investigate meaning-making mechanisms in current communicative artifacts, designs, and contexts; while attending to their immersive, aesthetic, and ideological dimensions. Each contribution details innovative aspects of empirical multimodality research, offering insights into challenges evolving from quantitative approaches, particular corpus work, results from eye-tracking and psychological experiments, and analyses of dynamic interactive experiences. The approaches and results presented foreground the inherent multidisciplinary nature and implications of multimodality, renegotiating concepts across linguistics, media studies, (social) semiotics, game studies, and design. With this, the volume will inform both current and future developments in theory, methods, and transdisciplinary contexts and become a landmark reference for anyone interested in the empirical study of multimodality.