A Multimodal Language Faculty: A Cognitive Framework for Human Communication

(Hardback)

Available Formats

Publishing Details

Classifications

Physical Properties

Description

Language has traditionally been held as an amodal system that flows into different forms like speech, writing, or signing; however, communication is multimodal by nature. We pair speech with gestures, use emoji with text, and combine writing with drawings and images in places from doodles to comics to advertising. Yet, the linguistic and cognitive theories maintaining the traditional amodal notion of language cannot account for the richness of this multimodal communication. What is needed is a new, multimodal paradigm of language. This book presents a model of a multimodal language faculty which heralds a re-organization of the structures of language and their guiding assumptions. It shows that the primary human expressive behaviors speaking, signing, drawingmay seem distinct, but actually decompose into similar cognitive building blocks, which coalesce into a multifaceted multimodal communicative system. The result is an account of human cognition where all communication whether speech, gesture, graphics, and their multimodal interactions arises as emergent activation states out of a singular cognitive architecture. The architecture put forward provides a 'grand unified theory' of language and communication, accounting for the insights of traditional linguistic formalizations, conceptual semantics, gesture studies, Peircean semiotics, visual language theory, and both unimodal and multimodal lexicons, all integrated into a single model. This overall approach directly confronts the traditional notions of language that have stood for at least two centuries with a new paradigm of acknowledging the multimodal nature of human communication, forcing us to reimagine what language is and how it works.

Author Bio

Neil Cohn is Associate Professor of Cognition and Communication at Tilburg University, Netherlands. Cohns books include The Visual Language of Comics (Bloomsbury, 2013) and the 2021 Eisner-nominated Who Understands Comics (Bloomsbury, 2020), which establish the linguistic and cognitive study of graphic communication. Joost Schilperoord is Assistant Professor of Cognition and Communication at Tilburg University, the Netherlands. He is best known for his research on language, graphics, multimodality, and cognition.