W OLFGANG H ALLET The Multimodality of Cultural Experience and Mental Model Constructions of Textual Worlds 1 Reader-cognition, Experientiality and Textual World Construction It seems to be a common experience and a consensual critical assumption that reading fiction implies some sort of imagining of the world that the literary text presents. Probably, this is what makes fiction fascinating and what makes readers read: encountering ‘people’ they normally don’t meet, moving through spaces they have never inhabited or visited, exploring worlds that are normally not accessible. The vast amount of implications of a theory of literary imagination can, of course, not be discussed here. Instead, the reader’s imagination of agents, events, spaces and a whole range of other constituents of the literary world is here regarded as a necessary precondition for successful acts of reading a literary text. This article therefore concerns itself with the factors and cognitive processes that eventually result in a reader’s imagination of the world in which imaginary characters live and act, think and feel, suffer and enjoy themselves. One must be aware, of course, that the ‘textual world’ is itself a metaphor, “one possible conceptualization among many others” (Ryan, Narrative 90), metaphorical concepts such as the text as ‘game’, ‘network’ or ‘assemblage’ (cf. 90). Since, after all, a literary text is hardly more than “a sequence of signs”, a linguistic realm “made of names, definite descriptions, sentences, and propositions” (91), it seems most evident that complex and intricate cognitive operations are required which, in a variation on Herbert Grabes’ early cognitive approach on mental character construction (Grabes, “Turning Words”; also cf. Grabes’ contribution in this volume), can turn ‘words on the page’ into a whole ‘real’ world. According to Ryan, such a textual world is defined by four essential features: “connected set of objects and individuals; habitable environment; reasonably intelligible totality for external observers, field of activity for its members” (Ryan, Narrative 91). The concept of the textual ‘world’, in other words, is necessarily holistic, it emphasizes totality and comprehensiveness and, generally speaking, shows all features of the real world. Therefore, readers must and will intuitively draw upon every possible experience from their own real world to understand the ‘words on the page’. It follows that the cognitive approaches available in literary criticism will probably have to be expanded beyond familiar grounds and single constituents of literary texts like the literary character or the causality of ac- W OLFGANG H ALLET 234 tions and events. A cognitive theory of the literary text that accounts for the totality of a reader’s construction of a textual world, this article argues, must integrate all the cognitive operations that are also involved in, and part of, the reader’s making sense of perceptions and experiences in the real world. In order to clarify the notion of, and the need for, a more holistic cognitive approach a bit further, it may be useful to briefly draw upon cognitive approaches to literary character that have gained a lot of ground after 2000, although an essay by Herbert Grabes on the reader’s cognitive activities that are involved in character construction dates back to as early as 1978. There are three major reasons for why it may be advisable to take the cognitive theory of literary character as a starting point. The first and most important one is that it has drawn attention to the fact that “understanding literary characters requires our forming some kind of mental representation of them”, “a complex interaction of what the text says about the characters and of what the reader knows about the world in general, specifically about people, and, yet more specifically, about ‘people’ in literature” (Schneider, “Toward a Cognitive Theory” 608). Schneider (Grundriß, “Toward a Cognitive Theory”) and Jannidis, in a more differentiated, systematized and historicizing approach, have proposed that the notion of mental representation of character is best conceptualized in terms of mental model construction. This notion will be taken up in section 4 of this essay, generalizing it beyond character construction and expanding it to other textual constituents and to the fictional world at large. The second aspect is the role of world knowledge which cognitive theories of literary character have introduced to explain how readers imagine literary characters. As Schneider (Grundriß, “Toward a Cognitive Theory”) has convincingly shown, the construction and the construal of character require various types of knowledge on the reader’s side. Yet, although Schneider explicitly states that “practically everything he or she knows about the world can be used in reception” (Schneider, “Toward a Cognitive Theory” 611), this cognitive approach to character, in a sense, remains rather limited, since eventually the types of knowledge that are considered relevant are all character-centred and personality-related. They mainly revolve around: • social knowledge structures, i.e. concepts of human behaviour, personality theories, social schemata or stereotypes; • literary knowledge structures, i.e. a reader’s literary experience and familiarity with genre conventions concerning character; • emotional and evaluative response to literary characters, e.g. likeability, empathy etc. (cf. Schneider, Grundriß 35-135; “Toward a Cognitive Theory” 612-613). As much as all of this definitely applies, it seems fairly obvious that the reader’s construction of a literary character must draw upon categories that are not included in these types of knowledge, like, e.g., the possession of a sensory and sensual apparatus, the materiality and corporeality of a given world or an individual’s need and the ability to make sense of his or her per- The Multimodality of Cultural Experience ... 235 ceptions. It is therefore advisable to use ‘experience’ as an additional dimension since this is a holistic category that can stand for an individual’s conceptualizations and mental appropriations of the world. For instance, in order to be able to construct a character properly it may be necessary for the reader not only to know what ‘joy’, ‘hunger’ or ‘pain’ mean, i.e. possess some knowledge about it, but to have at least a faint idea of how these physical and emotional states feel, i.e. to have experienced them. In other words: the ability to construct literary characters requires different kinds of knowledge and experience that refer to individuals as active, conscious and experiencing beings in the broadest sense. After all, any of these experiences, acts or feelings may occur in a literary text, and they are difficult to imagine if readers cannot draw on similar or comparable experiences in their own real world. In section 3, the notion of ‘experience’ will therefore be elaborated. Since characters in narrative fictions are embedded in physical-material and social environments in which they act and interact, suffer or enjoy themselves, environments which affect them in manifold ways, it is also necessary to account for an individual’s different ways of making sense of the world both in the literary text and in the reader’s world. There is a whole range of different modes of world-making that have been translated into the language of the literary text, and literary characters constantly engage in signifying processes and in attempts of making sense out of data and perceptions of all sorts, whether visual, acoustic, olfactory or tactile. In such acts of sensemaking, a literary character will be presented as drawing upon semiotic resources of all sorts, from colour, commodities and all kinds of objects to rooms, whole spaces and social situations. Understanding this world may be far more important in the act of constructing or empathizing with a literary character than the mental representation of the character itself in the narrow sense. Therefore, no matter how different or unusual and unprecedented acts of meaning-making in a literary text may be, understanding a literary character can hardly be separated from the reader’s own ways of making sense of perceptual data and signifying practices. Since the concept of multimodality has theorized different types of semiosis and their integration into single acts of meaning-making, this essay will attempt to integrate ‘multimodality’ into a cognitive approach to the cognitive construction of textual worlds. A short excerpt from Paul Auster’s novel Moon Palace may serve to illustrate all of what has been argued so far. There are two reasons for this choice: On the one hand Auster’s novel is a fine example of various bodily and physical experiences like hunger and blindness, enclosure in caves and small apartments or orientation in wide open spaces. On the other hand, long parts of this novel narrativize acts of signification through which the characters and Marco Stanley Fogg, the autodiegetic narrator, in particular, position themselves in their world and make sense of it. In the passage in question, it is the narrator’s job to identify and describe objects and the urban environment of New York to his employer, a blind old man in a wheelchair called Thomas Effing (a name he has given himself), who has appointed Fogg W OLFGANG H ALLET 236 mainly to write down his obituary and who later turns out to be his grandfather. In this passage, the novel foregrounds the narrator’s perceptions, sensory processes and sense experience as well as his attempts to find a language that makes them accessible to his master: I began to consider it as a spiritual exercise, a process of training myself how to look at the world as if I were discovering it for the first time. What do you see? And if you see, how do you put it into words? The world enters us through our eyes, but we cannot make sense of it until it descends into our mouths. […] I had seen these things before, I told myself, and how could there be any difficulty in describing them? A fire hydrant, a taxi cab, a rush of steam pouring up from the pavement - they were deeply familiar to me, and I felt I knew them by heart. But that did not take into account the mutability of those things, the way they changed according to the force and angle of light, the way their aspect could be altered by what was happening around them: a person walking by, a sudden gust of wind, an odd reflection. Everything was constantly in flux, and though two bricks in a wall might strongly resemble each other, they could never be construed as identical. More to the point, the same brick was never really the same. (122) It can hardly be overlooked that the focus in this part of the novel is on the shape, features and physical qualities of objects and their materiality. On the reader’s side, understanding this part of the novel therefore requires some familiarity with the forms, physical or material features and functions of these objects (fire hydrants, cabs) and even some sort of science-based knowledge (steam, force, angle) as well as sensory and perceptual experiences (fire, light, wind). In other words: The reader finds himself in a position analogous to that of the blind old man to whom Fogg’s descriptions are addressed. While the narrator attempts to find words that can best convey his perceptions, the reader, like the blind man, will be looking for experiences that match the narrator’s words: a taxi, a brick, a wall. Only if the reader, at least approximately, succeeds in recalling and representing these experiences and various types of schematic knowledge is she or he able to construct the fictional world in which this narrator lives, through which he moves on his daily excursions with the old man in the wheelchair, and in which he struggles to communicate his perceptions adequately to the old man. The mental representation of the textual world therefore depends on the experiences and the world knowledge that readers have at their disposal and on their ability to integrate them into a coherent and consistent whole: a mental model of the ‘world’ presented in the literary text. Presumably, the narrator’s experience as narrated in this episode is paradigmatic of the perceptions of narrators and literary characters in general. In order to understand what they perceive, sense, recognize and feel, readers must, at least to some extent, draw upon their own ways of perceiving the world and their ways of making sense of these perceptions. Therefore, ‘experientiality’ is a narratological category which, on the one hand, provides for the (re-)mediation or narrativization of experiences in literary texts and which, on the other hand, accounts for the reader’s cognitive (re- The Multimodality of Cultural Experience ... 237 )construction of these experiences in the reception process. The next section of this essay is therefore devoted to ‘experience’ as a narratological and a cognitive category. 2 Experientiality as a narratological category Whenever we read a narrative text we encounter all sorts of beings, objects, phemonena, spaces and places, actions and events. Therefore, any act of reading requires the activation of the reader’s world knowledge, i.e. all sorts of schemata, scripts and concepts which the reader uses to categorize and understand the actions, motives, occurrences and phenomena in the textual world. Otherwise we are not able to understand what we are reading (cf. Zerweck). In a complementary communicative act, the author anticipates the reader’s schematizations. This does, of course, not mean that the author simply provides the textual signals that match them, but the reader’s world knowledge and schemata will often be irritated, challenged, expanded and applied in unusual ways in the literary text. And yet, in order to be communicable, such unusual, innovative, disturbing or disruptive configurations must rely on the reader’s schemata that make up his cultural knowledge. How, then, can ‘experience’ in the broad sense be conceptualized in a narratological context so that it accounts for the reader’s active part in constructing the textual world? In Monika Fludernik’s ‘natural’ narratology, experientiality is the central paradigm. Independent of what the term comprises and how narrow or wide it is conceived, to her, ‘experientiality’ is the core of narrativization, i.e. the naturalization of textual information in a narrative text that makes it possible to construct a fictional world: ‘[N]atural’ narratology, as I envisage it, relies on a definition of narrativity as mediated human experientiality (which can be plotted on the level of action or on the level of fictional consciousness). […] Narrativity is then established by readers in the reading process on the basis of four levels of natural categories […] which are combined to ‘project’ the fictional world.” (Fludernik 36) The definition of the first level that Fludernik provides sounds very general and seems to include all acts of schematization mentioned above, including very basic ones. She formulates: “axiomatic natural parameters of real-life experience form the most basic experiential and cognitive level.” (43) But when she goes on to specify these natural parameters, she appears to be narrowing experientiality down to the actional and processual side of narratives, stating that her first level is identical to Ricoeur’s ‘Mimesis I’ and that it comprises the schema of agency as goal-oriented process or reaction to the unexpected, the configuration of experienced and evaluated occurrence, and the natural comprehension of observed event processes including their supposed causeand-effect explanations.” (43) W OLFGANG H ALLET 238 Fludernik’s definition of experientiality thus seems to be centred around the structuring and ‘recuperation’ of narrative information - data that constitute a text’s narrativity - and therefore around agency, sequentiality, goaloriented processes and reactions to or the comprehension of events and processes. No explicit mention is made of the phenomenological dimension of the narrative world (a self-restriction that resembles the exclusive focus on character and person in the cognitive theory of literary character) and the corresponding real world frames and schemata. Presumably, this may be owing to narratology’s notorious preoccupation with temporality and sequentiality and the neglect of other dimensions of narratives, such as space or materiality. But in order for ‘experientiality’ to really make sense as a narratological category that is able to comprehend the textual and the cognitive construction of a whole narrative world, a broader definition is required that incorporates all kinds of real world experiences, actually every possible category that is needed to cope with everyday life and to make sense of the real world. The obvious reason for this is that characters and agents in a narrative are situated or position themselves in or make sense of their world - the textual world - in recognizably similar ways as people in the real world do, or, to use a more cautious formulation, literary characters are presented as being challenged by phenomena and perceptions of their world and as attempting to make sense of them. One could even say that in most literary texts this is their main enterprise and concern, no matter whether they succeed or fail. Therefore, it seems quite obvious that narrativization and agency, which are experiential by definition in a cognitive approach, require the integration of experiences of time and space, of objects and phenomena as well as of the materiality of the world and of artefacts. This is, of course, on top of what has been said about the schematizations involved in character construction as conceptualized by Schneider, and in narrativization as theorized by Fludernik. Summing up the brief discussion of one of the most seminal theories of experientiality in the field of narratology, it can be claimed that a cognitive approach that seeks to conceptualize the factors and processes that are involved in, and contribute to, a reader’s mental ways of constructing and representing the fictional world cannot be constrained to the narrativization of events or processes and their ‘recuperation’ by the reader. Instead, it must be conceived of as a holistic category that incorporates virtually any number and type of phenomena, encounters and occurrences that an individual may experience in real life and any type of knowledge that is part of a reader’s world knowledge. This is a prerequisite in any reader-oriented cognitive approach because otherwise it cannot be explained how a reader transforms linguistic data from the literary text into an imagination of the literary characters’ thoughts, feelings and actions and the physical, social and cultural environment they inhabit. It can even be contended that a literary text is mainly either concerned with the characters’ attempts to make sense of their manifold encounters with objects and phenomena, objects, other beings and The Multimodality of Cultural Experience ... 239 actions in this environment, or the narrator is occupied with mediating such attempts to the reader. Of course it might be argued that such a wide narratological concept of experientiality is bound to be essentially mimetic. In fact, this is totally true; but it is hard to see how any cognitive approach could not, at least to a certain extent, be mimetic. Actually, mimesis in that sense is part of Fludernik’s narratology, too; it is even pivotal in her theory since otherwise the whole concept of ‘naturalness’ would be at stake: Mimesis is indeed to be located at the very core of a ‘natural’ narratology […], [it] needs to be treated as the artificial and illusionary projection of a semiotic structure which the reader recuperates in terms of a fictional reality. This recuperation, since it is based on cognitive parameters gleaned from real-world experience, inevitably results in an implicit though incomplete homologization of the fictional and the real worlds. (Fludernik 35) In accordance with Fludernik’s theory it must also be conceded that mental representations of fictional worlds, which necessarily resort to real world categories in order to be able to construct a coherent, plausible textual world, confine this narratological approach to narrative fiction in a realistic or representational mode. Non-representational and experimental narrative texts are a different case for which Fludernik explicitly accounts by conceiving alternative world-building strategies and interpretative naturalizations. […] Experimental fiction can be read as intertextual play with language and with generic modes, and this - since it projects an intentional meta-narrative function - is a mimetic strategy just like any other. […] [Such] experimental texts are therefore not mimetic in terms of reproducing, if in a different medium, a prototypical version of narrative experience, but are mimetic in their structured anticipation of readers’ attempts at reinterpreting them mimetically if only at a meta-meta-realist level of a self-reflexive, explicitly anti-mimetic language game. Phenomenology and language critique need not therefore be at odds but can be subsumed under the cognitive faculty of sense-making (35) With regard to modernist and post-modernist types of literature, such a broad concept of mimesis is, of course, indispensable. And in cognitive theories of literature it is the only way to explain why readers are able to understand literary texts of all kinds: sense-making “is by definition mimetic”, Fludernik rightly states, “since it always relies on natural categories of cognition” (36). 3 Multimodal semiosis and semiotic remediation A broad concept of experientiality that seeks to explain how fictional worlds are inhabited by characters, how they relate to the environments in which they live and how they make sense of the aspects of the fictional reality with which they are faced, requires closer investigation. Therefore, in this part of the essay the concept of multimodal semiosis is briefly introduced since it is W OLFGANG H ALLET 240 deemed to be suited to explain how individuals draw upon virtually all aspects of a world that they encounter in order to make sense of it and thus enable themselves to initiate, or participate in, meaningful actions and practices. The theory of multimodal semiosis seems to correspond with the need for a holistic concept of experientiality as well as for a concept of a whole world that is represented and evoked in the reader’s mind by the linguistic signs of a literary text. The main reason is that it accounts for the integration of virtually every resource that individuals may use to make meaning and turn any phenomenon that they encounter into something meaningful. In the theory of multimodality, a mode is a semiotic resource “used in recognisably stable ways as a means of articulating discourse” (Kress & van Leeuwen 25) and of producing cultural meaning, so that colour, a handwritten letter or a blackand-white photograph as well as a newspaper article or a ground plan are all regarded as semiotic modes (cf. Hallet, “Multimodal Novel”, “Multimodalität”, “Multimodality of Cultural Knowledge“), provided they are drawn into acts of meaning-making. In contrast to modes, media are defined as merely physical and material resources “used in the production of semiotic products and events, including both the tools and the materials used (e.g. the musical instrument and the air; the chisel and the block of wood)” (Kress & van Leeuwen 22). Therefore, modes are abstract semiotic concepts (like, e.g., genres) that “can be realised in more than one production medium” (22). For instance, narrative is a mode that can be realised in written words, in a film or in a cartoon. A concept of multimodality that is derived from different disciplines and fields of study - mainly discourse theory, semiotics, visual culture studies and art design - allows for an integrative approach that seeks to respond to the growing importance of visual and other modes in cultural processes of signification and to overcome the one-sided theorization of verbal communication as a monomodal approach. Instead, the concept of multimodality focuses on the integration of verbal and non-verbal symbolization in signifying processes: ‘meaning’, in this theory, is no longer explained as resulting solely from human language, but as a result of an integration of visual, auditory and other sensual modes and acts of meaning-making in individual as well as in cultural semiotic processes. Therefore, a multimodal theory of semiosis attempts to explain and describe how meaning is made across, and simultaneously through, a variety of different semiotic symbol systems, media and generic modes, and how a combination of modes and media can result in an integrated meaning. For instance, a wall poster may communicate a single coherent meaning through the simultaneous use of language, different font types and sizes of letters, colours, photographs and graphic elements. Evidently, such an approach contrasts strongly with monomodal concepts, in which “language was (seen as) the central and only full means for representation and communication” (45). This may partly explain the preoccupation with textual structures and narrativization as a verbal enterprise The Multimodality of Cultural Experience ... 241 in narratology and even in traditional cognitive approaches, and the neglect of all non-discursive processes and experiences that a literary text mediates in linguistic form. In contrast to this, a multimodal theory of signification defines modes as “semiotic resources which allow the simultaneous realisation of discourses and types of (inter)action” (21). Such resources may be very basic modes like colour or sound, but also more complex resources like textual or medial genres, or cultural artefacts like furniture, or even whole rooms; or they may be established social practices that cultural agents can draw upon in social interaction and communication (cf. 24ff.). ‘Narrative’, too, is a mode, because it allows discourses to be formulated in particular ways (ways which ‘personify’ and ‘dramatize’ discourses, among other things), because it constitutes a particular kind of interaction, and because it can be realised in a range of different media. (21f.). In all of these cases, the concept of multimodality allows for communicative and social practices that incorporate various modes and media in discursive acts of meaning-making or in the cultural negotiation of meaning. As a matter of fact, a closer inspection reveals that almost no act of communication (let alone discursive formation in the Foucauldian sense) is, or has ever been, monomodal. This applies all the more in the age of globalized televison networks, electronic multimedial communication and digital photography or videography. One of the implications of such an approach that is relevant in a cognitive theory of textual world construction, is an increased awareness of and attention to the materiality, physicality and sensousness of experience since the materiality of modes, i.e. the written word in printed form, the digital photograph, the newspaper cartoon etc., “interacts with the materiality of specific senses, even though modes are conventionalisations produced through cultural action over time, and therefore abstract in relation to any particular action” (28). This raises interesting questions, since in a literary text the material or physical quality of an object, the spatial dimension of movement and social interaction or the corporeal implications of perceptions - pain, feelings of well-being etc. - are all translated, or mediated, into verbal language. Gesture, colour or taste, for example, as experiential and semiotic categories, “may be fully articulated and yet not have a correspondingly articulated set of labels in language, spoken or written.” (28) Colour may be a case in point: How can the experience of colour be transformed into verbal language? Or, in the cognitive version: How can a person who has never experienced ‘blue’ or ‘red’ possibly understand the meaning of the words ‘blue’ and ‘red‘ and the perceptions that are connected with them? Evidently, colour is an almost exclusively experiential category because it is very difficult to define ‘red’ or ‘blue’ in verbal form without resorting to other colours in the spectrum, the grammar of colours, as it were. The same applies to many other experiences like weight, distance and time or any tactile qualities of materials. W OLFGANG H ALLET 242 Since a literary text engages in such semiotic translations, it must be regarded as a specific semiotic remediation practice and as a link in the chain of semiotic remediations that reaches from material resources, physical actions and situatedness of actions over human perception and thought (by the author, conceptualized and mediated in verbal discourse through a narrator) to reader-reception and thought, and, possibly, a reader’s discourse, action or social interaction. Interestingly enough, this is exactly the process in which the narrator in Moon Palace engages in the episode in the quotation above: His main concern is indeed how to ‘perceive’ properly (he must learn to see), how to conceptualize his perceptions (in an experiment on the subway he closes his eyes and tries to imagine the people whose voices he hears) and how to find an appropriate discursive form to communicate them to his mentor and employer: “Now I was being plunged into a world of particulars, and the struggle to evoke them in words, to summon up the immediate sensual data, presented a challenge I was ill prepared for.” (121). As a point in fact, Moon Palace stages this epistemological-linguistic challenge on the level of narrative discourse as a series of instructions and training periods during which Effing teaches Fogg how to develop an adequate language that conveys the world to him (cf. Rohr, “The World”, and Rohr, Die Wahrheit 19-74 and 263-291). The narrator (as much as Auster, the author) is well aware of “the immensity of this task” (Weisenburger 141), as his continuous reflexions on adequate ways of verbalizing reality demonstrate. Immediately before his master’s death, when the narrator, for one last time, describes the inventory and the objects in the bedroom to the blind man - “an unobtainable realm of ordinary miracles: the tactile, the visible, the perceptual field that surrounds all life” (Auster, Moon Palace 219) - he characterizes this challenging enterprise as a multimodal semiotic act in the above sense: In some sense, I worked harder for him in that room than I had ever worked before, concentrating on the minutest details and materials - the wools and cottons, the silvers and pewters, the wood grains and plaster swirls - delving into each crevice, enumerating each color and shape, exploring the microscopic geometries of whatever there was to see.” (219) By contrast, at the beginning of his appointment Fogg lacks almost every ability to give Effing “a precise account” (120) of any object that Effing would point to on their daily wheelchair journeys through New York, and as a consequence “the results were dreadfully inadequate” (121). At first, his attempts to find the appropriate language are “overly exact” (122), “piling too many words on top of each other” (123). As a result, his acts of signification are entirely overdetermined, “obscuring” (123) and “burying” the objects of description “under an avalanche of subtleties and geometric abstractions” or “lengthy catalogues” (123) of words and phrases. 1 Only gradually does Fogg learn that 1 Interestingly enough, Marie-Laure Ryan states exactly the same when discussing the immersive qualities of descriptions of space in narrative texts: “A description that mere- The Multimodality of Cultural Experience ... 243 [his] job was not to exhaust him with lengthy catalogues, but to help him see things for himself. In the end, the words didn’t matter. Their task was to enable him to apprehend the objects as quickly as possible, and in order to do that, I had to make them disappear the moment they were pronounced. (123) Eventually, Fogg learns that signifying needs blanks, “apertures of whiteness” (141) in which the meaning of signs and words rests: [T]he more air I left around a thing, the happier the results, for that allowed Effing to do the crucial work on his own: to construct an image on the basis of a few hints, to feel his own mind travelling toward the thing that I was describing for him. (123) Also, Fogg finally becomes aware that no single signification can ever be perfect, that signification remains a constant challenge even for the skilled and trained. He knows that he can never be “entirely satisfied with [his] efforts. The demands of words are too great for that; one meets with failure too often to exult in the occasional success.” (123) Of course it is not difficult to detect a precise description of the roles of the writer (with whom Fogg explicitly identifies) and the reader behind these words so that, as a result, writing and reading the novel perform the very acts of signifying that Fogg renders to his readers. Effing, the blind reader who cannot see what the writer (Fogg) sees and experiences, has to rely on the evocative power of his mediator’s words and on his own imagination. We could also say: Drawing upon his own lived experiences, he has to retranslate his mediator’s discursive remediations of perceptions into mental representations of the world. He thus cognitively constructs the people and the things that surround him, the world that he inhabits: the streets of New York, the buildings, the people, the cars and the cabs as well as his bedroom and his apartment. This holistic process that integrates all sorts of material and social resources in a semiotic act in order to create a meaningful, inhabitable world can be regarded as the mental model construction of a textual world that is created through Fogg’s words. 4 The Mental Model Construction of the Textual World As in all other acts of signification, a reader's response to a fictional narrative requires the cognitive ability to understand and interpret the real world, to process sensual data and perceptions into categories and experience-based knowledge, to apply social and cultural schemata, and to employ cognitive strategies to master the challenges provided by real life. These experiences and types of knowledge are required and activated in the reading process when words on the page are turned into mental worlds. The mind’s tool to ly accumulates details lets its object run through the reader’s mind like grains of sand through the fingers, thus creating the sense of being lost in a clutter of data.” (Ryan, Narrative 124) W OLFGANG H ALLET 244 cope with the task of creating a world that enables it to understand and to make sense of what is read, to transform all the data into a coherent whole that represents a ‘world’, is the formation of mental models. These models are, on the one hand, informed by the mind’s top-down schematizations (experiences, knowledge, concepts, scripts etc.), and, on the other hand, by textual data and signals that determine the boundaries of a given world, the selection of entities and objects, the emphasis on certain features, relations between elements and so forth. It is important to keep in mind, though, that a mental model is quite a holistic representation of (a part of) the actual world or a textual world; it fulfils a ‘unifying’ function that makes it possible to produce coherence and to make sense of diverging, often incomplete data by means of inference: [M]ental models play a central and unifying role in representing objects, states of affairs, sequences of events, the way the world is, and the social and psychological actions of daily life. They enable individuals to make inferences and predictions, to understand phenomena, to decide what action to take and control its execution, and above all to experience events by proxy; they allow language to be used to create representations comparable to those deriving from direct acquaintance with the world; and they relate words to the world by way of conception and perception. (Johnson-Laird 397) Since the reader’s mind must select the most important features of phenomena from the text and transform them into cognitive structures by resorting to his own experiential background, no two individual mental models will ever be identical. On the other hand, due to the shared meanings which all conceptualizations and schematizations represent, mental models are not totally subjective, but can be communicated intersubjectively. The other feature that is of interest in an experiential approach is the fact that the fictional world of the novel and the actions of a literary figure as well as the reader's own imagination of an alternative world (the dream of a better world, a vision, a nightmare etc.) have the same cognitive status as mental models based on real world experiences. This is why all mental models, no matter whether they represent an actual (real) or a textual world, incorporate real world experiences as required and why conceptions, ways of thinking and living, attitudes, values etc. from literary texts can enter real life models and become effective in real life. This explains the performative power of literary texts. The third point is that what is called evocation through words can, in terms of mental model construction, be regarded as the transformation of verbal symbols into cognitive representations. Again, in order for such a transformation to be conducted successfully, the whole range of experience and knowledge must be available if the text signalizes that a particular schema or type of knowledge is required to make sense of the text. This is why, as readers, we can ‘understand’ what literary characters do and why we can regard them as social, interactional and signifying models. The Multimodality of Cultural Experience ... 245 Marco Stanley Fogg in Auster’s novel is mainly presented as a person that engages in acts of meaning-making and signifying. Actually, since in long passages the novel also thematizes these processes, e.g. in the conversations between Fogg and Effing, and since the novel also presents Fogg’s ‘mind in action’ during these processes, the novel as a whole can be regarded as a model of signification. In particular, this concerns spaces, rooms and places all of which Fogg draws into semiosis, and in all of which he tries to define his own position, from a cave in Central Park and his apartment in which he assigns the meaning of furniture to boxes that are filled with books, and the streets of New York which he gradually appropriates semiotically in his wheelchair-journeys with Effing, to the deserts of the American West and, eventually, the West coast. This emphasizes that ‘narrativity’ comprises more than a chronological sequence of events, a change of status and agency. Rather, it must incorporate and narrate all those features of real worlds which the reader must draw upon if they want to make sense of the narrative text. Mental model construction comprises all modes of perception and signification and all kinds of cultural experience. Although readers will more or less ‘naturally’ construct mental models of the spaces represented in narrative fictions in order to “orient themselves on the map of the fictional world” and to “picture in imagination the changing landscape along the routes followed by the characters” (Ryan, Narrative 123), such mental models will presumably never be representations of abstract (empty) spaces or mental graphic maps (for examples of such maps, see Ryan, “Cognitive Maps”). Instead, they will represent a topography that is inhabited by agents and furnished with objects and entities; they will provide for actions, movement and change, and they will integrate all sorts of elements and features that are normally or may be part of a real world. A small empirical experiment in a seminar in which the students were asked to visualize their imagination of the most important elements in Chinua Achebe’s short story “Dead Men’s Path” suggests that readers, while imagining the spatial dimension of the textual world, will simultaneously represent the fictional topography as an inhabited world, including buildings, agents, objects, itineraries, actions and interactions and so forth (cf. Figure 1). W OLFGANG H ALLET 246 Handwritten inscriptions intgrated in the model • Figure, top left corner: "the white supervisor, far away from the reality of the village"; • cloud or bubble in the middle: "hopes and wishes of Michael Obi"; • lower path: "traditions"; upper path: "the traditional ways"; • flashes: "conflicts"; • top of school-building: "modern times"; • school yard: "the wife's influence" and "pride and arrogance"; • village: " the old village" and "poor people"; • fence: "the 'border' built by the modern school"; • bottom, right corner: "between hopes & wishes & reality". Fig. 1: A student’s visual model of "Dead Men's Path", with a transcription of inscriptions The student’s drawing in Fig. 1 is a striking example of the unifying role in the reading process that Johnson-Laird assigns to mental models: this visualization of the student’s mental model of Achebe’s story integrates a wide range of experiential modes and all sorts of semiotic resources upon which the student draws to transform the language of the story into a whole imagined world. These resources and experiences range from buildings and settlements to social practices and constellations (marriage, head of the school etc.). The integration of all these resources into a mental model that turns the signs of the literary text into a meaningful whole requires the full range of lived experiences and world knowledge, including an awareness of people’s imaginations and dreams (the inscription in the bubble). Narrative fiction, it can be concluded, mediates all sorts of experiences, knowledge and semiotic modes in linguistic and aesthetic form. In an act of imagined multimodal semiosis, the reader re-integrates (“recuperates”, as Fludernik says) all these modes and experiences into their imagination of the textual world as a more or less coherent, meaningful whole to make sense of the world (or a slice of it) and of the actions, perceptions, beliefs, thoughts and feelings of those that inhabit it. The Multimodality of Cultural Experience ... 247 5 Narratological Implications It is no easy task to identify possible implications for narratological and literary analysis. But the following aspects of a cognitive-semiotic approach to literature that defines ‘experientiality’ in terms of the full range of perceptual, sensual and cognitive experiences and schemata that are available to a reader and that he or she draws upon when reading narrative fiction, may come to mind: (1) It is impossible for readers not to engage in the formation of cognitive representations of the textual world, however restricted, limited or abstract it may be. This assumption rests on the fact that reading is a communicative act in which readers attempt to understand and make sense of what is being communicated. Readers’ acts of meaning-making principally rely on world categories, concepts and sensual or perceptual experiences when they plausibilize events and occurrences in the narrative world. This is the essence of a cognitive-experiential approach to narrative fiction and literature in general. (2) Mental representations of the textual world rely on a holistic concept of ‘the world’ and of experiencing it, since in everyday life it is a synthesis of a wide range of perceptions that contributes to our cognitive conceptualization of the world and to its mental representation. In particular, these experiences refers to • agency (‘character’, ‘social interaction’, ‘doing’, ‘responsibility, ‘dependence’, a ‘you’, third persons); • corporeality and physical experiences (hunger, pain etc.); • spatiality (experiences of distance, enclosure, empty spaces, movement, settling and appropriating places, travels etc.); • temporality (time, duration, chronology), • materiality and material qualities; • sensory experience and synaethesia (the senses: taste, tactile, olfactory, auditory, visual senses); • the laws of nature (gravity, cause-and-effect relations etc.); • cognition (including emotions). Incomplete as the list may be, it conveys the notion that all of these experiential qualities and types of knowledge and all of these different modes of perception and semiosis may simultaneously apply and be drawn upon in acts of reading in the reader’s attempts to make sense of the words in literary texts. Only a multimodal concept of experience can explain why readers can make sense of a narrative text by constructing a mental model that represents the textual world as a whole. For instance, it is almost impossible to imagine a colourless world - unless the literary text engages in the more or less explicit textual construction of such a world and presents textual signals that force the reader to imagine a world without colour. Still, a reader will have to resort to his knowledge of colour to imagine a colourless world. W OLFGANG H ALLET 248 (3) It can be assumed that character formation, and identity processes in particular, can better be conceptualized in terms of a multimodal experiential process (Hallet, “Plural Identities” 48-50; Hallet, Paul Auster 105-110). As Jürgen Schlaeger has shown, space may be one of the essential experiential categories that are drawn into meaning making processes and acts of signification. Identity processes are also often narrated in terms of physical-bodily transformations which require corporeality as an experiential category (Hallet, “Multimodality of Cultural Knowledge“ 106-108); also, as is prominently the case in Moon Palace, visual experiences are of paramount importance for all identity processes (Hallet, Paul Auster 130-133). (4) There is a growing number of novels that incorporate other semiotic modes in the narrative discourse by actually reproducing them, i.e. by integrating photographs, graphs, newspaper articles, e-mails etc. (cf. Hallet, “Multimodal Novel”). Examples are, e.g., Michel Oondatje’s Running in the Family, almost all of W. G. Sebald’s novels and stories, but also more recent novels by Mark Haddon (The Curious Incident) or Safran Foer (Extremely Loud and Incredibly Close). Thus, the reader, too, is compelled to integrate nondiscursive modes represented in the novel in graphic form into the mental construction of the textual world. (5) Possibly, such a multimodal concept that reads narrative discourse as a link in a chain of semiotic remediations is better suited to read narrative texts in an ethnographic way, in terms of a ‘thick description’, since it must take every single semiotic resource and all related signifying processes that are presented or represented in a narrative text into account. ‘Translational’ strategies, remediation practices, selections of particular dimensions of experiences, features of particular sensual and perceptual processes and their integration into narrative discourse or into identity processes could be very fruitful for cultural and cultural historical analysis, as, e.g., acts of looking and the art of seeing in Auster’s Moon Palace (Hallet, Paul Auster 130-133). (6) Presumably, the notion of multimodal experientiality is required to conceptualize all forms of intermediality. The notion of the musicalization or the visualization of fiction depends on the reader’s ability to incorporate visual or musical experience in the construction of the mental textual model. Otherwise they will miss the intermedial dimension of the narrative. (7) Finally, a holistic experiential concept will have to account for cases in which no or only little information about the textual world is available or in which the reader’s real world experiences are violated. If a literary text deviates from this experiential dimension of the reader’s construction of the textual world it will have to provide more or less salient textual signals to plausibilize and naturalize such a world and its implications. For instance, as long as no explicit mention is made of the colour of objects, nature, spaces and places or whole environments, the reader will imagine the textual world as a world in colours. A colourless (maybe not even black-and-white) textual world would require a number of respective textual signals to prevent the reader from applying ‘colour’ as an experiential dimension. Furthermore, The Multimodality of Cultural Experience ... 249 ‘colourlessness’ would be such a salient feature in a literary text that a particular significance and reflexive potential of such a textual feature can be assumed. Possibly, this is the point where a reader’s mental model would at least partly have to depart from experiential categories and allow for or actively construct a non-experiential alternative world, i.e. expand their ‘real’ everyday mind and add a non-actual dimension to it, transform it into a literary mind. Works Cited Achebe, Chinua. “Dead Men's Path.” 1953. Caught Between Cultures. Colonial and Postcolonial Short Stories. Eds. Ellen Butzko and Susanne Pongratz. Stuttgart: Klett, 2005. 118-121. Auster, Paul. Moon Palace. London: Faber and Faber, 1989. —. “Reznikoff x 2. 1: The Decisive Moment.” 1978. Collected Prose: Autobiographical Writings, True Stories, Critical Essays, Prefaces, and Collaboration with Artist. London: Picador, 2005. 373-388. Fludernik, Monika. Towards a 'Natural' Narratology. London: Routledge, 1996. Grabes, Herbert. “Wie aus Sätzen Personen werden … Über die Erforschung literarischer Figuren.“ Poetica 10 (1978): 405-428. —. “Turning Words on the Page into ‘Real’ People.” Style 38.2 (2004): 221-235. Hallet, Wolfgang. “Plural Identities: Fictional Autobiographies as Cultural Templates of Multitextual Self-Narration.” Narrative and Identity. Theoretical Approaches and Critical Analyses. Eds. Birgit Neumann, Ansgar Nünning and Bo Petterson. Trier: WVT, 2008. 37-52. —. “The Multimodal Novel. The Integration of Modes and Media in Novelistic Narration.” Narratology in the Age of Interdisciplinary Narrative Research. Eds. Sandra Heinen and Roy Sommer. Berlin/ New York: de Gruyter, 2008. [in press] —. “Multimodalität.” Metzler Lexikon Literatur- und Kulturtheorie: Ansätze - Personen - Grundbegriffe. Ed. Ansgar Nünning. 4th rev. and enl. ed. Stuttgart/ Weimar: Metzler, 2008. 521-523. —. “The Multimodality of Cultural Knowledge and Its Literary Transformations.” La conoscenza della letteratura. The Knowledge of Literature. Ed. Angela Locatelli. Bergamo: Bergamo University Press, 2008. 173-193. —. Paul Auster: Moon Palace. Stuttgart: Klett, 2008. Herman, David, ed. Narrative Theory and the Cognitive Sciences. Stanford, Ca.: CSLI, 2003. Jannidis, Fotis. Figur und Person: Beitrag zu einer historischen Narratologie. Berlin/ New York: de Gruyter, 2004. Johnson-Laird, P.N. Mental Models. Towards a Cognitive Science of Language, Inference and Consciousness. Cambridge et al.: Cambridge University Press, 1983. Kress, Gunther, and Leo van Leeuwen. Multimodal Discourse. The Models and Media of Contemporary Communication. London: Arnold, 2001. Rohr, Susanne. “The World as ‘Ordinary Miracle’ in William Dean Howell’s A Hazard of New Fortunes and Paul Auster’s Moon Palace.” Yearbook of Research in English and American Literature (REAL) 15 (1999). 93-110. —. Die Wahrheit der Täuschung. Wirklichkeitskonstitution im amerikanischen Roman 1889- 1989. München: Fink, 2004. W OLFGANG H ALLET 250 Ryan, Marie-Laure. Narrative as Virtual Reality. Immersion and Interactivity in Literature and Electronic Media. Baltimore and London: The Johns Hopkins University Press, 2001. —. “Cognitive Maps and the Construction of Narrative Space.”. Narrative Theory and the Cognitive Sciences. Ed. David Herman. Stanford: CSLI, 2003. 214-42. Schlaeger, Jürgen. “Selves for the Twenty-First Century.” Anglistentag 2005 Bamberg. Proceedings. Eds. Christoph Houswitschka, Gabriele Knappe and Anja Müller. Trier: WVT, 2006. 425-436. Schneider, Ralf. Grundriß zur kognitiven Theorie der Figurenrezeption am Beispiel des viktorianischen Romans. Tübingen: Stauffenburg, 2000. —. “Toward a Cognitive Theory of Literary Character: The Dynamics of Mental Model Construction.” Style 3.4 (2001): 607-640. Weisenburger, Steven. “Inside Moon Palace.” Beyond the Red Notebook: Essays on Paul Auster. Ed. Dennis Barone. Philadelphia, Penn.: University of Pennsylvania Press, 1995. 129-142. Zerweck, Bruno. “Der cognitive turn in der Erzähltheorie: Kognitive und 'Natürliche' Narratologie.” Neue Ansätze in der Erzähltheorie. Eds. Ansgar Nünning and Vera Nünning. Trier: WVT, 2002. 219-242.