Syncretic Semiotics of Hypermediality Sungdo Kim Preliminary remarks For the students of semiotics and text sciences the phenomenon of hypermediality or intermediality raises some crucial questions and even provokes some important epistemological transformations in the level of concepts of text and sign. In fact, the increasingly hypermedial nature of contemporary society demands a seamless activity of medial and semiotic translation. In a hypermedial era we see video, listen to radio and read text on the same page. Regretfully, we do not have a relevant semiotic theory for this new media environment. We have just a name, hypermedia, but little more. Hypermediality is embedded in the more general framework of intermediality which refers back to the entire cultural history of communication and media. The new digital technology and electronic media which have recently emerged through the integration of text, image, sound, and video, etc., can be considered as new modalities of communication and signification. For today’s web author, all kinds of sign-systems are available, be they text, sound, still or moving images. What emerges is a rhetoric of convergence, a blending of existing figures. Hypermediality is an interdisciplinary concept par excellence, which affects all branches of human and social sciences: critics, mediologists and linguists have all appropriated it in their respective domain of research in price of a definition each time modulated by the context and specific objectives of research. This transdisciplinarity made this concept a volatile and heterogenous element in a manner that is difficult to structure in the form of knowledge content (cf. Cotton and Oliver 1997). One must stress that hypermediality does not stand out uniquely among the possibilities of media convergence. It designates rather a mediatic ecology of an inedited genre. This new media ecology is founded upon an effective, accelerated and organic hybridization of textual and media practices. The new modality of intermedial relations offers a complete reversal of perspective, which takes away the consequences on the epistemological and cognitive level in the semiotic plan of media landscape. We are now attending to a semiotic reconfiguration of the network between diverse media. One of the objectives of this article is to take measure of the transformations fed by the introduction of the concept of hypermediality in several different fields of semiosis. This paper takes up the current status of hypermediality by examining cross cultural uses of word and image, both historically and currently (cf. Hocks and Kendrick [eds.] 2003). This study suggests that to attempt to characterize hypermedia as a new conflict zone between word and image or as a totally unedited phenomenon is to misconceive radically the dynamic interplay that already exists and has always existed between visual and verbal texts in world civilizations and to overlook insights into intermediality that hypermedia theories and practices can bring up. K O D I K A S / C O D E Ars Semeiotica Volume 29 (2006) No. 4 Gunter Narr Verlag Tübingen Sungdo Kim 292 <figure 1> The relationships among verbal texts and visual texts are complementary and dialogic, and have commonly appeared in East Asia’s visual traditions and the Middle Ages’ manuscripts. This study proposes a historic and comparative approach to hypermediality. The concept of syncretic semiotics: some epistemological cues in contemporary semiotics and media theories 1. An elementary definition of syncretic semiotics The concept of syncretic semiotics is not widely recognized or even understood today. In fact, contemporary semiotics has limited its principal domain to a single communication channel and a homogeneous system of signs. Therefore one might encounter difficulties in finding any cue in the actual semiotic scene, which would be attributable to the elaboration of this relatively unexplored area. I want here to sketch a few of the ways in which syncretic semiotics could explain the phenomena of the fusion of different sign systems. I argue that syncretic semiotics is a conceptual model which is able to assume a valid heuristic role in describing and categorizing the phenomenon of hypermediality. The scope of this research is potentially vast, since it proposes to attend to any interesting encounter or synthesis of different modalities of senses, and hence the description is limited to only some of its significant features. The idea of this inquiry has been conceived by questioning a powerful tradition that has dominated our thinking about communication and signification in the twentieth century. Let me explain very briefly what syncretic semiotics is. In simple terms, syncretic semiotics means a fusion or integration of different sign systems like opera or cinema. In the case of verbal communication, it includes not only verbal signs but also kinesics and proxemics like gestures, space management, facial expressions, and paralinguistic elements. However one can hardly find any serious study on this subject matter. In spite of its importance, few semioticians have shown interest in this new area. In fact the entry is absent in the major semiotic dictionaries. One exception is the dictionary of Greimassian Paris School in which syncretic semiotics is represented as a particular type of sign system which is a composite of several autonomous individual sign systems (Greimas and Courtés 1979: 344). The universality of syncretic phenomena of different systems of signs which have been discovered in different times and spaces is omnipresent and panchronic: from prehistoric graphism to hypermedia, via the illumination of the Middle Ages and East Asia’s rich tradition of the imagetext interaction. The list is not closed. For example, let me evoke an elegant page of Chinese-Korean polyscript of the 15th century which is a fascinating kind of syncretic semiotics (Figure 1). Syncretic Semiotics of Hypermediality 293 In sum, syncretic semiotics attempts to elaborate an integrative and holistic approach towards the various natures of sign production and understanding contrasted with homogeneous semiotic systems. 2. Some epistemological cues a) The concept of homology according to Benveniste A keynote regarding syncretic semiotics can be discerned from Benveniste’s semiological program. Although the great French linguist did not use the word, syncretic semiotics, his seminal article, “Semiology of language”, contains a very interesting concept which is relevant to my subject. Concerning the nature and the feasibility of relationships among semiotic systems, he proposed three kinds of relationships: the generative relationship, the relationship of homology, and the relationship of interpretation (Benveniste 1974: 51-63). Among these three, the second category concerns directly this work. In fact, the relationship of homology establishes a correlation between the parts of two semiotic systems and is defined by virtue of the connections we find or establish between two distinct systems. The kind of homology may have many variations: intuitive or rational, substantial or structural, conceptual or poetic (Benveniste 1974: 61). Benveniste provided three examples in his correspondence with Baudelaire which organized his poetic universe and the imagery which reflects it. Remember his phrase, “Les parfums, les couleurs, les sons se repondent.” (Fragrances, colors and sounds mutually respond.) The second example of a more intellectual nature might be very interesting for art historians as it concerns the homology that Panofksy sees between Gothic architecture and scholastic thought. However Benveniste does not provide any reason for this Panofskian view. The third example is the homology between writing and ritual gesture in China. Nevertheless, the fundamental element in his very brief and intuitive remark is that the homology established will serve as a unifying principle between two fields or will create new kinds of semiotic values. My contention is that syncretic semiotics is properly a new kind of semiotic value and it confirms the validity of this relationship. b) Definition of Paris School The bulletin of Paris School published a special issue on the subject and elaborated some theoretical issues (cf. Floch 1983). According to the editor, the mass media and the ritual, the speech and gestures of the patient belong to the syncretic sign system which are operated by different kinds of manifestant languages. The problem is that there is no theoretical convergence on the criterion of semiotic typology which the recognition of this semiotic plurality could imply. For instance, Peirce has considered the nature of signs according to the mode of relationship between a sign and its referent (cf. Peirce 1936-1956). (Perhaps, his concept of hypoicon anticipates syncretic semiotics given fact that every sign contains an icon, index, or symbol.) Eco (1975) presented his typology of sign systems according to the nature of substance of the signifier and that of the channel of sense organs. Hjelmslev (1961: 87-93) evaluated the nature of sign type according to the degree of scientificity and constituent numbers. In fact if we follow the theoretic stance of the great Danish linguist, the syncretic sign systems constitute their plan of expression with the elements which belong to the heterogenous sign systems. Sungdo Kim 294 However this definition remains incomplete. What does ‘the heterogenous’ mean? We have a tendency to think of the heterogeneity in relation to the linguistic model. In sum, we don’t have any elaborated definition of syncretic sign systems. This absence is partially due to the absence of the metalanguage which allows the description of syncretic texts as paralinguistic communication, dance, or cinema. There is no systematic notation of body language or any descriptive language which might make possible the first decoupage of cinema language. In addition, the question remains of the strategy of syncretic communication and its procedures of syncretism. The constitution of syncretic texts by reference to the plurality of manifestant languages is based on the global strategy of syncretic communication which consists of managing the discoursive contents of the syncretic textualization. The syncretic strategy is founded upon the discoursive competence of the enunciator. This strategy constitutes the plan of content of syncretic text, while the analysis of the procedures of syncretization is supposed to be located in the plan of expression of the same text. In short, this constitutes a connotative sign system. The inquiry on the object of this semiotic phenomenon might elucidate the overlapping of or the choice of relay between the different languages which establish the syncretic text. In this inquiry the factors of ideology and value should be considered. For instance, it would be worthy to reconsider the ideological function defined by Barthes (1978) in his seminal paper, “Rhetorics of Images”. According to his view, for any visual image is ambiguous, the linguistic message is one of the techniques to which every society refers to undermine to the terror of uncertain signs. An elaborate analysis of syncretic sign systems might be helpful to apprehend the nature of attitudes vis-à-vis the languages and signs and its reactions which a society adopts. In this regard, this analysis is perfectly adapted to the program of saussurian semiology, i.e., “the laws of transformation of sign in a society”. c) The notion of syncretism by Hjelmslev Hjelmslev wrote a section on “syncretism” in the framework of traditional grammar and modern phonology. On other hand, in his section on ‘connotative semiotics and metasemiotics’, he remarked that he operated his semiotic theory in supposing that the given text presents a structural homogeneity and that we could legitimately introduce in this text through the catalysis of a unique semiotic system. However, he noted that every text commonly contains drifts which are founded upon different systems. Various parts of a text can be composed of different stylistic forms (verse, prose, various blends of the two), different styles (creative style and the purely imitative, so-called normal, style), different value-styles, different media (speech, writing, gesture, flag code, etc), different tones (angry, joyful, etc), and different idioms (different vernaculars, different national languages, regional languages, physiognomies). Three points are worthy of being noted in his notion of semiotic syncretism: intrasemiotic translatability, solidarity of different styles, and semiotic hybridity. “Stylistic form, style, value-style, medium, tone, vernacular, national language, regional language, and physiognomy are solidary categories, so that any function of denotative language must be defined in respect of them all at the same time. By combination of a member of one category with a member of another category arise hybrids, which often have, or can easily be style provided with, special designations: belletristic style-a creative style that is a higher value-style” (Hjelmslev 1961: 116) Syncretic Semiotics of Hypermediality 295 Finally, a study on the procedures of syncretism and the syncretic plan will be required. This designates the procedures of incorporation of different manifestant languages on the syncretic plan of expression. The study of syncretic realization of the plastic sign system makes us aware of the necessity of the conditions of the syncretism of diverse languages. The task of studying the procedures of different kinds of semiotic natures is very difficult and may explain the judgment of Hjemlslev that the scientific analysis of connotative fact will be revealed as impossible. 1 According to him, in preparing the semiotic analysis one has proceeded on the tacit assumption that the datum is a text composed in one definite semiotic, not in a mixture of two or more semiotics. In other words, in order to establish a simple model situation we have worked with the premiss that the given text displays structural homogeneity, that we are justified in catalyzing one and only one semiotic system to the text. (…) This premiss, however, does not hold good in practice. On the contrary, any text that is not of so small extension that it fails to yield a sufficient bais for deducing a system generalizable to other texts usually contains derivatives that rest on different systems (Hjelmslev 1961: 115). 3. The implications of media theories for syncretic semiotics: interface with remediation theories Syncretic semiotics should be understood not as a technical concept or a particular branch of semiotics but rather a method to fill the gap between humanities and media theories in general. As indicated above, the syncretic semiotic model is to be elaborated in the semiotic scene. However if one broadens his perspective into media theories, it is possible to determine some epistemological convergences between the premise of syncretic semiotics and other media theories. In this section I will focus on the remediation theory of Bolter which could be incorporated into the program of syncretic semiotics. First of all we need a critical view of the explorations of the medium theory of communication technologies and incorporation of semiotics’ inquiry into the uses of meaning regarding texts and signs. The two fields have traditionally held oppositional views regarding concepts such as media/ medium, content, and interpretation and it is the purpose of this study to build a bridge between these two worlds. The traditional concern of semiotics has been the meaning implicit within particular texts, whereas the main contributions of medium theories deal with the impact of media or technologies on culture and human consciousness. Nevertheless, any moral or cognitive evaluation on the mixture or blending of different media is unnecessary here. As an example, the negative attitude of Jameson on this phenomenon, who reads this in the context of the cultural logic of post-capitalism, leads him in a consideration of television to discern two crucial moments in the television era (cf. Jameson 1991) . First, the phenomenon of intermedia is emergent and this fundamental and massif feature of audio visual media affects the representation of print. Second, a new kind of emotion is established with this displacement of experience and move to the new pattern of cognition which is totally different from that of the previous cognition. The remediation theory of Bolter is very relevant in syncretic semiotics. It means a formal logic by which new media refashions prior media forms. The double logic of remediation can be summarized as follows: “our culture wants both to multiply its media and to erase all traces of mediation: ideally, it wants to erase its media in the very act of multiplying them” (Bolter & Grusin 1999: 5). Sungdo Kim 296 Because of the rapid development of new digital media and the immediate response by traditional media it is important to understand the impact of remediation. In the effort to create a seamless moving image, film makers combine live-action footage with computer compositing and twoand three-dimensional computer graphics. In the effort to be up the minute and complete, television news assembles on the screen ribbons of text, photography, graphics, and even audio, without a video signal when necessary. In digital media today the practice of hypermediacy is most evident in the heterogeneous window style of world wide web pages. For instance, the CNN site is hypermediated - arranging text, graphics, and video in multiple planes and windows and joining them with numerous hyperlinks; yet the web site borrows its sense of immediacy from the televised CNN newscasts. At the same time televised newscasts are coming to resemble web pages in their hypermediacy (Bolter & Grusin 1999: 3-15). Bolter proposes to identify the same process through the last several hundred years of Western visual representation. He provides an excellent account of how new media inherit forms from older media and alter them. In fact, the logic of hypermediacy has a history as a representational practice and a cultural logic. Semiotic implications of hypermediality 1. From the techno-cumulative logic to the historic epistemology of writing From a semiotic point of view, hypermediality concerns, first of all, the multiplicity of signs: written texts, recorded speech, diverse sounds, fixed images and animated, formal codes, etc. However what is really innovating is the concentration of these signs on a single technical device (the program managing the processor, screen, keyboard, or printer). A semiotic syncretism and digital integration make possible some new modes of association between the signs of writing with a vast memory capacity and activation of these new signs. The whole question of the writing-reading of hypermediality and the complexity of the signifying spaces that the latter mobilize must be raised. At the same time, a tension is created between the freshness of a mode of manifestation of the written sign and the historic depth of what makes it interpretable. In the place of a cumulative logic that prevailed until now, we should be nourished with an epistemologico-semiotic reflection on the sense of the new intermedial syntax in avoiding an approach that is purely functional via the hypermedia. The hypermedia is not a simple sum of different media, but rather a creation of new message forms. The real problem is a redefinition of the writing (Jeanneret 2001: 386-394). What is the scope of this redefinition? Many scholars have recalled an anthropological mutation comparable to the invention of writing itself. Perhaps it is more reasonable to rediscover a buried force. Neither the association of different types of signs, nor the nonlinearity which characterize every written form, nor the virtuality of the sense related to the interpretation of the reader, or the consequence of the interactivity of the process of writingreading are totally new. Furthermore, some prophets have labeled this a denegation, for which the medieval commentary might have termed a hypertext, in which the multimedia animation may be a variant of the illustration and the computer screen may be a ‘volumen’. The hypermedia comes to reactivate some forgotten potentiality and richness of the writing through its search for newness within itself. Syncretic Semiotics of Hypermediality 297 2. The economy and phenomenology of writing signs Although the permanent evolution of signs might frustrate any prediction, it is not forbidden to make some observations. Firstly, the written sign receives some new dynamics of appearance-disappearance. Katherine Hayles (1999: 29-45) has called these the flickering signifiers. This property opens the possibility for realizing that complex texts contain diverse forms of chain, even as it poses the new question of how the reader can appropriate this complexity. Finally, we can observe a harmonization between text and image. The digitalization of information has produced an essential consequence concerning the writing. The computer can handle the writing in the modes of text (chain of characters) and image (a singular configuration). Such concern with the often fraught intersection of pictures and words is obviously nothing new. Nevertheless, their areas of reciprocity and friction, copresence and intractability, and equivalence and difference are among the most vexed issues in the study of cultural artifacts and artistic works. According to a general classification of different media there are three categories: traditional images, texts and technical images. Each of these media forms is created by man as an explanation of the world in order to ease his orientation in this world. Yet, each medium is possessed by the same deceitful dialectics: instead of representing the world, media present the world as it is perceived by humans. In other words, one could state that media prevail upon that which is seen through them: media determine our vision upon the world and not the other way around. The new code of writing not only brings with it a new form of consciousness, but also produces a new universe that is radically different from the one created by the traditional image (cf. Flusser 2004). There has been an ongoing dialectical struggle between images and writing. The images served between word and image have shifted. The images served as illustrators of the text, while the real work of communication was thought to be done by words. However, with the development of a series of audiovisual technologies influenced by modernist art, graphic artists associated with Bauhaus reversed the accepted relationship by subsuming words into image and so teaching us to regard words themselves as image. These graphic designers brought the modernist view into popular culture, by surrounding us in posters and magazine ads with examples of the word-as-image. Their work made the word immediate, sensually apprehensible by insisting on its visual form rather than its symbolic significance. Hypermedia continues this line of challenge to the dominance of the printed word, by claiming to provide a new kind of interaction between the user/ viewer and the digital application. Thus, we think of the world wide web as a multilinear writing space. In fact, many genres of print are being refashioned for presentation on the web. The hypermedia can in fact subsume many of the media forms of the twentieth century. The hypermedia’s eclectic character also means that it can borrow or imitate many styles of relationships between word and image that have characterized earlier forms. Through streaming media the web is now refashioning film and television as well. Film and television usually replace the word rather than absorb it into their images. Nevertheless, web designers still seem to prefer a different strategy borrowed from graphic design. In short, the world wide web and other new media challenge not only the form of the book, but also the representational power of the printed signs. It was hard to image that our culture would ever seek to replace all written communication with these audiovisual media. The hypermedia appears to accommodate both the verbal and the audiovisual, so that some Sungdo Kim 298 celebrate our return to a pictorial age in daring to suggest that the hypermedial form could fully replace written words. Secondly, the generation of the new kinesics might be underlined. The constituted complex text is accessible from the device of diverse nature, in which the typical example is the mouth symbolizing the multimodal interfaces. It concerns the means to intervene into the text without passing through the keyboard. These gestures are radically liberated from the alphabet code and actualize an interpretation founded essentially upon the reading of the image. In terms of the history of reading, we can say that these gestures provide a place for the new form of reading, namely a gestualized reading, after oralized reading and silent reading. In this context we can understand Flusser’s thesis that every medium has its own language of gestures. In this manner he speaks of the gesture of photography, of filming, of painting and so on. The core of every meaningful human action becomes the gestures that underlie it. Every important instrument and medium has its own specific set of gestures. Thirdly, the creation of the virtual image. The writing culture became rich in parallel with the new types of creations and manipulation of images, handled as geometric abstractions. This concerns the vectorial and virtual images. The represented objects are defined by an ensemble of parametered equations which are worn with the texture which precedes the illusion of reality. These images have permitted the creation of dynamic objects through writing, ones that the reader-actor can consider as analogues of the perceived world. 3. The plastic and social dimensions We might risk missing the nature of these changes in announcing a global conversion of the writing culture. Their appearance causes, on occasion, some experimentations that are limited by economic and cultural differences. The writing of hypermedia was socially preceded by its rewriting in some pioneer communities of avant-garde artists and then in the multimedia. Being immaterial and apt to permit all circulations, the hypermedia network frees the communication from the opacity of signs and might thereby inaugurate a society of minds absent any frontiers or differences. It corresponds to the notion of collective intelligence or the semantic ideography proposed by Pierre Lévy (1994). The integration of information seems to dispense a reflection about the nature of signs, even while requiring a deep insight into the specific forces of the written word, the visual image, and the speech. The association of all messages leads to a research into the principle of exhaustivity, even as it requires on the contrary the emergence of more supple codes capable of reducing the possibilities. We must consider the material and social approach of the text against the abstractions, in order to present a reflection that is much more adventurous about the semiotic nature and its possibilities of hypermediality. Two main factors need to be underlined to seize the nature of the image in hypermedia: the space of its medium and the temporality constrained by its reading. In terms of the initiative of the reader, the disjunction of the hypertext logic and hypermedia semiotics has no meaning. The real issue remains whether the central focus is our way of understanding the richness of the written signs or the initiative which could be shared in the cultural production. It is not certain that the hypermedia will have a creative and emancipating future but such a future is possible if the instability of the written sign, such as existing in the ideograms, enriched of the new dynamism of the plasticity and the rhythm, is Syncretic Semiotics of Hypermediality 299 recovered in the new type of texts, where the thoughts of vision, discourse and speech could mutually provoke each other, instead of being mutually flattened. This means that despite the opposition of all the metaphors which reduce the written signifier to immateriality, to an agglomeration of particles or a fluid element, our whole material culture of signs will be rejuvenated as a montage of images, places, gestures, and objects. Very often today, a space of lure and illusion quoting every place of memory among the most traditional ones is opened. One may call this the window of perspective, the architecture of the museum or the box where one can capitalize objects. Nevertheless, the challenge of the hypermedia is in becoming liberated from this cultural imaginary through the egocentrism of looking, delimitation of spaces, uniformization of cultural objects. We must return to the lively resources of writing in the context of the institution and frontiers of the social. Without this critical consciousness the writing of hypermediality could be a totalizing discipline of the actual media ecology. It is impossible to predict whether certain potentialities of the hypermedia, either as the omnipresence of the simulation or the acceleration of the rhythm of the message, by endangering the idea of intelligence of a text, will lead to a radical change of the writing form towards an unnamable. At least one thing is sure. The initiative of the users will pass through the recognition of visual elements of the screen. The writing will be nourished with its new plays on the spectacle, speech, and gesture towards the musical dimension of the rhythm. In this regard, the question of synesthesia can be raised: is it legitimate to claim synesthesia as one of the effects achieved by hypermedia? With the support of Morrison’s arguments, one might recognize that McLuhan’s notion of synesthesia as the simultaneous interplay of the senses in ratio cherished by the particular medium or media involved is missing in the theories of hypermedia, which dismiss all sensory phenomena to visual domain and overlooks the interplay between orality and literacy (cf. Morrison 2000). In fact McLuhan discovered the hidden connection between digital representations of reality and a sharpened ability to involve all the senses, but in a paradoxical way that returns modern consciousness to a preliterate mode of awareness. It is important to note that to Bolter synesthesia is a strictly a spatial phenomenon and exists purely a set of visual relationship between media, no matter what their origin. I think it is right to say that the primary reason for differences between Bolter and McLuhan lies in the fact that Bolter many proponents of hypermedia inspire their theoretical foundations from mono channel based semiotics which tend to treat all linguistic phenomena as static signs, whether they be truly static, as in writing or print, or dynamic, as in speech. Morrison also remarked that lacking an awareness of the kinesthetic interplay between the oral and the visual in reading, semioticians have neither a physiological nor an aesthetic theory that underlie any conceptions they might have of synesthesia. Here, it is worthy to evoke thoughts of André Leroi-Gourhan which are diametrically opposed to McLuhan interpretation of electronic media, he remarked that the fate of writing is now in a new stage in observing the audio-visual media. According to him the emergence of sound recording, films, and televisions in the past half-century forms part of a trajectory that began before the Aurignacian. From the bulls and horses of Lascaux to the Mesopotamian markings and the Greek alphabet, representative signs went from mythogram to ideogram, from ideogram to letter. Material civilization rests upon symbols in which the gap between the sequence of emitted concepts and their reproduction has become ever more narrow. This gap or interval is narrowed still further by the recording of thought and its mechanical reproduction (Leroi-Gourhan 1964: 261-300). Sungdo Kim 300 A radical change occurred, however with the coming of sound film and television, both of which address the faculties of sight, motion, and hearing at the same time and to induce the whole field of perception to participate passively. The margin for individual interpretation is drastically reduced because the symbol and its contents are almost completely merged into one and because the spectator has absolutely no possibility of intervening actively in the real situation thus recreated. The present stage is characterized simultaneously by the merging together of the auditive and the visual, leading to the loss of many possibilities of individual interpretation, and by a social separation between the functions of symbol making and of image receiving. Indeed, in primitive societies mythology and multidimensional graphism usually coincide. The forms of thought that existed during the longest period in the evolution of Homo sapiens seem strange to us today although they continue to underlie a significant part of human behavior. Our life is molded by the practice of a language whose sounds are recorded in an associated system of writing. A mode of expression in which the graphic representation of thought is radial is today practically inconceivable. Behind the symbolic assemblage of figures in the paleolithic art, there must have been an oral context with which the symbolic assemblage was associated and whose values it reproduced in space. Leroi-Gourhan believes that human potentiality would be more fully realized if we achieved a balanced contact with the whole of reality, therefore, we may ask ourselves whether the adoption of a regimented form of writing that opened the way to the unrestricted development of technical utilitarianism was not a step well short of the optimum. Cross-cultural analysis of semiotic syncretism: webpage, medieval illumination, East Asia’s painting 1. Illumination of Middle Ages The Middle Ages form a neatly defined historical period, but the outside limits for medieval manuscripts are determined by three crucial changes in the methods of book production: the use of parchment as a new matter of book, the appearance of codex, and the invention of printing. The first was the invention of the book as a more or less rectangular object with pages. Most ancient Roman literature was first written down on scrolls. The great change took place in the first century AD: the scroll gradually gave way to the book, or codex in the modern sense. This development coincides with the beginning of what are called the Middle Ages. The parchment, this new medium which would soon record the memory of a world of Middle Ages determine also the future of the book in its form as well as in its development. The codex of parchment represent in the West the form which, beyond the appearance of a new material (the paper) and the development of new techniques (printing and engraving), is still that of today’s books. If almost from its birth, printing constituted an industry capable of reproducing infinitely texts and images, the medieval book, because it is product of craftsmanship, is always a unique example, although this craftsmanship was organized until an extreme separation of tasks. The knowledge was transmitted thanks to the work of copyists who copied, commented, annotated and illumined existent texts. Manuscripts were the medium for the entirety of the Scriptures, liturgy, history, literature, law, philosophy and Syncretic Semiotics of Hypermediality 301 science from the classical and medieval ages. They preserved the major portion of medieval painting and all the arts of handwriting, bookbinding, and publishing (cf. de Hamel 1994). As artisans, these copyists invented some new graphic forms, and new writing forms. As artists, they create pieces of work by their illuminations. Because the illumination is more than a simple illustration of texts, the paintings which appeared on the manuscripts of parchment as early as the high Middle Ages participate actively in the history of written literature. In the end of Middle Ages, the advent of printing detached the painting from the text, which signaled the death of a millenary dialogue between texts and images, a dialogue which left in place the autonomous painting in one hand, and printed text (illustrated or not) in the other hand, which multiplied faithfully the same text, the same example. A number of parchment books reveal the continuity of their history and of their techniques, a history of another time, of another world. If the Egyptian papyrus of the high antiquity presents illustrations of multiple functions from the simple decoration of text to explanatory commentary, some papyrus of late Hellenistic Egypt integrates already their images in the colonies of writing. This constraint imposes images of reduced dimensions; this situation constitutes a specific feature of future illuminations of parchment. The Western papyrus, rare and often fragmentary, makes it very difficult to reconstitute the relations between text and image. The painting of parchment extended considerably from the 6 th century, in the Byzantine air. While the term of illumination insists on the light and the ornamental specificity of the image, the term of miniature refers to the red colour (de minimum, red pigment was utilized very early), to this meaning other meanings of small dimensions and minute detail were combined from the 19th century. Contemporary descriptions of manuscripts generally did not mention illumination. Some scholars just remarked on the function of decoration in a twelfth century book. The twelfth century was an age which delighted in the classification and ordering of knowledge. Its most admired writers, men like Peter Lombard and Gratian, arranged and shuffled information into an order that was accessible and easy to use. The fact that monks began making library catalogues at all reflects this fascination with order and accessibility of universal knowledge. It is essential to consider book illumination in these terms. It becomes easy to understand. Initials mark the beginnings of books or chapters (Figure 2). “They make a manuscript easy to use. A bigger initial is a visual lead into a more important part of the text. It helps classify the priorities of the text. Like the use of bright red ink for headings (…), coloured initials make a massive text accessible to the reader. A newspaper does this today with headlines of different sizes. In fact, a modern popular newspaper is a good example of thoroughly accessible text and uses very many of the devices of a twelfthcentury illuminated manuscripts: narrow columns, big and small letters, running-titles and, above all, pictures. These help explain a written text visually, they provide a reminder of a familiar image, they help the user to choose which sections to read next, they make for a satisfying page layout, and they can be amusing” (de Hamel 1994: 98) It is important to remark that the image is a graphic unity and that the page is a sphere, or rather, a format. Consequently between the image and the page are established symbolic relations of dependence. This is an affair of territory which marks the history of the book. In this regard the medieval manuscripts are celebrated where, according to diverse modes, the design illuminates the sacred texts: surprising paintings where the interlaces of letters with the images are commonly admitted. Sungdo Kim 302 <Figure 3> <Figure 2> Syncretic Semiotics of Hypermediality 303 I here signal these extraordinary initials of “Sacramentaire de Limoges” 2 painted in 12th century. The letters in question participate in grotesque sensibility which desire that the contortions of lines make up the alphabets. These are the Latin words, Te igitur (So you), which the illuminator distributed in two pages and represented in a way that the reader must make a real effort to discern these letters (T-E/ I-G-I-T-U-R) among the monstrous animal images which form the body of these letters (Fresnault-Deruelle 1999: 139). This is a claimed singular continuity that later the standardization of characters of the printing has forbidden. It is in the 20 th century that the wedding of the letter and image will be again celebrated (Figure 3). 2. Syncretic semiotics of the East Asian visual tradition In the representational art of the West, the pictorial space may be formal, but it is usually not rigidly codified. At least from the Renaissance to the 19th century, paintings have accomplished the strategy of transparency - to provide the viewer with a window onto the real world. Artists and their viewers supposed that the space of the picture should reflect the space of nature. In a verbal text, where the visual space is wholly conventional, learning to read means learning the conventions of that space. Pictorial space and verbal space are therefore apparent opposites: one claims to reflect a world outside of itself while the other is arbitrary and self-contained. The situation becomes more complex when painters put words into the space of their pictures - not a continuous practice in Western art, but common in East Asia’s visual traditions. In these traditions, the juxtapositions of word and image created a subtle harmony. In Chinese writing, for example, there was an intimate relationship between image and text. Chinese writing could blend smoothly with illustrative images both visually and conceptually. In this regard, François Cheng provides an excellent explanation. Peindre, écrire : deux chemins pour eux également libérateurs. Deux chemins qui à leur yeux se distinguent à peine. Il n’est pas exagéré de dire que l’écriture idéographique, qui exploite comme aucune autre la richesse visuelle, la charge imaginaire du signe, a été au départ la grande chance de la peinture chinoise, et comme son tremplin. Les idéogrammes chinois, composés de traits subtilement assemblés, constituent un inépuisable répertoire de formes signifiantes auxquelles l’artiste, de par sa maîtrise du pinceau, peut prêter à volonté un surcroît de force et de beauté (Cheng 2004: 10). The privileged relations that text and image hold in the universe of East Asia are, in part, due to the importance of the notion of space in the tradition where the writing and the painting have never lost their original proximity, and in another part, due to the development of the xylography which allowed the maintenance of the absolute flexibility of the mise en page (Figure 4). While in the West the passage to the printed book might upset and rigidify the treatment of the image, in East Asia the printing tradition shows a remarkable continuity with the manuscript tradition, the impression of the text and that of the image belonging all together to the same procedures of fabrication. As mentioned, the notion of space is essential in the Chinese writing: each character occupies virtually a square and every copied text is inscribed in the rigorous geometry. Sometimes, the Chinese copyist is pleased to make images with text. In many cases, the Sungdo Kim 304 <Figure 4> images are peacefully accompanied by legends and commentaries. One cannot find out any discontinuity between the manuscript tradition and the printed tradition, because the plate of impression is engraved after a manuscript established to this effect. The same liberty is offered to the copyist to dispose on his paper-plate, text, diagram, geometric figure or design. The osmosis is total between text and image, whatever the different modalities are: images integrated to text, text included in the image, etc. Let me consider the lettered painting of East Asian tradition. The Chinese civilization was privileged in the ideographic system in its most bizarre principle which dominated the rule of composition of its characters. Issued from the association of a key (silent sign) and of a phonogram (sonorous sign) that their reciprocal contamination gathers in a new unity, the Chinese ideophonograms are led to the meaning by the combination of heterogenous elements. Their optical proximity entailed to a mutual metamorphosis in a third through the act of their lecture. It is this archaic visual law of inductive neighborhood which they discovered that is also founded in the writing of the Chinese letters which exalt in their landscape paintings. They restituted its original magnetic virtuality to the medium of their compositions and they enriched that medium so that this virtuality contributed to the graphic expressionism from the birth of the writing. First of all, calligraphy, the first phase towards the creation of a spectacle, guided the gaze and the thought beyond the words and their substance. The representation of natural elements which, treated themselves as calligraphies became in their turn bearing the graphic memory of the sense. The words seem to be trying to transform the world of the picture into a writing space, while at the same time the picture invites the viewer to consider the words as images or abstract shapes rather than conventional signs. Here are two examples. Among the numerous puns of letters (moji asobi) attested in Japan, the moji-e (image composed of letters) is distinguished by its discretion and perennity. Practiced in particular in the era of Edo (1603-1868) this genre consists in figures realized with the aide of characters, kana and kanji. Every image is composed of a small scene which will present familiar persons realized by the moji-e (Figure 5). For another example, let me introduce Munjado, Korean ideograph painting. This mid 19th century Korean folk painting represents the Confucian values of honor and justice. Ink and slight color on paper, it is matted with silk and framed with a traditional Korean rounded frame. The ideograph, called munjado, is a stylized graphic rendering of a Chinese character. Chinese characters, rather than Korean Hangul characters, were used in Korean arts for many years after the country’s official language became Hangul. Munjado can be considered as a typography of the East (Figure 6). Syncretic Semiotics of Hypermediality 305 <Figure 6> <Figure 5> Sungdo Kim 306 In “Remediation” Bolter and Grusin (1999) examine historical developments in print, visual, and electronic media, beginning with the Renaissance. They analysed the ways in which the new media repurposed older media, and they provided a cultural and historic sensibility by complicating oversimplified categorizations of the differences between visual images and textual utterances by directing attention to the existence of very different styles and strategies of approach in the visual media. In this process they distinguish between two approaches to visual media. One style corresponds to the traditional view of visual media in suggesting immediacy and transparency and in presenting a unified visual space. The other style openly presents signs of mediation. Termed “hypermediacy” by Bolter and Grusin, this style brings awareness of the media along with some distancing from it. My project here extends this historical perspective to other traditions in the history of writing. I would argue that the style of visual media seen here in the visual traditions of East Asia is a hypermediated style. 3. A new semiotic landscape of hypermedia 3 In hypermedia one has a spectrum of print styles that reflects every possible visual relationship between word and image: from traditional nonfiction in which the word dominates the image to art books or advertising billboards in which the image dominates. This spectrum is too varied to explore in detail here. The main point is that the relationship between word and image is becoming increasingly dynamic and unstable, and that this dynamic instability is especially apparent in the graphic forms of various popular webpages. The emergence of the ‘new’, that is, the hypermedia, has created a situation in which the relationship of the textual and the visual is being problematized in ways even more radical than that which was announced by the discussions accompanying the rise of visual culture. The hypermedia morphology has modified the structure of the verbal and written text in such a way that writing seems indeed to have recovered its fundamental visuality. Reading even plain texts on a screen is not just reading, it is experienced as looking at characters whose visual materiality is foregrounded to the very discomfort of the reader. It is a very surprising fact that hypermedia writing shares qualities with both post literate and preliterate picture writing. By combining alphabetic writing with images sound diagrams, hypermedia designers are defining the computer as a new writing space that hesitates between intuitive and abstract modes of representation. An electronic text, such as a web page, may now be a dissemination of alphabetic signs among picture elements with various sizes and functions, static or animated elements that address the writer and reader without reference to speech. Therefore, the verbal text must compete for the reader’s attention with a variety of pictorial elements, any or all of which may be in motion. Although pictures and verbal text have been combined in previous technologies of writing, the spoken word has perhaps not faced such determined visual competition since the introduction of the phonetic principles. We can see the selective and animated qualities of the computer’s picture writing in the now familiar desktop, graphical user interface. The crucial element of the desktop GUI is the icon, which, although often named, is above all a picture that performs or receives an action. These actions give the icon its meaning. As elements in a true picture writing, icons do not merely remind the user of documents and programs, but function as documents and programs. Syncretic Semiotics of Hypermediality 307 I have identified a process of reciprocal remediation between visual image and verbal text. Graphics in printed publications are being used to replace text. They seem to bubble out of the prose and appear before our eyes, transforming us from readers into viewers. This process, as old as rebus poetry, is renewed for us by the techniques of animation and interactivity in hypermedia. The reciprocal process is equally old, in which prose tries to represent images, sounds or sensory experience. In ancient rhetoric, this technique was called ekphrasis, and in ancient rhetorical practice and still today, we can discern the ambivalence that belongs to ekphrasis as a strategy for remediation. Ekphrasis sets out to rival visual art in words, to demonstrate that words can describe vivid scenes without recourse to pictures. Ekphrasis also indicates that the writer is preoccupied with the visual, for in order to rival the visual, the prose must become descriptive in an effort to find the equivalent of what is naturally a visual experience. The breakout of the visual, the ekphrastic impulse, is at its most vigorous in the hypermedia writing space, where new media designers and authors are also redefining the balance between word and image. Like the older visual media of photography, film, and television, hypermedia remediate the book, the newspaper, and the magazine by offering a space in which images can break free of the constraint of words and present their own narrative. Designers of hypermedia employ images as well as sound in an effort to provide a more authentic or immediate experience than words alone can offer. This strategy of remediation cuts deeply into the history of writing itself - beyond alphabetic writing to earlier forms. Hypermedia can be regarded as a kind of picture writing, which refashions the qualities of both traditional picture writing and phonetic writing. In short, the problems raised by the use of illustrations in contemporary electronic writing are not totally new. On the contrary, they continue discussions linked with the emergence of visual culture, and they reactualize the still ongoing fundamental debates on the status and positions of the image itself. For all these reasons, this paper has adopted a more historical and comparative point of view, focusing on general problems of visuality. Conclusion To briefly enunciate some of the arguments presented here, five points can be made. Firstly, in terms of communication theory, modern semiotics shows a reductionist model in postulating the ontological autonomy of sign and in ignoring the multichannel communication. Secondly, from the history of sign systems, namely, from the archeology and evolutionary perspective, the syncretic sign systems are very common and frequent phenomena. I furnished many evidences from cross cultural sources (illuminations, lettered paintings, webpages, etc.) Thirdly, in terms of the biological mechanism of human perception, human beings rely on the unity and synthesis of different senses to understand the external phenomena. This point is essential, because it is associated with the common senses or synesthesia which might be a capital component of syncretic semiotic program. The importance of synesthesia (the physiological sensation on a part of the body other than the stimulated one) can never be emphasized enough in the study of the sign-exchange processes of social interaction, either between fully-equipped individuals or in situations of reduced interaction. Sungdo Kim 308 Fourthly, I want to mention the hypermedia effect which could be characterized by the fusion of different sign systems. This phenomenon of the synthesis of aurality, visuality, textuality and tactility which is manifested in digital language could suggest a return of our senses to the primitive age, as McLuhan has stated. In other terms, this phenomenon signifies the synaesthesic recovery of ear and eye. In the same vein the great prehistorian, André Leroi-Gourhan, interprets the multimedia language as a recovery of concrete contact with things which were lost after the invention of alphabets which are based on abstractions of the human mind. Finally, syncretic semiotics, being very flexible, can be applied to the theory of remediation. Like the principle of remediation, syncretic semiotics does not accept the logic of ‘ceci tuera cela’: one new medium will kill the old one. This envisages the coexistence and harmony of different media by introducing a theory of intermediality. Syncretic semiotics must apprehend multimediality, intermediality, and hypermediality. In this regard we should broaden this perspective into the humanistic traditions concerning the multimedia. The aim of this essay has been to draw attention to some aspects of the fusion and syncretism of different kinds of sign systems which are manifested in different media. For this I have adopted the perspective of syncretic semiotics and underlined a historic and comprehensive sensitivity to the problematics of hypermediality. In contrast to the revolutionary assumptions about hypermedia as breaking completely with precedents in print and other older forms of media, I tried to show some of the historical continuities and other cultural universals. To understand the cultural effects of any new hypermedia forms that might be developed, however, we need to understand better the ways that language interacts with other media. Such interaction is by no means a totally new phenomenon but in fact fundamental to the Middle Ages and East Asia. In the fields of semiotics and hypermedia theories, the inherent multimodal aspects of the history of writing have relatively been disregarded. As I begin to address the relation of text and visuals in the multimedia culture, it is the duty of syncretic semiotics to examine the cross cultural roots of such multimodal and hypermedial representations. Reference Alpers, Svetlana 1983: The Art of Describing, Chicago: University of Chicago Press. Barthes, Roland 1978: Image, Music, Text, New York: Hill and Wang. Benveniste, Emile 1974: Problèmes de Linguistique Générale II, Paris: Gallimard. Birkerts, Sven 1995: The Gutenberg Elegies: The Fate of Reading in the Electronic Age, Winchester, MA: Faber and Faber. Bolter, Jay David 1991: Writing Space: The Computer, Hypertext and the History of Writing, Hillsdale, NJ: Lawrence Erlbaum. Bolter & Grusin 1999: Remediation: Understanding New Media, Cambridge, MA: MIT Press. Burwick, Frederick (ed.) 1999: “Lessing’s Laokoon, Context and Reception”, in: Poetics Today 20.2 (1999). Cheng, François 2004: Toute beauté est singulière, Paris: Phébus. Christin, Ann-Marie 1995: L’image écrite, Paris: Flammarion. Christin, Ann-Marie 2000: Poétique du blanc. Vide et intervelle dans la civilisation de l’alphabet, Leuven: Peeters/ Vrin. Christin, Ann-Marie (ed.) 2001: Histoire de l’écriture, Paris: Flammarion. Derrida, Jacques 1967: De la grammatologie, Paris: Éd. de Minuit. Cotton & Oliver 1997: Understanding Hypermedia, London: Phaidon Press Limited. Syncretic Semiotics of Hypermediality 309 Eco, Umberto 1975: A theory of semiotics, Bloomington: Indiana University Press. Eisenstein, Elizabeth 1991: Le Révolution de l’imprimé dans l’Europe des premiers temps modernes, Paris: La Découverte. Fabre, Daniel (ed.) 1997: Par écrit. Ethnologie des écritures qoutidiennes, Paris: Éd. de la Maison des Sciences de l’Homme. Floch, Jean-Michel (ed.) 1983, “Sémiotique Syncrétiques”, in: Actes Sémiotiques - Bulletin VI, 27, Paris: EHESS- CNRS. Flusser, Vilém 2004: Writings, Minneapolis: University of Minnesota Press. Fresnault-Deruelle, Pierre 1999: “L’image et la page”, in: Anne Zali (ed.) 1999. Gelb, Ignace J. 1973: Pour une théorie de l’écriture, Paris: Flammarion. Goody, Jack 1979: La Raison graphique, Paris: Éd. de Minuit. Greimas, A.J. and Courtés, J. 1979: Sémiotique. Dictinnaire raisonné de la théorie du langage, Paris: Hachette. Greimas, A.J. and Courtés, J. 1979: “Sémiotique”, in: Greimas, A.J. and Courtés, J. 1979. Griolet, Pascal 2001: “L’écriture au Japon”, in: Christin (ed.) 2001: pp.122-141. de Hamel, Christopher 1994: A History of Illuminated Manuscripts, New York: Phaidon Press. Hayles, Katherine 1999: How We Become Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics, Chicago: University of Chicago Press. Hayles, Katherine 1999: “Virtual Bodies and Flickering Signifiers”, in: Hayles (1999): pp.25-49. Hjelmslev, Louis 1961: Prolegomena to a Theory of Language, Madison: The University of Wisconsin Press. Hocks, Mary E. and Kendrick, Michelle R. (eds.) 2003: Eloquent Images: Word and image in the age of new media, Cambridge, MA: MIT Press. Jameson, Frederic 1991: Postmodernism, or, The Cultural Logic of Late Capitalism, Durham, NC: Duke University Press. Jeanneret, Yves: “Écriture et multimédia”, in: Christin (ed.) 2001: pp.386-394. Kress, Gunther 1998: “Visual and Verbal Modes of Representation in Electronically Mediated Communication: The Potentials of New Forms of Text”, in: Snyder (ed.) 1998: 53-79. Landow, George P. 1992: Hypertext: The Convergence of Contemporary Critical Theory and Technology: Baltimore, Johns Hopkins University Press. Lanham, Richard A. 1992: The Electronic Word: Democracy, Technology and the Arts, Chicago: University of Chicago Press. Leroi-Gourhan, André 1964: Le Geste et la Parole,2 vol., Paris: Albin Michel. Levinson, Paul 1999: Digital Mcluhan: A Guide to the Information Millennium, New York: Routledge. Lévy, Pierre 1994: L’intelligence collective: Pour une anthropologie du cyberspace, Paris: La Découverte. Manovich, Lev 2001: The Language of New Media, Cambridge, MA: MIT Press. McLuhan, Marshall 1964: Understanding Media: The Extensions of Man, New York: McGraw Hill. Mitchell, W.J.T. 1985: Iconology, Chicago: University of Chicago Press. Mitchell, W.J.T. 1994: Picture Theory, Chicago: University of Chicago Press. Morrison, James C. 2000: “Hypermedia and Synesthesia”, in: Proceedings of the Media Ecology Association, Inaugural Convention, Fordham University, New York, NY, June 16-17, 2000. Peirce, Charles S. 1936-1956: Collected papers, vols. 1~6 ed., Charles Hartshorne and Paul Weiss, vols. 7-8 ed., Arthur W. Burks, Cambridge, Mass.: Harvard Univ. Press. Pigeot, Jacqueline 2001: “L’écriture dans l’image”, in: Christin (ed.) 2001: pp.142-144. Simon-Oikawa, Marianne 2001, “Des images en écriture”, in: Christin (ed.) 2001: pp.145-147. Snyder, Ilana (ed.) 1998: Page to Screen: Taking Literacy into the Electronic Era, London: Routledge. Tufte, Edward 1997: Visual Explanations: Images and Quantities, Evidence and Narrative, Cheshire, CT: Graphics Press, 1997. Zali, Anne (ed.) 1999: L’aventure des écritures: La page, Paris: Bibliothèque nationale de France. Notes 1 “The denotative semiotic, by which we mean a semiotic none of whose plan is a semiotic. Through a final broadening of our horizon, to indicate that there are also semiotics whose expression plan is a semiotic and semiotics whose content plane is a semiotic. The former we shall call connotative semiotics, the latter metasemiotics.” (Hjelmslev 1961: 114). Sungdo Kim 310 2 Sacramentaire de Limoges Limousin, born in 12nd century, BNF, Manuscripts occidental, Latin 9438, Fos 59v- 60. 3 This section refers basically to the observations of Bolter’s work (Bolter 1991: 47-98).