Mediation vs. Representation and Transmediation vs. Media Representation

My main distinction, which is rarely highlighted in intermedial relations studies, is between mediation and representation. If these two notions are conflated, it becomes difficult to discern certain important media transformation stages and aspects (^{Elleström, 2013}).

Mediation, as I define it, is a pre-semiotic, physical realization of entities (with material, sensorial, and spatiotemporal qualities, as well as semiotic potential) perceived by human sense receptors within a communication context.

Representation, as discussed here, is the creation of meaning in the perceptual and cognitive acts of reception. To say that a media product represents something is to say that it triggers a certain kind of interpretation. This interpretation may be more or less hardwired in the media product and the way one perceives it, but it never exists independently of the recipient’s cognitive activity. When something represents, it calls forth something else; the representing entity makes something else, the represented, present to the mind. As noted by Charles Sanders ^{Peirce (1960}), a sign, or a representamen, stands for an object.¹ Representation, the very essence of the semiotic, is often a result of mediation.

The notion of mediation thus foregrounds the material realization of the medium, whereas the notion of representation highlights the semiotic conception of the medium. This distinction is helpful for analyzing complex relations and processes. In practice, however, mediation and representation are deeply interrelated. Every single representation is based on a specific mediation’s distinctiveness. Furthermore, some kinds of mediation facilitate certain sorts of representation while rendering other sorts impossible. For example, vibrating air emerging from vocal cords and lips, perceived as sound, is well suited for iconic representation of bird song, though such sound cannot possibly form a detailed, three-dimensional iconic representation of a cathedral.

Hence, I use the term mediate to describe the process of a technical medium realizing potentially meaningful sensory configurations: a book page can mediate, say, a poem, a diagram, or a musical score. If equivalent sensory configurations (that is, those that have the capacity to trigger corresponding representations) are mediated for a second (or third, or fourth) time by another kind of technical medium, they are transmediated: the poem that was seen on the page can later be heard when it is transmediated by a voice. In other words, the content of the poem is represented again by a new kind of sensory configuration (not visual, but auditory signs), mediated by another kind of technical medium (not a book page emitting photons, but sound waves generated by vocal cords).

The concept of transmediation involves not only re-mediation, that is, repeated mediation, but also repeated mediation of equivalent sensory configurations by another kind of technical medium (please note that the term remediation, as used here, should not be understood in Bolter and Grusin’s open-ended sense). Hence, the composite term transmedial remediation would be more accurate for the concept in question. For the sake of simplicity, I prefer the brief term transmediation.

All transmediation involves some degree of transformation: the equivalent sensory configurations and the corresponding representations that they trigger may be only slightly different and clearly recognizable, but they may also be profoundly transformed (for example, musical narratives based on literature differ very much from their sources).

Transmediation is the first kind of media transformation. Also, media representation, the second kind of media transformation, involves modification in the transfer process. Such media representations as, for instance, a news article describing a documentary, or a photograph depicting a dance performance, should be understood as potentially representing both a medium’s form and its content; media representation is at hand when one medium presents another medium to the mind. A medium, something that represents, becomes itself represented.

The distinctive features of transmediation and media representation can be seen in Figure 1. The M circles should be understood as media products or, more specifically, as technical media mediating sensory configurations. The C, which is placed within an M, should be understood as media characteristics represented by the sensory configurations. A circle and its interior is thus a depiction of both the mediation and the ensuing representation. The T arrows represent the transfer acts between two media products: the source medium (M1) and the target medium (M2). M2 is thus a new technical medium mediating more or less different sensory configurations compared to M1.

Source: Own elaboration.

Figure 1 Transmediation and media representation 

In the case of transmediation, the target medium (M2) represents the same content (C1) as the source medium (M1); in the case of media representation, the target medium (M2) represents the source medium (M1), which means that the source medium constitutes the media characteristics of the target medium (C2 = M1). As represented content of M2, M1 still represents C1. In other words: In the first media transformation case, the target medium (M2) transmediates (represents again) the source medium characteristics (M1). In the second case, the target medium (M2) represents the source medium (M1). In both cases, the source medium’s characteristics (C1) must be understood to remain the same, to a certain extent, after the transfer from M1 to M2.

Consequently, media representations may often be understood also as transmediations if they include, to some degree, a repeated mediation of equivalent sensory configurations. There is no contradiction between a target medium representing, on the one hand, a source medium, and, on the other hand, mediating sensory configurations equivalent to those of the source medium. This might be inferred from the diagram in Figure 1. A photograph representing a drawing of three kittens is obviously a medium representing another medium, but it also clearly includes repeated mediation of not only equivalent, but actually very similar (visual) sensory configurations by another technical medium. An auditory, verbal description of a drawing such as “I bought a drawing of three cute kittens” is also a case of media representation, but since it includes repeated mediation of equivalent sensory configurations by another technical medium (the voice is able to produce symbolic signs that represent substantial parts of the objects represented by the iconic signs on the paper: the notion of three kittens), it also includes transmediation.

Both these examples may be understood as comparatively complex instances of media representation and it is clear that if a medium is represented in some detail, the represented source medium characteristics become transmediated by the transfer target. However, a very simple verbal representation such as “I bought a drawing” is a media representation, but not a transmediation; C1 (in this case, the represented kittens) is not represented again. Hence, in pure media representation, only M1, the empty shell of the source medium, so to speak, is transferred to M2. In pure transmediation, only C1, the source medium content, is transferred to M2. Often, both M1 and C1 are transferred, which means that both media representation and transmediation are present. To be strict, then, the diagram representing media representation in Figure 1 actually depicts media representation including transmediation. Pure media representation should be depicted as in Figure 2.

Source: Own elaboration.

Figure 2 Pure media representation 

Other distinctions that should be kept in mind are that both transmediation and media representation can involve, on the one hand, specific media products (which has been assumed so far), and, on the other hand, general characteristics of qualified media. Qualified media is a term I use to denote all kinds of abstract media categories-both artistic and non-artistic-that are historically and communicatively situated, meaning that their properties differ depending on time, culture and aesthetic preference. Qualified media include such categories as music, painting, television programs, and news articles. A qualified medium is made up of a cluster of concrete media products (^{Elleström, 2010}). Media products can represent other specific media products, as well as general characteristics of qualified media, both of which can be transmediated (Elleström, 2014). While a novel may describe a particular piece of music, it may also discuss and, hence, represent music in general. A poem may transmediate characteristics of a specific musical piece, but it may also transmediate general musical characteristics, such as formal traits. Hence, the diagrams in Figure 1 can also be extended to illustrate transmediation and media representation involving general media characteristics, in which case M must be understood as the idea of a qualified medium and C as general media characteristics.

Certain types of complex, specific media product representations are commonly called ekphrasis. Whereas an ekphrasis is typically understood to be a poem representing a painting, the notion has been extended substantially during the last decades (^{Yacobi, 1995}; ^{Clüver, 1997}; ^{Bruhn, 2000}; ^{Sager Eidt, 2008}). Ekphrasis, in turn, is only the tip of the iceberg of media representation.

The general term for transmediation of media products is “adaptation” (^{Elleström, 2013}). While the archetypal adaptation is a novel-to-film transmediation, the term has not been reserved exclusively for this type of transfer (^{Elliott, 2003}; ^{Hutcheon, 2006}; ^{Bernhart, 2008}; ^{Urrows, 2008}; ^{Schober, 2013}). Furthermore, far from all types of transmediation of specific media products tend to be called adaptation. Transmediations from libretti, scores, scripts and so forth, and transmediations from written, visual and verbal text to oral, auditory and verbal text (aloud readings of texts), or the other way round, to mention only a few examples, are very seldom referred to as “adaptation” (see, however, ^{Groensteen, 1998}).

The Transmedial Basis

So far, transmediation and media representation have been discussed without really asking how these phenomena are at all possible. Which features are involved in the transformational processes encompassing several media and how are they related? Initially, it must be restated that no medium can fully transmediate or represent all media. Both qualified media and individual media products have dissimilar basic properties; these differences set the limits for what can be transmediated or represented. In addition, transmediated and represented media characteristics are not equally transmedial; while certain traits are almost universally present in the media landscape, others can only be marginally transformed to fit other media.

I refer to this wide range of media features as the transmedial basis. The questions are: which characteristics can be transmediated or represented by other media, and why? As media characteristics are often results of contextualization and complex interpretive practices, this question can be treated systematically only to a certain extent.

The most elementary transmedial basis consists of what I have called elsewhere the “four modalities of media”-the material, the sensorial, the spatiotemporal and the semiotic (^{Elleström, 2010}). A modality should be understood as a category of related characteristics that are basic in the sense that all media can be described in terms of all four modalities. All individual media products, and all conceptions of qualified media, may be understood as specific combinations of modes of the four modalities. The modes of the modalities do not cover all media characteristics-far from it-but they constitute a sort of a skeleton upon which all media are built.

The four modalities of media and, more specifically, the modes of the four modalities, thus constitute an essential transmedial foundation. The flat surface, being a mode of the material modality, is an aspect of printed novels, etchings, posters, television news and so forth, and is a prerequisite for comprehensive and close transmediation of, say, a graphic novel to a motion picture. The audible, a mode of the sensorial modality, is an aspect of radio theatre, opera, ordinary conversation, gamelan music and many other types of qualified media. The best way of faithfully representing sound media characteristics is to produce similar sounds.

Temporality, which must be understood as a mode of the spatiotemporal modality, is an aspect of recited poetry, theatre and television commercials, but not of oil paintings and printed tourist brochures. While all media are perceived in time, only some media are temporal in themselves. Transmediation often involves media that are either temporal or non-temporal. Graffiti is easily and faithfully transmediated by still photographs, whereas stills can only partially transmediate dance. However, some qualified media, such as most written, visual literature, are conventionally decoded in fixed sequences, which makes them second-order temporal, so to speak, and hence well-suited for transmediation into temporal media, such as motion pictures.

Iconicity, a mode of the semiotic modality, is a vital aspect of creating meaning in media, such as newspaper advertisements, statistical graphs, rock music, and scholarly figures (such as the ones in this article). Iconic structures create meaning on the ground of resemblance; similarities can be established over both sensorial and spatiotemporal borders. For example, visual traits may depict auditory or cognitive phenomena, and static structures may depict temporal phenomena; that is, a graph may depict both changing pitch and altering financial status. In general, iconicity interacts with the two other main modes of the semiotic modality: indexicality (meaning created by contiguity) and symbolicity (meaning created by conventions). This well-known trichotomy originates in Peirce.²

The modes of the modalities are clearly necessary for identifying media similarities and differences and, consequently, essential for delineating processes of transmediation and representation, though they do not definitely determine their limits. It may be the case that shared modality modes facilitate extensive transmediation and representation, while some media are very difficult to transmediate or represent if the target medium does not possess vital modality modes. Nevertheless, due to the brain’s cross-modal capacities, transmediation and media representation over modality mode borders are, to a certain extent, possible, common and, indeed, productive.

To summarize, it is the material, sensorial, spatiotemporal and semiotic differences between source medium and target medium that allow for inventive alterations of media products into new creations. By the same token, modal differences make it impossible to transfer vital information without transforming it, as in news reports that include chains of interconnected media.

While the modes of the four media modalities are basic and universal transmedial characteristics, it is not the modality modes that are transferred in the processes of media transformation; rather, they are deeply integrated fundamentals that are required for forming what I call “compound media characteristics” (^{Elleström, 2014}). Many media products share modality modes, but each individual media product has a distinct set of compound media characteristics created by the arrangement of all details in the full medial expression, as discerned and construed by the media product’s perceiver. Compound media characteristics should be understood as features of media products that are apprehended and formed when a structuring and interpreting mind makes sense of the mediated sensory configurations.

Compound media characteristics may be strongly linked to certain modality modes, to a specific media product, or to a qualified medium; they may also be transmedial to a considerable degree, meaning that they can successfully be transferred among many kinds of qualified media. However, compound media characteristics can never be fully transmedial: the modal differences among dissimilar media always make a difference. Nevertheless, the mind has the capacity of recognizing similarities that bridge media variances.

Compound media characteristics that can be transferred among media can roughly be divided into aspects of form and content, which should be understood as a coarse distinction between comprehensive media characteristics and more confined elements.

Form includes all kinds of structure, manifested sensorially in what can be seen, heard or otherwise perceived, or as cognitive configurations. There is a wide range of aspects or derivations of structure: pattern, rhythm, balance, proportion, relation, repetition, similarity, and contrast, to mention only a few. In spite of their (sometimes) inaccessible nature, these notions have extraordinary longevity, which must mean that they correspond to basic perceptual inclinations and fulfil vital cognitive needs.

Other formal compound media characteristics are the intricate qualities of style and perspective. Lately, Werner Wolf has provided several volumes on formal transmedial characteristics, such as description and metareference (Wolf & Bernhart, 2007; ^{Bantleon, Thoss & Wolf, 2009}). One complex transmedial characteristic that I have investigated myself is irony (^{Elleström, 2002}).

When it comes to content, some compound media characteristics are directly perceptible from the material interface of the media product as the appearance of symbolic and iconic microstructures, such as visual or auditory words and sentences, and visual or auditory iconic details. Other compound media characteristics that have the nature of content are products of cognitive activity triggered by processes of representation and conceived as situations, spaces, places, persons, animals, objects, motifs and the like. All these characteristics are transmedial to a certain extent.

Narration, which includes aspects of both form and content, is one of the most important compound media characteristics. While narration is traditionally associated with literature and motion pictures, it has increasingly gained the status of a fundamental cognitive notion. Narration can be said to be an offspring of structured perception and spatial thinking. To narrate, and to interpret in terms of narration, is a way of creating meaning in sequential form. Narration is not limited to specific material or sensorial modes. Our two most cognitively developed senses, sight and hearing, are both well suited to narration, and all types of spatiotemporal configurations may display traits that can be connected to narrative sequences. Naturally, however, media that are temporal on the material level, such as movies and music, and those that are based on conventionally sequential sign systems, such as oral and printed literature, have an advantage when it comes to forming developed narratives. Furthermore, media that rely on advanced, symbolic sign systems (primarily verbal language), such as literature and motion pictures, are well suited to outline complex narratives. Certainly, many kinds of narratives can be transferred among media. The phenomenon has been investigated by Marie-Laure ^{Ryan (2004}), among others, who has explored what she calls “transmedial narratology”.

Plots and stories are two kinds of narrative sequential structures that can be more or less fully transferred among media. In addition, the storyworld, which includes an elaborated virtual space and should count as form, can at least partly be transferred among different media, as can portions of narratives, such as relations among particular characters or other figurations. Linda ^{Hutcheon (2006}) has listed a number of features that are certainly compound media characteristics and should be understood as transmedial narrative content: characters, motivations, consequences, events, symbols, and themes, among others.

It is clearly impossible to create an exhaustive list of transmedial compound media characteristics. Furthermore, media characteristic complexity makes neat classification very difficult. Nevertheless, the notion of compound media characteristics cannot be dispensed with if the idea of transfer among media shall be fully understood.

A Basic Formula for the Transfer of Media Characteristics

On the basis of Figure 1, I would like to propose a formula for the rudimentary traits of media transformation; this involves recapitulating the central issues presented so far. Since “trans” means “across” or “beyond”, the term must be understood to represent a spatiotemporal notion: compound media characteristics are transferred from one place to another. First, we read a novel in a book and then we see a motion picture and recognize it as more or less the same story. First, there is a sculpture placed on the square and then we see photos and read newspaper descriptions of it. Hence, media transformation can be captured in the formula “A compound transmedial Characteristics is Transferred from a source Medium to a target Medium”, or “C is T from M1 to M2”.

The transfer is either a transmediation or a media representation or a combination of the two. In either case, it involves some sort of transformation. However, in what follows, I will mainly discuss transmediation in light of one practical example to make the rudimentary aspect survey as clear as possible: William M. Thackeray’s 1844 novel The Luck of Barry Lyndon being transmediated into Stanley ^{Kubrick’s 1975} motion picture Barry Lyndon. This is a typical example of adaptation that I will refer to briefly in order to illuminate some standard transmediation traits.

Vis-à-vis Barry Lyndon, how should C, T, M1 and M2 be understood? M1 is the source medium, the “first place”, and M2 is the target medium, the “second place”. When thinking of media characteristic transfer in the most straightforward way, M1 and M2 are two particular media products, as in the case of Barry Lyndon, where the novel’s central, compound characteristics are transmediated by the movie. However, transmediation also occurs in cases where either M1 or M2 is a qualified medium or a submedium (genre) rather than a particular media product, which may be illustrated with the same example. One of the traits of the movie is voice-over narration. When seeing Barry Lyndon as a version of the novel, this voice-over is part of the general transmediation of The Luck of Barry Lyndon, yet one may also say that the voice-over is simply a novelistic trait of the motion picture. Indeed, there are many movies with voice-over that cannot be understood as transmediations of particular literary works. Nevertheless, they can be understood within the framework of the formula “C is T from M1 to M2”, the difference being that M1 is a genre of written literature, a qualified medium, and M2 stands for particular media products. The formula might then be “C is T from MQ1 to MP2”, where MQ means qualified medium and MP means a specific media product.

Furthermore, both M1 and M2 may be qualified media or genres (“C is T from MQ1 to MQ2”). It makes perfect sense to talk about, say, “novelistic traits in movies”, and indeed the transformation of The Luck of Barry Lyndon to Barry Lyndon may be understood as a particular instance of novelistic traits in movies.

As explained earlier, C represents transmedial compound media characteristics, such as form, structure, rhythm, narration, material, theme, motif, and so forth. Obviously, characteristics that are not transmedial cannot be transmediated by other media and not all kinds of transmedial characteristics can be transmediated by all media. In Barry Lyndon, a certain narrative form (the rise and fall of an ambitious man), many verbal microstructures, several characters, themes, and motifs, and probably many other characteristics can be said to be transmediated from the novel.

However, the book and the cinema or television screen are entirely different technical media, so the transmediation necessarily involves several modal changes: the auditory mode is added, the novel’s conventional sequentiality is transformed to the material temporality of the movie, the degree of iconicity of the visual surface increases dramatically, and so forth. Yet, vital narrative form aspects survive the transmediation from conventional sequentiality to material temporality: the verbal micro-structures are easy to transfer from the visual to the auditory; many of the qualities of the characters in the novel can also be expressed in the movie, since verbal language and images overlap to a large degree when it comes to what they can represent.

Finally, T is the transfer; it includes both transmediation and media transformation, of course, although the focus is on transmediation for the moment. I will here pay attention to three further aspects of T that are valid for both transmediation and media transformation.

The first aspect concerns the “thickness” of the T arrow and includes differences between what might perhaps be called more and less complete transfers, strong and weak transfers, and so forth. Conversely, one may perceive differences between degrees of transformation; a less complete transfer is likely to include a higher degree of transformation.

As we have seen in the Barry Lyndon example, a particular transmediation may involve several compound media characteristics. The more characteristics it involves, the closer the target medium is to the source medium. If very few transmedial characteristics are involved, it might be questioned whether there is a point in treating it as a transmediation at all. As noted, transmedial characteristics are, as a rule, more or less modified by the modal changes involved in a media transformation, which certainly has an effect on the perceived transfer strength.

The second aspect concerns the “direction” of the T arrow. In the straightforward standard transfer, the arrow points from M1 to M2 (as in Figures 1 and 2). This is how media representation must always be understood: one particular media product (M2) represents other media products or qualified media (M1). There is no question about what represents and what is being represented. In this respect, transmediation is more complicated. When both M1 and M2 are qualified media or submedia (genres), there is not always a point in saying that the compound media characteristics are definitely transferred from one place to another; the truth might rather be that they “circulate”, or go back and forth between MQ1 and MQ2 as in the development of forms and motifs in modern literature and film. The T arrow sometimes points in many directions.

The third aspect concerns the “extension” of the T arrow. Individual transfers must also be seen as parts of more far-reaching and complex networks involving many specific media products (MP3, MP4, and so on). The T arrow may be part of arrow chains, perhaps with weak and strong links or thin and thick branches. There may also be several source media that are transformed to one new target medium. The number of potential transmediation variations is probably endless. Figure 3 illustrates one example: two qualified media that are transmediated to one media product, as when an advertisement borrows traits from both concrete poetry and comics, or when a photograph has the appearance of both a classicist painting and a scene from theatre drama.

Source: Own elaboration.

Figure 3 An example of transmediation 

After this rather formalistic presentation of distinctions, diagrams, and formulas not intended to capture the phenomenon of media transformation in strict and endless subdivisions, but rather to make possible methodical analyses of a multi-faceted area that no doubt escapes neat classifications, I want to emphasize that there is necessarily a strong subjectivity element in all media transformation discussions. Hermeneutics can never be escaped. When finding traces of other media in media, whether they are specific media products or qualified media, it sometimes simply makes sense to say that some media should be treated as source media because they are recognizable in other media, which may then be treated as target media. Ultimately, theoretical analysis is nothing without interpretation.