8
Multimedia as an Indian Meal
theory
In the late 1950s, the emergence of the first programmable screens and lighting devices inspired artists from various disciplines to conduct creative experiments.
The interest in combining light, sound, and images to create new expressive approaches began to permeate all traditional art genres—from classical concerts to industrial exhibitions.
This phenomenon was then called intermedia—a term proposed by the artist and theorist Dick Higgins in 1965 to describe artistic forms on the border between different genres of representation. He used it to denote practices where the boundaries between painting, sound, theater, text, and technology were blurred.
Up until the 1980s, "intermedia" remained an area of active exploration for new ways of communication, including in exhibition and performative venues.
The Greek architect and composer Iannis Xenakis, in his musical performances and exhibition projects, particularly vividly combined behavioral scenarios, light, unusual locations, and sound.
The interest in combining light, sound, and images to create new expressive approaches began to permeate all traditional art genres—from classical concerts to industrial exhibitions.
This phenomenon was then called intermedia—a term proposed by the artist and theorist Dick Higgins in 1965 to describe artistic forms on the border between different genres of representation. He used it to denote practices where the boundaries between painting, sound, theater, text, and technology were blurred.
Up until the 1980s, "intermedia" remained an area of active exploration for new ways of communication, including in exhibition and performative venues.
The Greek architect and composer Iannis Xenakis, in his musical performances and exhibition projects, particularly vividly combined behavioral scenarios, light, unusual locations, and sound.
пример
One of the first multimedia projects was the Philips Pavilion, created for the 1958 Brussels World’s Fair. The architect Le Corbusier and the composer Iannis Xenakis conceived the pavilion as a single audiovisual experience: the hyperbolic architecture, sound, light, and projections worked as a unified system.
Viewers entered the dome-shaped space, where Edgar Varèse's audio work Poème électronique played, accompanied by hundreds of slide projections and pulsating light, all controlled by a pre-programmed scenario. The pavilion became an important milestone in the history of intermedia, influencing the development of multimedia scenography and installation.
Viewers entered the dome-shaped space, where Edgar Varèse's audio work Poème électronique played, accompanied by hundreds of slide projections and pulsating light, all controlled by a pre-programmed scenario. The pavilion became an important milestone in the history of intermedia, influencing the development of multimedia scenography and installation.
theory
Building on the phenomenon of intermedia, we propose to look at multimedia more broadly than just as a technology for the audiovisual digital presentation of data. In this chapter, we will consider multimedia as an approach to combining different formats, genres, and channels with the aim of enriching the viewer's experience.
8.1
Multi-sensoriality
In Indian cuisine, there is a kind of set meal—it is called a thali. A thali is served on a metal tray divided into many sections. Into each compartment, food with different taste characteristics is placed: warm and cold, sour and sweet, spicy and bland, bitter and savory. It is believed that the combinations presented on the tray should cover the entire spectrum of a person’s taste receptors, including tactility, as a thali is eaten with the hands.
example
This image of an Indian meal helps us to vividly represent the principle of multimedia.
theory
We define multimedia as the synergistic effect from the impact of different types of information, forms of representation, and communication channels. The juxtaposition and mutual influence of all components of a multimedia concept allow for the engagement of different channels of human perception and make the viewer’s experience voluminous and rich.
There is a theory about the extended system of senses. This system suggests that human perception is regulated not only by the basic six senses but also by more specific nuances of feeling. It is these finer settings of impact that are important for designing the viewer experience. Here are a few examples that help to illustrate the principle of the extended spectrum of senses.
There is a theory about the extended system of senses. This system suggests that human perception is regulated not only by the basic six senses but also by more specific nuances of feeling. It is these finer settings of impact that are important for designing the viewer experience. Here are a few examples that help to illustrate the principle of the extended spectrum of senses.
Proprioception — the sense of the position of the body in space and its parts relative to each other
1
2
3
Equilibrioception — the sense of balance, equilibrium, perception of acceleration
Kinesthesia — the sensation of body movement, muscular effort
Thermoception — the sensation of heat and cold
4
Sense of vibration — the perception of the body's oscillatory movements
5
8
Temporal perception — the sense of time: rhythm, duration, pause, waiting
Interoception — the perception of the body's internal states: heartbeat, hunger, breathing, anxiety
Empathic perception — the ability to read the emotions of other people, the work of mirror neurons
7
6
When creating an exhibition, we are designing a person's experience in space. Therefore, all features of the interaction of bodies in a physical environment are significant for us. For example, a person's ability to feel the passage of time. You have probably noticed that two video clips of the same duration can be perceived completely differently: one will seem drawn out, the other will fly by like a spark. Our perception in space is influenced by rhythms, contrasts, and nuances, expressed in the characteristics of both the environment itself and its content.
Material
1
2
3
Lighting
Dimensions
Shapes
4
8.1.1
Examples of physical space settings:
Intonation
1
2
3
Scale
Medium (communication channel)
Position in space
4
8.1.2
Examples of content settings:
example
The main exhibition of the 60th Venice Biennale, curated by Adriano Pedrosa, is titled "Stranieri Ovunque" ("Foreigners Everywhere"), which symbolizes the theme of marginality and foreignness.
The central image of the exhibition is the neon signs "Foreigners Everywhere" in multiple languages, created by the Claire Fontaine collective. They reflect the idea of the universality of foreignness, countering xenophobia and nationalist narratives.
The project itself combines visual, sound, and installation elements, which allows it to evoke in the visitor a sense of empathy, reflection, and engagement, thereby setting up an empathic perception and questions of identity and belonging.
The central image of the exhibition is the neon signs "Foreigners Everywhere" in multiple languages, created by the Claire Fontaine collective. They reflect the idea of the universality of foreignness, countering xenophobia and nationalist narratives.
The project itself combines visual, sound, and installation elements, which allows it to evoke in the visitor a sense of empathy, reflection, and engagement, thereby setting up an empathic perception and questions of identity and belonging.
theory
The multisensoriality approach allows for the most expressive delivery of each value message, simultaneously acting on different sensory channels and levels of perception—physical, emotional, and intellectual.
8.2
The Opposition of Text and Context
The Opposition of Text and Context
Can an explanation on a label or an accompanying sound fundamentally change our perception and understanding of an adjacent object? The answer would likely be yes.
An object possesses its own value and undoubtedly carries information. We can enjoy the elegance of its form, the skill of its craftsmanship, notice an unusual mark on its side, and speculate on its origin. We can be surprised if we cannot figure out its purpose. However, all the information related to the life of this object is inaccessible to us. Being the fruit of human labor or a force of nature, it is only revealed in dynamics—in its life cycle. To understand an object, its vital essence is important to us: How did it appear? Who owned it? Why was it invented? What did it witness?
Detachment from physicality, place, and time leads to complexity or incompleteness of perception.
We are faced with a task: to surround the object with an additional shell of hints that will help to read it. This is the task of creating context. We can imagine a gradient of additional layers—from a small inscription with its name, period, and geography of origin to a performance that allows one to fully experience the object’s function.
One can go further: an exhibition exhibit does not necessarily have a material, tangible form. Can the concept of adherence or despair be an exhibit? After all, if we are telling a story, not only actions but also relationships are important to us.
In many cases, it becomes significant for us to fill the temporal, historical, cultural, or religious gap between the exhibit and the viewer. We can call this the tactility of knowledge—the ability of an object to be a conductor, like an electrical one, to conduct knowledge through itself.
Museum storage limitations can increase this distance. A display case, glass, a safety distance, special lighting, or a temperature regime create a barrier environment in our relationship with the exhibit.
By designing the context, we can convey atmosphere, history, and saturate the air with value-based characteristics. Creating context is not limited to the stylization of an era or the reconstruction of a place. Try to consider two vectors of contextualization in an exhibition: the object-based and the non-material.
An object possesses its own value and undoubtedly carries information. We can enjoy the elegance of its form, the skill of its craftsmanship, notice an unusual mark on its side, and speculate on its origin. We can be surprised if we cannot figure out its purpose. However, all the information related to the life of this object is inaccessible to us. Being the fruit of human labor or a force of nature, it is only revealed in dynamics—in its life cycle. To understand an object, its vital essence is important to us: How did it appear? Who owned it? Why was it invented? What did it witness?
Detachment from physicality, place, and time leads to complexity or incompleteness of perception.
We are faced with a task: to surround the object with an additional shell of hints that will help to read it. This is the task of creating context. We can imagine a gradient of additional layers—from a small inscription with its name, period, and geography of origin to a performance that allows one to fully experience the object’s function.
One can go further: an exhibition exhibit does not necessarily have a material, tangible form. Can the concept of adherence or despair be an exhibit? After all, if we are telling a story, not only actions but also relationships are important to us.
In many cases, it becomes significant for us to fill the temporal, historical, cultural, or religious gap between the exhibit and the viewer. We can call this the tactility of knowledge—the ability of an object to be a conductor, like an electrical one, to conduct knowledge through itself.
Museum storage limitations can increase this distance. A display case, glass, a safety distance, special lighting, or a temperature regime create a barrier environment in our relationship with the exhibit.
By designing the context, we can convey atmosphere, history, and saturate the air with value-based characteristics. Creating context is not limited to the stylization of an era or the reconstruction of a place. Try to consider two vectors of contextualization in an exhibition: the object-based and the non-material.
Formulate the key value messages about the objects that you want to convey.
1
2
3
8.2.1
What materials can become the carriers of these messages?
Through what forms of representation can they be shown?
The Material Objects World
It may turn out that for this you will need additional expressive means: for example, data from related disciplines or works by contemporary artists.
Formulate which non-material concepts and values are important in the theme of the exhibition.
1
2
8.2.2
Think through which materials and forms of representation they can be expressed.
Non-material Values
The search for formats and genres of representation is not accidentally included in the chapter on multimedia—it is the wide choice of expressive means and media levels that allows for the creation of living, rich contexts.
8.3
Represen-tation
Representation
What basic types of information do we use?
When we create a system of multimedia representation, we assign a medium and a channel to each unit of content. This assignment is based on key principles:
This means that material that exists in a text format at the research stage does not necessarily have to be presented as text. It can become sound or an image. The choice of representation form is based on preserving the main semantic message of each unit of exhibition material.
At the research stage, we form a repository of diverse materials: objects, archives, documents, recordings. Our task is to assemble a cohesive narrative from these elements and to construct the future dialogue between the material and the viewer.
We lay everything we have out on a conceptual table and determine in what place, in what form, and on what medium each semantic unit will appear in the exhibition.
Thus, from large volumes of text, specific levels of representation emerge. For example, letters can be presented in the genre of a reading and placed in headphones, while photo captions can be turned into stickers on the floor that guide the viewer through the space.
When choosing a method of representation, we determine:
- Text
- Sound
- Image (static / dynamic)
When we create a system of multimedia representation, we assign a medium and a channel to each unit of content. This assignment is based on key principles:
- expressiveness
- efficiency
- economy (of attention, time, space)
- unity (of composition at all levels)
This means that material that exists in a text format at the research stage does not necessarily have to be presented as text. It can become sound or an image. The choice of representation form is based on preserving the main semantic message of each unit of exhibition material.
At the research stage, we form a repository of diverse materials: objects, archives, documents, recordings. Our task is to assemble a cohesive narrative from these elements and to construct the future dialogue between the material and the viewer.
We lay everything we have out on a conceptual table and determine in what place, in what form, and on what medium each semantic unit will appear in the exhibition.
Thus, from large volumes of text, specific levels of representation emerge. For example, letters can be presented in the genre of a reading and placed in headphones, while photo captions can be turned into stickers on the floor that guide the viewer through the space.
When choosing a method of representation, we determine:
- the material
- the scale
- the medium
- the position in space
8.4
More than Text
Academic research is currently mostly text-oriented. This means that its primary medium of presentation is text. Nevertheless, many researchers are already forming a demand for the presentation of scientific material in other formats. This demand is partly based on the academy’s desire to see an applied result of its research work—and generally to see the final effect of its labor.
We are already seeing academic research being turned into documentary plays and museum exhibitions. But the transformation of text into other formats and genres requires us to transform the text itself. On the one hand, we face the task of poetization and the search for a figurative language, the search for audiovisual means to display research materials. On the other hand, in this process, it is very important not to lose the scientific depth and broad perspective of the topic’s consideration. Furthermore, in exhibition products, we are making a transition from a professional scientific audience to a lay, unprepared one. This will require simplification, openness, and a manifestation of care for the viewer from us.
We are already seeing academic research being turned into documentary plays and museum exhibitions. But the transformation of text into other formats and genres requires us to transform the text itself. On the one hand, we face the task of poetization and the search for a figurative language, the search for audiovisual means to display research materials. On the other hand, in this process, it is very important not to lose the scientific depth and broad perspective of the topic’s consideration. Furthermore, in exhibition products, we are making a transition from a professional scientific audience to a lay, unprepared one. This will require simplification, openness, and a manifestation of care for the viewer from us.
ram case
Иmperфекtionz
How did we turn a book into an art project? In 2017, Ellen Rutten, a professor of Russian literature and culture at the University of Amsterdam, published the book Sincerity after Communism: A Cultural History. In it, the author explores the phenomenon of "new sincerity"—an artistic and social movement that emerged in the late USSR and was reinterpreted in post-Soviet Russia. She shows how the striving for honesty and emotional directness permeated literature, media, architecture, art, and mass culture. In particular, one such manifestation was the acceptance—and sometimes elevation—of the aesthetics of imperfection. Based on these and other, later studies by Ellen, we created an art project about imperfections in the format of a digital dictionary and a series of events.
How did we turn a book into an art project? In 2017, Ellen Rutten, a professor of Russian literature and culture at the University of Amsterdam, published the book Sincerity after Communism: A Cultural History. In it, the author explores the phenomenon of "new sincerity"—an artistic and social movement that emerged in the late USSR and was reinterpreted in post-Soviet Russia. She shows how the striving for honesty and emotional directness permeated literature, media, architecture, art, and mass culture. In particular, one such manifestation was the acceptance—and sometimes elevation—of the aesthetics of imperfection. Based on these and other, later studies by Ellen, we created an art project about imperfections in the format of a digital dictionary and a series of events.
Users could browse through different concepts within the chapters set out in Ellen’s research and determine their attitude towards them. This form created a new way for a wide audience to interact with the research’s problematics: viewers could reflect on different manifestations of imperfections and record their conclusions through a voting mechanic. The dictionary was illustrated with original images created with the help of artificial intelligence. This approach rhymed with the project’s theme: we used a technology created to "improve" images to visualize imperfections.
Use the scrollbar to read the case
As part of the project, a series of discussion-performances was also held at the New Stage of the Alexandrinsky Theatre. During these events, imperfections were illustrated through performance: experts from a wide variety of disciplines discussed how the attitude towards imperfection is formed in their field, while the audience voted on their personal perception of, for example, gray hair or cacophony.
theory
8.4.1
5
Conduct a search for object-based exhibits.
Carry out serious editorial and artistic work with the text.
6
4
Develop additional audiovisual materials.
Highlight key narrative plots.
Create a system of imaginative keys.
3
2
1
Formulate the main problematics.
To transition from an academic text to an exhibition, it is necessary to:
After these tasks are solved, the main process becomes the introduction of a variety of expressive forms of statement in the exhibition. For each semantic unit, we determine the genre, medium, style, and communication channel.
example
This photograph was taken on one of our first expeditions. The inscription reads: "You don’t love me—well the hell with you. God will punish you with sulfuric acid." It was written in chalk on the wall of an abandoned wooden house.
Now let’s imagine that we send the same text as a message in a messenger to a close friend—it is perceived as a joke and brings a smile. But if we print it on a T-shirt and go out on the street in it, it becomes a manifesto and a challenge.
As soon as a text is placed on a different medium and in a different environment, its meaning and intonation change. The same applies to any unit of content.
Now let’s imagine that we send the same text as a message in a messenger to a close friend—it is perceived as a joke and brings a smile. But if we print it on a T-shirt and go out on the street in it, it becomes a manifesto and a challenge.
As soon as a text is placed on a different medium and in a different environment, its meaning and intonation change. The same applies to any unit of content.
theory
The form of representation does not change the text, but it produces messages that differ in meaning — depending on exactly how it comes into the viewer’s field of vision. When choosing the intonations and meanings for a particular form of representation, we start from those messages that we consider significant for the exhibition. What exactly do we want to say to the viewer?
8.5
The Concept of Air
We strive to create a variety of representation forms, to use rich combinations of media, and to engage a person's different receptors.
Imagine that in an exhibition we hear the sound of a river, read letters fluttering in the wind, see a small projection of a person in a field in a darkened corner. What is the exhibit here? What determines the route of our movement and reading?
We say that we design not the method of displaying exhibits, but the air between them. Because it is in this indivisible space that sensations are born, and therefore—the assimilation of knowledge occurs.
Our typographer colleagues say that good work with a font is not only the aesthetics of the black letter's form on a white sheet, but also the white space that results from subtracting the letter's shape from the paper.
Such a metaphor gives us a hint about the significance of air in an exhibition. By creating air, we do not draw clear boundaries between a collection item, labels, architecture, navigation, equipment, and other elements of the space. Everything becomes part of a single statement and must have meaning and value.
These elements are connected through a unified narrative style and the artistic concept of the scenography. To explain this concept more clearly, let's turn to the theater.
In a theatrical production, we observe the development of a plot on stage. It happens according to a script and with the participation of actors. The action unfolds in specially created sets, often accompanied by sound, light, and movement. All these elements do not exist on their own—they are subordinate to a unified system that expresses a common idea through sound, color, forms, style, and composition. The creators of the play—the director, composer, artist, set designer, actors—work together to build an expressive interaction of all elements and through it, to convey a memorable experience to the viewer.
Similarly, in an exhibition, it is important to create a unified system of expressive means and stylistic solutions, coordinating them with each other and subordinating them to a single concept.
When working with sensitive topics, scenography also becomes a tool for moving away from the direct fixation of traumatic events and circumstances. Multimedia layers, rhymes between fact and image, and the combination of narrative strategies allow us to convey not actions, but atmosphere. Not a document, but a sensation.
Stylistic unity is responsible for unifying all elements in an exhibition.
Imagine that in an exhibition we hear the sound of a river, read letters fluttering in the wind, see a small projection of a person in a field in a darkened corner. What is the exhibit here? What determines the route of our movement and reading?
We say that we design not the method of displaying exhibits, but the air between them. Because it is in this indivisible space that sensations are born, and therefore—the assimilation of knowledge occurs.
Our typographer colleagues say that good work with a font is not only the aesthetics of the black letter's form on a white sheet, but also the white space that results from subtracting the letter's shape from the paper.
Such a metaphor gives us a hint about the significance of air in an exhibition. By creating air, we do not draw clear boundaries between a collection item, labels, architecture, navigation, equipment, and other elements of the space. Everything becomes part of a single statement and must have meaning and value.
These elements are connected through a unified narrative style and the artistic concept of the scenography. To explain this concept more clearly, let's turn to the theater.
In a theatrical production, we observe the development of a plot on stage. It happens according to a script and with the participation of actors. The action unfolds in specially created sets, often accompanied by sound, light, and movement. All these elements do not exist on their own—they are subordinate to a unified system that expresses a common idea through sound, color, forms, style, and composition. The creators of the play—the director, composer, artist, set designer, actors—work together to build an expressive interaction of all elements and through it, to convey a memorable experience to the viewer.
Similarly, in an exhibition, it is important to create a unified system of expressive means and stylistic solutions, coordinating them with each other and subordinating them to a single concept.
When working with sensitive topics, scenography also becomes a tool for moving away from the direct fixation of traumatic events and circumstances. Multimedia layers, rhymes between fact and image, and the combination of narrative strategies allow us to convey not actions, but atmosphere. Not a document, but a sensation.
Stylistic unity is responsible for unifying all elements in an exhibition.
A system of stylistic unity allows for:
8.5.1
Establishing and maintaining connections between different fragments and episodes of the exhibition.
3
Creating an expressive language that forms the atmosphere and intonation of the exhibition.
2
1
Retaining the viewer's attention, giving them confidence that they are still inside a single narrative.
The color scheme.
1
2
3
Consistent navigation, built with a unified logic.
The formal composition: the approach to organizing all elements in the space.
The approach to visual materials (photos, videos, illustrations).
4
The approach to sound materials (audio tracks with speech, music, soundscapes).
5
The approach to graphic materials and texts (navigation, explications, labels, captions, etc.).
Materials and tools: textures, proportions, and physical characteristics of objects.
7
6
8.5.2
What features of stylistic unity are important:
We create the air