Multimodality

Have you ever wondered how to analyse communication in different ways to understand its meaning? This is where the term multimodality comes in handy. As a key concept in linguistics, multimodality is often considered when analysing discourse (written/spoken text or images) as a way to understand not only the written meaning of discourse but also other elements that contribute to the meaning.

This article will explore the definition of multimodality and look at the different semiotic modes and mediums (semiotics is the study of signs and symbols and how they are used to create meaning). We will look at an example of a multimodal text and consider the effect of different modes. Finally, we will also look at the use of multimodal teaching and how it can affect learning.

Multimodality meaning

Multimodality refers to the use of more than one mode of communication in a text to create meaning. Although communication has always been multimodal, multimodality in discourse analysis is a relatively recent approach; linguists began to consider a multimodal approach around the 1960s.

One person who took an interest in multimodality was Gunther Kress. Alongside linguist Theo van Leeuwen, he contributed to the study of multimodality and was well known for writing books on the topic that were instrumental to the understanding and development of multimodal analysis. Because of this, his work set the foundation for the study of multimodality and influenced the works of future linguists. Perhaps his work may influence you too!

When we look at multimodality, we take into account:

• Modes
• Mediums

Types of multimodality: modes

Modes refer to the different ways that we can create meaning. Gunther Kress¹ defines a mode as:

'[...] a socially and culturally shaped resource for making meaning. Image, writing, layout, speech, moving images are examples of different modes'.

Fig. 1 - Different modes create meaning.

People communicate in different ways, so it is important to be aware of the different modes used in communication to fully understand the meanings that are being conveyed. These can be broken down into five categories:

1. Linguistic

2. Visual

3. Aural

4. Gestural

5. Spatial

It is important to note that a text does not need to contain all of the above modes to be considered multimodal; it can contain two or more.

Let’s explore these in more detail.

Linguistic mode

The linguistic mode focuses on the meaning of written or spoken language in communication. For example, this includes: choice of words, vocabulary, grammar, structure etc.

Visual mode

The visual mode focuses on the meaning of what can be seen by a viewer. This includes: images, symbols, videos, signs, etc. It also includes aspects of visual design, such as colour, layout, font type and size, etc.

Aural mode

The aural mode focuses on the meaning of what can be heard by a listener. This includes: sound effects, music, voice. This can be realised through tone, pitch, speed, volume, rhythm etc.

The gestural mode focuses on the meaning of communication through movement. This includes: facial expressions, gestures, body language, interactions between people. These are all examples of non-verbal communication, as meaning is conveyed without the use of speech.

Spatial mode

The spatial mode focuses on the meaning of communication through physical layout. This includes: position, spacing, the distance between elements in a text, proximity between people/objects, etc.

Types of multimodality: mediums

Another type of multimodality lies in mediums. Mediums are the forms in which the different modes are carried, i.e. the ways that modes can be accessed by people.

Modes are influenced by the type of medium they are carried by. The medium of a text can determine the purpose of a text and its target audience. Different types of mediums include, but are not limited to:

• Books

• Newspapers

• Radio

• Television/Film

• Billboards

• Theatre

• Websites

• Social media posts

There are so many different mediums… can you think of any more?

It is also important to note that there can be more than one medium for a text. For example, an online newspaper combines the form of a newspaper with the layout/features of a website for easy, global access. Also, online newspapers have the advantage of being updated with news in real-time, so news can be reported quickly.

Examples of multimodality: text and analysis

When analysing a multimodal text, you should identify the medium the text is in. For example, is it a book? A sign? A film? This will give you an idea of the function of the text and who it is aimed at.

You should then identify the different modes that make up the text and what each mode communicates. The modes convey a message that can be interpreted by looking at each mode individually and also together; as communication doesn’t only depend on a single mode.

For example:

Fig. 2 - A sign is multimodal. Each mode conveys different meanings.

What can this text tell us?

• The medium of this text is a sign; particularly a road sign. This tells us that it is aimed at drivers, and has the purpose of giving instruction to them.

• The linguistic mode - the words ’slow down’ let us know of the action to be taken by the driver. They deliver a clear message in an imperative manner (ordering the driver to do something).

• The visual mode - the font of the text is large, bold and evenly spaced out, making it easy to read for passing drivers. This stands in contrast to the orange background of the sign, which is bright and will grab the attention of the driver. Orange could also be associated with a warning, to signify caution.

• The spatial mode - in terms of where the sign is in relation to other objects, it is situated right by the road, so will be visible to drivers and will be able to caution them to avoid speeding or accidents on the road.

It is also important to note that there can be a difference between the denotation and connotation of certain signs. Denotation refers to the literal definition of something, whereas connotation refers to the secondary cultural meanings we apply to something.

Taking a look at the sign above, we can gather the following:

Another example is the use of orange:

Multimodality is not just considered when analysing texts… It is also used as a teaching method.

Multimodal teaching methods

Multimodal teaching methods refer to different ways of teaching that use different sensory modes. This is an important way of teaching as it allows students to discover their preferred way of learning and focuses on adapting learning to fit their individual styles.

Multimodality in teaching methods focus on these four modes:

• Auditory

• Kinaesthetic

• Reading/Writing

The visual way of learning refers to people who retain information through seeing things. For example, they may prefer to use graphs, pictures, charts flashcards, etc. to be able to visualise the information and remember it.

The auditory way of learning refers to people who retain information through hearing. Those types of learners may prefer to participate in listening exercises in which they hear information and repeat it.

The kinaesthetic way of learning is suitable for people who retain information through physical activity. For example, they may prefer to show someone how to do something instead of simply explaining it through writing.

The reading/writing way or learning refers to people who retain information through reading texts and writing things down. Learners with this style of learning may prefer to answer questions from a textbook or eBook.

Why is multimodal teaching important?

The ability to convey information in more than one way through multimodality enables creativity and flexibility for teachers, as they can combine different learning styles and adapt them to better suit their students. This encourages students to explore the different ways of learning and find the style that allows them to retain information the most effectively.