The McGurk Effect: Hearing Words with Your Eyes?

Due to the McGurk effect, the brain leads us to hear an unspoken syllable when lip-reading. Is this positive or negative? Can we take advantage of this phenomenon?

Written and verified by the psychologist Maria Fatima Seppi Vinuales.

Last update: 12 March, 2023

Seeing skin and feeling the softness in the body. Seeing a plate of food and evoking that flavor, feeling that we’re actually tasting the food. Similarly, the McGurk effect tells us that it’s actually possible to listen with our eyes. Let’s see what it’s all about.

What is the McGurk effect?

The McGurk effect is the integration between auditory and visual stimuli in oral language. To understand what it is, let’s think of an example. We are in front of a person who repeats the syllable “pa” over and over again. That is the sound that reaches our brain, accompanied by a lip movement.

The person then continues to repeat the syllable “pa”. However, what our brain interprets is that what he or she is now saying is “fa”. What has happened?

The McGurk effect happened! What the person is now seeing is another lip movement that corresponds to the second syllable, even though the person is still saying the first syllable. If we close our eyes, we would hear the correct thing.

The McGurk effect indicates that prominence is given to what we see.

What we are actually doing in the example is lip-reading. The final sound would be given or guided by a mixture of lip-reading and previous experience.

What is it named after?

The McGurk effect is so named because it was first proposed in 1976 by psychologists Harry McGurk and John MacDonald, when they conducted several experiments. In them they studied the imitation patterns that children followed when learning speech.

In that case, the conclusion they reached was that, on occasions, visual information prevailed in the interpretation, ignoring the information that came through the auditory pathways. In other words, what we see influences what we hear.

In turn, many researchers report that the brain, in certain circumstances, operates according to a principle of causal inference. That is, it interprets contextual signals (it knows or associates that a certain lip movement corresponds to a certain sound), calculating the probabilities that certain messages are presented together.

Characteristics of the McGurk effect

Among the characteristics of the McGurk effect is that of anticipation, which allows us a certain economy of thought. That is, the brain makes available its previous visual experiences to understand what is being talked about and what is being heard.

Lip-reading can sometimes be useful, especially in noisy environments. However, it can also lead to confusion, as the brain would deceive us into believing that we are hearing something based on lip movement alone.

Many of these perceptual phenomena or deceptions also occur in optical illusions.

Another known fact is that the McGurk effect, with respect to audiovisual perception, decreases over the course of development. For example, since the region of the brain that deals with auditory information develops earlier than the one that deals with visual perception, then children rely more on sounds than adults.

Therefore, children are less prone to the confusions of the effect in question. That is, over the course of growth, the way sensory information is processed varies.

Also, some of the influencing factors are visual distractions, familiarity, and syllable structure. Some diseases may also play a negative role, such as aphasia, Alzheimer’s, and specific language disorders.

Finally, there’s another curious aspect to the McGurk effect in languages. Its occurrence varies according to the language analyzed.

The visual and the auditory working together

Through the study of the brain, it’s possible to know that the visual and auditory systems evolved hand in hand to facilitate the development of speech. Beyond its importance per se, the visual system is an important ally and mediator of the auditory system, collaborating in the comprehension of messages.

For example, it helps the recognition of sounds that may be difficult to distinguish. It also allows us to complete the information received, providing greater security and confidence in relation to what we hear.

The auditory and visual systems work together as a team. One strengthens and supports the action of the other.

The importance of knowing how the brain works is key to enhancing its use, and to be able to communicate and adapt to certain contexts, and to cope with phenomena that affect perception, such as aging.