Exploring the Eyes from a Scientific Perspective

The movement of human eyes mirrors the cognitive processes occurring in the brain. Today, neuroscientists can precisely monitor their parameters with millisecond accuracy. Video-oculography holds the key to understanding numerous phenomena related to reading, perception, and language production processes. IQ.HSE interviewed Andriy Myachikov, Leading Research Fellow at the HSE Institute for Cognitive Neuroscience (ICN), about the method of video-oculography, or, in scientific parlance, eye-tracking, and its applications in psycholinguistic research.

Andriy Myachikov,
Leading Research Fellow, Centre
for Cognition and Decision Making,
HSE Institute for Cognitive Neuroscience

— How can the examination of eye movements benefit scientists?

— Perhaps I should begin by noting the extent to which we can control the movement of our eyes. We can easily direct our gaze at one thing or another—these processes are indeed within human control. However, a substantial number of eye movement parameters elude conscious access, as we are not even aware of them.

This aspect is significant for scientists because, through the study of oculomotor behaviour, we can trace the processes which are implicit . For instance, if a flash of light suddenly occurs next to you, you will undoubtedly pay attention to it—not because you consciously decide, 'Oh, I'm going to look this way now.' Instead, this decision will be automatic, as from an evolutionary standpoint, it is crucial to react promptly to the sudden appearance of various objects, such as potential predators or falling rocks. In other words, the eyes serve not only as a 'mirror of the soul' but also as a kind of window into the brain.

Thus, by employing the method of video-oculography or eye-tracking, we can identify, track, and measure various actions and perception processes that are beyond direct and constant control.

Compared to some other animals, humans have relatively poor hearing and are not very sensitive to odours, but we possess fairly good eyesight. Therefore, the eyes serve as an essential source of perceptual information.

Eye movements can be broken down into numerous parameters. Our research is based on what is known as the eye-mind hypothesis. It posits a direct link between the direction of our gaze—the focus of our attention, on one hand, and the cognitive processes occurring in our brain at that moment, on the other. Hence the simplest explanation of the relevance of eye-tracking for both scientists and practitioners who apply the science such as engineers, designers, architects, and others.

For instance, eye-tracking enables researchers to determine the optimal arrangement, in terms of visual accessibility, of controls on the control panel in cars or airplanes. Developers of websites and applications can leverage eye-tracking to position various icons or links strategically, ensuring they attract attention and are user-friendly.

— Do these studies primarily use heat maps?

— Yes, as far as applied research is concerned, it is often sufficient to employ so-called temperature or heat maps. They appear as areas varying in colour from red to green, indicating approximately where a viewer focuses their gaze and to what extent.

In practice, it looks as something like this: the study subjects examine information on a computer screen, while their eye movements are recorded and used to generate a temperature map, which is overlaid onto the website page. The red areas indicate places where the viewers looked more frequently. While a person may not always recall precisely where they were looking, a heat map will clearly show this.

— But perhaps this method is not sufficient for fundamental research? What other eye movement parameters are of interest to scientists?

— In scientific research, particularly in psycholinguistics, we are interested in more detailed and precise parameters, including, of course, those which are not under conscious control at all. Conventionally, we categorise eye movements into static and dynamic. Static eye movements include fixation and gaze. When a person focuses on an object displayed on a screen, it constitutes a static parameter, and we can measure some of its components, eg for how long an individual has been looking at the object.

If there are ten objects on the screen, and the task is to locate, for example, the letter X among the letters P , we can measure the search time, reaction time, and the frequency of fixations, indicating how many times a person returns their gaze to an object. It is possible to aggregate fixations and measure the gaze duration—providing insight into how much time a person spent looking at a particular stimulus throughout the entire presentation period. However, fixations are not entirely static either.

— Does the eye move during fixation?

— Indeed, even though it may appear that you are fixating on a single point, this is not entirely accurate. Our eyes are constantly in motion. In one of our papers, we analyse a dynamic eye fixation parameter known as ocular drift. When looking at an object on the screen, the eyes actually appear to drift around it. Researchers have seldom investigated this parameter—which, naturally, is not under conscious control—in the context of cognitive processes.

This parameter is interesting, since it can be used to track the most implicit processes. That is, before a person shifts their attention and, consequently, their gaze to any new location, minute changes in the fixation position are already occurring, preceding the actual gaze movement—saccade. Thus, it is possible to identify the earliest implicit signs of attention shift and the accompanying eye movements.

— What are the dynamic parameters of eye movements?

— For example, the saccades mentioned earlier, which represent a rapid coordinated movement of the eyes in a specific direction. When, for instance, you focus on the cursor while I move it to another point on the screen and your eyes follow the cursor to that point—that is a saccade.

This parameter is also of great interest to researchers. During an experiment, we can tell the participant that an object is about to appear on the right side of the screen. Their brain then begins to plan the upcoming saccade, and various parameters related to how quickly the saccade will be prepared and launched can be measured.

Interestingly, during the saccade, a person remains virtually blind. This is attributed to the phenomenon known as change blindness. In a well-known experiment, subjects viewed photographs displayed on a screen, and while they were scanning the picture, the original image was replaced by a modified version—perhaps two individuals sitting on a bench 'exchanged heads.' When such modifications occurred during a saccade, many study subjects failed to notice the change.

In general, saccades are a highly interesting phenomenon for scientists to examine, since it is possible to measure reaction time, saccade speed, amplitude, and more. Interestingly, many saccade parameters are traditionally considered ballistic, meaning that they are predetermined at initiation and cannot be changed once the saccade is launched—that is, during the saccadic eye movement.

The ballistic parameters of saccades are sometimes described as 'cognitively impenetrable,' indicating that they cannot be influenced by cognitive processes such as understanding the meaning or significance of a perceived stimulus, eg a word.

However, recent studies have shown that this is not entirely accurate. For example, in one study, subjects were instructed to read and memorise 'up words' and 'down words,' such as cloud or worm , spatially associated with upper or lower locations. Then, a visual stimulus was presented either in a compatible location (eg in the upper part of the screen following the word cloud ) or in an incompatible one.

We often refer to such conditions as congruent and non-congruent. When a visual stimulus was presented in the congruent part of the screen, saccades to this image appeared to extend over it, landing slightly higher. This suggests that the spatial meaning of the word influenced the amplitude of the saccade, causing an increase, contrary to the previous belief that amplitude was one of the 'cognitively impenetrable' ballistic parameters of saccades.

— Is the dilation or constriction of the eye's pupil included in eye-tracking research?

— Yes, that is another significant and intriguing aspect of eye movement. In addition to the pupil's obvious response to factors such as the amount of light and the brightness of the perceived signal, its size also correlates with various cognitive parameters, eg the complexity of the task performed by the subject. While the neurocognitive nature of this phenomenon is not entirely clear, the phenomenon itself allows for the exploration of various aspects by examining the parameters of pupil dilation and constriction. This applies not only to basic research but also to applied research involving web pages, applications, and more.

— What is the time scale of investigating eye movement parameters?

— We can assess eye movement parameters at the millisecond level. Technically, it works like this: an eye-tracker generates a large text file in which each line reflects the parameters of eye movement during a separate millisecond. Thus, every millisecond, we can determine whether the eye is in fixation or undergoing a saccade. If there is a saccade, we can see its angle, speed, and amplitude.

— How accessible is eye-tracking as a research method?

— It is a relatively low-cost and accessible method. There is no need to directly tap into the brain by attaching things to the head, placing a person in an MRI scanner, or investing millions in equipment installation and staff training.

It is worth noting that existing technologies allow researchers to co-register—ie simultaneously register—eye movements and brain activity. We can track all this on a single timeline with maximum accuracy and understand the ongoing processes in the brain associated with the accompanying eye movements.

This makes eye-tracking a versatile method that does not require significant expenditures when employed in fundamental, applied, or clinical research. Eye movement is traditionally used, for example, as a correlate of certain mental disorders and brain dysfunctions. 

— How are modern eye trackers designed?

— In the early days of eye movement studies, eye-tracking was a cumbersome and painful procedure for the subject, who had to wear contact lenses connected to wires enabling the researcher to track the motion of the eyeball. Modern optical eye trackers are non-invasive. They can be portable, such as glasses with an accompanying recording device carried in a pocket or backpack. A person wearing such glasses can be instructed to visit a store or view a website.

Eye trackers can be used in schools to conduct studies involving children. Or these devices can be applied in healthcare settings to conduct research with patients, including persons with limited mobility. Returning to the question of the method's accessibility, we have the flexibility to choose when and where to apply it. Its application does not require a large space or specific conditions, such as sound insulation.

Regarding the technology itself, there are various methods for recording eye movements. Primarily, we use optical eye trackers. They offer the maximum possible accuracy, a crucial factor in psycholinguistic research. Externally, such a device resembles a computer with a recording camera underneath. Most commonly, head support is also used. During a study, various stimuli are presented to the subject on the screen, and their eye movements are simultaneously recorded.

— You and your colleagues recently published a paper on eye-tracking methods in psycholinguistics. Could you tell us how eye-tracking is used in psycholinguistic research?

— Eye-tracking has traditionally been used to analyse the reading process, with the assumption that various eye movement parameters are linked to the comprehension and perception of the content being read.

In such studies, scientists aim to understand, for example, which words capture the eye's attention for a longer duration, and conversely, which are fixated for shorter periods, which words lead to so-called regressive saccades, where a person returns to a word multiple times to clarify its meaning, and which ones are skipped altogether. The more predictable a word, the more likely it is to be skipped by the eye.

It is worth noting that many words may not be fixated upon by the eye at all, even though they have been read—the brain seems to take them for granted, so to speak. This phenomenon is particularly intriguing from the perspective of understanding how the attention and perception systems operate.

In reading studies employing the eye-tracking method, it is possible to analyse, for example, the frequency of a particular phonological structure of a word or the word itself—whether it is common or rare. Thus, 'table' is a high-frequency word, while 'idiosyncrasy' is not.

It is also possible to analyse word length, complexity, functionality, spelling, and even syntax—for example, whether a particular word is appropriate in a given syntactic structure.

— In addition to reading, what other areas of psycholinguistics are studied using the eye-tracking method?

— These include areas related to perception, language production, as well as language learning and the phenomenon of bilingualism. Here is an example of how research on language perception is conducted. A person is presented with a number of objects on the screen alongside a sentence related to these objects.

Let's say, the images of a mum, a hat, and a bus are displayed on the screen. Then a sentence is pronounced gradually: 'The mum... has ... a hat,' and we observe the distribution of eye movements as the viewer processes the sentence related to the images on the screen.

At the sound of the words 'The mum ... has ...,' the subject often looks at the image of a hat before hearing the next word, because their brain makes the relevant prediction. Furthermore, by adding an interesting 'competing' image to the screen, such as a cat—which sounds similar to hat—we can observe these two images really competing for access to cognitive processing for approximately 50 to 100 milliseconds.

— Language production was the subject of your dissertation. Could you tell us how research is conducted in this area?

— In a much similar way, by employing the visual-world paradigm. A person is presented with a picture and asked to describe it. Certain interacting objects in the picture can be highlighted and made more prominent, eg by increasing their brightness or size.

I was particularly interested in the syntactic structure of the sentence produced by the person, specifically the choice of the subject and object. Would they construct their sentence as 'A boy kisses a girl' or vice versa: 'A girl kisses a boy'? In other words, how would they decide, in producing the sentence, who kisses whom. Or perhaps their sentence would be, 'A boy and a girl kiss each other.' Or another example of alternative sentence structures: 'Petya fought with Vasya,' 'Vasya fought with Petya,' and ‘Petya and Vasya had a fight.'

The choice of the sentence subject can be influenced by what is more prominent and attracts attention. In a study conducted at HSE University in collaboration with Mikhail Pokhoday and Yury Shtyrov, we demonstrated that the choice of word order in a sentence in the Russian language depends on the distribution of the speaker's attention. The attention directed at an object determines its earlier mention in a sentence and increases the likelihood of this word being chosen as a primary member in a sentence, such as the subject.

In other words, directing the speaker's attention to the boy in the picture makes it more likely for the speaker to describe the situation as 'a boy kissing a girl,' and the reverse is also true. It might seem trivial, but in fact, from the perspective of the linguistic theory of language production, it is not.

Thanks to such experiments, we can observe a direct relationship between the distribution of the speaker's attention and their choice of word order or syntactic framework. This essentially means that we can understand how language is constructed at the level of the attention system. From a practical perspective, the insights from such studies can be applied to teaching languages, writing advertising texts, and more.

— Have eye-tracking methods provided any intriguing insights into bilingualism?

— There has been an interesting study by American scientists involving bilingual Russian-English speakers, ie Russian-speaking individuals living in the United States. The study proceeded as follows: the subjects were presented with four objects on the screen, including, eg, a postage stamp, a felt-tip pen (marker), and a couple of other items. The participants were then instructed to move a stamp or marker from one square to another using the mouse cursor. The noteworthy aspect is that the English word marker shares several phonemes with marka, the Russian word for 'stamp'.

When the instruction was given in English, eg: 'move the marker to the green square,' once the participants heard mar, the initial sounds of the 'unfolding' word marker, their eyes began shifting between the English word marker and the Russian marka. This suggests that in bilingual individuals, the lexical items in both languages were activated simultaneously and competing cross-linguistically until it was clear to the brain that the target word was marker, not marka.

It is important for us to understand how these processes occur in monolinguals (speakers of just one language) and in bilinguals. Eye movement analysis makes it possible to assess the structure and nature of monolingual and bilingual lexicons. Research reveals that the words of a new language become integrated into the already existing lexicon. Indeed, no distinct areas in the brain are specifically allocated for storing English words or Russian words. Instead, there is a shared repository where the new words are integrated, incorporating rules that align with the lexicon already existing in the mind.

Thus, if we already know the Russian word marka and acquire the English word marker, the new word is integrated in such a way that these words begin to interact. The aforementioned experiment involved cross-linguistic competition between these words, demonstrating that they were co-activated—that is, activated simultaneously.

— What ongoing studies at HSE University, including those conducted within the framework of the strategic project 'Human Brain Resilience: Neurocognitive Technologies for Adaptation, Learning, Development and Rehabilitation in a Changing Environment,' use the eye-tracking method?

— A wide range of studies are led by Anna Izmailova of the HSE ICN Centre for Cognition and Decision Making. Her interests include reading processes and various aspects of learning a second foreign language. In her studies, Anna is working on designing methodologies for optimising language learning, and these studies actively incorporate eye-tracking techniques.

ICN HSE operates Unique Equipment, a cutting-edge scientific facility supported by a government grant. Research conducted using this facility also includes eye-tracking. Among other things, we continue to investigate the relationship between eye movements and attention, as well as the speaker's choice of word order and syntactic structure. In addition, we are actively involved in the World-class Science Centres (WCRC) project.
IQ

Author: Marina Selina, December 20, 2023