Cognitive load

Cognitive Load in NoldusHub

NoldusHub uses pupil sizes and ambient light fluctuations to estimate cognitive load. Ambient light fluctuations are estimated from the light intensities of the participant screen when using a remote eyetracker bar. When using head-mounted eyetrackers, the forward facing camera is used to measure ambient light intensity. Eyetrackers are also used to measure pupil sizes. Therefore, cognitive load is only calculated when using an eyetracker and capturing the participant screen or using forward facing cameras. Cognitive load calculations require heave usage of system resources. For that reason, calculations are performed on the entire session data set, immediately after session recording. The resulting cognitive load measures will be available in replay immediately after. 

What is Cognitive Load?

To a large extent, humans show behaviors that require little conscious preparation. Such behaviors include eating, walking, simple decision taking like choosing which food to eat and driving the daily commuting route. In contrast, some behaviors require effortful thinking and planning such as “complex problem solving”, “visualizing sequences of actions”, and “planning a social event with demanding and fully booked colleagues”. When performing tasks in the latter category, humans process information consciously while using working memory resources. We refer to “cognitive load” as the sum of those resources used per time unit.

Pupil responses to light

The pupillary light reflex is a rapid unconscious constriction of the pupil in response to bright light (Figure 1). This reflex regulates the amount of light stimulating the retina in order to protect the retina from being overstimulated. When the amount of light striking the retina decreases, the opposite reflex occurs: the pupil dilates. 
A prototypical response of the pupil to a pulse of light that strikes the retina is demonstrated by Cortez et al (2017). When stimulating the retina with a short-lived light pulse, the pupil diameter decreases within approximately one second to a minimum diameter that, in size, reflects the intensity of the light pulse, and then gradually recovers to its original baseline diameter 5 seconds after presentation of the light pulse.

Figure 1. Pupillary constriction in reaction to a light pulse. BPD = baseline pupillary diameter, CA = constriction amplitude, RA5 = redilation amplitude at five seconds (Cortez et al, 2017).

Pupil responses to mental processes

Interestingly, humans can also display pupil dilation under constant light conditions, i.e. independent of light conditions. Light-independent pupil dilations are observed in response to cognitive processes such as paying attention (e.g., Kahneman, 1973), experiencing emotions (e.g., Bradley et all, 2008), taking decisions (van Slooten et al, 2018), using working memory and performing demanding cognitive tasks (Krejtz et al, 2018). Hence, independent of light conditions, pupil diameter responses can be observed due to cognitive- and/or emotional challenges.

Measuring cognitive load magnitude

NoldusHub obtains continous measures of pupil diameter from eyetrackers. In order to asses the effects of mental processes on pupil diameter, first pupil diameter variations due to light changes need to be isolated from the overall pupil diameter. This is achieved by measuring the intensity of light striking the eye, and calculating the anticipated contribution of this light to pupil diameter changes using the single-pulse-induced pupil constriction function (Figure 1). For this, NoldusHub uses a mathematical operation called deconvolution. NoldusHub obtains estimates of light intensities from:

1. The captured screen video, in case a remote eyetracker is used attached to this screen
2. The forward facing camera integrated in head-mounted eyetracker glasses (Tobii Glasses 3)

Once the effects of light on pupil diameter are accounted for, remaining pupil diameter variation can entirely be attributed to mental processes, provided that the experimental task is geared towards triggering mental processes only. For that, NoldusHub uses an algorithm based on the work by Gollan & Ferscha (2016) and Stolte, Gollan & Ansorge (2020). This algorithm calculates the relative intensity and duration of cognitive load from observed pupil responses, in line with observed responses for 21 human subjects (Gollan & Ferscha 2016, Stolte et al 2020, see Figure 2). This study demonstrated that pupil responses are distinctive for different cognitive load states during the active use of working memory. A concurrent visual search instruction also induces large pupil diameter changes, but these did not differ between different cognitive load conditions. Hence, pupil size responses only distinguish between the cognitive load conditions when the task requires working memory use.

Figure 2. Changes in pupil diameter as a function of memory load and search difficulty in a combined Visual Working Memory – Visual Search task (Stolte, Gollan & Ansorge, 2020). Significant Working memory load effects on pupil size are observed during a visual memorization phase (gray area1-2 seconds after stimulus presentation) and during visual recollection phase (5.5 – 6.5 seconds after stimulus presentation). Instead no main effects of search difficulty are observed during Visual Search phase (2.2 – 4.3 seconds after stimulus presentation), although an expected overall pupil diameter increase is observed in this period due to the visual attention that is raised by the visual search instruction. 

What if pupil data is partly missing?

If during a session, the participant's eyes are closed or the pupils are not visible to the eyetracker, no pupil data will be available. The cognitive load algorithm will then continue using the last known pupil diameter until the next valid pupil size measure is available. The longer pupil diameters are not available, the less reliable the cognitive load estimate will be. The reduced reliability will then be reflected by a decreasing "cognitive load validity" measure. So, although cognitive load estimates are still produced, the validity measure will eventually become zero. Cognitive load estimates accompanied by low validity measures should not be used for data interpretation.  

Cognitive load signals available in NoldusHub

NoldusHub versions 1.4 and 1.6 produce Cognitive load estimates and Cognitive load validity measures. These are calculated during data acquisition and stored for later inspection in replay mode. Cognitive load measures can be exported as from v1.6.

The interpretation of cognitive load measures

Both "cognitive load" and "cognitive load validity" are normalized measures, ranging from 0 – 1. Hence, cognitive load estimates are relative within each participant, not absolute. The proper use of cognitive load measures therefore is to compare states within each participant. Ideally, each participant starts of with a short easy cognitive load task followed by a difficult cognitive load task, followed by the actual experimental task that invokes the cognitive load states of interest. Validity scores indicate the validity of the cognitive load estimates. Low scores approaching zero indicate low validity, scores approaching 1 indicate high validity.

The term "internal validity" refers to the extent to which one can be confident that causal relationships established in an experiment cannot be explained by other factors than those that are being investigated. Since pupil dilatations can be triggered by various cognitive and emotional processes, the internal validity of cognitive load estimates produced by NoldusHub entirely depends the extent to which the experimental task precludes the occurrence of other processes that invoke pupil dilations. Therefore, it is essential to observe a strict hygiene in task design: Experimental conditions should be designed in such way that pupil responses can be compared between episodes in which task demands only varied with respect to the cognitive-load task parameters.

For example, to assess whether experienced drivers rely more on automated information processing (i.e. low cognitive load) during driving than novice drivers, they may be tasked to conduct a simulated ride in a crowded city. As a control condition, they may watch a documentary on the same screen. Differences in cognitive load between both conditions are expected to be lower for experienced drivers than for novice drivers. However, if the experienced drivers pay more attention to entertaining or upsetting events on the sidewalks than novices, they may react more emotionally. Such unforeseen emotional responses will confound the actual differences in cognitive load between experienced and novice drivers. A well-designed experiment should prevent such confounds from happening to assure pure cognitive load assessment. In the words of the late Daniel Kahneman, Nobel Prize winner and pioneer in the research of pupil responses to mental processes (Kahneman, 1973):

Alternatively, natural experiments may include uncontrolled conditions that invoke different cognitive and emotional responses. For such experiments, the validity of cognitive load estimates in relation to the cognitive load conditions of interest is improved by scoring uncontrolled environmental event episodes for analysis. For instance, episodes with funny or distressing stimuli should be identified with markers indicating the stimulus intensity. During data analysis, these scores can then be taken into account as covariate.

Literature

• Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J. (2008). The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology, 45(4), 602-607.
• Cortez, M. M., Rea, N. A., Hunter, L. A., Digre, K. B., & Brennan, K. (2017). Altered pupillary light response scales with disease severity in migrainous photophobia. Cephalalgia, 37(8), 801-811.
• Gollan, B. & Ferscha, A. (2016). Modelling pupil dilation as online input for estimation of cognitive  load in non-laboratory attention-aware systems. COGNITIVE 2016-The Eighth International Conference on Advanced Cognitive Technologies and Applications.
  Kahneman, D. (1973). Attention and Effort. New York: Prentice-Hall.
• Krejtz, K., Duchowski, A. T., Niedzielska, A., Biele, C., & Krejtz, I. (2018). Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze. PLoS ONE, 13(9), e0203629.
• van Slooten, J. C., Jahfari, S., Knapen, T., & Theeuwes, J. (2018). How pupil responses track value-based decision-making during and after reinforcement learning. PLoS Computational Biology, 15(5), e1007031.
• Stolte, M., Gollan, B., & Ansorge, U. (2020). Tracking visual search demands and memory load through pupil dilation. Journal of Vision, 20(6), 1-19.