Using Eye Movements to Evaluate the Cognitive Processes Involved in Text Comprehension

1. Properties of Eye Tracking Equipment

Eye trackers vary regarding how eye movements are measured, how frequently the eye position is measured (sampling rate), accuracy of determining eye position, how much head movement is allowed, and ease of use. The importance of these factors varies depending on the type of research being performed and the participants being tested. For instance, in most studies of reading, high accuracy is needed to determine which word is being fixated. As a second example, tolerance of head movements and ease of use is crucial when using children as participants.

The research described here was conducted using an SR Research EyeLink 1000 eye tracker (SR Research Ltd). A picture of the eye tracking system is presented in Figure 1. The EyeLink system tracks eye movements by measuring changes in pupil position in a video image. This is done by shining a dispersed infrared light (which is not visible to participants) onto the subjects' eyes and recording the infrared reflection (image) from one eye (or both eyes) with a high resolution infrared sensing video camera. The infrared light source and video camera are positioned beneath the monitor that is used to display the stimuli. Infrared light is used to avoid spurious reflections from normal spectrum lights. The infrared light produces a bright spot where the pupil is located (the lights enters the pupil and reflects off the retina to brighten the pupil) and a pinpoint reflection on the surface of the eye called the corneal reflection. The video image is digitized so that horizontal and vertical movements of the pupil (the bright spot) in the video frame can be measured. The corneal reflection is a stationary reflection that does not move unless the head is moved (because it is a reflection off the surface of the eye, it does not move when the eyes move). Measuring the corneal reflection provides a means to distinguish small head movements, which lead to movement of the corneal reflection, from eye movements alone, which do not lead to movement of the corneal reflection. To minimize head movements and to keep the participant in focal range of the video camera, participants place their heads on a forehead and chin rest while reading text presented on a computer monitor. Several critical features of eye tracking systems are described below.

1. Sampling rate. Sampling rate refers to how many times per second eye position is measured. The sampling rate for the EyeLink 1000 system is 1,000 Hz, which means eye position is measured 1,000 times/sec. Common sampling rates are 1,000 Hz, 500 Hz, 250 Hz, and 60 Hz (the video refresh rate/frequency of many computer monitors).
   Note: When studying reading, the goal is to accurately measure the location and duration of fixations and saccades. During normal adult reading, fixation durations typically vary from about 100-800 msec, with the average being approximately 250 msec (for college-aged readers). Saccades typically range in duration from about 10-20 msec when readers move their eyes from one word to the next. Very large saccades, such as moving from the end of one line to the beginning of the next line, may be as long as 60-80 msec in duration. Higher sampling rates produce better temporal accuracy (also called temporal resolution) when measuring the duration of fixations and saccades. Specifically, the average temporal error will be approximately half the duration of the time between samples. For example, a sampling rate of 1,000 Hz (sampling eye position every 1 msec) will lead to an average error of 0.5 msec and a sampling rate of 60 Hz (sampling eye position every 16.7 msec) will lead to an average error of approximately 8 msec. An 8 msec error might be considered too large to study the duration of saccades, but not too large to study the duration of fixations. Thirty years ago most reading research was conducted using eye trackers with 60 Hz sampling rates. Most research on reading is now performed using eye trackers capable of sampling at 500 Hz or 1,000 Hz.
   During reading, the goal is to focus both eyes on the same location; therefore, common practice is to record eye movements from one eye. Some eye tracking systems allow tracking of both eyes simultaneously. The advantage of tracking both eyes is that the eye with the best tracking accuracy can be selected for the final analysis. The disadvantage of tracking both eyes is that the sampling rate is generally reduced by a factor of two (i.e. a sampling rate of 1,000 Hz for one eye is reduced to 500 Hz when recording from both eyes).
2. Accuracy. Accuracy refers to how well the calculated fixation location matches actual fixation location. This is expressed in degrees of visual angle (a half circle has 180º of visual angle). The average accuracy of the EyeLink 1000 system is 0.25-0.5º of visual angle. To put this in perspective, when looking at a 17-20 in computer monitor at a normal viewing distance, the width of the monitor covers 20-30º of visual angle.
   Note: The degree of accuracy needed depends on the research goals. If the goal is to measure which character on a line is fixated, then character position accuracy is needed. If the goal is to measure which word on a line is fixated, then word position accuracy is needed. In the research described here, text was displayed so that 3 characters equaled approximately 1° of visual angle. The measurement is approximately 3 characters because the text was displayed in proportional font (i.e. characters differed in width, such as the character i being narrower than the w). To obtain character position accuracy, the eye tracker must determine fixation location to 1/3° (the width of approximately one character) across a 30° horizontal range (the width of the computer display). To obtain word position accuracy, the eye tracker must determine fixation location within a 1° range for words of 3 characters in length. Eye trackers tend to be slightly less accurate at measuring large vertical eye movements (e.g. moving from the bottom of the display to the top) because the pupil can be partly occluded by the eye lids and eye lashes. This problem can be substantially reduced or eliminated by double-spacing the texts, which makes it easier to discern which line of text is being read. For 17-20 in computer monitors, double-spacing produces approximately 2.5° of vertical separation between lines, well within the accuracy range of the EyeLink 1000 and most current generation eye trackers.
3. Head movement. The allowable head movement for the EyeLink 1000 system is 25 mm x 25 mm x 10 mm (horizontal x vertical x depth). That is, participants can make head movements of ±12.5 mm left/right, ±12.5 mm up/down, and ±5 mm in/out from the head position in which initial calibration (explained below) was performed without compromising accuracy. The left/right and up/down restrictions are needed to keep the eye within the video camera's field of view. The in/out restriction is needed to keep the eye in the video camera's focal range. Using a combination forehead/chin rest easily keeps movements within this range.
   Note: If larger head movements are needed, such as if the display consisted of three monitors side-by-side that required head movements to look at each monitor (as in a driving simulator), a "head mounted" version of the eye tracker is available that does not require a forehead/chin rest. For the head mounted system, the cameras used to track eye position are mounted on an adjustable headband so that participants can freely move their heads. A separate camera that points forward records the scene being viewed. Eye movements are determined relative to the scene being viewed. The drawback of head mounted system is that sampling rate is reduced to 500 Hz (maximum) or less, accuracy tends to be slightly less because large head movements can introduce error, and setup times tend to be slightly longer because the position of the eye movement cameras must be adjusted for each participant. The software for operating the head mounted eye tracker is essentially identical to the EyeLink 1000.
4. System setup time. The EyeLink 1000 typically can be set up and calibrated in 5 min or less, which is typical of video-based eye trackers. This process is further defined in the following procedure section.

2. Stimulus Preparation

When comparing eye movements for stimuli taken from two or more conditions, the stimuli need to be matched on features that are known to influence eye movements. The metaphor texts used here illustrate several important properties that should be controlled when comparing how two stimuli are read.

1. Key words should be matched on average word length (in number of characters) and word frequency (typically expressed as occurrences/million words) across conditions. This is critical because fixation duration increases as word frequency decreases and the probability of fixating a word increases as word length increases^10,13. In the metaphor passages, content words in familiar and unfamiliar metaphors were matched on average word length and word frequency.
2. Key words should be presented in similar positions within texts and sentences. This is important because words at the end of sentences are typically read slower than earlier words in the sentences and readers tend to read faster as they progress through a passage and then slow down on the last sentence^11,12. In the metaphor passages, all metaphors were presented at the end of the third sentence. Presenting some metaphors at the beginning of sentences and others at the end of sentences would lead to variation in reading times not associated with the metaphors themselves.
3. Key phrases should be roughly matched on word length and structure. This is important because sentence length and syntactic structure influence reading time, the number of fixations, and the likelihood of regressions¹³. In the metaphor passages, familiar and unfamiliar metaphors had the same number of words and structure (X is a Y).
4. The context immediately preceding key words should be roughly equated on the number of words, format, and processing difficulty. Equating contexts across conditions is necessary because contextual constraint influences the fixation durations for subsequent words^1,14. In the metaphor passages, the first context sentence always related to the first key word in the metaphors (love and fisherman) and the second context sentence always related to the second key word in the metaphors (flower and spider).
5. Key words or phrases should not be the last word or phrase of a passage. This is important because readers read the ending of a text slower than earlier parts of the text, which is called the passage wrap-up effect¹². Adding a concluding sentence also allows for processing spillover to be measured. Spillover refers to processing difficulty that carries over from one sentence to the subsequent sentence. In the metaphor passages, a neutral conclusion sentence followed the metaphors. If readers did not understand the meaning of a metaphor, they might move on to the next sentence hoping for cues about the meaning of the metaphor. As such, the conclusion sentence was intentionally neutral to remove meaning cues.
   Although the stimulus control properties were described here in terms of metaphors, they apply to most any study of text comprehension or any study that manipulates linguistic stimuli. Consider our prior example in which readers solve word-based math problems that vary in complexity (e.g. high versus low complexity). One would not want the high complexity problems to include more uncommon (very low frequency) words than the low complexity problems because mathematical complexity would be confounded with word frequency. Of course, the goal of the experiment dictates what features need to be controlled. For instance, if the goal of the experiment is to examine how sentence structure influences processing, then sentence structure must be manipulated. To return to our math problem experiment, one might want to explore how different grammatical structures influence the difficulty of solving the problems. For example, sentences containing key details of the problems could be written in active or passive voice. The pattern of eye movements on these key sentences could be measured as well as the influence on determining the correct solutions.

3. Running the Experiment

1. Participants should begin by completing an informed consent that describes the general procedure. Behavioral research such as the experiment described here is commonly approved by an institution's behavioral IRB (Institutional Research Board) as opposed to a medical IRB because video-based eye tracking equipment does not make contact with the eye and is approved as a Class 1 LED device that is safe under all conditions. If eye tracking procedures are combined with medical procedures, such as recording eye movements while undergoing an fMRI, medical IRB will be necessary.
2. Participants should turn off or mute all distracting electronic devices.
3. The height of the chinrest should be set so that the eye being monitored is roughly centered in the video display.
4. Participants should adjust the seat height so they are able to comfortably rest their chin on the chinrest and their forehead against the forehead rest. Participants tend to slouch in the chair as they relax, which tends to pull their forehead away from the forehead rest. This can increase the vertical error in the eye tracking record. This problem can be minimized by making sure participants begin with their chins slightly above the height of the chin rest so that they can rest their chin on the chin rest.
5. Participants should be told how the eye tracker is adjusted and set up before beginning the experiment. Displaying instructions on the monitor gives participants an opportunity to see what the display looks like before beginning the experiment and to further adjust their position on the forehead/chin rest if needed.
6. Make sure that only that eye being recorded is visible in the camera display. This will prevent the tracker from "shifting" to the other eye should participants make a large head movement. If both eyes are in the field of view of the camera, shifting can occur if participants move their heads far enough so that the image of the eye being recorded is moved out of the camera's field of view and the other eye is moved into the camera's view. The eye tracker will then "search" for a new pupil reflection. The tracker will shift back to the original eye when the head is returned to the starting position, but the shifting causes a temporary loss of eye position.
7. Focus the camera (the image will be shown on the experimenter's display). Proper focus increases the ability to detect and track the pupil and corneal reflection.
8. Adjust the infrared sensitivity threshold of the video camera. The EyeLink system has an "auto threshold" function that correctly sets the threshold for the vast majority of participants. If large areas of infrared reflection are visible close to the eye, the threshold can be manually reduced. At this point the eye tracker should detect the pupil and corneal reflection and begin tracking eye position (indicated by cross-hairs over the pupil and corneal reflection).
9. Ensure that the pupil and corneal reflection are being tracked across the entire surface of the display by having participants look at each corner of the computer monitor. If the pupil or corneal reflection is lost at the edges of the display, tilting the participant's head by moving the chin rest base forward or backwards usually resolves the problem. For participants who wear glasses, the frames sometimes occlude part of the video image of the eyes when looking at extreme horizontal or vertical angles. This is only a problem if the pupil and corneal reflection cannot be tracked across the entire area in which the stimuli will be shown. The eye tracker can be calibrated (described next) over a smaller range if necessary to compensate for this problem.
10. Calibration is the process used to set the eye tracking software to accurately track eye movements. This is done by recording eye position while participants fixate a set of nine fixation points (black dots) displayed on the monitor at known locations. The fixation points are presented in a random order. The number of fixation points can be varied depending on how much of the display the stimuli will occupy. If passages fill most of the display, calibration should use a 9-dot formation (top left, top center, top right, middle left, middle center, middle right, bottom left, bottom center, bottom right). If only one line of text is presented in the vertical center of the display, the calibration range may be reduced to the central region of the display.
11. During calibration, instruct participants to fixate each dot until is disappears and try to not predict the dot's movements. If participants move their eyes in an attempt to predict the next location of the dot, the movement of the dot can be controlled manually to ensure participants are fixating each dot before the next dot is shown.
12. Validate the calibration. During validation, participants fixate the same nine points as during calibration. The calculated fixation locations are then compared to the known fixation locations to determine the degree of visual error in calculated fixation locations. At this point the software displays information about the degree of visual error for each fixation point, the average error across all points, and the maximum error across all points.
13. If the average error exceeds 0.5º of visual angle, the setup of the eye tracker must be checked and the calibration process repeated. The average error combines vertical and horizontal errors; therefore, an acceptable average error of 0.3º could, for example, reflect a combination of small horizontal and vertical errors, a combination of small horizontal errors (e.g. 0.1º) and large vertical errors (e.g. 0.6º), or a varying pattern of errors. Consequently, researchers should examine the horizontal and vertical displacement for each calibration point and establish a threshold of acceptable error based on the experiment being performed. For example, if the stimuli are single-line sentences that will appear in the center of the screen, vertical accuracy is less important because there is only one line of text. The prior example of 0.1º horizontal errors and 0.6º of vertical errors might be acceptable for single-line displays. When using multi-line passages, vertical and horizontal accuracy are both critical.
14. Begin the experiment after obtaining an acceptable calibration. Instruct participants to not talk when stimuli are displayed. Talking causes the head to move up and down when resting on the chin rest and this decreases eye tracking accuracy.
15. Begin by presenting a small number of practice trials so that participants become comfortable with the eye tracker, response controller (if used), and the format of the stimuli.
16. Before each trial a fixation point (often called a drift correction point) is displayed where the first word of the text will be located. Instruct participants to fixate the drift correction point before each trial. If the visual error when fixating the drift correction point exceeds the maximum allowable error (0.5º), the system will not allow the trial to begin. At this point recalibration is needed. This ensures a consistently accurate track throughout the experiment. Recalibration typically takes less than one minute because the system is already set to track participants' eyes.
17. Ask participants if they have any questions after completing the practice trials. Participants need to remove their heads from the forehead/chin rest to ask questions. Tracking accuracy should be rechecked after participants return to the forehead/chin rest because their heads will not be in exactly the same position. This can be done by having participants look at drift correction point and comparing the calculated position to the actual position, which is displayed on the experimenter's monitor. For the majority of participants, recalibration is typically not necessary after coming off and then returning to the forehead/chin rest.
18. If at any time participants need to take a break or the quality of the track has degraded (usually due to participants repositioning themselves in their chair), calibration should be checked and recalibration performed as needed. Participants tend to relax their seating positions (slouch) during an experiment, which can change the angle of the head. This can reduce tracking accuracy and lead to the need to recalibrate. Including short breaks every 15-20 min in longer experiments minimizes this problem.
19. Participants should be debriefed after completing the experiment.