We present a protocol for a behavioral analysis of adults (ages 18 to 70-year-old) engaged in learning processes, undertaking tasks designed for Self-Regulated Learning (SRL). The participants, university teachers and students, and adults from the University of Experience, were monitored with eye-tracking devices and the data were analyzed with data-mining techniques.
Behavioral analysis of adults engaged in learning tasks is a major challenge in the field of adult education. Nowadays, in a world of continuous technological changes and scientific advances, there is a need for life-long learning and education within both formal and non-formal educational environments. In response to this challenge, the use of eye-tracking technology and data-mining techniques, respectively, for supervised (mainly prediction) and unsupervised (specifically cluster analysis) learning, provide methods for the detection of forms of learning among users and/or the classification of their learning styles. In this study, a protocol is proposed for the study of learning styles among adults with and without previous knowledge at different ages (18 to 69-year-old) and at different points throughout the learning process (start and end). Statistical analysis-of-variance techniques mean that differences may be detected between the participants by type of learner and previous knowledge of the task. Likewise, the use of unsupervised learning clustering techniques throws light on similar forms of learning among the participants across different groups. All these data will facilitate personalized proposals from the teacher for the presentation of each task at different points in the chain of information processing. It will likewise be easier for the teacher to adapt teaching materials to the learning needs of each student or group of students with similar characteristics.
Eye-tracking methodology applied to behavioral analysis in learning
Eye-tracking methodology, among other functional uses, is applied to the study of human behavior, specifically during task resolution. This technique facilitates monitoring and analysis during the completion of learning tasks1. Specifically, the attention levels of students at different points of the learning process (start, development, and end) in different subjects (History, Mathematics, Science, etc.) can be studied with the use of eye-tracking technology. In addition, if the task includes the use of videos with a voice that guides the learning process, Self-Regulated Learning (SRL) is facilitated. Therefore, the implementation of eye-tracking technology in the analysis of tasks to which SRL (that include the use of videos) is proposed as a significant resource to understand how learning is developed2,3,4. This combination will also mean that the differences between instructional methods (with or without SRL, etc.) may be checked with different types of students (with or without prior knowledge, etc.)5. In contrast, the presentation of multi-channel information (simultaneous presentation of both auditory and visual information, whether verbal, written, or pictorial) can facilitate both the recording and the analysis of relevant versus non-relevant information from the above-mentioned variables6. Students with prior knowledge exposed to multimedia learning channels appear to learn more effectively than those with little or no prior knowledge. Students with high levels of prior knowledge of the subject matter will integrate textual and graphical information more effectively7. This functionality has been observed in the learning of texts8 that include images9. Eye-tracking technology offers information on where attention is focused and for how long. These data give insight into the development of a learning process in a more precise way than through the simple observation of the resolution process during the completion of a task. Also, the analysis of these indicators facilitates the study of whether the student develops deep or superficial learning. Furthermore, the relationship between these data and the learning results facilitates the validation of the information obtained with eye-tracking technology4,10. In fact, this technique together with SRL are increasingly used in Higher Education and in Adult Education11 learning environments, both on regulated and on non-regulated courses12.
Eye-tracking technology offers different metrics: distance, speed, acceleration, density, dispersion, angular velocity, transitions between Areas of Interest (AOI), sequential order of AOI, visits in the fixations, saccades, scan path and heat map parameters. However, the interpretation of these data is complex and requires the use of supervised (regression, decision trees, etc.) and unsupervised (k-means cluster techniques, etc.)13,14 data-mining techniques. These metrics can be applied for monitoring the behavior of the same subject over time or for a comparison between several subjects and their performance with the same task15, by analyzing the difference between participants with previous knowledge versus no previous knowledge16. Recent research11,17 has revealed that novice apprentices fixate longer on the stimuli (i.e., there is a greater fixation frequency while similar scan-path patterns are recorded). The average duration of fixation was longer for experts than for novices. The experts presented their focus of attention on the middle points of the information (proximal and central), differences that may also be seen in the visualization points within the AOI on the heat maps.
Interpretation of metrics in eye tracking
Recent studies18 have indicated that information acquisition is related to the number of ocular fixations on the stimuli. Another important metric is the saccade, which is defined as the rapid and sudden movement of a fixation with an interval of [10 ms, 100 ms]. Sharafi et al. (2015)18 found differences in the number of saccades, depending on the information coding phase of the student. Another relevant parameter is the scan-path, a metric that captures the chronological order of the steps that the participant performs for the resolution of the learning task within the AOI defined by the researcher18. Similarly, eye-tracking technology can be used to predict the participant's level of understanding, which appears to be related to the number of fixations. Recent studies have indicated that variability in gaze behavior is determined by the properties of the image (position, intensity, color, and orientation), the instructions for performing the task, and the type of information processing (learning style) of the participant. These differences are detected by analyzing the student's interaction with the different AOI19. Quantitative20 (frequency analysis) and/or qualitative or dynamic21 (scan path) techniques can be used to analyze the data collected from the different metrics. The former techniques are analyzed with traditional statistical techniques (frequency analysis, mean difference, variance difference, etc.) and the latter are analyzed with Machine Learning techniques (Euclidean distances with string-edit methods21,22, and clustering17). The application of these techniques facilitates clustering, by considering different characteristics of the subjects. One study17 found that the more expert the student, the more effective the spatial and temporal information processing strategy that is implemented. A descriptive table of the measurement parameters that were used in this study can be consulted below in Table 1.
Table 1: Most representative parameters that can be obtained with the eye-tracking technique, adapted from Sáiz, Zaparaín, Marticorena, and Velasco (2019).20 Please click here to download this Table.
Application of the eye-tracking methodology to the study of the learning process
The use of the technological advances and the data-analysis techniques described above5 will add greater precision to behavioral analysis of learners during problem solving in the different phases of information processing (task initiation, information processing, and task resolution). It will all facilitate individual behavioral analysis, which will in turn permit the grouping of students with similar characteristics24. Likewise, predictive techniques (decision trees, regression techniques, etc.)25 can be applied to learning, related both to the number of fixations and to the task-resolution results of each student. This functionality is a very important advance in the knowledge of how each student learns and for the proposal of personalized learning programs within different groups (people with or without learning difficulties26). Therefore, the use of this technique will contribute towards the achievement of personalization and optimization of learning27. Life-long learning must be understood as a cycle of continuous improvement since the knowledge of society is constantly advancing and progressing. Evolutionary psychology indicates that resolution skills and effectiveness in information processing decrease with age. Specifically, saccade frequency, amplitude, and speed of eye movements among adults have been found to decrease with age. In addition, at older ages, attention is focused on the lower areas of visual scenes, which is related to deficits in working memory14. Nevertheless, activation increases in the frontal and prefrontal areas at an older age, which appears to compensate for these deficits in task resolution. This aspect includes the level of previous knowledge and the cognitive compensation strategies that the subject can apply. Experienced participants learn more efficiently, since they manage attention more effectively, due to the application of automated supervision processes28. In addition, if the information to be learned is imparted through SRL techniques, the aforementioned deficiencies are mitigated17. The use of such techniques means that visual tracking patterns are very similar, both in subjects without prior knowledge and in subjects with prior knowledge7.
In summary, the analysis of multimodal-multichannel data on SRL obtained with the use of advanced learning (eye-tracking) technologies is key to understanding the interaction between cognitive, metacognitive, and motivational processes, and their impact on learning29. The results and the study of differences in learning have implications for the design of learning materials and intelligent tutoring systems, both of which will enable personalized learning that is likely to be more effective and satisfactory for the student30.
In this research, there were two investigation questions asked: (1) Will there be significant differences in the learning results and in the ocular fixation parameters between students and expert versus non-expert teachers in Art History differentiating students with official degrees versus students with non-official degrees (University of Experience – Adult education)? and (2) Will clusters of each participant with learning results and ocular fixation parameters coincide with the type of participants (students with official degrees, students with non-official degrees (University of Experience – Adult education) and teachers)?
This protocol was performed in compliance with the procedural regulations of the Bioethical Committee of the University of Burgos (Spain) nº Nº IR27/2019. Prior to their participation, the participants had been made fully aware of the research objectives and had all provided their informed consent. They received no financial compensation for their participation.
1. Participant recruitment
Table 2. Characteristics of the sample. Please click here to download this Table.
2. Experimental Procedure
Table 3. Interview questionnaire. Please click here to download this Table.
Figure 1. Process of eye-tracking calibration Please click here to view a larger version of this figure.
Figure 2. Crossword puzzle to check the acquired knowledge. Please click here to view a larger version of this figure.
Figure 3. Phases of the experimental procedure. Please click here to view a larger version of this figure.
The 36 participants recruited for the present study were from three groups of adults (students from the university of experience, university professors, and undergraduate and master's degree students) with ages ranging between [18 and 69] years (Table 2). The protocol was tested over 20 months at the University of Burgos. An outline of the development can be seen in Table 4.
Table 4. Outline of the development of the learning behavior analysis protocol. Please click here to download this Table.
First, the gaze position parameters of the fixations were analyzed (Table 5). In this study, using a video, the start and end time was the same for all participants: start 0 ms and end 1:14 s, duration 1:14 s.
Table 5. Fixations parameter results. Please click here to download this Table.
Figure 4 shows a graph of the tasks developed by the three groups with respect to the fixation parameters. The group of men and women in each of the participating groups (University of Experience Students, University Teachers and Graduate & Master's Students) each completed the tasks in different ways.
Figure 4. Graph of the three groups and their development of the task related to the fixation parameters. Please click here to view a larger version of this figure.
A two-factor fixed-effects ANOVA (type of participant and previous knowledge) was then applied to check whether there were significant differences in the parameters of fixation positions between the three groups (University of Experience students, University professors and University students). No significant differences were found in any of the fixation parameters, but a trend towards differences for Fixation Average Pupil Size Y, Fixation Average Pupil Diameter, and Fixation Dispersion X was noted, although with low effect values (see Table 6).
Table 6. Two-factor fixed-effects ANOVA (type of participant and prior knowledge) and effect value. Please click here to download this Table.
Subsequently, the k-means cluster was applied to study whether there were different groupings in the initial research group (University of Experience students, University professors and University students) with respect to the results in the parameters of fixation positions, previous knowledge and the crossword puzzle results. Three clusters were found (Table 7). A visualization of the clusters can be seen in Figure 5.
Table 7. Final Cluster Centers. Please click here to download this Table.
Then, a cross table was prepared between the values of the cluster of group membership assigned to each participant with respect to the category type of participant (University of Experience students, University professors and University students) (Table 8). Figure 5 shows the position of the participants within the clusters with respect to the three groups for the fixation position parameters.
Table 8. Participant * Cluster Number of Case Crosstabulation. Please click here to download this Table.
A two-factor fixed-effect ANOVA, "participant group" and "background", was performed for the following eye-tracking measurement parameters: Dwell Time, Glance Duration, Diversion Duration, Glance Count, Fixation Count, Average, and Duration obtained in the task start phase (Slide 1) and in the task end phase (Slide 5) (Table 6). Significant differences were found depending on the background knowledge variable in Diversion Duration 1 (analyzing the input, dwell and output time for each stimulus inserted in each AOI). It can therefore be concluded that the way of entering, remaining, and exiting in the different AOIs was different, depending on the variable "participant group" during the initial phase of information access (F2,32 = 4.07, p = 0.03, η2 = 0.23). Differences were also found in the Average Fixation Duration parameter (longer fixations refer to the participant spending more time analyzing and interpreting the information content within the different AOIs (F2,32 = 3.53, p = 0.04, η2 = 0.21). Bonferroni's mean difference test was applied to establish group membership, with which it was established that they were between the group (University of Experience students) and the group of University Professors [mean difference = 0.04, p = 0.04 CI 95% (0.03-2.75)]. The means were higher for group 1 (University of Experience students) where participants spent more time analyzing and interpreting the AOIs during the data entry phase (see Table 9 and Figure 6).
Table 9. Two-factor fixed-effects ANOVA (type of participant and previous knowledge) and effect value for the eye-tracking measurement parameters: Dwell Time, Glance Duration, Diversion Duration, Glances Count, Fixation Count, Average Fixation and Duration. Please click here to download this Table.
Figure 6. Graph of the three groups and their development of the task involving fixation parameters with respect to the prior knowledge variables at the start and at the end of task processing. Please click here to view a larger version of this figure.
Based on the results found in this study, the development of a personalized learning program was proposed to improve task-resolution learning outcomes. This program was focused on the work with the participants grouped in cluster 3, as they obtained scores of 3 points out of 5 in the test to check the learning results, which represented 85.43% of the total participants. Participants from the three study groups (University of Experience Students; University Teachers and Graduate & Master's Students) were found in this cluster. The program will focus on the reinforcement of the concepts worked on in the video for which an extension and specification of the concepts will be made.
Figure 5. Cluster analysis of the fixation parameters in the three groups (Experience university students, University professors, and University students). Please click here to view a larger version of this figure.
SUPPLEMENTARY FILES
The authorizations for recording the video from which images have been used to reference the observation of functional abilities in children with learning difficulties are attached.
The research results indicated that the average fixation duration on the relevant stimuli was longer among participants with previous knowledge. Likewise, the focus of attention on this group is on the middle points of information (proximal and distal)7. The results of this study have revealed differences in the way participants processed the information. Furthermore, their processing was not always linked to the initial grouping (University of Experience Students, University Teachers and Graduate & Master's Students). These differences were found with respect to the analysis of the participants' visual position on the X-axis in the fixations on the images presented in the learning video. In cluster 2, the highest frequencies of Position X fixation, Position Y fixation, Average Pupil Diameter fixation, X- and Y-axis fixation dispersion were found. This fact coincides with the fact that the members of cluster 2 were themselves members of the University Teachers group, had previous knowledge of the subject, and obtained the highest score in the results verification test. Likewise, it is relevant to point out that most of the participants in the study (85.43%), regardless of their group of origin, were in cluster 3 where they obtained the lowest results in the verification test.
Another relevant aspect of this protocol is the presentation of the information in a video with SRL. This form of presentation, on the one hand, guides and focuses the attention and on the other hand minimizes the individual actions of each participant, which is to say that it unifies the way of accessing the information. Where necessary, it is an aspect that can compensate possible deficits or difficulties in processing. Evidence of this statement is that no significant differences were detected in learning outcomes between the three groups, with the average performance interval [3.60, 4.86] established at 5. Significant differences were only found in the parameters of Diversion Duration of the processing (a parameter used to analyze the input, the permanence and the output time for each stimulus inserted in each AOI) and of Average Fixation Duration (this parameter refers to longer fixations, which indicates that the participant spends more time analyzing and interpreting the content of the information within the different AOIs) at the initial moment of the processing but not at the end of it. The longest times in these parameters were detected among the group of the Graduate & Master's Students and the University of Experience Students in this study. These results are supported by the conclusions of other studies4,12,15. Finally, this protocol can be applied to study differences during information processing, depending on the coding phase of the student.
Based on the above, the first conclusion is that the use of the eye-tracking methodology during the completion of the tasks provided useful data for the study of information processing1,2. Likewise, the data extracted from eye-tracking technology and analyzed with unsupervised learning techniques (clustering) facilitated the knowledge of clusters according to the pre-determined parameters11,15. This aspect is very relevant for the study of information processing in each participant and for the proposal concerning personalized educational responses3,4,5,6,22,23 which is expected to lead to more effective learning23.
It is also important to point out that the experimental results reaffirmed the results of other research with respect to: differences in processing according to novice-expert variables and participant age, and the use of SRL materials that minimize the effects of these variables and increases the expected performance of the participants7,8,9,10.
However, it is necessary to treat any generalization of these results with caution, since convenience sampling was applied, and the work was focused on specific content relating to Art History. Therefore, the sample will be extended in future research and the findings will be checked in other disciplines.
As indicated in the introduction, the presentation of information in a multimodal way through various information channels (auditory, visual or both) with SRL methodology, together with the use of eye-tracking technology and machine-learning techniques, is key to understanding the way in which the participants process the information so as to offer personalized learning design according to the educational needs of each user and, in consequence, promote successful learning and its development among all students29,30.
In sum, the use of the eye-tracking technique is not a methodology of usual application in education frameworks due fundamentally to cost factors from material to personal resources. However, its use is starting to increase little by little and it is important to see its usual use in the observation of tasks in autoregulated learning. The advantage of this eye-tracking technology is that it permits the recording of the learner's interaction with the task which has got advantages of reliability and validity, in this case, on the simple observation of the learning process. In addition, the eye-tracking methodology offers different techniques of registered information visualization and this information in data bases can also be analyzed with computer programs or more powerful tools. Therefore, it opens a wide variety of possibilities for investigation in natural contexts.
The authors have nothing to disclose.
The work has been developed within the Project "Self-Regulated Learning in SmartArt Erasmus+ Adult Education" 2019-1-ES01-KA204-095615-Coordinator 6, funded by the European Commission. The video of the task completion phase had the prior informed consent of Rut Velasco Sáiz. We appreciate the participation of teachers and students in the task implementation phase.
iViewer XTM | iViewer | ||
SMI Experimenter Center 3.0 | SMI | ||
SMI Be Gaze | SMI |