The current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs.
Learning disabilities (LDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations for their chronological age in the areas of reading, writing, and/or mathematics. Each of the disorders making up the LDs involve different deficits; however, some commonalities can be found within that heterogeneity, such in terms of learning self-regulation and metacognition. Unlike in early ages and later educational levels, there are hardly any evidence-based evaluation protocols for adults with LDs. LDs influence academic performance but also have serious consequences in professional, social, and family contexts. In response to this, the current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs. The assessment is carried out through analysis of the on-line learning process using a variety methods, techniques, and sensors (e.g., eye tracking, facial expressions of emotion, physiological responses, concurrent verbalizations, log files, screen recordings of human-machine interactions) and off-line methods (e.g., questionnaires, interviews, and self-report measures). This theoretically-driven and empirically-based guideline aims to provide an accurate assessment of LDs in adulthood in order to design effective prevention and intervention proposals.
Specific learning disorders (SLDs) encompass disorders of those who have difficulty learning and using academic skills, exhibiting performance below expectations for their chronological age in the areas of reading, writing, and/or mathematics1,2. There are different estimations of prevalence rates depending on the age, language and culture analyzed but they are between 5% and 15%1,3. Within the global category of neurodevelopmental disorders in the Diagnostic and Statistical Manual of Mental Disorders (5th Ed.)1, it is also necessary to focus on the incidence of Attention-Deficit/Hyperactivity Disorder (hereinafter ADHD) as it is a common disorder that has given rise to various controversies about how to approach it in recent years. Based on the DSM-51, it can be defined as a pattern of persistent behaviors of inattention and/or hyperactivity-impulsivity. Likewise, autism spectrum disorder (hereinafter ASD) is a category in the same manual that includes students who present neurodevelopmental disorders as a result of multifactorial dysfunctions of the central nervous system, which result in qualitative dysfunctions in three fundamental areas of the development of the person: social interaction, communication and interests and behaviors1,2.
On these lines, a new concept has emerged moving away from the sense of deficit and offering a more positive approach to these disorders to be consistent with current ideas of neurodevelopmental difficulties as highly coexistent and overlapping4. From these new models, it is understood that the skills involved in high-level cognitive processes, which allow managing and regulating one's behavior in order to achieve a desired goal, are crucial for self-regulation and, therefore, for activities of daily living, including the academic ones5. In the context of adulthood, neurodiversity has evolved to include various types of difficulties, including ADHD and ASD, as well as dyslexia, dyspraxia, and/or dyscalculia. Accordingly, we are approaching this neurodiversity from a broad conception of learning difficulties (LDs). The increase in students with this diversity enrolled in postsecondary education is well documented and is due, in part, to the increase in high school graduation rates for students with disabilities6, but at the same time, there is less research about the learning process of these students than necessary7.
Each of the disorders approached in isolation involve different deficits and manifestations; however, some commonality can be found within that heterogeneity in terms of LD, such as metacognitive, self-regulatory, and emotional malfunctioning8,9,10,11. Three fundamental foundations in the literature of learning in general, and LDs in particular, that represent the basis of successful learning and play an essential role in these well-known difficulties at the academic level12. As well as this, other approaches understand that there could be a certain commonality between deficits in executive functions, such as problems in automatic processing or working memory, that occur in different disorders such as ADHD and reading disorders13 or ADHD and ASD5. However, there is still work to be done in this field, since not all studies reach the same conclusions about these points in common in relation to executive functions. It could be due to the variations presented by the samples from which the studies are based and the evaluation procedures of the executive functions used in the investigations5,14.
In educational terms, this diverse mix affects not only the quality of learning, due to the fundamental nature of the affected functions, but also phenomena such as school dropout, change of degree, etc., with economic implications for governments and universities15. The dropout rate for students with LDs is higher than for students in the general population16 but also higher than the dropout rates for any other category of psychological disabilities except for those students with emotional disturbances17. In contrast, the number of students with LDs who are accessing post-compulsory education (vocational training, college, etc.) is increasing15, specifically in higher education19,20,21,22. Moreover, one might well assume that there are many more students with LD than those who officially pass through student services and typically make up the prevalence statistics23.
These difficulties are not always detected during childhood, especially in adults born before these disorders were considered in the regular academic system, and the symptoms of these disorders persist throughout people’s lives and cause difficulties in work, education and personal lives24. Research has shown that although people might overcome some of their difficulties, most continue to exhibit struggles with learning during adulthood and their persistence is still problematic at those higher educational levels25.
Paradoxically, unlike in previous educational levels and earlier ages, there are hardly any evidence-based instruments or evaluation protocols for adults with LDs. Despite the proliferation of diagnostic tools to evaluate LDs during childhood, the availability of valid, reliable instruments and methodologies for the adult population is significantly limited24. A recent literature review about learning disabilities in higher education found that most of the information collected in this regard is done through interviews, and only occasionally are self-report questionnaires used26. Self-report methodology and interviews, although valuable, are not enough to accurately assess metacognitive, self-regulation, and emotional skills processes, in fact, among others, because of the process nature. The importance of scales and interview methodology for measuring those processes is undeniable27,28, but so too are the associated problems of validity29 and incongruence with other innovative methods of assessment30. An additional problem in the detection of LDs is the bias in the diagnosis of the disorder due to the absence of comprehensive assessment protocols. The fact that professionals do not have a reference protocol based on objective variables is frequently causing many false positive and false negative cases of LDs31.
In response to both scarcity of instruments for adults and the need to improve existing methodology, the current study proposes a multimodal evaluation protocol focused on metacognitive, self-regulation, and emotional processes, which make up the basis of the difficulties in adults with LDs. In line with the current literature, we propose a move toward integrative and multichannel measurement32,33. The assessment is carried out through an analysis of the on-line learning process using several methods, techniques, and sensors (e.g., hypermedia learning environment, virtual reality, eye tracking, facial expressions of emotion, physiological responses, log files, screen recordings of human-machine interactions) and off-line methods (e.g., questionnaires, interviews, and self-report measures). This mixed methodology provides evidence of the deployment of target processes before, during, and after learning that can be triangulated to enhance the understanding of how students learn and where the problem lies, if there is one34.
The evaluation protocol is carried out over two sessions. The sessions can be done in one sitting or may need partial applications depending on the person. The first is focused on the detection or confirmation of LDs and what specific kind of disorder we are facing, and the second is designed to go into the metacognitive, self-regulation, and emotional processes of each individual case in depth.
Session 1 is intended to be a diagnostic or confirmation assessment of the participant’s learning disabilities: SLD, ADHD and/or ASD (high functioning) to determine what type of specific problems the participants have. This assessment is essential for two reasons. 1) Adults with Learning Disabilities rarely have accurate information about their dysfunctional behavior. Some of them suspect that they have a LD but have never been evaluated. Others may have been assessed when they were children but do not have any reports or further information. 2) There may be discrepancies with previous diagnoses (e.g., a previous dyslexia diagnosis as opposed to a current diagnosis of attention deficit and slow processing speed; previous ASD diagnosis in contrast to current limited intellectual ability, etc.). The participant is interviewed, and questionnaires and standardized tests are applied. This session here is carried out by therapists with experience in diagnosing developmental and learning difficulties in the research and clinical context in different offices of a Spanish Psychology Faculty. The session begins with a structured interview that collects biographical information along with the presence of symptoms related to SLDs that are referred to in the DSM-51. Following that, the reference intellectual ability test WAIS-IV35 is used in case of exclusion criterion implementation and because it provides very valuable information for learning difficulties from the scales “work memory” and “processing speed”36. Additionally, the PROLEC SE-Revised Test37 is extensively used to evaluate reading disabilities (lexical, semantic and/or syntactic processes of reading), one of the most prevalent and disabling difficulties for learning in current academic contexts, which overlaps with other disorders such as ADHD38. This evaluation collects reading accuracy, speed and fluency along with reading disabilities, and more importantly, in which reading process the failure occurs37 (this test has been evaluated with pre-university students. Currently, there are no tests in Spain that are adapted to the general adult population, so this test was selected because it is the closest to the target population). Then, we screen symptoms of ADHD through the World Health Organization Adult ADHD Self-Report Scale (ASRS)39 and refine the evaluation of this disorder, introducing multimodality with a cutting-edge virtual reality continuous performance test for the evaluation of attentional processes and working memory in adults, the Nesplora Aquarium31,40. This test is a very useful tool when diagnosing ADHD in adults and adolescents over 16 years old in an ecological scenario, providing objective, reliable data. It evaluates selective and sustained attention, impulsivity, reaction time, auditory and visual attention, perseverance, quality of attentional focus, motor activity, work memory and cost of change of task. Additionally, along with the WAIS-IV35 as a whole for collecting information about the participant’s intellectual ability, we pay special attention to the scales “work memory” and “processing speed” because they are related to learning difficulties and the results of these scales are used in the final decision. Finally, we include the Autism Spectrum Quotient (AQ-Short)41 in the protocol, the short version of the reliable AQ-Adult from Baron-Cohen, Wheelwright, Skinner, Martin and Clubley42.
Session 2 focuses on a multimodal assessment of the participant’s learning process. The key to understanding complex learning lies in understanding the deployment of students’ cognitive, metacognitive, motivational, and affective processes43. To that end, participants work with MetaTutor, where the use of metacognitive and cognitive strategies deployed are observed while they are learning. MetaTutor is a hypermedia learning environment that is designed to detect, model, trace, and foster students’ self-regulated learning while learning different science topic44. The design of MetaTutor is based on extensive research by Azevedo and colleagues43,45,46,47 and belongs to a new trend in the measurement of SRL, the so called third wave, which is characterized by combined use of measurement and advanced learning technologies33. The use of MetaTutor also provides multimodal trace data, incorporating measures such as, eye tracking, emotional physiological responses (galvanic skin response (GSR) and facial expressions of emotions)48, log-data and questionnaires. All these measures are combined to reach a deeper understanding of the participants SRL and metacognition.
Eye tracking provides an understanding of what attracts immediate attention, which target elements are ignored, in which order elements are noticed, or how elements compare to others; electrodermal activity lets us know how emotional arousal changes in response to the environment; facial-emotion-recognition allows the automatic recognition and analysis of facial expressions; and data logging collects and stores the student´s interaction with the learning environment for further analysis. Concerning the questionnaires, the Mini International Personality Item Pool49 informs about a range of activities and thoughts that people experience in everyday life assessing each of the five major personality traits (extraversion, agreeableness, conscientiousness, neuroticism and openness). The Connotative Aspects of Epistemological Beliefs50 provides information about participants’ beliefs about knowledge. The Rosenberg Self-esteem scale shows how the participants feel about themselves overall51. The Emotion Regulation Questionnaire52 provides information about participants’ emotion regulation. The Achievement Emotions Questionnaire (AEQ)53 informs about emotions typically experienced at university.
In short, assessing LDs during adulthood is particularly difficult. Education and experience allow many adults to compensate for their deficits and later show undifferentiated or masked symptoms, on which scientific knowledge is still scarce. Taking into account the critical research gap that arises, this current work aims to ensure theoretically-driven, empirically-based guidelines for accurate assessment of LDs during adulthood in order to design effective prevention and intervention actions.
To help readers decide whether the method described is appropriate or not, it is necessary to specify that the protocol is not suitable for people with intellectual disabilities because their diagnosis invalidates the diagnosis of learning difficulties. In addition, due to the singularities of the equipment used and the format of showing the learning content, it is still not possible to evaluate people with motor disabilities (upper limbs, neck and/or face), hearing or visual impairment. Nor would it be suitable for participants with severe psychiatric disorders. It would require the use of drugs that could alter information processing or the physiological expression of emotions.
The research ethics committee of the Principality of Asturias and the University of Oviedo approved this protocol.
1. Session 1: diagnosis assessment
NOTE: In this session of the protocol, evaluation tests from different publishers are used, which have their own specific application and interpretation manuals. Since these tests, or other similar ones, are widely known by the scientific community in the field of psychology and education, the procedure to apply them is not detailed step by step (for example, given the aim of this paper, it does not make sense to detail each step of the WAIS-IV35 application).
2. Session 2: multimodal assessment
NOTE: Session 2 must be done between 1 and 7 days after Session 1.
3. Logoff
4. Analysis of learning difficulties
This section illustrates the representative results obtained from the protocol, including an example of conjoint results of Session 1 and an example of each source of information from Session 2.
The results about disorders are collected in Session 1 through diagnostic tests taking into account the procedures and cut-off points specified for the diagnostic assessment of participants’ learning difficulties (SLD, ADHD, and ASD). The expert committee decides whether the participant has learning disabilities or is at risk of having them or not (see an example of decision making in Figure 1). If the participant exhibits learning disabilities and takes part in Session 2, data from alternative sources are collected.
During Session 2 the protocol collects results from five different sources: participants´ GSR, face emotions, eye-movements, questionnaires and log-data.
Firstly, we obtain a measure of the GSR as an indication of emotional arousal during learning session (calm/excited)56. Learning disabilities are linked to anxiety in adults, and several studies have found that students with learning disabilities from first grade to university report higher anxiety symptoms, acting as a factor in decreased performance57,58,59. However, there is no one-to-one relationship between understanding and remediation; every case needs to be analyzed individually by the expert committee taking into account each participant’s specific baseline. Figure 2 shows two paradigmatic cases that can show us whether anxiety regulation is a key point for intervention.
Secondly, we obtain a recording of the participant’s face throughout the session that show us the different emotions they were feeling during the learning process to consider the theoretical relationship with metacognition and self-regulation. There is a variety of facial-emotion-recognition software to gather that information. In the current protocol, we use a tool55, which includes emotion recognition, returning the confidence across a set of emotions for each face in the video (disgust, fear, anger, happiness, contempt, neutral, sadness, and surprise). These emotions are understood to be cross-culturally and universally communicated with specific facial expressions60. Participants tended to experience all the detected emotions during the session, but we can obtain a general index for each giving information about the general trend. Positive activating emotions such as happiness, surprise and enjoyment, are thought to promote both intrinsic and extrinsic motivation, facilitating use of flexible learning strategies, and fostering self-regulation. Conversely, negative deactivating emotions, such as boredom and sadness, are posited to uniformly reduce motivation and the effortful processing of information, producing negative effects on learning outcomes. For neutral deactivating and negative activating emotions, such as anger, fear, contempt, and disgust, the relationships are presumed to be more complex. Specifically, anger and fear can undermine intrinsic motivation, but can induce strong extrinsic motivation to invest effort to avoid failure, meaning that the effects on students’ learning need not be negative53 (see Figure 3). The results indicate the degree of coincidence with one of the analyzed emotions, assigning values between 0 and 1 to each of them.
Thirdly, we use data from eye-tracking. Eye-trackers capture gaze information in terms of fixations, and saccades (Figure 4). In the current protocol, we are interested in analyzing fixations, particularly the proportion of fixation time and pattern of fixations. For that purpose, we defined seven Areas of interest (AOIs) in the MetaTutor interface for self-regulation assessment (labeled with rectangles in Figure 5): AOI1 Timer, AOI2 Goal and Sub goals, AOI3 Agent/avatar for scaffolding, AOI4 Table of Contents, AOI5 Text Content, AOI6 Image Content, AOI7 Learning Strategies Palette.
In terms of assessment for concise intervention guidance, we can infer the following.
Fixations in AOI1 denote time management and/or resource management strategies. Reduced or massive fixations in AOI1 denote incorrect time management skills. It should be checked promptly.
Fixations in AOI2 denote planning, setting and prioritizing goals and sub-goals. Previous studies show that this particular AOI, along with the AOI7, is especially important for assessing learning with MetaTutor61. As this information is concise, short and visual, the proportion of fixations should not be very high (Figure 6).
Fixations in AOI3 Agent show that the participant is taking advantage of the prompts and feedback which the agents provide during the interaction in response to participants’ goals, behaviors, self-evaluations, and progress. It is worth noting that a lack of fixations on the Agent AOI must be considered carefully, because learners may not always need to look at an agent to process its audio prompts and feedback61. This AOI should be checked occasionally. Avatars do not speak frequently, so there should be a small percentage of fixations compared to other areas, but it would reflect that they have established an interaction with the agent (Figure 6).
Fixations in AOI4 and/or transitions between text and image/graph (AOI5 and AOI6) point to participants’ strategy-use for coordinating informational sources (COIS), associated with conceptual gains45. The length of fixations on texts and images indicate integration processes contributing to accurate mental representations of the information presented62. COIS are operationalized as a sequence of two transitions between eye fixations on text and image/graph areas (e.g., text/graph/text). AOI4 should be checked with some frequency. As the information is clear, short and visual, the proportion of fixations should not be very high. The highest proportion of fixations should be in AOI5 and AOI6. The subject should spend most of their time reviewing the content (i.e. the written texts) and spend a notable amount of time on the images and graphs to coordinate and integrate both sources of knowledge (Figure 6).
Fixations on AOI7 indicate the use of cognitive strategies (taking notes, writing a summary, making an inference) and metacognitive strategies (activating prior knowledge, evaluating content relevance, assessing understanding and knowledge)63. It is reasonable for the participant to review the available resources or learning strategies with some frequency (Figure 6).
For the subsequent analysis, it is necessary to focus on data related to students interacting with MetaTutor, excluding the parts of the interaction during which participants watch system tutorials. The collected data can be noisy and needs expert validation. The main source of noise is due to participants looking away from the screen, which the eye-tracker interprets as invalid data; in this case, it is advisable to remove the corresponding segments from gaze data. Figure 6 shows a participant with metacognitive malfunctioning and a participant with an adaptive use of strategies at this level.
Fourthly, questionnaires are analyzed together with the rest of the information and are scored according to the authors’ instructions. They provide data at the participant level of self-esteem and emotional regulation. A favorable level of self-esteem or correct emotional regulation strategies facilitates learning processes64. To see examples of interpretation (Figure 7).
Finally, all interactions of learners with content, agents, and the learning environment are recorded in logs for further detailed analysis following the scheme in Figure 8. The MetaTutor log data provides us with a wide range of possibilities for determining, among other things, the number of times that learners deployed self-regulatory learning strategies (e.g., note-taking, summaries, monitoring progress toward goal, content evaluation, judgments of learning, feelings of knowing, planning, prior knowledge activation, etc.), whether these strategies were self or externally generated by the external scaffolding, and the time each participant spent viewing material in MetaTutor that was relevant/irrelevant to their current active sub-goal65,66. Pattern Mining, Process Mining, Association Rules, and other potential approaches67,68 would provide a measure of students’ use of cognitive and metacognitive monitoring and regulation throughout the learning session.
Figure 1. Example of making decision points of Session 1. This case shows a participant that has had learning problems since childhood, mostly in reading processes. The expert can see that these reading disabilities are more significant in lexical and syntactic processes (b). In addition, it is observed that the participant does not have any motor, sensory or mental disability. It is observed that the participant has a normal intellectual ability and is not at risk in relation to autism spectrum disorder or ADHD (a) omissions, commissions and reaction time, in visual and auditory channels, are less than 60, so are in the normal range). In this case, reading problems are detected and exclusion criteria are not observed, so it is considered that the participant has SLD due to reading disabilities. Please click here to view a larger version of this figure.
Figure 2. Results of a stable activation level and unstable activation level during learning session. This image represents the results of two participants. Participant A with stable activation levels and participant B with unstable activation levels during the learning session since the participant B line is more irregular and with many peaks. Please click here to view a larger version of this figure.
Figure 3. Image of emotion recognition. a) Example of neutral emotion; b) Example of sadness emotion; and c) Example of happiness emotion trend. In the yellow circle it is possible to see the emotion trend. Please click here to view a larger version of this figure.
Figure 4. Example showing transition data between text and graph (AOI5 and AOI6) during a MetaTutor learning session. Circles and lines represent areas of fixation and transitions between areas. Please click here to view a larger version of this figure.
Figure 5. Areas of interest (AOIs) of the MetaTutor interface for the self-regulation assessment: AOI1 to AOI7. AOI1 Timer, AOI2 Goal and Sub-goals, AOI3 Agent, AOI4 Table of Contents, AOI5 Text Content, AOI 6 Image Content, AOI7 Learning Strategies Palette. Please click here to view a larger version of this figure.
Figure 6. Proportion of fixations in the MetaTutor interface AOIs expressed as a percentage. a) Example of a participant deploying self-regulation malfunctioning; b) Example of a participant deploying self-regulatory behaviors. Proportion of fixations in each area (values between 0 and 1). a) Real data from a participant that spends more than 80% of the time reading the written text (AOI5) he underuses the resources designed to help him understand that content (AOI6); he hardly reviews the content scheme to check what he has already learned and what is left to learn (AOI4); neglects learning objectives and sub-goals (AOI2) and he rarely reviews the palette of learning strategies (AOI7). In addition, he does not monitor the time assigned to the task (AOI1) and ignores the avatars that try to help him (AOI3); b) Real data from a participant that spends half the time (50% approximately) reading the written text (AOI5) and frequently reviews the graph designed to help him to understand the content (AOI6). Although he spends most of his time on content, he reviews the content scheme frequently to check what he has learned and what he has left to learn (AOI4); he pays attention to learning objectives and sub-objectives (AOI2) to ensure that he is reaching them and he goes to the learning strategies palette (AOI7) when needed. In addition, he monitors the time without worrying too much about it (AOI1) and establishes some interaction with agents (AOI3). Please click here to view a larger version of this figure.
Figure 7. Example of interpretation of the questionnaires results. In graphic left) Rosenberg self-esteem scale51, higher scores indicate higher self-esteem (minimum = 10; maximum = 40). In graphic right), Emotion Regulation Questionnaire52, cognitive Reappraisal (minimum = 7; maximum = 42); Expressive Suppression (minimum = 4; maximum = 28). Higher scores indicate higher use of reappraisal or suppression strategies. Cognitive reappraisal is a form of change at the cognitive level that helps one to interpret a situation that provokes emotions in another way, thereby changing their emotional impact (using reappraisal strategies help one to think about negative situations and about some alternative construal to resolve them). Expressive suppression is a form of response modulation that involves inhibiting ongoing emotion-expressive behavior (recurrent users of suppression strategies should have less understanding of their moods, view them less favorably, and manage them less successfully). Please click here to view a larger version of this figure.
Figure 8. Log data processing. This image represents the management of log data. The system collects the raw interaction data between the student and MetaTutor, then performs data preprocessing to subsequently apply Learning Analytics and/or Data Mining technics for discovering, analyzing or visualizing the complete learning process. Please click here to view a larger version of this figure.
The current protocol proposes a multimodal evaluation focused on metacognitive, self-regulation, and emotional processes, which make up the basis of the difficulties in adults with LDs.
Session 1 is essential because it is intended to be a diagnostic assessment of the participant’s learning disabilities. Note that this session here is carried out by therapists with experience in diagnosing developmental and learning difficulties in the research and clinical context. We use these tools in Spain, so researchers from other countries should select tests adapted to their population. The significance of the method with respect to existing methods is that many of the scales for ADHD, SLDs and ASD were designed for use in children, with neuropsychological testing and neuroimaging being the better, but less realistic, alternative to this paucity of instruments24. Additionally, all the aforementioned disabilities are usually evaluated through their specific symptoms in isolation, without taking into account well-known commonalities found in LDs, such as metacognitive, self-regulatory, and emotional malfunctioning. In any case, most of the knowledge about metacognition, self-regulation and emotions is based on self-reported data at early or adult ages. However, self-reports of any kind are vulnerable to various types of biases69 and several times no correlations between physiological and self-reported data have been found in LD samples70.
For this reason, Session 2 of the protocol is critical. It focuses on the core processes of learning (metacognitive, self-regulation, and emotional behavior), the significance of the method compared to alternative methods is that it is a multimodal assessment of the participant’s learning process providing multichannel trace data. The tool that makes the integration of all those sources of information possible is MetaTutor43, a metacognitive tool based on advanced learning technology and one of the best representatives and most well-known lines of research of the so-called third wave of self-regulation measurement33.
Regarding galvanic skin responses, the majority of psychophysiological studies of LD subjects have focused on one of three related topics: arousal, orienting, and attention. In this protocol, arousal provides a unique framework for understanding emotion and cognition that cannot be provided by static measures like self-reports71. With facial expressions, previous research has indicated that academic emotions are significantly related to students' motivation, learning strategies, cognitive resources, self-regulation, and academic achievement72. When it comes to eye movements, we know the value of gaze data in predicting student learning during interaction with MetaTutor61 and multiple researchers have suggested that the duration of fixations indicate deeper cognitive processing during learning73. The questionnaires provide complementary information about participants’ performance during the learning session in MetaTutor, their perceptions of themselves as learners and their behavior when they learn. Finally, the log data is an additional source of information about participants’ self-regulatory processes. After the collection of raw data and data preprocessing, emerging Learning Analytics and Educational Data Mining techniques let us discover, analyze and visualize, or to put it another way, dive into the learning process74,75,76.
This mixed methodology provides evidence of the deployment of target processes before, during, and after learning that can be triangulated to enhance our understanding of how adults with LDs learn and where problems lie.
This proposal is a protocol, which means a procedure and system of instruments, so it is advisable to remember that the proposed measures do not have the same value in isolation as they do when they form part of the whole, and therein lies the interest in this proposal. The objective is to converge those data streams, to understand how adults with LDs monitor and control their cognitive, metacognitive, and affective processes during learning.
Although this protocol is an effective toolbox for screening and diagnosis by the practicing psychologist, it is not without limitations. Diagnosis of adult LDs is particularly difficult. Education and experience allow many adults to compensate for their deficits and these adults subsequently show individual characteristics on testing24. As the results indicate, it is difficult to provide accurate cut-off points from some of the data sources (e.g., GSR, log data, etc.) as a general rule in the target population.
Another challenge, rather than limitation, is about the complexity in dealing with the resulting complex, noisy, messy data, which needs the involvement of experts from different domains such as psychologists, physiologists, computer and educational scientists, etc. As recently noted by Azevedo and Gašević77 we need to integrate a complex mosaic of theoretical models and frameworks from the psychological, educational, instructional, and computational sciences. In addition to this, instrumentation errors, internal and external validity, ecological validity versus experimental rigor, converging data channels, and inferences about process data are only some of the methodological issues that result from collecting multimodal multichannel data that researchers must address77,78.
Nonetheless, the future direction of this methodology surpasses the goal of assessment, currently the possibility is open to use real-time multimodal multichannel data to design preventive interventions based on Adaptive Hypermedia Learning Environments79 or provide learners with real-time, intelligent, adaptive scaffolding (modeling cognitive strategies, regulating metacognition via an artificial agent, prompting emotion regulation, introducing visualization tools to discover hidden processes, etc.)77,80.
Finally, LDs should be tracked over their lifetimes; the longitudinal course of SLDs, ADHD and ASD and their long-term sequelae are only beginning to be explored21. We hope that widespread use of this theoretically-driven, empirically-based guideline will help to identify the population of adults with LDs and spur deeper understanding of these disorders in order to design effective prevention and intervention actions.
The authors have nothing to disclose.
This manuscript was supported by funding from the National Science Foundation (DRL#1660878, DRL#1661202, DUE#1761178, DRL#1916417), the Social Sciences and Humanities Research Council of Canada (SSHRC 895-2011-1006), the Ministry of Sciences and Innovation I+D+i (PID2019-107201GB-100), and the European Union through the European Regional Development Funds (ERDF) and the Principality of Asturias (FC-GRUPIN-IDI/2018/000199). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or Social Sciences and Humanities Research Council of Canada. The authors would also like to thank members of the SMART Lab at UCF for their assistance and contributions.
AQUARIUM | Nesplora | ||
Eye-tracker RED500 Systems | SensoMotoric Instruments GmbH | ||
Face API | Microsoft | ||
GSR NUL-217 | NeuLog |