An experimental design was developed to investigate the real-time influences of an examination experience to assess the emotional realities students experience in higher education settings and tasks. This design is the result of a cross-disciplinary (e.g., educational psychology, biology, physiology, engineering) and multi-modal (e.g., salivary markers, surveys, electrodermal sensor) approach.
Over the past ten years, research into students' emotions in educational environments has increased. Although researchers have called for more studies that rely on objective measures of emotional experience, limitations on utilizing multi-modal data sources exist. Studies of emotion and emotional regulation in classrooms traditionally rely on survey instruments, experience-sampling, artifacts, interviews, or observational procedures. These methods, while valuable, are mainly dependent on participant or observer subjectivity and is limited in its authentic measurement of students' real-time performance to a classroom activity or task. The latter, in particular, poses a stumbling block to many scholars seeking to objectively measure emotions and other related measures in the classroom, in real-time.
The purpose of this work is to present a protocol to experimentally study students' real-time responses to exam experiences during an authentic assessment situation. For this, a team of educational psychologists, engineers, and engineering education researchers designed an experimental protocol that retained the limits required for accurate physiological sensor measurement, best-practices of salivary collection, and an authentic testing environment. In particular, existing studies that rely on physiological sensors are conducted in experimental environments that are disconnected from educational settings (e.g., Trier Stress Test), detached in time (e.g., before or after a task), or introduce analysis error (e.g., use of sensors in environments where students are likely to move). This limits our understanding of students' real-time responses to classroom activities and tasks. Furthermore, recent research has called for more considerations to be covered around issues of recruitment, replicability, validity, setups, data cleaning, preliminary analysis, and particular circumstances (e.g., adding a variable in the experimental design) in academic emotions research that relies on multi-modal approaches.
Psychologists have long understood the importance of humans' emotions in elucidating their behaviors1. Within the study of education, Academic Achievement Emotions (AEE) has become the focus of emotion research2. Researchers that use AAE argue that the situational contexts students find themselves in are important to consider when examining students' emotions. Students may experience test-related, class-related, or learning-related emotions that involve multi-component processes, including affective, physiological, motivational, and cognitive components. AEE is expressed in two forms: valence (positive/negative) and activation (focused/unfocused energy). Positive activating emotions, such as enjoyment, may increase reflective processes like metacognition, whereas positive deactivating emotions such as pride may result in low levels of cognitive processing. Negative activating emotions such as anger and anxiety may spark engagement, whereas negative deactivating emotions such as hopelessness may dampen motivation3,4,5. Academic emotions contribute to how we learn, perceive, decide, respond, and problem-solve2. To regulate academic emotions, an individual must possess self-efficacy (SE)6,7,8, which is their confidence in their ability to employ control over their motivation, behavior, and social environment6. Self-efficacy and academic emotions are interrelated, where lower self-efficacy is tied to negative deactivating emotions (e.g., anxiety, anger, boredom) and higher self-efficacy is tied to positive activating emotions (e.g., happiness, hope, excitement)6,7,8. SE is also believed to be strongly tied to performance6,7,8.
Research that has examined classroom emotions have relied on self-reports, observations, interviews, and artifacts (e.g., exams, projects)9,10. Although these methods provide rich contextual information about students' classroom experiences, they have significant limitations. For example, interviews, observations, and self-reports rely on individuals' introspections10. Other methods have sought to examine academic emotions more proximally than prior researchers, such as those based on experience-sampling approaches where researchers ask students to report on their emotions during the school day11. Although this research allows us to report students' emotions more accurately, this work relies on self-report methods and does not allow for real-time reporting as students have to pause their work on the exam to address the experience survey.
Recently, researchers have begun to address concerns about self-report measures through the use of biological or physiological measures of emotion9, that combined with other instruments or techniques such as surveys, observations, or interviews, consists of a multi-modal form of data collection for educational and psychological research12. For example, biological techniques, including salivary biomarkers, are being used to understand the role biological processes have on cognition, emotion, learning, and performance13,14,15. For cognitive processes, androgens (e.g., testosterone) have been linked to different spatial recognition patterns in adults and children16,17 whereas hypothalamic-pituitary-adrenocortical hormones (e.g., cortisol) and adrenergic hormones (e.g., salivary α-amylase or sAA) are linked to stress responsiveness amongst individuals18,19,20.
Electrodermal activity (EDA) represents a physiological measure of the activation of the autonomic nervous system (ANS) and is linked to increased activation of the system, cognitive load, or intense emotional responses21,22,23. In examination activities, EDA is affected by physical mobility21,22, bodily and ambient temperatures24,25,26,27, and verbalization of thoughts28, as well as sensitivity and degree of connectivity of the analog-digital electrodes to the skin29.
Although these can be limitations to using EDA, this technique can still provide valuable insight into what happens during near-real-time examinations and can serve as a promising tool to explore AEE and by extent, self-efficacy. As a result, an accurate picture of students' AEE can be obtained through a combination of survey methods, to determine the valence of emotion, and physiological and biological data, to measure the activation of that emotion. This paper builds upon a previous publication on examination activities30 and expands the scope of that work to include multi-modal approaches (using experience-sampling surveys, EDA sensors, and salivary biomarkers) in an examination scenario. It is essential to mention that the protocol described below allows for multiple participant data to be collected at the same time within a single experimental setting.
Procedures were approved by the Institutional Review Board (IRB) under a general review at Utah State University for studies on human subjects and use of these constructs. The typical results include two semesters of an engineering statics course, each with a slightly different experimental setup, at a western institution of higher education in the United States. Practice exams, whose content paralleled the actual exams, were developed by the course instructor and were used for our study. Please note that the protocol outlined below describes concurrent steps, and some steps may overlap.
1. Considerations for Experimental Designs and Integration of Disciplinary Practices
As researchers consider experimental designs of this nature, disciplinary knowledge and approaches must be integrated in a way that complements and sustains the main research goal. As new instruments and methods are added, additional validation considerations are needed. In this work, we will explore an experimental study where surveys and electrodermal sensors were used for one of the semesters (experimental design A), and salivary biomarker collection (i.e., cortisol and sAA) was added to the subsequent semester (experimental design B). Below are the considerations for the two setups:
2. Setup and cleaning pre- and post-experiment
3. Increasing ecological validity in light of surveys, electrodermal sensors, and salivary biomarkers
4. Considerations for data processing and analysis
contains the 16 zip files. Each zip file contains all of the EOL quizzes for a given Core chapter. They will want to unzip this file, and then each zip file gets loaded individually into Canvas.
In this study, we were interested in studying the influences of self-efficacy, performance, and physiological (EDA sensors) and biological (sAA and cortisol) responses of undergraduate engineering students as they took a practice exam. The data shown is a representative subset of samples: (a) one that considered surveys and electrodermal sensors (experiment design A) and (b) one that included the same exam along with the salivary biomarker data (experiment design B). While we collected emotions data in this study, we will not present it, as our goal was to demonstrate granular data in real-time rather than at prescribed timepoints at the beginning, middle, or end of the exam, which is where emotions data was collected.
As shown in Figure 4, the degree of difficulty of the exam according to the collective response of students was compared across the experimental designs. Also, the mean EDA as a function of students´ reported self-efficacy scores before completing the exam questions was plotted. Even though the degree of difficulty was the same for the two designs, opposing differences in the mean EDA values were found between the correct and incorrect responses across different self-efficacy scores. For experimental design A (EDA sensors and surveys), mean EDA increased for a mid-SE score for students who responded incorrectly to the exam questions compared to students who responded the questions correctly (p < 0.001). For experimental design B (EDA sensors, surveys, and salivary biomarkers), mean EDA values varied where an opposite effect was found for low SE scores (p < 0.05) and high SE scores (p < 0.01), respectively.
To understand any potential salivary influences, the mean EDA as well as cortisol and sAA assay values for set data points in the exam (beginning, middle, end, and 20-minutes after the exam) were normalized (Figure 5) for experimental design B. It is important to note that the mean EDA values for this table were truncated at 60-s intervals during the pre-set timeframe to allow for comparisons between each salivary marker. The data suggest that EDA levels decreased from beginning to the end of the exam, and these levels recovered by the 20-minute mark after the exam. These trends were paralleled in the cortisol and sAA data. Statistical significance, as determined through ANOVA, was found between EDA and sAA at the beginning and middle of the exam (p < 0.05 for both times) whereas EDA and cortisol showed significance between the middle and end of the exam (p < 0.01 and p < 0.05, respectively). By the 20-minute mark, EDA and sAA (p < 0.01) and cortisol and sAA (p < 0.05) began to show significance between each other.
Figure 1. Experimental setup when using surveys and electrodermal sensors to study examination experiences. The image shows Experimental Design A (sensors and survey) and B (sensors, survey, and salivary biomarkers). Please click here to view a larger version of this figure.
Figure 2. A schematic representation of how participants can fit and start the electrodermal sensor. Image A (in the left) shows the placement of the start button on the sensor while Image B (on the right) shows the placement of the EDA electrodes on the wrist of the participant. Please click here to view a larger version of this figure.
Figure 3. Representation of an experimental timeline when surveys, salivary biomarkers, and electrodermal sensors are included. Please click here to view a larger version of this figure.
Figure 4. Degree of difficulty. Degree of difficulty of the exam according to collective student performance and mean EDA as a function of self-efficacy scale ranking by participants for the correct and incorrect responses for experimental design A (A and B) and experimental design B (C and D). N = 15 participants per design; data is reported as mean ± standard error of the mean (represented in the error bars); dashed lines on panels A and C represent the limits for moderate ranges of difficulty (between 0.3 to 0.8)52; *p < 0.05, **p < 0.01, and ***p < 0.001, implying a statistically significant difference. Please click here to view a larger version of this figure.
Figure 5. Normalized sAA, cortisol and mean EDA. Normalized sAA, cortisol and mean EDA for experimental design B compared at 60-s intervals at prescribed time periods during the exam (beginning, middle, end, 20 minutes after). N = 15; data is reported at mean ± standard error of the mean (represented in the error bars); *p < 0.05 and **p < 0.01, implying a statistically significant difference. Please click here to view a larger version of this figure.
Although physiological measures have been used in many authentic learning contexts, it is critical to design a study environment that is mindful of the limits of the current technology. Our design balances the need for an authentic testing environment and accommodates the technology. Comfortably limiting participant movement, reducing unintended interruptions, and timestamping participants' testing responses are all critical steps within the protocol.
The space and expense of the electrodermal sensor devices may make the study impractical for researchers with limited research funds. However, once purchased, these sensors have unlimited uses. Salivary biomarkers must be processed in a laboratory and have significant per-sample pre- and post-processing expenses. It is also important to consider the particular laboratory conditions and equipment used, as alternate salivary assay validation methods may be needed to identify inter- and intra-assay percentages of CV.
The protocol is a significant step forward in the application of multi-modal approaches in the study of academic emotions. The protocol maximizes the precision of EDA measurements by timestamping participant responses while replicating an authentic testing environment, which enables more objective real-time studies of student coursework and classroom studies, addressing a constraint that limited prior research studies focused on learning and performance. It is possible to modify the technique to include online learning activities that require keystroke capture. It is also possible to use the protocol for deception studies in where the difficulty of the test or present text-based prompts are pre-designed to influence students' expectations for the test.
The authors have nothing to disclose.
This material is based upon work supported in part by the National Science Foundation (NSF) No. EED-1661100 as well as an NSF GRFP grant given to Darcie Christensen (No. 120214). Any opinions, findings, and conclusions or recommendations expressed in this material do not necessarily reflect those of NSF or USU. We want to thank Sheree Benson for her kind discussions and recommendations for our statistical analysis.
Author contributions in this paper are as follows: Villanueva (research design, data collection and analysis, writing, editing); Husman (research design, data collection, writing, editing); Christensen (data collection and analysis, writing, editing); Youmans (data collection and analysis, writing, and editing); Khan (data collection and analysis, writing, editing); Vicioso (data collection and analysis, editing); Lampkins (data collection and editing); Graham (data collection and editing)
1.1 cu ft medical freezer | Compact Compliance | # bci2801863 | They can use any freezer as long as it can go below -20 degrees Celsius; these can be used to store salivary samples for longer periods of time (~4 months) before running salivary assays. |
Camping Cooler | Amazon | (any size/type) | Can be used to store salivary samples during data collection |
E4 sensor | Empatica Inc | E4 Wristband Rev2 | You can use any EDA sensor or company as long as it records EDA and accelerometry |
EDA Explorer | https://eda-explorer.media.mit.edu/ | (open-source) | Can be used to identify potential sources of noise that are not necessarily due to movement |
Laptops | Dell | Latitude 3480 | They can use any desktop or laptop |
Ledalab | http://www.ledalab.de/ | (open-source) | Can be used to separate tonic and phasic EDA signals after following filtration steps |
MATLAB | https://www.mathworks.com/products/matlab.html | (version varies according to updates) | To be used for Ledalab, EDA Explorer, and to create customized time-stamping programs. |
Salivary Alpha Amylase Enzymatic Kit | Salimetrics | # 1-1902 | For the salivary kits, you should plan to either order the company to analyze your samples and/or go to a molecular biology lab for processing |
Salivary Cortisol ELISA Kit | Salimetrics | # 1-3002 | For the salivary kits, you should plan to either order the company to analyze your samples and/or go to a molecular biology lab for processing |
Testing Divider (Privacy Shields) | Amazon | #60005 | They can use any brand of testing shield as long as they cover the workspace |
Web Camera | Amazon | Logitech c920 | They can use any web camera as long as it is HD and 1080p or greater |