A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences

Summary

An experimental design was developed to investigate the real-time influences of an examination experience to assess the emotional realities students experience in higher education settings and tasks. This design is the result of a cross-disciplinary (e.g., educational psychology, biology, physiology, engineering) and multi-modal (e.g., salivary markers, surveys, electrodermal sensor) approach.

Abstract

Over the past ten years, research into students' emotions in educational environments has increased. Although researchers have called for more studies that rely on objective measures of emotional experience, limitations on utilizing multi-modal data sources exist. Studies of emotion and emotional regulation in classrooms traditionally rely on survey instruments, experience-sampling, artifacts, interviews, or observational procedures. These methods, while valuable, are mainly dependent on participant or observer subjectivity and is limited in its authentic measurement of students' real-time performance to a classroom activity or task. The latter, in particular, poses a stumbling block to many scholars seeking to objectively measure emotions and other related measures in the classroom, in real-time.

The purpose of this work is to present a protocol to experimentally study students' real-time responses to exam experiences during an authentic assessment situation. For this, a team of educational psychologists, engineers, and engineering education researchers designed an experimental protocol that retained the limits required for accurate physiological sensor measurement, best-practices of salivary collection, and an authentic testing environment. In particular, existing studies that rely on physiological sensors are conducted in experimental environments that are disconnected from educational settings (e.g., Trier Stress Test), detached in time (e.g., before or after a task), or introduce analysis error (e.g., use of sensors in environments where students are likely to move). This limits our understanding of students' real-time responses to classroom activities and tasks. Furthermore, recent research has called for more considerations to be covered around issues of recruitment, replicability, validity, setups, data cleaning, preliminary analysis, and particular circumstances (e.g., adding a variable in the experimental design) in academic emotions research that relies on multi-modal approaches.

Introduction

Psychologists have long understood the importance of humans' emotions in elucidating their behaviors¹. Within the study of education, Academic Achievement Emotions (AEE) has become the focus of emotion research². Researchers that use AAE argue that the situational contexts students find themselves in are important to consider when examining students' emotions. Students may experience test-related, class-related, or learning-related emotions that involve multi-component processes, including affective, physiological, motivational, and cognitive components. AEE is expressed in two forms: valence (positive/negative) and activation (focused/unfocused energy). Positive activating emotions, such as enjoyment, may increase reflective processes like metacognition, whereas positive deactivating emotions such as pride may result in low levels of cognitive processing. Negative activating emotions such as anger and anxiety may spark engagement, whereas negative deactivating emotions such as hopelessness may dampen motivation^3,4,5. Academic emotions contribute to how we learn, perceive, decide, respond, and problem-solve². To regulate academic emotions, an individual must possess self-efficacy (SE)^6,7,8, which is their confidence in their ability to employ control over their motivation, behavior, and social environment⁶. Self-efficacy and academic emotions are interrelated, where lower self-efficacy is tied to negative deactivating emotions (e.g., anxiety, anger, boredom) and higher self-efficacy is tied to positive activating emotions (e.g., happiness, hope, excitement)^6,7,8. SE is also believed to be strongly tied to performance^6,7,8.

Research that has examined classroom emotions have relied on self-reports, observations, interviews, and artifacts (e.g., exams, projects)^9,10. Although these methods provide rich contextual information about students' classroom experiences, they have significant limitations. For example, interviews, observations, and self-reports rely on individuals' introspections¹⁰. Other methods have sought to examine academic emotions more proximally than prior researchers, such as those based on experience-sampling approaches where researchers ask students to report on their emotions during the school day¹¹. Although this research allows us to report students' emotions more accurately, this work relies on self-report methods and does not allow for real-time reporting as students have to pause their work on the exam to address the experience survey.

Recently, researchers have begun to address concerns about self-report measures through the use of biological or physiological measures of emotion⁹, that combined with other instruments or techniques such as surveys, observations, or interviews, consists of a multi-modal form of data collection for educational and psychological research¹². For example, biological techniques, including salivary biomarkers, are being used to understand the role biological processes have on cognition, emotion, learning, and performance^13,14,15. For cognitive processes, androgens (e.g., testosterone) have been linked to different spatial recognition patterns in adults and children^16,17 whereas hypothalamic-pituitary-adrenocortical hormones (e.g., cortisol) and adrenergic hormones (e.g., salivary α-amylase or sAA) are linked to stress responsiveness amongst individuals^18,19,20.

Electrodermal activity (EDA) represents a physiological measure of the activation of the autonomic nervous system (ANS) and is linked to increased activation of the system, cognitive load, or intense emotional responses^21,22,23. In examination activities, EDA is affected by physical mobility^21,22, bodily and ambient temperatures^24,25,26,27, and verbalization of thoughts²⁸, as well as sensitivity and degree of connectivity of the analog-digital electrodes to the skin²⁹.

Although these can be limitations to using EDA, this technique can still provide valuable insight into what happens during near-real-time examinations and can serve as a promising tool to explore AEE and by extent, self-efficacy. As a result, an accurate picture of students' AEE can be obtained through a combination of survey methods, to determine the valence of emotion, and physiological and biological data, to measure the activation of that emotion. This paper builds upon a previous publication on examination activities³⁰ and expands the scope of that work to include multi-modal approaches (using experience-sampling surveys, EDA sensors, and salivary biomarkers) in an examination scenario. It is essential to mention that the protocol described below allows for multiple participant data to be collected at the same time within a single experimental setting.

Protocol

Procedures were approved by the Institutional Review Board (IRB) under a general review at Utah State University for studies on human subjects and use of these constructs. The typical results include two semesters of an engineering statics course, each with a slightly different experimental setup, at a western institution of higher education in the United States. Practice exams, whose content paralleled the actual exams, were developed by the course instructor and were used for our study. Please note that the protoc…

Representative Results

In this study, we were interested in studying the influences of self-efficacy, performance, and physiological (EDA sensors) and biological (sAA and cortisol) responses of undergraduate engineering students as they took a practice exam. The data shown is a representative subset of samples: (a) one that considered surveys and electrodermal sensors (experiment design A) and (b) one that included the same exam along with the salivary biomarker data (experiment design B). While we collected em…

Discussion

Although physiological measures have been used in many authentic learning contexts, it is critical to design a study environment that is mindful of the limits of the current technology. Our design balances the need for an authentic testing environment and accommodates the technology. Comfortably limiting participant movement, reducing unintended interruptions, and timestamping participants' testing responses are all critical steps within the protocol.

The space and expense of the electrode…

Materials

1.1 cu ft medical freezer	Compact Compliance	# bci2801863	They can use any freezer as long as it can go below -20 degrees Celsius; these can be used to store salivary samples for longer periods of time (~4 months) before running salivary assays.
Camping Cooler	Amazon	(any size/type)	Can be used to store salivary samples during data collection
E4 sensor	Empatica Inc	E4 Wristband Rev2	You can use any EDA sensor or company as long as it records EDA and accelerometry
EDA Explorer	https://eda-explorer.media.mit.edu/	(open-source)	Can be used to identify potential sources of noise that are not necessarily due to movement
Laptops	Dell	Latitude 3480	They can use any desktop or laptop
Ledalab	http://www.ledalab.de/	(open-source)	Can be used to separate tonic and phasic EDA signals after following filtration steps
MATLAB	https://www.mathworks.com/products/matlab.html	(version varies according to updates)	To be used for Ledalab, EDA Explorer, and to create customized time-stamping programs.
Salivary Alpha Amylase Enzymatic Kit	Salimetrics	‎# 1-1902	For the salivary kits, you should plan to either order the company to analyze your samples and/or go to a molecular biology lab for processing
Salivary Cortisol ELISA Kit	Salimetrics	# ‎1-3002	For the salivary kits, you should plan to either order the company to analyze your samples and/or go to a molecular biology lab for processing
Testing Divider (Privacy Shields)	Amazon	#60005	They can use any brand of testing shield as long as they cover the workspace
Web Camera	Amazon	Logitech c920	They can use any web camera as long as it is HD and 1080p or greater