This article focuses on the experimental elicitation of pain through heat (thermal) and electrical stimulation while recording physiological, visual, and paralinguistic responses. It aims at collecting valid multimodal data for analyzing pain based on its intensity, quality, and duration.
The assessment of pain relies mostly on methods that require a person to communicate. However, for people with cognitive and verbal impairments, existing methods are not sufficient as they lack reliability and validity. To approach this problem, recent research focuses on an objective pain assessment facilitated by parameters of responses derived from physiology, and video and audio signals. To develop reliable automated pain recognition systems, efforts have been made in creating multimodal databases in order to analyze pain and detect valid pain patterns. While the results are promising, they only focus on discriminating pain or pain intensities versus no pain. In order to advance this, research should also consider the quality and duration of pain as they provide additional valuable information for more advanced pain management. To complement existing databases and the analysis of pain regarding quality and length, this paper proposes a psychophysiological experiment to elicit, measure, and collect valid pain reactions. Participants are subjected to painful stimuli that differ in intensity (low, medium, and high), duration (5 s / 1 min), and modality (heat / electric pain) while audio, video (e.g., facial expressions, body gestures, facial skin temperature), and physiological signals (e.g., electrocardiogram [ECG], skin conductance level [SCL], facial electromyography [EMG], and EMG of M. trapezius) are being recorded. The study consists of a calibration phase to determine a subject’s individual pain range (from low to intolerable pain) and a stimulation phase in which pain stimuli, depending on the calibrated range, are applied. The obtained data may allow refining, improving, and evaluating automated recognition systems in terms of an objective pain assessment. For further development of such systems and to investigate pain reactions in more detail, additional pain modalities such as pressure, chemical, or cold pain should be included in future studies. Recorded data of this study will be released as the “X-ITE Pain Database”.
Pain is a very personal and unpleasant sensation that is perceived differently by everyone. It lasts from seconds to months and may vary in its quality (throbbing, sharp, burning, etc.). If treated inadequately, pain influences physical and psychological functions of the body, reduces the quality of life, and bears the risk of becoming a chronic condition. In clinical care, the accurate assessment of pain intensity and quality is highly relevant to provide successful pain management1,2. Gold standard methods for assessing pain, such as the visual analog scales (VAS), the numeric rating scale (NRS), or the McGill Pain Questionnaire3, rely on self-reports of patients and, thus, only work sufficiently with cognitively and verbally unimpaired persons. Consequently, all those established methods lack validity and reliability when it comes to neonates4, delirious, somnolent, sedated, or ventilated patients5, or people suffering from dementia6,7. In addition to or as an alternative to self-report scales, methods to measure pain through observation by trained personnel (e.g., the Zurich Observation Pain Assessment8 or the Abbey Pain Scale9) have been developed in recent years. Nevertheless, even these tools suffer from limitations in reliability and validity, as even trained raters cannot guarantee an objective assessment. Furthermore, the application is often too time-consuming for clinical staff when pain assessment should be done on a regular basis.
Several research teams have focused on developing automated pain recognizing systems, which allow for measuring pain by means of physiological, visual, and/or paralinguistic signal sets as new approaches for assessing and monitoring pain and its intensities objectively. Previous studies show promising results in detecting and differentiating pain10,11,12,13,16,17,18 or discriminating pain from basic emotions14,15 based solely on one of the signal sets10,11,12,13,14,15 as well as on a combination/fusion16,17,19 of the sets. The abovementioned modalities react almost autonomously to stressful stimuli such as pain. Using them has the advantage that they do not require a person’s ability to report her/his pain. Such individuals would greatly benefit from an objective pain recognition system which incorporates such modalities. Data sets consisting of reactions to elicited pain provide precious information for analyzing pain patterns and developing practical applications for detecting and monitoring pain. Amongst others, Walter et al.20 created the “BioVid Heat Pain Database”, a multimodal database that is publicly available and provides data from short-time induced painful heat stimuli and corresponding psychophysiological and visual reactions. The “SenseEmotion Database” of Velana et al.21 includes biosignals, videos, and paralinguistic information from volunteers affected by phasic heat pain and emotional stimuli.
While these databases are well suited for examining pain reactions, they are mostly based on one specific pain model. As pain differs in its quality (supposedly depending on the pain model) and in its duration, it also may differ in its physiological, visual, and paralinguistic correlates. To the best of the authors’ knowledge, no multimodal studies or databases exist that combine two or more pain models and vary pain stimuli in intensity and duration in order to not only detect pain patterns but also distinguish between pain qualities.
This paper provides a protocol on how to conduct a complex psychophysiological experiment to elicit pain and simultaneously record physiological responses (e.g., ECG, EMG of Musculus trapezius, corrugator supercilii, and zygomaticus major, SCL) as well as video (e.g., facial expressions, body gestures, facial skin temperature) and audio data. Participants are stimulated with short (phasic) and longer lasting (tonic) heat and electrical pain stimuli that differ in intensity. A calibration phase prior to the experiment determines pain thresholds for each subject individually.
The study aims at collecting multimodal data for investigating pain (patterns) regarding intensity, quality and length by means of statistical methods, machine learning algorithms, etc. Additionally, the already collected data is planned to be published for academic research purposes under the name “X-ITE (Experimentally Induced Thermal and Electrical) Pain Database”. It may extend existing databases, such as BioVid Heat Pain and SenseEmotion20,21, and contribute to the further development, improvement, and/or evaluation of automated pain recognition systems in matters of validity, reliability, and real-time recognition.
The rest of the paper is organized in the following way. The protocol describes how to carry out the pain elicitation study step-by-step. Then, the representative results present the outcome of the experiment. Finally, the discussion covers critical steps, limitations, and benefits of the study followed by suggestions for future extensions.
The study was conducted in accordance with the ethical guidelines laid down in the World Medical Association Declaration of Helsinki (ethical committee approval was granted: 196/10-UBB/bal) and approved by the ethics committee of the University of Ulm (Helmholtzstraße 20, 89081 Ulm, Germany).
1. Subject recruitment and selection
2. General preparations of the pain elicitation experiment
NOTE: The pain elicitation experiment consists of two temporally successive parts: the calibration part and the pain stimulation part. The calibration part determines a participant’s individual pain threshold and pain tolerance level in terms of thermal and electrical stimuli. The pain stimulation part performs the pain induction adjusted to the individual thresholds. Each part of the experiment takes place in a different room: the calibration room and the experimental room. The calibration room also serves as a monitoring room for the experimenter during the pain stimulation part (see Figure 1).
Figure 1: Schematic representation of room setup. The right side shows the calibration/monitoring room where the calibration part takes place. Later on, it also serves as a signal monitoring room during the pain stimulation part, which follows the calibration part. The left side shows the experimental room where the pain stimulation part takes place. Both rooms are connected by a conduit pipe, which the thermode, the electrodes’ cable of the electrical stimulator and computer wires can be passed through. Please click here to view a larger version of this figure.
3. Calibration of electrical pain threshold and tolerance (parts 1 and 2)
NOTE: Only one experimenter should conduct the calibration part to minimize the social effects on pain sensitivity. Choose an experimenter with the same sex as the participant to minimize cross-sex effects on pain sensitivity23. Part 1 determines pain threshold and tolerance in terms of short (phasic) electrical stimuli and part 2 in terms of longer lasting (tonic) electrical stimuli. Those values serve as a basis for calculating the phasic and tonic electrical pain stimuli applied in the pain stimulation part.
4. Calibration of thermal pain threshold and tolerance (parts 1 and 2)
NOTE: The thermal pain calibration is divided into two parts. Part 1 determines pain threshold and tolerance in terms of short (phasic) thermal stimuli and part 2 does so in terms of longer lasting (tonic) thermal stimuli. Those values serve as basis for calculating the phasic and tonic thermal pain stimuli applied during the pain stimulation part.
5. Preparation of the pain stimulation experiment
Figure 2: Schematic representation of camera and microphone setup. The frontal face camera, thermal camera and microphone are set up approx. 1 m above the head of the participant. A side view camera captures both sides of the face with the help of a mirror. A body view camera mounted to the wall allows for the recording of body movement. Please click here to view a larger version of this figure.
NOTE: Due to a small experimental room, combining a side view camera with a mirror is a very elegant solution to capture both sides of the subject’s face with just one camera.
Figure 3: Graphical illustration of the pain stimulation part. (A) Exemplary pain elicitation script with randomized phasic (blue) and tonic (red) pain stimuli. (B) Excerpt from the pain elicitation script above: Three phasic stimuli with a duration time of 5 seconds and subsequent pauses. The duration of pauses varies between 8 and 12 seconds. (pH1, pH2, pH3 = phasic heat pain with intensity 1, 2, 3; tH1, tH2, tH3 = tonic heat pain with intensity 1, 2, 3; pE1, pE2, pE3 = phasic electrical pain with intensity 1, 2, 3; tE1, tE2, tE3 = tonic electrical pain with intensity 1, 2, 3; s = seconds). Please click here to view a larger version of this figure.
6. Pain stimulation
Pain is perceived differently by any person and may express itself diversely in facial expressions, paralinguistic and/or physiological signals. The design of this study is suitable to analyze pain responses in numerous ways with respect to the underlying aims. The obtained data may allow answering research questions, such as: Are there specific pain response patterns? Do they differ regarding pain model and duration?
A total of 134 subjects participated in our experiment. The sex ratio was 50/50. We divided them in the following age groups: 1) 18-29 years (N = 49, 23 men, 26 women), 2) 30-39 years (N = 45, 23 men, 22 women), 3) 40-50 years (N = 40, 21 men, 19 women). The average age of all subjects was 31.4 (SD = 9.7), of all men = 33.4 (SD = 9.3) and of all women = 32.9 (SD = 10.2) years. The study took place at the Department of Medical Psychology of the University of Ulm, Germany.
The main outcome of this protocol is a data set of audio, video and psychophysiological signals reflecting the subjects’ responses to pain stimuli. Table 1 provides a general overview on the technical features of the recorded signals and on the numbers of induced pain stimuli in the study.
Technical Features | ||||
Signal: | Sampling Rate: | Attributes: | ||
Audio | 44100 Hz | Mono, MP3 320 kbps | ||
Camera 1 (face, frontal view) | 25 Hz | Color video: resolution 1384 x 1032, HEVC encoded with libx265 (CRF 16, preset medium) |
||
Camera 2 (face, side view) | 25 Hz | Color video: resolution 1620 x 840, HEVC encoded with libx265 (CRF 16, preset medium) |
||
Body Camera | ca. 30 Hz | Color video: resolution 1500 x 600, HEVC encoded with libx265 (CRF 16, preset medium); Depth video: resolution 500 x 200, lossless encoding |
||
Thermal Camera | ca. 120.8 Hz | Surface temperature video: resolution 120 x 160, grayscale MPEG-4-AVC encoded with libx264 (CRF 0, preset veryfast), encoded temperature range 26.5-52.0 °C (steps of 0.1) |
||
ECG | 1000 Hz | Hardware filtered via BioPac: 35 Hz LP, 0.5 Hz HP, 50 Hz notch filter |
||
SCL | 1000 Hz | Hardware filtered via BioPac: 10 Hz LP, no HP, no notch filter |
||
EMG M. trapezius | 1000 Hz | Hardware filtered via BioPac: 500 Hz LP, 10 Hz HP, no notch filter |
||
EMG M. corrugator supercilii | 1000 Hz | Hardware filtered via BioPac: 500 Hz LP, 10 Hz HP, no notch filter |
||
EMG M. zygomaticus major | 1000 Hz | Hardware filtered via BioPac: 500 Hz LP, 10 Hz HP, no notch filter |
||
Stimuli | Thermal | Electrical | ||
Subjects: | Phasic Stimuli (5 s): | Tonic Stimuli (60 s): | Phasic Stimuli (5 s): | Tonic Stimuli (60 s): |
Per subject | 90 (30 per intensity) | 3 (1 per intensity) | 90 (30 per intensity) | 3 (1 per intensity) |
All (N = 134) | 12060 (4020 per intensity) | 402 (134 per intensity) | 12060 (4020 per intensity) | 402 (134 per intensity) |
Men (n = 67) | 6030 (2010 per intensity) | 201 (67 per intensity) | 6030 (2010 per intensity) | 201 (67 per intensity) |
Women (n = 67) | 6030 (2010 per intensity) | 201 (67 per intensity) | 6030 (2010 per intensity) | 201 (67 per intensity) |
Table 1: Technical features and number of induced stimuli. The upper half (Technical Features) shows the sampling rates and attributes of the specific signals. The lower half (Stimuli) shows the numbers of the induced specific (thermal/electrical) pain stimuli for one subject, for all subjects and for each gender. (MP3 = Moving Picture Experts Group Layer-3 Audio, kbps = kilobits per second, HEVC = High Efficiency Video Coding, CRF = Constant Rate Factor, MPEG-4-AVC = Motion Picture Experts Group Layer-4 Video Advanced Video Coding, Hz = Hertz, °C = degrees Celsius, s = seconds, ECG = Electrocardiogram, SCL = Skin Conductance Level, EMG = Electromyography, LP = low-pass filter, HP = high-pass filter, M. = Musculus).
A secondary outcome concerning the calibration phase of the study is presented in Table 2. It shows the mean stimulation temperatures and currents of pain intensities 1 and 3 (as calculated in step 5.11 of the protocol) for all subjects and additionally for the male and female subgroup.
Stimuli | Thermal [in °C] mean (SD) | Electrical [in mA] mean (SD) | ||||||
科目 | pH1 | pH3 | tH1 | tH3 | pE1 | pE3 | tE1 | tE3 |
All (N = 134) | 44.03 (2.25) | 49.17 (1.20) | 42.50 (2.14) | 47.76 (1.02) | 1.63 (0.94) | 5.64 (2.72) | 1.69 (1.12) | 5.70 (2.59) |
Men (n = 67) | 44.56 (2.18) | 49.48 (0.89) | 43.11 (1.98) | 47.93 (1.04) | 1.94 (1.01) | 6.83 (3.02) | 1.96 (1.16) | 6.90 (2.72) |
Women (n = 67) | 43.51 (2.74) | 48.87 (1.39) | 41.89 (2.14) | 47.59 (0.98) | 1.32 (0.75) | 4.45 (1.70) | 1.43 (1.01) | 4.51 (1.80) |
Table 2: Mean stimulation temperatures and currents of pain intensities 1 and 3. (pH1, pH3 = phasic heat pain with intensity 1, 3; tH1, tH3 = tonic heat pain with intensity 1, 3; pE1, pE3 = phasic electrical pain with intensity 1, 3; tE1, tE3 = tonic electrical pain with intensity 1, 3; °C = degrees Celsius; mA = milliampere, SD = standard deviation).
If all steps of the protocol are conducted carefully and no technical problems occur (in terms of computer or recording device crashes, etc.), a successful outcome may look similar as depicted in Figure 4. All signals are of high quality and not affected by external sources of interferences. The participant is clearly visible in every camera.
Figure 4: Example data from a successful experiment. The figure depicts recorded signals a few seconds before, during and after an intense pain stimulus. All signals are non-filtered and synchronized in time. For clarity, only representative screenshots of the video signals are shown here. (EMG = Electromyography, SCL = Skin Conductance Level, ECG = Electrocardiogram, M. = Musculus, s = seconds). Please click here to view a larger version of this figure.
However, unexpected incidents may cause the data to become noisy or corrupted. Besides computer or recording device crashes, the coming-off of electrodes (especially reusable electrodes with small diameter which are attached by means of double-sided adhesive collars) mostly leads to unusable signals. As an example for a sub-optimal data set, Figure 5 shows the moment when an EMG electrode comes off and renders the corresponding signal useless.
Figure 5: Example data from a sub-optimal experiment. The red circle indicates the time one of the EMG electrodes (M. zygomaticus major) fell off the subject’s cheek. This might have been due to sweat or head movement. From this moment on, the signal was lost. (EMG = Electromyography, SCL = Skin Conductance Level, ECG = Electrocardiogram, M. = Musculus, s = seconds). Please click here to view a larger version of this figure.
Due to ethical guidelines, the maximum intensities of thermal and electrical stimuli had to be restricted. Regarding thermal calibration control part (see supplementary file), 37 subjects (31 men, 6 women) reached the given cutoff of 50.5 °C (ratio = 37/134 = 27.61 %). As for thermal calibration part 1, 60 participants (39 men, 21 women) reached the cutoff of 50.0 °C (ratio = 60/134 = 44.78 %) and concerning part 2, 57 persons (37 men, 20 women) reached the cutoff of 49.5 °C (ratio = 57/134 = 42.54 %). The cutoff for both electrical calibration parts was 25 mA. None of the 134 subjects reached it.
As we plan to publish the data (see next paragraph), the data sets of participants who have reached the cutoffs will additionally be marked and their subjective pain ratings for the corresponding cutoffs will be included.
We would like to point out that the main focus of the protocol is obtaining multi-modal signals for analyzing thermal and electrical pain. Therefore, no other results are discussed here. After checking and excluding data sets due to missing data or rejected written consent for data sharing, the data sets of this study will be made available under the name “X-ITE Pain Database”. For further information on when and how to obtain the X-ITE Pain DB please visit https://github.com/philippwerner/pain-database-list.
Supplementary File 1. Please click here to download this file.
The presented protocol focuses on the experimental elicitation of thermal (heat) and electrical pain while recording physiological, visual and paralinguistic signals. This novel approach, combining two pain models with different stimuli intensities and two different stimuli durations (phasic and tonic), offers a broad perspective about the psychophysiological patterns and expressions of pain. However, for the realization of this protocol several steps need to be considered.
In general, if working with pain stimuli it is crucial to ensure the safety of the subjects. All pain stimuli have to be highly controlled and should only be carried out by experienced experimenters.
Furthermore, for recording and collecting reliable and high quality data, the proper attachment of devices (electrodes), the perfect functioning of recording devices and a smooth communication between computers is highly recommended. All sources of interferences should be eliminated or reduced to a minimum. To guarantee consistency between participants, it is important to provide standardized instructions and unvarying experimental conditions.
According to our experience, finding suitable participants who meet all criteria and are willing to receive numerous painful stimuli, takes a long time and is quite challenging. In addition to that, the monetary compensation has to be high enough to attract subjects to the study. Especially persons between 30 and 50 years are hard to find. This may be because the experiment is too long (ca. 4 hours, including arrival and departure) and they have to take half a day off from work.
Because the safety of the participants is of top priority, pain induction may need to be restricted. Due to ethical guidelines, the stimulus intensities must not exceed certain levels to prevent burns and unconsciousness in terms of thermal and electrical pain induction, respectively. A general cutoff of intensities may result in a ceiling effect as some subjects may reach the intensity limits before feeling intolerable pain. In this study, approximately 42% (considering thermal calibration part 1 and 2) of the participants reached the thermal cutoffs (see representative results). As they did not reach their “real” pain tolerances, their physiological responses to the highest thermal stimuli might behave differently in contrast to physiological responses of subjects who reached them. If so, mixing these two groups could influence classification results in terms of pain recognition.
An important point to address is the pain modalities in this experiment. Participants are subjected only to thermal and electrical pain stimuli (due to the fact that these are highly controllable in an experimental setting). Thus, if examining pain patterns regarding quality, findings may not translate to other pain modalities such as pressure, chemical or visceral pain.
The same consideration on transferability of results applies to the study sample. The protocol is ethically restricted to healthy adults. For example, it does not include children or cognitively and verbally impaired persons. Furthermore, in our study only European people participated. Also here, analytical results may not apply to groups not considered in this experiment.
Another limitation may concern the Hawthorne effect24: The subjects are aware that they are being filmed/observed in the study. This might change their behavior.
Compared to existing pain databases, the protocol provides significant advantages for analyzing pain response patterns as it combines two pain models and two time courses (phasic and tonic): Besides the intensity and duration of pain, it also considers the quality of pain. As thermal pain is described differently than electrical pain (e.g., burning vs. sharp), it may also differ in the pain reactions. If so, those findings could link a pain response pattern to the underlying source of pain. Furthermore, the study is multi-modal to widen the range of pain investigation opportunities: Employing 5 psychophysiological signals, 2 face (front/side) camera signals, 1 body view camera signal, 1 thermal camera and 1 audio signal, pain may be analyzed and assessed more precisely.
For a more complex investigation of pain response patterns, future extensions of this method should include more biosignals such as electroencephalography (EEG), body temperature and respiration. It would also be of great benefit to employ controlled pressure as a further pain model. Researchers aiming at automatic pain recognition via data gathered with this protocol should further test promising machine learning models with clinical control groups.
The authors have nothing to disclose.
The authors would like to thank Verena Friedrich, Maria Velana, Sandra Gebhardt, Romy Bärwaldt, and Tina Daucher for their precious help in conducting the study. In addition, a special thank you goes out to Dr. Stefanie Rukavina for her scientific support. This research was part of the DFG/TR233/12 (http://www.dfg.de/) “Advancement and Systematic Validation of an Automated Pain Recognition System on the Basis of Facial Expression and Psychobiological Parameters” project, funded by the German Research Foundation.
PATHWAY Model ATS | Medoc Ltd., Ramat Yishai, Israel | Thermal Stimulator | |
30 mm x 30 mm ATS Thermode | Medoc Ltd., Ramat Yishai, Israel | Thermode | |
PATHWAY Software Arbel 6.3.7.22.1 | Medoc Ltd., Ramat Yishai, Israel | Thermal Stimulator Software | |
Digitimer DS7A Current Stimulator | Digitimer Ltd., Hertfordshire, UK | Electrical Stimulator | |
Inquisit 5 | Millisecond Software, Seattle, WA, USA | Software for triggering electrical stimuli | |
Analogue-To-Digital Converter | Wissenschaftliche Werkstatt Elektronik, University of Ulm, Ulm, Germany | custom built | |
BIOPAC MP150 System | BIOPAC Systems, Inc., Goleta, CA, USA | Biosignal Recording Hardware | |
AcqKnowledge Software 4.1.1 | BIOPAC Systems, Inc., Goleta, CA, USA | Biosignal Recording Software | |
NTG-2 Dual Powered Directional Condenser Microphone | RØDE Microphones, Silverwater, Australia | Audio Recording Microphone | |
Kinect v2 | Microsoft, Redmond, WA, USA | Body View Camera | |
AV Pike F-145C | Allied Vision Technologies GmbH, Stadtroda, Germany | Face Camera (frontal view) | |
AV Prosilica GT 1600C | Allied Vision Technologies GmbH, Stadtroda, Germany | Face Camera (side view) | |
PIR uc 180 Thermal Camera | InfraTec GmbH, Dresden, Germany | Thermal Face Camera | |
Synchronization Hardware | Werkstatt, IIKT, University of Magdeburg, Magdeburg, Germany | custom built | Hardware triggering of cameras, trigger signal is recorded by BIOPAC and Audacity |
Recording and Synchronization Software | Philipp Werner, Neuro-Information Technology, University of Magdeburg, Magdeburg, Germany | custom software | Real-time recording, offline video encoding, and offline synchronization |
Examination Couch | ClinicalCare GmbH, Bremen, Germany | ||
Ag-AgCl Electrodes EL254 / EL254S (Reusable, 4mm recording diameter) | BIOPAC Systems, Inc., Goleta, CA, USA | Used to record EMG M. corrugator and M. zygomaticus | |
Ag-AgCl Electrodes BlueSensor P (Disposable, skin contact size: 34 mm diameter, measuring area 154 mm2) | Ambu GmbH, Bad Nauheim, Germany | Used to record ECG and EMG M. trapezius. Also used for electrical stimulation | |
Audacity 2.1.2 | Dominic Mazzoni (Audacity) | Audio Recording Software | |
Cold Gel Pack | C+V Pharma Depot GmbH, Versmold, Germany | ||
Panthenol 50mg/g | ratiopharm GmbH, Ulm, Germany | Ointment | |
Alumnium Profiles | item Industrietechnik GmbH, Solingen, Germany | Used to install all cameras and microphone | |
Electrode Gel GEL1 | BIOPAC Systems, Inc., Goleta, CA, USA | ||
ELPREP Skin Preparation Gel | BIOPAC Systems, Inc., Goleta, CA, USA |