Pilot Testing
10,123 Views
•
•
Vue d'ensemble
Source: Laboratories of Gary Lewandowski, Dave Strohmetz, and Natalie Ciarocco—Monmouth University
In any experiment researchers have the challenge of creating experiences for participants that are consistent (i.e., reliable) and authentic (i.e., valid). Yet there are many ways to manipulate any one variable. For example, if you want participants to feel sad, you can have them think of their own sad memory, watch a sad video, or read a sad story.
Researchers must find the best way to operationalize a psychological construct in order to produce the most effective manipulation possible. Often, before running the main study, researchers will pilot test (i.e., try out) their manipulations to check their effectiveness.
This video demonstrates how to operationalize the same independent variable (acute stress) in three different ways. Specifically, this study seeks to identify the best sound (static, ticking clock, or crying baby) to play during a difficult task (solving complex math problems) to optimally manipulate stress.
Psychological studies often use higher sample sizes than studies in other sciences. A large number of participants helps to ensure that the population under study is better represented and the margin of error accompanied by studying human behavior is sufficiently accounted for. In this video, we demonstrate this experiment using just three participants, one for each condition. However, as represented in the results, we used a total of 120 (40 for each condition) participants to reach the experiment’s conclusions reflected in the Results section.
Procédure
1. Define key variables.
- Create an operational definition (i.e., a clear description of exactly what a researcher means by a concept) of stressful sound.
- For the purposes of this experiment, a stressful sound is any noise that creates a feeling of tension, immediacy, or anxiety within participants.
- This will be manipulated through three different sounds: static, ticking clock, and a crying baby.
- For the purposes of this experiment, a stressful sound is any noise that creates a feeling of tension, immediacy, or anxiety within participants.
- Create an operational definition (i.e., a clear description of exactly what a researcher means by a concept) of acute stress.
- For purposes of this experiment, acute stress is defined as the stress or feeling of tension and strain resulting from recent demands or pressures.
- In order to measure this accurately, ask participants about their own stress levels using a straightforward question.
- For purposes of this experiment, acute stress is defined as the stress or feeling of tension and strain resulting from recent demands or pressures.
2. Conduct the study.
- Meet the student/participant at the lab.
- Provide participant with informed consent, a brief description of the research (concentration on a task), a sense of the procedure, an indication of potential risks/benefits, the right of withdrawal at any time, and a manner to get help if they experience discomfort.
- Run the static condition.
- Tell the participant “I’m going to give you a series of math problems that should be easy to solve. Your job is to complete as many as possible in the 2-min time limit. Please try to concentrate and ignore any sounds you may hear.”
- Give the participant the math problem sheet (Figure 1), start the timer (set for 2 min), play the static sound, and say “you may start.”
Figure 1: Math task. Examples of math problems given to participants
- Give the participant the dependent variable.
- Give the participant a measure that asks him/her to indicate how he/she currently feels (Figure 2). The item stressed appears embedded within several other distractor items (i.e., items not related to the present study but included to make the true purpose of the study less obvious). Feelings will be rated on a 1-7 scale, ranging from ‘not at all’ to ‘very much.’
Figure 2: Survey on feelings. Participants rates their feelings on a scale from 1-7 (not at all to very much)
- Give the participant a measure that asks him/her to indicate how he/she currently feels (Figure 2). The item stressed appears embedded within several other distractor items (i.e., items not related to the present study but included to make the true purpose of the study less obvious). Feelings will be rated on a 1-7 scale, ranging from ‘not at all’ to ‘very much.’
3. Debrief the participant.
- Tell the participant the nature of the study.
- “Thank you for participating. In this study I was trying to determine what type of sound led participants to experience the most stress. There were three conditions. Everyone worked on the same math problems for two min, but each group did so while hearing static, a ticking clock, or a crying baby. We hypothesized that the group who listened to the crying baby would report the most acute stress.”
- Explain explicitly why deception was necessary for the experiment.
- “We want to tell you about the deception we used in this study. We used deception by telling participants that the study was about concentration, which wasn’t true, but we didn’t want participants knowing that the study was actually about stress because it may have led to unnaturally increasing stress levels. We also indicated that the math problems were easy, when in fact they were not. We chose difficult problems to increase demands on participants, which generally creates a sense of stress. In both cases, deception was necessary because we wanted to get participants’ natural reactions. If participants were to know the true reasoning and hypothesis behind the study, they may perform in an unnatural way by trying to purposefully disprove the experimenter’s hypothesis. Because of the nature of the deception, it is quite natural for participants to not realize that they were being deceived.”
4. Run steps 2 & 3 above two additional times—once for ticking clock and once for crying baby.
- Each condition has a unique participant with everything being exactly the same except for the sound played.
Before administering any experiment it is beneficial to perform a smaller preliminary analysis, or a pilot study, to test and refine the experimental procedure. Pilot testing is the assessment of one aspect of a pilot study.
In psychological experiments, researchers have the challenge of creating experiences for participants that are consistent and authentic; however, there are many ways to manipulate any one variable. For example, if you want participants to feel sad you can have them think of their own sad memory, watch a sad video, or read a sad story.
Researchers must find the best way to modify a psychological construct in order to produce the most effective manipulation. Often, before running the main study, researchers will “pilot test” their manipulations to check their effectiveness.
This video demonstrates how to design and execute a pilot test with an example that seeks to identify the best sound to play during a difficult task to maximally manipulate acute stress.
Before performing a pilot test, proper experimental design is essential. This process includes creation of operational definitions, or clear descriptions of specific concepts.
In this experiment, several stressful sounds are being piloted as acute stress inducers for the participant while they complete a difficult task.
For the purposes of this experiment, the operational definition of a stressful sound is any noise that creates a feeling of tension, immediacy, or anxiety within participants.
This will be manipulated through three different sounds—static, ticking clock, and crying baby—which are subjected to three experimental groups.
Next an operational definition is created for acute stress. For the purposes of this experiment, acute stress is defined as the stress or feeling of tension and strain resulting from recent demands or pressures.
Here, pressure is applied through administration of complex math problems.
In order to measure acute stress accurately, participants will be asked about their own stress levels using a straightforward question on a survey.
The item “stressed” appears embedded within several other distractor items and is rated on a 1 to 7 scale.
Distractor items are not related to the present study but are included to make the true purpose of the study less obvious.
From this scale, the experimenter will evaluate the different sounds to find which one most consistently results in the highest level of acute stress in the participant.
To conduct the study, first meet the participant at the lab. Provide the participant with informed consent. This is a brief description of the research, a sense of the procedure, an indication of potential risks and benefits, the freedom of withdrawal at any time, and a manner to get help if they experience discomfort.
To run the condition, tell the participant that they will receive a series of math problems that should be easy to solve, and they are to complete as many as possible in the 2 min time limit. Tell the participant to try to concentrate and ignore any sounds they may hear.
Next, give the participant the math problem sheet. Start the timer and immediately play the sound being tested. Now, indicate to the participant that they may start.
Following the math task, measure the dependent variable of acute stress by giving the participant a form that asks them to indicate how they currently feel.
After the experiment, debrief the participant by telling them the nature of the study. Specifically, explain that to determine what type of sound led to the most stress, three different groups worked on the same math problems for 2 min while listening to one of three sounds.
Further debrief the participant by explaining that deception was used in this experiment by indicating that the study was about concentration and that the math problems were easy.
Explain that in both cases, deception was necessary to capture the participants’ natural reaction. Divulging that the study was actually about inducing stress and that difficult math problems would be used would likely increase initial stress levels.
In a large-scale pilot study, 40 participants were tested for each sound used. This number of participants was necessary to ensure that the results are reliable and reflective of the greater population.
After collecting data from 120 people, an analysis of variance comparing the static, ticking clock, and crying baby conditions was performed to see how they influenced stress level. The numbers presented here reflect the mean reported stress levels that participants indicated on the 1 to 7 scale for the “stressed” item in each condition.
As seen in the figure, the crying baby condition reported the most stress as hypothesized.
Now that you are familiar with how researchers conduct pilot experiments, let’s take a look at how these data are incorporated into future studies.
For example, based on the results of this pilot study, researchers expanded upon the type of math problems given—easy or difficult—to induce different levels of stress and then examined the effects on relationship behaviors.
The results indicated that those under more stress were more likely to pay attention to alternate partners and were less likely to give their own partner compliments.
In another study, pilot data were collected to test the feasibility of using a multimedia mobile phone program for youth smoking cessation intervention.
As the program’s content was found to be appropriate and technically easy to distribute to the target audience, the study was designed for a larger population of youth and revealed its benefits to help youth quit smoking.
You’ve just watched JoVE’s introduction to pilot testing. Now you should have a good understanding of how to modify a variable to determine the best manipulation for an experiment. The video demonstrated how to conduct a pilot test, as well as how to evaluate the results, and concluded with examples of larger studies informed by pilot tests.
Résultats
The researcher used 40 participants per condition, and as a result, collected data from 120 participants overall. Numbers above reflect the mean reported stress levels that participants indicated on the 1-7 scale for the stressed item in each condition. This multi-group experiment showed how researchers can operationalize the same construct in multiple ways.
A large number of participants is necessary to ensure that the results are reliable. If this research were conducted using just a few participants, it is likely that the results would have been much different and not reflective of the greater population.
After collecting data from 120 people, an analysis of variance (ANOVA) comparing the static, ticking clock, and crying baby conditions was performed to see how they influenced stress level. As shown in Figure 3, the crying baby condition reported the most stress as hypothesized.
Figure 3: The effects of different noises on stress levels. Shown are the average stress levels reported by condition.
Applications and Summary
The use of a pilot test helps researchers determine the most effective way to manipulate stress. With this knowledge, researchers can use the best manipulation in their future study.
For example, researchers manipulated stress by having participants do easy or difficult math problems to determine how stress influenced relationship behaviors.1 The results indicated that those under stress were more likely to pay attention to alternate partners and were less likely to give their own partner compliments.
Another study of stress used an entirely different method for manipulating stress.2 In this study, researchers induced stress by having participants immerse their arm in cold water to see how stress influenced long-term memory. Results indicated that exposure to stress led to worse performance on long-term memory tasks.
References
- Lewandowski, G. W., Jr., Mattingly, B. A., & Pedreiro, A. Under pressure: The effects of stress on positive and negative relationship behaviors. Journal of Social Psychology. 154, 463-473. doi: 10.1080/00224545.2014.933162 (2014).
- Trammell, J. P., & Clore, G. L. Does stress enhance or impair memory consolidation? Cognition and Emotion. 28 (2), 361-374. doi:10.1080/02699931.2013.822346 (2014).
Transcription
Before administering any experiment it is beneficial to perform a smaller preliminary analysis, or a pilot study, to test and refine the experimental procedure. Pilot testing is the assessment of one aspect of a pilot study.
In psychological experiments, researchers have the challenge of creating experiences for participants that are consistent and authentic; however, there are many ways to manipulate any one variable. For example, if you want participants to feel sad you can have them think of their own sad memory, watch a sad video, or read a sad story.
Researchers must find the best way to modify a psychological construct in order to produce the most effective manipulation. Often, before running the main study, researchers will “pilot test” their manipulations to check their effectiveness.
This video demonstrates how to design and execute a pilot test with an example that seeks to identify the best sound to play during a difficult task to maximally manipulate acute stress.
Before performing a pilot test, proper experimental design is essential. This process includes creation of operational definitions, or clear descriptions of specific concepts.
In this experiment, several stressful sounds are being piloted as acute stress inducers for the participant while they complete a difficult task.
For the purposes of this experiment, the operational definition of a stressful sound is any noise that creates a feeling of tension, immediacy, or anxiety within participants.
This will be manipulated through three different sounds—static, ticking clock, and crying baby—which are subjected to three experimental groups.
Next an operational definition is created for acute stress. For the purposes of this experiment, acute stress is defined as the stress or feeling of tension and strain resulting from recent demands or pressures.
Here, pressure is applied through administration of complex math problems.
In order to measure acute stress accurately, participants will be asked about their own stress levels using a straightforward question on a survey.
The item “stressed” appears embedded within several other distractor items and is rated on a 1 to 7 scale.
Distractor items are not related to the present study but are included to make the true purpose of the study less obvious.
From this scale, the experimenter will evaluate the different sounds to find which one most consistently results in the highest level of acute stress in the participant.
To conduct the study, first meet the participant at the lab. Provide the participant with informed consent. This is a brief description of the research, a sense of the procedure, an indication of potential risks and benefits, the freedom of withdrawal at any time, and a manner to get help if they experience discomfort.
To run the condition, tell the participant that they will receive a series of math problems that should be easy to solve, and they are to complete as many as possible in the 2 min time limit. Tell the participant to try to concentrate and ignore any sounds they may hear.
Next, give the participant the math problem sheet. Start the timer and immediately play the sound being tested. Now, indicate to the participant that they may start.
Following the math task, measure the dependent variable of acute stress by giving the participant a form that asks them to indicate how they currently feel.
After the experiment, debrief the participant by telling them the nature of the study. Specifically, explain that to determine what type of sound led to the most stress, three different groups worked on the same math problems for 2 min while listening to one of three sounds.
Further debrief the participant by explaining that deception was used in this experiment by indicating that the study was about concentration and that the math problems were easy.
Explain that in both cases, deception was necessary to capture the participants’ natural reaction. Divulging that the study was actually about inducing stress and that difficult math problems would be used would likely increase initial stress levels.
In a large-scale pilot study, 40 participants were tested for each sound used. This number of participants was necessary to ensure that the results are reliable and reflective of the greater population.
After collecting data from 120 people, an analysis of variance comparing the static, ticking clock, and crying baby conditions was performed to see how they influenced stress level. The numbers presented here reflect the mean reported stress levels that participants indicated on the 1 to 7 scale for the “stressed” item in each condition.
As seen in the figure, the crying baby condition reported the most stress as hypothesized.
Now that you are familiar with how researchers conduct pilot experiments, let’s take a look at how these data are incorporated into future studies.
For example, based on the results of this pilot study, researchers expanded upon the type of math problems given—easy or difficult—to induce different levels of stress and then examined the effects on relationship behaviors.
The results indicated that those under more stress were more likely to pay attention to alternate partners and were less likely to give their own partner compliments.
In another study, pilot data were collected to test the feasibility of using a multimedia mobile phone program for youth smoking cessation intervention.
As the program’s content was found to be appropriate and technically easy to distribute to the target audience, the study was designed for a larger population of youth and revealed its benefits to help youth quit smoking.
You’ve just watched JoVE’s introduction to pilot testing. Now you should have a good understanding of how to modify a variable to determine the best manipulation for an experiment. The video demonstrated how to conduct a pilot test, as well as how to evaluate the results, and concluded with examples of larger studies informed by pilot tests.