Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury

The manuscript describes three tasks used to assess emotion recognition abilities of people with TBI. The goal of the described method was to compare response accuracy for facial and vocal emotional cues in isolation, to perception of these cues occurring simultaneously within meaningful situational context. Film clips were included in the current study because their approximation of everyday situations made them more ecologically valid than isolated representations of emotion expressions. When carrying out this protocol with people with brain injury, it is important to consider the individual's visual scanning skills and ensure that the stimuli and response sheets are fully visible. To assist with this, response sheets were created that consisted of one line for each stimulus, and used alternated color coding for each line (e.g., Line 1 = gray; Line 2 = white; Line 3 = gray; etc.).

The DANVA2 is fully available on-line. However, the on-line version of the DANVA-Faces was not used because it is pre-set to have each face shown for only two seconds. Given the speed of processing issues often experienced by people with TBI, we were concerned that errors made on this version may be more indicative of speed of processing problems, than reflective of facial emotion recognition. Our presentation version of the DANVA-Faces increases the time each stimulus is shown to 5 seconds.

The stimuli used in the current study provided important insight into the emotion recognition abilities of people with TBI. However, there are many ways in which the stimuli may be improved for use in future studies of this nature. The stimuli used to assess recognition of isolated facial emotion expressions were from the DANVA-Faces task of the DANVA2. The benefit of using the DANVA-Faces stimuli is that they are standardized and used previously with people with TBI^3,34. The 24 items include both high and low intensity portrayals of emotion by a small group of speakers so participants are exposed to the same speaker producing variable expressions. In addition, the DANVA-Faces task includes colored photographs versus the black and white images often used. Not only are colored images more contemporary, McDonald and Saunders¹⁵ argue they are likely less abstract to people with TBI. The primary issue with the stimuli used in the DANVA-Faces task, is that the emotion expressions are static. As previously discussed, static images provide people more time to interpret the emotion being portrayed, lack the temporal cues that occur when the face is moving^12,13,15, and are not processed in the same area of the brain as dynamic ones^17-23. Thus, using only static facial emotion expressions when assessing emotion recognition does not allow for appropriate generalization to real-life processing. In addition, research has shown that we can identify emotions in photographs more accurately than moving images, so tests using static images may overestimate abilities^13,35,36. Therefore, future studies need to include isolated dynamic facial emotion stimuli to more accurately assess how people with TBI interpret emotion portrayed in only the face. These stimuli could potentially be created using the film clips by selecting frames in which the targeted emotion was portrayed, deleting the sound, and cropping the film to focus on only the head and neck of the character. This would eliminate additional contextual cues, including gestural cues and body language.

The vocal emotion expressions used in the current study were also taken from the DANVA2. Similar to the DANVA-Faces task, the DANVA-Voices task includes 24 items of both high and low intensity. Each item on the DANVA-Voices uses the same semantically neutral sentence (I'm going out of the room now, and I'll be back later). The repetition of a single sentence allows listeners to focus on the non-verbal emotion cues (i.e., tone of voice) and ignore the verbal content. This is in contrast to stimuli used in previous studies that included different (semantically neutral) verbal content for each item¹⁵. Unlike the results of the current study, McDonald and Saunders¹⁵ reported that people with TBI had significant difficulty identifying emotion in isolated vocal emotion expressions when compared to isolated facial emotion expressions. The verbal content of the vocal stimuli used in that study varied for each item, so it is possible that participants were focusing on interpreting the verbal content, which was semantically neutral, and thus, meaningless. This assertion is supported by studies that have shown that people with (and without) TBI are more likely to use verbal content versus the non-verbal acoustic cues to interpret a vocal emotion expression^37-39. Future studies should further explore the degree to which people with TBI make use of the nonverbal versus verbal content in isolated vocal expressions to determine whether the semantic information conveyed in the verbal message is more dominant than the emotion conveyed through tone of voice. This could be accomplished by creating stimuli with a limited number of sentences, both meaningful and neutral in verbal content. These sentences could then be combined with either congruent or incongruent nonverbal vocal cues. Sentences that are congruent in verbal and nonverbal emotion content are likely to lead to faster and more accurate identification of the emotion expressed⁴⁰. However, it is unclear which cue participants will focus on when the verbal and non-verbal content is incongruent, and whether one channel would dominate throughout the task, or if the dominant cue will be dependent on the emotion portrayed through each channel.

Stimuli in the DANVA2 were created by presenting speakers with a short vignette aimed to elicit a target emotion and then instructing those speakers to express that emotion via the face or voice. It has been suggested that such methods lead to exaggerated and stereotypical expressions that may lead to overstated recognition abilities⁴¹. The film clips on the other hand included facial and vocal expressions that occurred with situational context, making them more ecologically valid than the use of isolated expressions. It is important to note that while not semantically neutral, the verbal content did not specify the emotional state of the characters in any way. The use of film clips of this nature with people with TBI is novel. While results of the current study suggest that the additional contextual information included in the film clips greatly facilitated emotion recognition, it is possible that participants were influenced by the additional cues present (e.g., body posture) and/or the cinematic design of the clips (e.g., pacing, lighting, and framing)^42,43. Participants may also have been influenced by the presence of background music, so results need to be interpreted and generalized carefully. The influence of affective background music should be further investigated in future studies by comparing film clips without and without this cue to ascertain if the presence of congruent background music facilitates emotion recognition. While this information would not necessarily be generalizable to everyday social interactions, it could have important implications for treatment of emotion recognition deficits. It is also possible that participants in the current study were relying on the body posture and gestures of the characters in the film clips to facilitate emotion perception. Again, finding film clips that manipulate the presence and absence of such cues would be important for exploring which cues people with TBI make most use of when interpreting emotions in everyday situations. However, identifying commercial film clips that manipulate each of these extraneous factors to varying degrees poses a significant challenge. Instead, future research might focus on creating high quality film clips within various situational settings show characters engaged in meaningful interactions. These clips could be controlled for the presence and absence of contextual cues, postures, gestures, and background music. In addition, the clips could be used to assess recognition of dynamic isolated facial expressions as described earlier, recognition of isolated vocal emotion expressions, these two cues combined, and then the addition of contextual information. This would provide much needed insight into how people with TBI perceive emotion and might allow for better generalization to real-life settings.

Other methodological considerations for future studies like the one presented here are the number of target emotions and the response format chosen. The current study included four primary emotions: happy, sad, angry, and fearful. Neutral was also evaluated in the film clips task. This selection of target emotions includes only one positive emotion (i.e., happy) making it difficult to compare response accuracy for positive versus negative emotions. Also, such a limited set of emotions makes it easier for participants to rely upon process of elimination when making their selections, a strategy that could not be reliably used in everyday interactions. Future studies should expand upon the number of emotions included and should also include isolated facial and vocal expressions portraying neutral. Further, angry film clips were not well identified by either group, suggesting that the film clips themselves may have been poorly chosen. The average identification rate of the angry film clips in the previous validation study by Zupan and Babbage^{Zupan, B. & Babbage, D. R. Emotion elicitation stimuli from film clips and narrative text. Manuscript submitted to J Soc Psyc} was only 73% compared to 85 – 90% for the remaining four categories. Previous studies using film clips also report difficulty selecting angry clips^33,44,45. It remains unclear why it is more challenging to select film clips that lead to high identification of angry. It may be because angry is a more complicated emotion that serves a range of social functions (e.g., expressing dislike versus asserting authority), requiring the perceiver to evaluate a situation in terms of its relevance to their own belief systems⁴⁶.

In addition, future studies should consider whether the valence and intensity of the film clip shown influences participant responses, particularly for neutral clips. While the current study showed no effect of the order in which the 15 film clips were shown to participants, it is still possible that some clips may have led to greater feelings of empathy, resulting in a carry-over effect when the next clip was played. While we did have participants complete a questionnaire to evaluate their general capacity for empathy (see Neumann, Zupan, Malec and Hammond⁴⁷) We did not specifically ask participants to indicate whether or not they emphasized with the characters, and if so, to what degree. These questions may provide greater insight into the why our results indicated participants with TBI were more likely to provide negative responses to neutral stimuli.

The current study used a forced-choice response format, a commonly used paradigm in emotion recognition studies. IT is possible that the limited options constrained participants to select a single emotion when they perceived the presence of multiple emotions⁴⁸. Additionally, they may have selected an emotion they did not perceive rather than choose the alternative response (i.e., I don't know). Using a free-choice response format with people with severe traumatic brain injury could result in additional and/or more problematic issues (e.g., word generation difficulties; long streams of explanations versus a single word). Therefore, it might be best to first adopt a forced-choice format that allows an additional free-choice word or comment to be added before moving onto the next item.

The use of film clips in the current study was a novel approach to assessing emotion recognition by people with TBI. It was hypothesized that people with TBI would find the multimodal nature of the film clips more complex, and thus more challenging than isolated facial and vocal cues. However, results suggested that rather than increase processing demands, the presence of numerous affective cues, including contextual information, facilitated processing. Yet, in everyday situations where a multitude of cues are presumably present, emotion recognition is a significant issue for people with TBI. Thus, while the film clips are more ecologically valid than isolated emotion expressions, they are not fully representative of the cues available in everyday social interactions. It could be that acting as an observer versus a participant allows for better integration of the available emotion cues.