Memorization-Based Training and Testing Paradigm for Robust Vocal Identity Recognition in Expressive Speech Using Event-Related Potentials Analysis

Human vocal streams are rich in information, such as emotion^1,2, health status^3,4, biological sex⁵, age⁶, and, more importantly, the individual vocal identity^7,8. Studies have suggested that human listeners have a robust capacity to recognize and differentiate their peers’ identities through voices, overcoming within-speaker variations around speaker identity’s average-based representation in the acoustic space⁹. Such variations are brought about by acoustic manipulation (fundamental frequency and vocal tract length, i.e., F0 and VTL) that corresponds to no clear pragmatic intentions⁹, emotion prosodies¹⁰, and vocal confidence that conveys speakers’ feeling of knowing¹¹. Behavioral experiments have focused on many factors that influence listeners’ performance in recognizing the talkers, including language-related manipulations^8,12,13, participants-related characteristics such as music experience or reading ability^14,15, and stimuli-related adaptions like backward speech or nonwords^16,17; more can be found in literature reviews^18,19. A few recent experiments have investigated how individual variation of the speaker identity representation might undermine the recognition accuracy, considering aspects including high versus low emotional expressiveness¹⁶ and neutral versus fearful prosodies⁵; more possible scenarios open for further investigation, as suggested by a review²⁰.

For the first research gap, the study proposes that the neurological underpinnings of speaker identification have yet to fully explore how within-speaker variation challenges listeners’ brain activities. For example, in an fMRI-based speaker recognition task by Zäske et al., participants’ right posterior superior temporal gyrus (pSTG), right inferior/middle frontal gyrus (IFG/MFG), right medial frontal gyrus, and left caudate showed reduced activation when correctly identified as old versus new talkers, regardless of the linguistic content being the same or different²¹. However, an earlier electroencephalography (EEG) study by Zäske et al. did not observe this old/new effect when speaker identity variation was introduced through different texts²². Specifically, a larger, late positive component (LPC) ranging from 300 to 700 ms, detected at the Pz electrode when listeners encountered their familiar trained talker expressing the same text (i.e., hearing a replay with non-varied linguistic content), was absent when the talkers delivered new texts.

In support of the assertion made by Zäske et al.²¹, this study suspects that an old/new effect can still be observed despite differences in linguistic content between training and testing sessions in event-related potential (ERP) analyses. This rationale stems from the notion that the absence of the old/new effect in Zäske et al.²², under conditions where different texts were used, may be attributed to the lack of an additional check session during the training task to ensure thorough and effective identity learning, as suggested by Lavan et al.²³. Consequently, the first objective of the study is to examine and validate this hypothesis. This study aims to test this by adding a checking session to the training-testing paradigm²².

Another key question this study aims to address is the robustness of speaker identification in the presence of speech prosody. Previous behavioral studies have suggested that listeners particularly struggle to recognize talkers across different prosodies, which indicates a modulatory role of prosodic context – listeners underperformed in the different training-testing prosody conditions. This study aims to test this by exposing listeners to recognize familiar talkers in either confident or doubtful prosodies²⁴. This study expects that the observed ERP differences will help explain how speech prosody influences identity recognition.

The core objective of the current study is to investigate the robustness of the old/new effect in speaker recognition, specifically examining whether there are differences in recognizing talkers in confident versus doubtful prosodies. Xu and Armony¹⁰ performed a behavioral study using a training-testing paradigm, and their findings suggest that listeners cannot overcome prosodic differences (e.g., trained to recognize a talker in neutral prosody and tested on fearful prosody) and can only achieve accuracy lower than chance level¹⁰. Acoustic analysis indicates that speakers expressing varied emotive states are associated with VTL/F0 modulation; for example, confident prosody is characterized by lengthened VTL and lower F0, whereas the opposite is true for doubtful prosody^11,24. Another piece of evidence comes from the study by Lavan et al.²³, which confirmed that listeners can adapt to VTL and F0 changes of the speaker and form average-based representations of the talkers. This study reconciles that, from a behavioral data perspective, listeners are likely to still recognize the talker’s identity across prosodies (e.g., trained to recognize one in confident prosody but tested in doubtful prosody; reported in a separate manuscript in preparation). Yet, the neural correlates of speaker identification, specifically the generalizability of the old/new effect observed by Zäske et al.²², remain unclear. Hence, the current study is committed to validating the robustness of the old/new effect in confident versus doubtful prosodies as contexts for testing.

The study introduces a departure from previous research paradigms in old/new effects studies. While past research focused on how old/new talker recognition influences perception, this study extends this by incorporating two confidence levels (confident versus doubtful) into the paradigm (thus, a 2+2 study). This allows us to investigate speaker recognition within the contexts of confident and doubtful speech prosodies. The paradigm enables the exploration of the robustness of old/new effects. The analyses of memory effects and regions of interest (ROI) within both confident and doubtful speech contexts serve as evidence for this investigation.

Altogether, the study aims to update the understanding of the EEG correlates of voice recognition, with the hypotheses that the enlarged LPC of the EEG old/new effect is observable even when 1) the linguistic content is not the same, and 2) with the presence of confident versus doubtful prosody. This study investigated the hypotheses through a three-step paradigm. Firstly, during the training phase, participants established associations between three voices and their corresponding names. Subsequently, in the checking phase, they were tasked with identifying the name corresponding to a voice from a selection of three candidates. This checking, following Lavan et al.²³, aims to overcome insufficient old speaker familiarization, which led to the unobserved old/new effect when the text in the training and testing phases differed⁶, and talkers could not recognize talkers across neutral and fearful prosodies¹⁰. Finally, in the testing phase, participants distinguished between three old and three new speaker voices in a two-alternative forced-choice task, with speech prosody presented as either confident or doubtful. EEG data were collected using a 64-channel EEG system and underwent preprocessing before analysis. Statistical analysis and event-related potential (ERP) analysis were conducted in RStudio, while MATLAB was utilized for brain topography analysis.

Regarding design details, this study proposes a speaker identity learning experiment that controls for the talker’s height, which is related to VTL and influences impressions of who is talking²³. This aspect also influences social impressions, such as perceived dominance²⁵, and such higher-level impression formation might interact with decoding speaker identity²⁶.