A Method to Study Adaptation to Left-Right Reversed Audition
PREPARAÇÃO DO INSTRUTOR
CONCEITOS
PROTOCOLO DO ALUNO
All methods described here have been approved by the Ethics Committee of Tokyo Denki University. For every participant, informed consent was obtained after the participant received a detailed explanation of the protocol.
1. Setup of the Left-Right Reversed Audition System
- Setup of the Reversed Audition System without a Participant
- Prepare a linear pulse-code-modulation (LPCM) recorder, binaural microphones, and binaural in-ear earphones.
- Connect the left and right lines of the microphones crossly to the LPCM recorder so that left-right reversed analogue sound signals are digitalized. Moreover, connect the left and right lines of the earphones straight through to the recorder so that the reversed digitalized signals are immediately played.
NOTE: In the case of employing the binaural earphone-microphones as binaural earphones, do not use the earphone parts in order to reduce the spillover of the sounds that go through the microphone parts. - Put the bodies of the microphones and the earphones together for each ear with slight isolation by sound proofing materials, and cover the microphones with dedicated windscreens for suppressing the wind noise.
- Insert rechargeable batteries and a large-capacity high-speed memory card into the LPCM recorder and turn it on. Set the recording conditions properly in such a manner that the sound signals are recorded on the memory card as an LPCM format at a sampling rate of 96 kHz with a 24-bit depth.
- Place the body of the system into a pocket-sized bag.
- Setup of the Reversed Audition System with a Participant
- Instruct a participant to insert the earphones of the reversed audition system tightly into the ear canals.
- Disconnect the lines for the left and right microphones and connect the dominant-ear side of the microphone straight through to the recorder.
- Instruct the participant to take off and put on the dominant-ear side of the system repetitively while adjusting the sound volume of the recorder to make the subjective loudness of direct (normal) and indirect (reversed) sounds equal (as close as possible).
- Check the loudness for the non-dominant ear as well, and connect all the lines of the system back again.
- Place the system into the participant’s pocket, fix the cords on the participant’s clothes appropriately to prevent them from becoming entangled, and pick up unwanted noises.
2. Validation of the Left-Right Reversed Audition System
NOTE: Perform the following steps to validate the left-right reversed audition system, irrespective of experiments studying adaptation to left-right reversal.
- Validation of the Sound Source Localization of the Reversed Audition System
- Locate a digital angle protractor whose initial direction is defined as 0° at the center of an anechoic room, and assume a virtual circle centered at this point with a radius of 2 m. Along the virtual circle, mark 72 possible sound sources at every 5° from -180° to 175° in a clockwise manner, and set up plane-wave speakers at these points directed towards the center of the circle.
- Set up a video camera near the center of the room to record the display of the digital protractor.
NOTE: Since the display of the protractor moves with the protractor’s body, the field of view of the video should be large enough to cover all the possible areas. Moreover, the video camera should be carefully placed in order to not disturb the participant’s sitting position and the sound presentation. - Prepare for two sessions of sound source localization: in the first session, the participant does not put on the reversed audition system. In the second session, the participant puts on the equipment, calibrates it, and checks the system (as explained in step 1.2) as quickly as possible.
- Guide the participants to sit comfortably and blindfolded at the center of the circle facing a 0° sound source and wait for the experiment to start.
- Conduct two sessions of sound source localization. In both sessions, have the participant use the protractor to indicate the perceived sound direction as precisely as possible without moving the head.
- For each session, start video-recording the angle display of the protractor, and present 1000-Hz sounds at 65-dB sound pressure level (SPL) from any of the sound sources: the sound at one location is randomly switched to the sound at another location every 10 s in such a way that each location is used once.
NOTE: Here we use MATLAB with the Psychophysics Toolbox16,17,18. Although this toolbox is commonly used to present sounds, any reliable stimulation software can also be used. - After each session, stop the video-recording and instruct the participants to take a break for sufficient amount of time.
- Read the trial-by-trial perceptual angles displayed on the protractor from the recorded video, and evaluate the spatial performance of the reversed audition system by comparing the perceptual angles in the normal and the reversed conditions against the physical angles defined by the direction of sound sources.
- Validation of the Delay of the Reversed Audition System
- Put the reversed audition system on a desk in a calm room with no participants.
- Disconnect a line to the left microphone, and place a plane-wave speaker and the left earphone as close as possible to the right microphone.
- Start recording direct (normal) sounds from the speaker and indirect (reversed) sounds from the left earphone simultaneously through the right microphone.
- Present 1-ms click sounds from the speaker with a moderate inter-stimulus interval at 65-dB SPL.
- After a sufficient number of trials, stop presenting and recording the sounds.
- In order to confirm the symmetrical configuration of the system, repeat the same steps above using the right earphone and the left microphone.
- Read the recorded sound data using software (e.g., MATLAB) and evaluate the difference between the onset timings of the direct (normal) sounds and indirect (reversed) sounds, which corresponds to a potential delay caused by the time spent passing through the electrical path in the system.
3. Studying the Adaptation to Left-Right Reversed Audition
- Procedure of the Exposure to Reversed Audition
- Remind the participants repeatedly of their right to quit the exposure at any time.
NOTE: Stop the exposure as soon as possible if the participant reports sickness or if an observer notices any sign that the participant wants to quit the exposure for any reason. - Prepare a sufficient number of spare rechargeable batteries and large-capacity high-speed memory cards to allow the participant to replace them at anytime.
- Instruct the participant to wear, calibrate, and check the reversed audition system by themselves during the exposure period, as explained in step 1.2. Perform the same procedure each time the participant wears the system after each interruption.
- Instruct the participant to perform daily-life activities while wearing the system continuously for approximately a month, except while sleeping, bathing, neuroimaging, and other emergency times. In these cases, ask participants to remove the system and immediately insert earplugs into their ears to prevent recovery of adaptation.
NOTE: Although it is ideal for the participant to wear the system all day and night, it is strongly recommended that the system not be worn while sleeping and bathing in order to prevent unexpected loud noises and electrical shocks, respectively. - Replace the batteries and memory cards routinely before battery exhaustion and memory overcapacity, respectively. Remove the system and replace it with earplugs quickly in a silent place without producing any sound.
- When a participant needs to move around outside, drive the participant in a car, accompany the participant on the move, or ask them to use safe means of transportation for acts performed alone.
NOTE: Great care should be taken by the researcher in order to not endanger the participant’s safety during the exposure period, especially when the participant goes outside. Prohibit the participant from performing any dangerous behaviors. - In order to facilitate adaptation, instruct the participant to experience situations involving high auditory input, such as walking in a shopping mall or a campus, having a conversation with more than two persons, and playing 3D video games, for as long as possible.
- Instruct the participant to keep a diary or provide a subjective report to an observer as frequently as possible about perceptual and behavioral changes, experienced events, and anything that the participant notices.
- After the target exposure period, instruct the participant to take off the reversed audition system.
NOTE: It is also important to follow up about the perceptual and behavioral changes in order to examine the recovery process from the adaption to left-right reversed audition.
- Remind the participants repeatedly of their right to quit the exposure at any time.
- Neuroimaging During the Exposure to Reversed Audition
- Instruct the participant to train on a task that will be used during the neuroimaging experiments as sufficiently as possible.
- For example, train the participant to perform a selective reaction time task in two conditions, compatible and incompatible15. The compatible condition consists of responding immediately to the right-ear sound with the right index finger and to the left-ear sound with the left index finger. The incompatible condition consists of responding immediately to the right-ear sound with the left index finger and to the left-ear sound with the right index finger.
- Use 1000-Hz sounds at 65-dB SPL for 0.1 s with an inter-stimulus interval of 2.5 – 3.5 s, which appears pseudorandomly on either ear side.
- Before the exposure to reversed audition, conduct a neuroimaging experiment under the trained task.
- For example, record either MEG or EEG responses, as well as the left and right finger responses under the selective reaction time task15. The task consists of two compatible and two incompatible blocks that are alternatively arranged with an inter-block interval of at least 30 s, and with sounds appearing 80 times for each block through the inserted earphones with plastic ear tubes.
NOTE: Although a 122-channel MEG system was used in Aoyama and Kuriki15, a multi-channel EEG system is also suitable for this protocol. - For the MEG/EEG recording, set the sampling rate at 1 kHz and the analog recording passband at 0.03 – 200 Hz.
- For example, record either MEG or EEG responses, as well as the left and right finger responses under the selective reaction time task15. The task consists of two compatible and two incompatible blocks that are alternatively arranged with an inter-block interval of at least 30 s, and with sounds appearing 80 times for each block through the inserted earphones with plastic ear tubes.
- During approximately a 1-month exposure to reversed audition, conduct neuroimaging experiments under the trained task every week without the reversed audition system in exactly the same way as in the pre-exposure experiment (step 3.2.2).
NOTE: The system is removed immediately before and put on immediately after each experiment. - One week after the exposure, conduct a neuroimaging experiment under the trained task in exactly the same way as the pre-exposure experiment (step 3.2.2).
- Analyze the collected data before, during, and after the exposure to left-right reversed audition.
- For example, after rejecting the epochs contaminated with eye-related artifacts, removing the offset in the pre-stimulus interval, and setting the low-pass filtering at 40 Hz, average the MEG/EEG data from 100 ms before to 500 ms after the sound onset for the stimulus-response compatible and incompatible conditions15.
- Using an MNE software package19,20, estimate the sources of the brain activity with dynamic statistical parametric maps (dSPMs) overlaid on cortical surface images and quantify the intensities of brain activity with minimum-norm estimates (MNEs) for each time point of the averaged data.
- Calculate the auditory-motor functional connectivity from single-trial zero-mean MEG/EEG data from 90 to 500 ms after the sound onset for each condition
NOTE: Here we use MATLAB with the Multivariate Granger Causality Toolbox21. - For the behavioral data, calculate the mean reaction times for the stimulus-response compatible and incompatible conditions.
- Instruct the participant to train on a task that will be used during the neuroimaging experiments as sufficiently as possible.
A Method to Study Adaptation to Left-Right Reversed Audition
Learning Objectives
The representative results shown here are based on Aoyama and Kuriki15. The present protocol achieved left-right reversed audition with high spatiotemporal accuracy. Figure 1 shows the sound source localization in directions over 360° before and immediately after putting on the left-right reversed audition system (Figure 1A), in six participants, as indicated by the cosine similarity. As shown in Figure 1B, the perceptual angles in the normal condition were quite well correlated with the physical angles (positive correlation, adjusted R2 = 0.99). The perceptual angles in the reversed condition were also well correlated with the physical angles (negative correlation, adjusted R2 = 0.96; see also Figure 4 in Aoyama and Kuriki15), although there existed a slight perceptual bias toward the counterclockwise rotation, especially for sounds coming from the right-front and the left-back directions. Notably, the perceptual angles in the reversed condition were more correlated with the oppositely arranged perceptual angles in the normal condition (adjusted R2 = 0.98) than the physical angles, as shown in Figure 1C. Furthermore, a potential delay of the system was estimated to be a constant 2 ms. The present protocol also achieved a natural wearing appearance, like listening to music with a mobile music player, thereby avoiding any stress of being noticed by other individuals.
Figure 1: Sound source localization in 360° directions, before and immediately after putting on the left-right reversed audition system, in six participants. (A) The constructed left-right reversed audition system. (B) Cosine similarity between perceptual angles and sign-regulated physical angles in the normal (blue) and reversed (red) conditions plotted against (unregulated) physical angles, respectively. While the physical angles are directly used for the cosine similarity in the normal condition, the signs of physical angles are inverted in the reversed condition. (C) Cosine similarity between perceptual angles in the reversed condition and oppositely arranged perceptual angles in the normal condition plotted against physical angles (purple). This figure has been modified from Aoyama and Kuriki15. Please click here to view a larger version of this figure.
The present protocol revealed perceptual changes to the reversed audition from a relatively early stage during the approximately 1-month exposure. Although a feeling of strangeness was reported just after the exposure, it began to decrease within a week of the exposure and continued to drop further over time. Mirror-image sounds were gradually perceived as normal, which also occurred with visual information and movements. One week after the end of the exposure period, all changes returned to the pre-exposure level. The present protocol detected not only perceptual but also behavioral and neural changes underlying the adaptation. Figure 2 shows changes in behavioral and neural responses during the selective reaction time task over the exposure time in a representative participant. As shown in Figure 2A, the mean reaction times for response-incompatible sounds were overall longer than those for response-compatible sounds from the pre-exposure period to the third week, but became slightly shorter in the fourth week. This relative inversion followed the transient elongation of the mean reaction times irrespective of compatibility in the second week. After the exposure, all mean reaction times returned to the initial level. The MNE intensities of the left and right N1m components exhibited similar trends to the mean reaction times, as shown in Figure 2B, although the compatible-incompatible relationship was inversed. The N1m components are distinct auditory evoked fields observed at about 90 ms after sound onset, and their source was confirmed to be located in the bilateral superior temporal planes using dSPMs. Overall, the intensities in the stimulus-response compatible conditions were higher than those in the incompatible conditions from the pre-exposure period to the third week, but were slightly lower in the fourth week. This relative inversion followed the transient enhancement of the intensities irrespective of compatibility and laterality in the second week. After the exposure, they returned to the initial levels.
Figure 2: Behavioral and neural responses during the selective reaction time task in a representative participant. (A) Mean reaction times for stimulus-response compatible and incompatible conditions. (B) Left and right auditory N1m intensities for stimulus-response compatible and incompatible conditions, as evaluated by minimum-norm estimates. Yellow zones indicate a period exposed to left-right reversed audition. This figure has been modified from Aoyama and Kuriki15. Please click here to view a larger version of this figure.
Furthermore, the present protocol revealed changes in the functional connectivity across the left and right auditory and motor areas during the selective reaction time task in two participants, as shown in Figure 3. The functional connectivity was tested with the Granger causality test at a threshold of p < 0.05. Initially, these auditory-motor areas communicated with each other irrespective of stimulus and response. However, after exposure to the reversed audition, the auditory-motor connectivity became unstable. Notably, in the second week, the auditory-motor connectivity was disrupted drastically, especially in the right motor-to-auditory feedback and left-to-right motor communication. Immediately after that, the connectivity recovered at the level of the first week, and returned to the initial level after the exposure.
Figure 3: Auditory-motor functional connectivity as tested by Granger causality tests during the selective reaction time task in two participants. Red, yellow, and no arrow(s) indicate the number of participants who showed significance at a threshold of p < 0.05 (N = 2, 1, and 0, respectively). LM and RM denote left and right motor areas, respectively, and LA and RA denote left and right auditory areas, respectively. This figure has been modified from Aoyama and Kuriki15. Please click here to view a larger version of this figure.
List of Materials
| Linear pulse-code-modulation recorder | Sony | PCM-M10 | |
| Binaural microphones | Roland | CS-10EM | |
| Binaural in-ear earphones | Etymotic Research | ER-4B | |
| Digital angle protractor | Wenzhou Sanhe Measuring Instrument | 5422-200 | |
| Plane-wave speaker | Alphagreen | SS-2101 | |
| Video camera | Sony | HDR-CX560 | |
| MATLAB | Mathworks | R2012a, R2015a | R2012a for stimulation and R2015a for analysis |
| Psychophysics Toolbox | Free | Version 3 | http://psychtoolbox.org |
| Insert earphones | Etymotic Research | ER-2 | |
| Magnetoencephalography system | Neuromag | Neuromag-122 TM | |
| Electroencephalography system | Brain Products | acti64CHamp | |
| MNE | Free | MNE Software Version 2.7, MNE 0.13 |
https://martinos.org/mne/stable/index.html |
| The Multivariate Granger Causality Toolbox | Free | mvgc_v1.0 | http://www.sussex.ac.uk/sackler/mvgc/ |
Preparação do Laboratório
An unusual sensory space is one of the effective tools to uncover the mechanism of adaptability of humans to a novel environment. Although most of the previous studies have used special spectacles with prisms to achieve unusual spaces in the visual domain, a methodology for studying the adaptation to unusual auditory spaces has yet to be fully established. This study proposes a new protocol to set-up, validate, and use a left-right reversed stereophonic system using only wearable devices, and to study the adaptation to left-right reversed audition with the help of neuroimaging. Although individual acoustic characteristics are not yet implemented, and slight spillover of unreversed sounds is relatively uncontrollable, the constructed apparatus shows high performance in a 360° sound source localization coupled with hearing characteristics with little delay. Moreover, it looks like a mobile music player and enables a participant to focus on daily life without arousing curiosity or drawing attention of other individuals. Since the effects of adaptation were successfully detected at the perceptual, behavioral, and neural levels, it is concluded that this protocol provides a promising methodology for studying adaptation to left-right reversed audition, and is an effective tool for uncovering the adaptability of humans to a novel environments in the auditory domain.
An unusual sensory space is one of the effective tools to uncover the mechanism of adaptability of humans to a novel environment. Although most of the previous studies have used special spectacles with prisms to achieve unusual spaces in the visual domain, a methodology for studying the adaptation to unusual auditory spaces has yet to be fully established. This study proposes a new protocol to set-up, validate, and use a left-right reversed stereophonic system using only wearable devices, and to study the adaptation to left-right reversed audition with the help of neuroimaging. Although individual acoustic characteristics are not yet implemented, and slight spillover of unreversed sounds is relatively uncontrollable, the constructed apparatus shows high performance in a 360° sound source localization coupled with hearing characteristics with little delay. Moreover, it looks like a mobile music player and enables a participant to focus on daily life without arousing curiosity or drawing attention of other individuals. Since the effects of adaptation were successfully detected at the perceptual, behavioral, and neural levels, it is concluded that this protocol provides a promising methodology for studying adaptation to left-right reversed audition, and is an effective tool for uncovering the adaptability of humans to a novel environments in the auditory domain.
Procedimento
An unusual sensory space is one of the effective tools to uncover the mechanism of adaptability of humans to a novel environment. Although most of the previous studies have used special spectacles with prisms to achieve unusual spaces in the visual domain, a methodology for studying the adaptation to unusual auditory spaces has yet to be fully established. This study proposes a new protocol to set-up, validate, and use a left-right reversed stereophonic system using only wearable devices, and to study the adaptation to left-right reversed audition with the help of neuroimaging. Although individual acoustic characteristics are not yet implemented, and slight spillover of unreversed sounds is relatively uncontrollable, the constructed apparatus shows high performance in a 360° sound source localization coupled with hearing characteristics with little delay. Moreover, it looks like a mobile music player and enables a participant to focus on daily life without arousing curiosity or drawing attention of other individuals. Since the effects of adaptation were successfully detected at the perceptual, behavioral, and neural levels, it is concluded that this protocol provides a promising methodology for studying adaptation to left-right reversed audition, and is an effective tool for uncovering the adaptability of humans to a novel environments in the auditory domain.