Investigating the Effect of Visual Imagery and Learning Shape-Audio Regularities on Bouba and Kiki

Tor&#248; Graven; Clea Desebrock

doi:10.3791/59954

JoVE Journal > Behavior

行動学

Investigating the Effect of Visual Imagery and Learning Shape-Audio Regularities on Bouba and Kiki

Published: September 13, 2019

doi:

10.3791/59954

Torø Graven¹, Clea Desebrock¹

¹Department of Experimental Psychology,University of Oxford

概要

The aim of the presented protocol is to investigate the role of visual imagery in the bouba/kiki-effect, whether training in noticing the bouba/kiki shape-audio regularities affects the bouba/kiki-effect and the recognition of individual bouba and kiki shapes, and finally what mental images these regularities produce.

Abstract

This article presents a protocol for investigating the role of visual imagery in the bouba/kiki-effect, whether training in noticing the bouba/kiki shape-audio regularities affects the bouba/kiki-effect and the recognition of individual bouba and kiki shapes, and finally what mental images these regularities produce. To generate bouba/kiki shape-audio regularities, there were two types of shapes (filled; outlined) and two types of audio (word; non-word sound). Three groups of individuals participated in three experiments: Blind, blindfold, and vision. The experiments were conducted in fixed order across participants, with no break between them. In Experiment 1 (pre-test-post-test design with three repeated within-group measures) the participants were asked to pick out the shape they associated with the auditory bouba/kiki; in Experiment 2 (within-subject design), to name one shape and some audio (sometimes congruous; sometimes incongruous) as ‘bouba’ or ‘kiki;’ and in Experiment 3 (post-test only design), to draw the shape they associated with the auditory bouba/kiki. The results suggest that the blindfold-group draw upon visual imagery to solve new problems, but not long term; that training in noticing bouba/kiki shape-audio regularities affects the bouba/kiki-effect and the recognition of individual bouba and kiki shapes, but differently in each experimental group; and that all experimental groups create mental images of the most characteristic shape feature of bouba (curve) and kiki (angle). In fact, the effect of visual imagery is robust across tasks, but not long term; the effect of learning shape-audio regularities is robust long term, but not across tasks. The presented protocol is appropriate for investigating the effect of visual imagery and learning shape-audio regularities, when they occur and how robust they are; in specific individuals and groups of individuals. This protocol is unique in that it keeps under control both the visual imagery and the sensory information during training and testing.

Introduction

95% of the world’s population shows an auditory-visual bouba/kiki-effect, associating the visually rounded shape with the spoken word, ‘bouba;’ and the visually angular shape with the spoken word ‘kiki;’ and they do this even when they have not had any experience with either the shape or the word¹. The bouba/kiki-effect precedes language learning; it occurs across languages¹^,²^,³^,⁴^,⁵, with both bouba and kiki (cf. people’s preference for visual curves over visual angles⁶^,⁷^,⁸^,⁹), and it depends on the combination of vowels and consonants (e.g., it does not occur with ‘bibi’ and ‘kuku’.)¹^,⁵^,¹⁰^,¹¹^,¹². This shape-word association occurs with other shape and word pairs as well, as long as they are mainly curved, as bouba, and mainly angular, as kiki¹¹^,¹³; and/or they have the same combination of vowels and consonants as ‘bouba’ and ‘kiki,’ for example: ‘Baluma’ and ‘takete,’ ‘maluma’ and ‘takete,’ ‘uloomo’ and ‘takete,’ ‘maa-boo-maa’ and ‘tuh-kee-tee’¹^,³^,⁴^,¹⁰^,¹¹^,¹²^,¹⁴. In fact, people associate the rounded visual shapes with the spoken words containing continuant consonants (such as /l/ and /m/) and open back vowels (such as /ɑ:/, and /ɔ:/), and the visual angular shapes with the spoken words containing plosive consonants (such as /k/ and /t/) and close front vowels (such as (/ε/ and /I/)¹¹^,¹²^,¹⁵^,¹⁶. They are influenced more by the consonants than the vowels, especially by the voiceless ones (e.g., /k/ and /t/)¹¹^,¹²^,¹⁵^,¹⁶. Indeed, it seems the features of the global shapes – their curves and angles¹⁷ – and the sound of the spoken words – their melody¹⁷ – are the most crucial, as opposed to the global shapes and words themselves.

One study has investigated the bouba/kiki-effect with tactile bouba and kiki instead of the visual bouba/kiki shapes and found that participants who were visually impaired showed a significantly less robust bouba/kiki-effect (~64%) than the fully sighted (~90%)¹⁸. This study argued that the significantly less robust bouba/kiki-effect amongst the participants who were blind and partially sighted was caused by a lack of visual imagery: The fully sighted participants had noticed regularities in their environment that are not easily accessed when vision is impaired¹⁸. It is not clear from this study, however, whether visual imagery is necessary for the bouba/kiki-effect to occur: Only six of the 42 participants with a visual impairment were congenitally totally blind¹⁸, thus had no visual imagery at all. Furthermore, the other participants, none of them blindfolded, may have observed the experimenter’s lip movements when announcing the bouba/kiki word – rounded lip movements when announcing ‘bouba’ and angular lip movements when announcing ‘kiki’¹^,¹⁸ – showing a tactile-visual-auditory bouba/kiki-effect instead of drawing upon any visual imagery. Additionally, any effect of noticing tactile-auditory regularities was not investigated, for example, by comparing the bouba/kiki-effect on trial one and trial four instead of calculating the effect across all four trials. Moreover, the tactile bouba and kiki were rather different across trials: In trials one and two they were 3D and 2D shapes (curved v angular); in trials three and four, identical in shape (circle) and dissimilar in texture (smooth v rough; smooth v spiky)¹⁸. In a related vein, one study has investigated a kinesthetic-auditory bouba/kiki-effect in blindfolded (fully sighted) individuals and found that, after a two-minute training period of holding a robotic stylus programmed to draw trajectories of the bouba and kiki shapes, 82% showed the bouba/kiki-effect¹⁹. It is not clear from this study, however, whether the bouba/kiki-effect occurred because of the training period: The study did not include a pre-test, nor a control group.

To this end, we have investigated the role of visual imagery in the bouba/kiki-effect, whether training in noticing tactile/visual-auditory bouba and kiki regularities affected the bouba/kiki-effect and the recognition of individual tactile/visual bouba and kiki shapes, and finally what mental images these shape-audio regularities produced¹⁷. To generate bouba/kiki shape-audio regularities – not merely, for example, kiki-shape/kiki-word particularities¹ – this study included two types of tactile/visual bouba/kiki shapes (filled; outlined) and two types of auditory bouba/kiki ¹⁷. Three groups of individuals participated in three experiments: Blind (N = 12), blindfold (N = 12), and vision (N = 12). The experiments were conducted in fixed order across participants and with no break between them, to keep under control the participants’ amount and type of experience with bouba and kiki. In Experiment 1 (pre-test-post-test design with three repeated within-group measures) the participants were asked to pick out the tactile/visual shape they associated with the auditory bouba/kiki; in Experiment 2 (within-subject design), to name one tactile/visual shape and some audio (sometimes congruous; sometimes incongruous) as ‘bouba’ or ‘kiki;’ and in Experiment 3, (post-test only design) to draw the tactile/visual shape they associated with the auditory bouba/kiki¹⁷. Overall, this study suggested that the effect of visual imagery is robust across tasks, but not long term within each task, whereas the effect of learning shape-audio regularities is robust long term within each task, but not across tasks¹⁷. This article presents the protocol from this study¹⁷. The presented protocol is appropriate for investigating the effect of visual imagery and learning shape-audio regularities, when they occur and how robust they are; in specific individuals and groups of individuals. This protocol is unique in that it keeps under control both the visual imagery and the sensory information during training and testing.

Protocol

The Medical Sciences Inter-Divisional Research Ethics Committee (IDREC) University of Oxford provided approval for this protocol (Ref No: MS-IDREC-C1-2015-200, R46287/RE002, and R42687/RE004).

1. Design and conditions

Conduct three experiments in the following fixed order with all participants: Experiment 1, Experiment 2, and Experiment 3, with no break between them. Make sure that all participants in Experiment 2 participated in Experiment 1 and all participants in Experiment 3 also in Experiment 2 (and Experiment 1).
Conduct Experiment 1 – a pre-test-post-test design, with three experimental groups: Blind, blindfold, and vision; and three repeated within-group measures. Present the trials in fixed order to all participants. (Total number of trials: 8.)
1. Conduct the pre-test (Trial 1) to test the instant tactile/visual-auditory bouba/kiki-effect.
2. Conduct the first repeated within-group measure (Trial 4) to test the effect of repeated tactile/visual shape (either filled or outlined) and varied audio (word and non-word sound).
3. Conduct the second within-group measure (Trial 5) to test the effect of new tactile/visual shape (either outlined or filled) and varied audio (word and non-word sound).
4. Conduct the third within-group measure and post-test (Trial 8) to test the effect of varied tactile/visual shape (filled and outlined) and varied audio (word and non-word sound).
5. Randomly assign which type of tactile/visual bouba/kiki shapes to present in trials 1-4 (either filled or outlined). Present the other type in trials 4-8. Then, randomly assign which auditory word (either bouba or kiki) to present in Trial 1, and which auditory bouba/kiki (either bouba or kiki and either word or non-word sound) to present in each one of Trials 2-8.
  NOTE: Make sure that in total, there are four bouba and four kiki trials; four filled and four outlined tactile/visual shape trials; four auditory word and four auditory non-word sound trials.
Conduct Experiment 2 – a within-subject design, with three experimental groups: Blind, blindfold, and vision; and two within-group measures. Present the trials in fixed order to all participants. (Total number of trials: 8)
1. Test the instant recognition of congruous (bouba/kiki + bouba/kiki) shape-audio (Trial 1).
2. Test the recognition of (a) congruous (bouba/kiki + bouba/kiki), and (b) incongruous (bouba/kiki + kiki/bouba) shape-audio [across four trials per type of congruousness (Trials 1-8)].
3. Randomly assign which type of tactile/visual bouba/kiki shape (either filled or outlined) to present in trials 1-8. Then, randomly assign which congruous tactile/visual shape and auditory non-word sound (either bouba + bouba or kiki + kiki) to present in Trial 1, and finally which type of congruousness (either congruous or incongruous tactile/visual bouba/kiki shape + auditory bouba/kiki non-word sound) to present in each one of Trials 2-8.
  NOTE: Make sure that in total, there are four tactile/visual bouba and four tactile/visual kiki trials; four filled and four outlined tactile/visual trials; four congruous and four incongruous (shape + non-word sound) trials. (The auditory bouba/kiki word is not included in this experiment.)
Conduct Experiment 3 – a post-test only design, with three experimental groups: Blind, blindfold, and vision. Present the trials in fixed order to all participants (Total number of trials: 4).
1. Test what mental images the training in noticing shape-audio regularities (i.e. Experiment 1 and Experiment 2) have produced, in four separate trials: Bouba word, bouba non-word sound, kiki word, and kiki non-word sound.
2. Randomly assign which type of auditory bouba/kiki (either bouba or kiki and either word or non-word sound) to present in each one of Trials 1-4.
  NOTE: Make sure that in total, there are one auditory bouba word trial, one auditory bouba non-word sound trial, one auditory kiki word trial, and one auditory kiki non-word sound trial.

2. Participants

Recruit one group of participants who are congenitally blind; that is, blinded less than four months after birth²⁰^,²¹^,²²: Categories 5: total blindness; and 4: Light perception (perceiving a light source) and light projection (perceiving where a light source is situated)²². Make sure they are all braille readers.
Recruit one group of fully sighted participants, at twice the size of the group of participants who are congenitally blind. Make sure they all have normal/corrected to normal vision.
NOTE: Make sure that all participants are between 18 and 70 years of age, with normal/corrected to normal hearing, no cognitive delay or impairment, nor any physical disabilities, and that they are all naïve to the bouba/kiki-effect.
Randomly assign half the group of the fully sighted participants to the blindfold-group and the other half to the vision-group.
NOTE: Participants, type of tactile/visual bouba/kiki shape(s), type of auditory bouba/kiki, and congruousness are all randomly assigned to one of two categories: Blindfold or vision; filled or outlined; word or non-word sound; bouba or kiki; congruous or incongruous. Thus the random assignment can be done either by a computer program or manually (e.g., by tossing a coin).

3. Materials

Produce the tactile/visual materials
1. Transfer the visual bouba/kiki shapes (30 x 23 mm) to swell paper, either by printing them from a personal computer or by photocopying then from a picture. Make sure to transfer two types of visual bouba/kiki shapes: One filled and one outlined [(1.5 pt. line). Cf. Figure 1].
2. Pass the swell paper through a heat fuser. Make sure the tactile bouba/kiki shapes rise to ~0.5 mm height. Alternatively, order the tactile/visual bouba/kiki shapes from a transcription service for people who are visually impaired (with a PDF file attached).
3. Glue the swell paper onto foam board [white (5 mm thick)]. Use spray glue.
4. Cut the swell paper and foam board into size (50 x 50 mm), that is, to create four separate tactile/visual picture cards: Bouba filled, bouba outlined, kiki filled, and kiki outlined (cf. Figure 1). Use a knife for cutting wallpaper.
5. Cut a carton plate/foam board for covering the tactile/visual bouba/kiki shapes into size (297 x 210 mm). Use a knife for cutting wallpaper. Alternatively, order the A4 format.
6. Cut copy paper/plastic embossing film into size (210 x 148 mm). Use a guillotine. Alternatively, order the A5 format.
7. Cut a rubber mat into size (230 x 150 mm). Use a knife or pair of scissors. Alternatively, order the Moon hand frame and take it apart to uncover the rubber mat.

Figure 1: Two types of bouba/kiki shapes: Filled and outlined Please click here to view a larger version of this figure.

Produce the auditory materials
1. Record the spoken bouba (cf. Audio 1) and kiki words. Save in the MP3 format: One file for the bouba word and one for the kiki word.
2. Create the bouba and kiki non-word sounds, by mimicking the duration and frequency (i.e. melody) of the spoken words ‘bouba’ and ‘kiki.’ Use a sound wave generator and pitch bend. Save in the MP3 format: One file for the bouba sound (cf. Audio 2) and one for the kiki sound.
3. Install the VLC app for iOS on a smartphone. Transfer the MP3 files for bouba and kiki to the VLC app. Save them as four separate files: Bouba sound, bouba word, kiki sound, and kiki word.

Audio 1: Bouba word Please click here to download this file.

Audio 2: Bouba sound Please click here to download this file.

4. Procedure and scoring

Set up the test session
1. Conduct the three experiments in a quiet room that is neutral in color, and with all participants individually. Ask the participants in the blindfold-group to put on their blindfold, before setting up the tactile/visual-auditory materials, and make sure they keep their blindfold on throughout and between all three experiments.
Set up the tactile/visual materials
1. Place the rubber mat directly in front of the participant. Then, place (Experiment 1) the two tactile/visual bouba and kiki shapes (Experiment 2), the one tactile/visual bouba or kiki shape and (Experiment 3), the copy paper/plastic embossing film on top of the rubber mat.
  NOTE: In Experiment 1, make sure to always present bouba on the left and kiki on the right; and in both Experiment 1 and Experiment 2, to always present them in the same orientation (cf. Figure 1): To enhance the participants’ ability to form mental images of the bouba and kiki shapes without interference from their position and/or orientation on the rubber mat. Make sure not to inform the participants about this fixed presentation of the tactile/visual bouba and kiki shapes. In the vision-group, always set up the test material behind a carton plate/foam board, to prevent these participants from taking a sneak peek at what is in store.
Set up the auditory materials
1. Place the MP3 player directly in front of the participant (on the other side of the rubber mat), with the loudspeaker closest to them.
  NOTE: Do not repeat back verbally the auditory information from the MP3 player: To prevent the participants in the vision-group from observing the lip movements on the bouba/kiki word-trials¹, and thus keep under control across all experimental groups the amount and type of sensory information.
Conduct Experiment 1, Experiment 2, and Experiment 3
1. Invite the participants who will be exploring and drawing the tactile shapes (i.e. the blind and blindfold-group) to explore the rubber mat (by touch): For example, for size and texture, without any of the tactile/visual picture cards and/or copy paper or plastic embossing film placed on top.
Conduct Experiment 1
1. Explain that two picture cards will be presented on the rubber mat, each one picturing a tactile/visual shape. Then, explain that some audio will be played from the MP3 player and that this audio will correspond to one of the tactile/visual shapes.
  NOTE: Do not inform the participants about the two types of tactile/visual bouba/kiki (filled and outlined) and the two types of auditory bouba/kiki (sound and word).
2. Place the two tactile/visual picture cards on the rubber mat (cf. Figure 1).
3. In the blind and blindfold-group, ask the participant to make a fist using both hands and to place their fists on top of the presented tactile/visual picture cards, guiding them if needed. Place the left hand on top of the participant’s fists.
4. Ask the participant to explore the two tactile/visual shapes, the blind and blindfold-group by touch and the vision-group by vision, when the hand on top of their fists (blind and blindfold-group) / the carton plate or foam board (vision-group) is removed, and to say ‘yes’ when they have finished their exploring.
5. Remove the left hand from the participant’s fists (in the blind and blindfold-group) / the carton plate or foam board (in the vision-group).
6. As soon as the participant says ‘yes,’ cover the tactile/visual picture cards.
7. Play the audio from the MP3 player.
  NOTE: Play the auditory information from the MP3 player only once and do not repeat it back orally to the participants.
8. Ask the participant to signal which tactile/visual shape they associate with the audio, for example, by saying ‘left/right,’ ‘curvy/angular’ or ‘flower/star’, and note the shape given. Do not respond to the participant’s answer.
9. Score whether the participant’s answer is correct or incorrect.
  NOTE: Repeat 4.5.2–4.5.9 for all eight trials.
Conduct Experiment 2
1. Explain that one picture card, picturing a tactile/visual shape, will be presented on the rubber mat and that some audio will be played from the MP3 player. Then, explain that the audio will be played one time per picture card, as soon as the participant starts exploring the tactile/visual shape.
  NOTE: Do not inform the participants about the two types of tactile/visual bouba/kiki (filled and outlined), the one type of auditory bouba/kiki (sound), and the two types of congruousness (congruous and incongruous).
2. Place the tactile/visual picture card on the rubber mat.
3. In the blind and blindfold-group, ask the participant to make a fist using both hands and to place their fists on top of the presented tactile/visual picture card, guiding them if needed. Place the left hand on top of the participant’s fists.
4. Ask the participant to explore the tactile/visual shape, the blind and blindfold-group by touch and the vision-group by vision, when the hand on top of their fists (blind and blindfold-group) / the carton plate or foam board (vision-group) is removed, and to listen to the audio. Then, ask them to name the tactile/visual shape and the audio as either ‘bouba’ or ‘kiki’: One name only²³^,²⁴^,²⁵, when they have finished their exploring.
5. Remove the left hand from the participant’s fists (in the blind and blindfold-group) / the carton plate or foam board (in the vision-group) and play the audio from the MP3 player.
  NOTE: Play the auditory information from the MP3 player only once and do not repeat it back orally to the participants.
6. As soon as the participant names the tactile/visual shape and the audio, cover the picture card and note the name given. Do not respond to the participant’s answer.
7. Score whether the participant’s answer is correct or incorrect.
  NOTE: Repeat 4.6.2–4.6.7 for all eight trials.
8. Across the congruous and incongruous trials, count a participant as ‘correct’ when they have recognized ≥75% of the tactile/visual bouba and kiki shapes.
Conduct Experiment 3:
1. Explain that some copy paper/plastic embossing film will be placed on top of the rubber mat, instead of the tactile/visual picture card(s), and that the participant will be asked to draw on it. In the blind and blindfold-group, invite the participant to try out the different drawing equipment and to decide for themselves which they prefer: The copy paper or plastic embossing film and rollerball pen or saddleback style. Next, explain that some audio will be played from the MP3 player and that the participant will be drawing the tactile/visual shape they associate with this audio, one shape per segment of audio when asked.
  NOTE: Do not inform the participants about the auditory information (bouba and kiki), nor the two types of auditory bouba/kiki (sound and word).
2. Place the copy paper/plastic embossing film on the rubber mat.
3. In all three experimental groups (blind; blindfold; vision), ask the participant to make a fist using both hands while holding the rollerball pen/saddleback style and to place their fists on top of the copy paper/plastic embossing film, guiding them if needed. Place the left hand on top of the participant’s fists.
4. Ask the participant to draw the shape that they associate with the presented audio when the hand on top of their fists is removed. In the blind and blindfold-group, ask the participant to draw a tactile shape; and in the vision-group, a visual shape. Ask the participant to say ‘yes’ when they have finished their drawing.
5. Remove the left hand from the participant’s fists.
6. Play the audio from the MP3 player.
  NOTE: Play the auditory information from the MP3 player only once and do not repeat it back orally to the participants.
7. As soon as the participant says ‘yes,’ remove their tactile/visual drawing from the rubber mat and label it in the bottom right corner with the correct trial number. Do not respond to or provide any comments on the participant’s answer/drawn shape.
  NOTE: Repeat 4.7.2–4.7.7 for all four trials.
8. Scan all tactile/visual drawings (1:1 size) into the jpg format immediately after each test session.
  NOTE: Experiment 3 is scored by a group of scorers after all drawing data have been collected.
9. Produce a questionnaire for scoring the drawing data (cf. Figure 2):
  1. Scale down all scanned drawings to include 12 pictures per page in the landscape A4 format.
  2. Randomly assign bouba/kiki trials, sound/word trials, and blind/blindfold/vision drawers.
  3. Add trial number in the upper left corner of each picture.
  4. Add ‘bouba’ and ‘kiki,’ and ‘blind,’ ‘blindfolded’ and ‘sighted’ under each picture.
10. Recruit a group scorer, above 18 years of age; with normal/corrected to normal hearing and vision; with no cognitive delay or impairment, nor any physical disabilities; that are all naïve to this study, and that show a visual shape-printed word bouba/kiki-effect.
11. Ask the scorers to answer two multiple-choice questions per picture in the questionnaire:
  (1) ‘Is the pictured shape “bouba” or “kiki;”’ and (2) ‘Is the pictured shape drawn by someone who is blind (since birth), blindfolded, or fully sighted?’¹⁷.
12. Depending on the number of pictures in the questionnaire, ask half of the group of scorers to score half the questionnaire, and the other half to score the rest.
13. For each trial, count ‘correspond,’ ‘not correspond,’ and ‘inconclusive,’ that is:
  1. ‘Correspond’ when (i) ≥75% of the scorers agreed on which tactile/visual shape (bouba or kiki) was pictured, and (ii) this shape corresponded to the presented auditory bouba/kiki in this trial.
  2. ‘Not correspond’ when (i) ≥75% of the scorers agreed on which tactile/visual shape (bouba or kiki) was pictured, and (ii) this shape did not correspond to the presented auditory bouba/kiki in this trial.
  3. ‘Inconclusive’ when <75% of the scorers agreed on which tactile/visual shape (bouba or kiki) was pictured. Remove the ‘inconclusive’ pictures/participant drawings from the statistical analyses.
14. Across trials, count a participant as ‘correspond’ when ≥75% of their drawn mental images: That is, that tactile/visual shapes correspond to the presented auditory bouba/kiki.
15. For each trial, count ‘recognized,’ ‘not recognized,’ and ‘inconclusive,’ that is:
  1. ‘Recognized’ when (i) ≥66.6% of the scorers agreed on which experimental group (blind, blindfold or vision) had drawn the pictured shape, and (ii) this was correct in this trial.
  2. ‘Not recognized’ when (i) ≥66.6% of the scorers agreed on which experimental group (blind, blindfold or vision) had drawn the pictured shape, and (ii) this was not correct in this trial.
  3. ‘Inconclusive’ when <66.6% of the scorers agreed on which experimental group (blind, blindfold or vision) had drawn the pictured shape. Exclude the ‘inconclusive’ participant recognitions from the statistical analyses.
16. Across trials (and with ‘inconclusive’ participant recognitions removed), count a participant as ‘recognized’ when recognized in ≥75% of all trials.

Figure 2: Examples from the questionnaire for scoring the drawing data Please click here to view a larger version of this figure.

Representative Results

The bouba/kiki-effect

Six of the 12 participants who were congenitally blind (50%), nine of the 12 who were blindfolded (75%), and 10 of the 12 who were fully sighted (~83%) showed an instant tactile/visual-auditory bouba/kiki-effect: That is, both the blindfold and vision-group were significantly above the chance level (of 50%): χ²(1. N = 12) = 3.00, p = 0.08 and χ²(1. N = 12) = 5.33, p = 0.02 (Experiment 1, Trial 1)¹⁷. No significant difference was found by Fisher’s exact test between the three experimental groups (Experiment 1, pre-test)¹⁷. (Cf. Table 1.)

When it comes to the long term tactile/visual-auditory bouba/kiki-effect: On the first repeated within-group measure, nine of the participants who were congenitally blind showed the bouba/kiki-effect along with nine who were fully sighted and seven who were blindfolded: The blind and vision-group significantly above the chance level (of 50%): χ²(1. N = 12) = 3.00, p = 0.08 and χ²(1. N = 12) = 3.00, p = 0.08 (Experiment 1, Trial 4)¹⁷. On the second repeated within-group measure, 11 participants in the blind and vision-group showed the tactile/visual-auditory bouba/kiki-effect: Both experimental groups were again significantly above the chance level (of 50%): χ²(1. N = 12) = 8.33, p = 0.00 and χ²(1. N = 12) = 8.33, p = 0.00; and seven in the blindfold-group (Experiment 1, Trial 5)¹⁷. Finally, on the third repeated within-group measure and post-test, nine of the 12 participants who were congenitally blind (75%), six of the 12 who were blindfolded (50%), and all of the 12 participants who were fully sighted (100%) showed the tactile/visual-auditory bouba/kiki-effect; both the blind and vision-group were again significantly above the chance level (of 50%): χ²(1. N = 12) = 3.00, p = 0.08 and χ²(1. N = 12) = 12.00, p = 0.00 (Experiment 1, Trial 8)¹⁷. Fisher’s exact test found a significant difference between the blindfold and vision-group [p = 0.01. (Experiment 1, post-test)]¹⁷. (Cf. Table 1.)

Participant group	Instant bouba/kiki- effect	Long term bouba/kiki-effect
Participant group	Pre-test	Repeated measure 1	Repeated measure 2	Repeated measure 3/post-test
Blind	50.0%	75.0%	91.7%	75.0%
Blindfold	75.0%	58.3%	58.3%	50.0%
Vision	83.3%	75.0%	91.7%	100.0%

Table 1: The instant and long term bouba/kiki-effect

The recognition of bouba and kiki shapes

Eleven of the 12 participants who were congenitally blind (~92%), nine of the 12 who were blindfolded (75%), and all of the 12 who were fully sighted (100%) instantly recognized the congruous tactile/visual and auditory bouba/kiki; all three experimental groups were significantly above the chance level (of 50%): χ²(1. N = 12) = 8.33, p = 0.00, χ²(1. N = 12) = 3.00, p = 0.08, and χ²(1. N = 12) = 12.00, p = 0.00 (Experiment 2, Trial 1)¹⁷. (Cf. Table 2.)

Long term, 11 participants in the blind-group recognized the tactile bouba/kiki shapes together with congruous audio and 10 participants with incongruous audio: Both types of congruousness recognized significantly above the chance level (of 50%): χ²(1. N = 12) = 8.33, p = 0.00 and χ²(1. N = 12) = 5.33, p = 0.02 (Experiment 2, Trial 1-8)¹⁷. Nine participants in the blindfold-group recognized the tactile shapes together with congruous audio and eight participants together with incongruous audio; in other words, the congruous shape and audio were significantly above the chance level (of 50%): χ²(1. N = 12) = 3.00, p = 0.08 (Experiment 2, Trial 1-8)¹⁷. All 12 participants in the vision-group recognized the visual bouba/kiki shapes together with congruous audio and six participants with incongruous audio: The congruous shape and audio recognized significantly above the chance level (of 50%): χ²(1. N = 12) = 12.00, p = 0.00 (Experiment 2, Trial 1-8)¹⁷. (Cf. Table 2.)

Participant group	Instant recognition of bouba/kiki shape	Long term recognition of bouba and kiki shapes
Participant group	Congruous shape and audio	Congruous shape and audio	Incongruous shape and audio
Blind	91.7%	91.7%	83.3%
Blindfold	75.0%	75.0%	66.7%
Vision	100.0%	100%	50.0%

Table 2: The instant and long term recognition of bouba and kiki shapes

The mental images of bouba and kiki

Eight of the 12 participants who were congenitally blind [~73% (with one ‘inconclusive’ participant drawing removed)], eight of the 12 who were blindfolded [~89% (with three ‘inconclusive’ participant drawings removed)], and eight of the 12 who were fully sighted [80% (with two ‘inconclusive’ participant drawings removed)] instantly drew a mental image: A tactile/visual shape that corresponded to the presented auditory bouba/kiki (Experiment 3, Trial 1)¹⁷. Both the blindfold and vision-group were significantly above the chance level (of 50%): χ²(1. N = 9) = 5.44, p = 0.02 and χ²(1. N = 10) = 3.60, p = 0.06 (Experiment 3, Trial 1)¹⁷. (Cf. Table 3.)

Regarding the long-term mental images of bouba and kiki: 11 participants in the blind-group, eight in the blindfold-group, and 12 in the vision-group drew tactile/visual bouba/kiki shapes that corresponded to the presented auditory bouba/kiki (Experiment 3, Trial 1-4)¹⁷. Both the blind and vision-group were significantly above the chance level (of 50%): χ²(1. N = 12) = 8.33, p = 0.00 and χ²(1. N = 12) = 12.00, p = 0.00 (Experiment 3, Trial 1-4)¹⁷. Fisher’s exact test found a significant difference between the blindfold-group and the vision-group [p = 0.09 (Experiment 3, Trial 1-4)]¹⁷. (Cf. Table 3.)

Participant group	Instant mental image of bouba/kiki	Long term mental images of bouba and kiki
Blind	72.7%	91.7%
Blindfold	88.9%	66.7%
Vision	80.0%	100.0%

Table 3: The instant and long term mental images of bouba and kiki

Furthermore. ~83% of all participant drawings included the most characteristic shape feature of the global bouba and kiki shapes: Curve and angle, respectively (Experiment 3, Trial 1-4)¹⁷. The participant drawings differed in the quantity of curves/angles (e.g., one and five angles for the kiki word: cf. Figure 2, Trials 1, 6, and 9), and in the direction of the curves/angles [i.e. horizontal, vertical or diagonal: cf. Figure 2, Trials 2, 5, and 8 (bouba sound)], but typically did not include the global bouba/kiki shape [Experiment 3, Trial 1-4 (cf. Figure 1; Figure 2)]¹⁷. Finally, the experimental group was recognized in ~43% of the scores’ answers: Five participants in the blind-group, three in the blindfold-group, and five in the vision-group; no experimental group was significantly above the chance level [of 33.3% (Experiment 3, Trial 1-4)]¹⁷.

The effect of visual imagery and learning

The presented protocol succeeded in investigating the role of visual imagery in the bouba/kiki-effect, whether training in noticing tactile/visual-auditory bouba and kiki regularities affected the bouba/kiki-effect and the recognition of individual tactile/visual bouba and kiki shapes, and finally what mental images these shape-audio regularities produced¹⁷. By including one experimental group with no visual experience (i.e. the blind-group) and two experimental groups with visual experience (i.e. the blindfold and vision-group), it was possible to test the effect of visual imagery; and by including one experimental group with no visual experience and extensive tactile experience (i.e. the blind-group) and one experimental group with extensive visual experience and no tactile experience (i.e. the blindfold-group), it was possible to test the effect of the training in noticing tactile-auditory regularities. The results clearly suggest that the blindfold-group drew upon visual imagery to solve new tactile-auditory problems, but not long term (Experiment 1–3); that training in noticing bouba/kiki shape-audio regularities affected the tactile/visual-auditory bouba/kiki-effect (Experiment 1) and the recognition of individual tactile/visual bouba and kiki shapes [i.e. together with congruous audio (Experiment 2)], but differently in each experimental group (Experiment 1-2), and that all experimental groups created mental images of the most characteristic tactile/visual-auditory shape feature of bouba (curve) and kiki [(angle) Experiment 3. Cf. Table 1; Table 2; Table 3]¹⁷. Moreover, by including repeated within-group measures of instant and long term effect (Experiment 1-3), it was possible to test when the effect of visual imagery and learning shape-audio regularities actually occurred; and by including three different tasks (i.e. the tactile/visual-auditory bouba/kiki-effect in Experiment 1, recognizing individual tactile/visual bouba/kiki shapes together with congruous and incongruous audio in Experiment 2, and drawing mental images of the bouba/kiki audio in Experiment 3), to test how robust these effects were. The results clearly suggest that the effect of visual imagery was not robust long term within each task, especially when using haptic touch [cf. the blindfold-group (Experiment 1-3)]; whereas the effect of learning shape-audio regularities was, but not across tasks [cf. the blind and vision-group (Experiment 1-3). Cf. Table 1; Table 2; Table 3.]¹⁷

Discussion

The presented protocol is appropriate for investigating the effect of visual imagery and learning shape-audio regularities, when they occur and how robust they are; in specific individuals and groups of individuals.

This protocol would be appropriate for testing not only the accuracy but also the exploration time: From when the experimenter removes their left hand from the participant’s fists (in the blind and blindfold-group) / the carton plate or foam board (in the vision-group) to when the participant says ‘yes.’ In the statistical analyses, the longest exploration time per trial per experimental group would be divided into four and the number of ‘fast,’ ‘medium’ and ‘slow’ participants in each experimental group counted: ‘Fast’ using ≤25% of this exploration time, ‘medium’ using 26-75%, and ‘slow’ using ≥75%. Across trials, the participants would be counted as in Experiment 2: That is, as ‘fast’ if counted as ‘fast’ in ≥66.6% of all trials, as ‘medium’ if counted as ‘medium’ in ≥66.6% of all trials, and so on. (For ‘inconclusive’, i.e. the participant cannot be counted as ‘fast,’ ‘medium’ or ‘slow’, cf. Experiment 3.) It would be possible to test the exploration time for correct and incorrect answers separately, depending on the sample size and/or the number of correct and incorrect answers. Whether testing accuracy, exploration time or both, there are critical protocol steps that must be followed: The three experiments should be conducted in fixed order, and their trials presented in fixed order, for all participants. The participants should not be made aware that the test materials in all three experiments include bouba and kiki. Further, the auditory information from the MP3 player should be played only once and not repeated back orally to the participants (all three experiments). In Experiment 1 and Experiment 2, the tactile/visual bouba and kiki should be presented in the same position and orientation. For the vision-group (in all three experiments), the test materials should be set up behind a carton plate/foam board. Finally, in the blindfold-group, the participants should not remove their blindfold until Experiment 3 has been completed. These steps are to ensure that the amount and type of information about the tactile/visual and the auditory bouba and kiki is kept under control across all experimental groups. In addition, it is crucial in Experiment 2 to present the auditory bouba/kiki as soon as the participant starts exploring the tactile/visual bouba/kiki. This will ensure that multimodal (congruous or incongruous) information is presented. Moreover, this protocol would also be appropriate for testing other groups of individuals. To integrate these, such participant groups could be added as separate experimental groups: For example, a color perception-group and a shape perception-group;²² a sudden vision loss-group and a progressive vision loss-group;²⁰^,²¹ and/or specific groups of individuals with a hearing impairment and/or an Autism Spectrum Disorder¹³ could be added.

With this protocol, it would be possible to establish a group norm (for accuracy and/or exploration time) for specific groups on each instant and long term effect, including each repeated within-group measure, and/or task; then to test individuals against their group norm: Whether any particular individual or the entire group is in need of further training in noticing the shape-audio regularities. A group norm for each instant and long term effect, including each repeated within-group measure, and/or task would also make it possible to determine what kind of training is needed to notice the shape-audio regularities: For example, more trials with the tactile/visual-auditory bouba/kiki-effect (Experiment 1) and/or more trials with congruous and incongruous tactile/visual shape and audio (Experiment 2). Indeed, it would be possible to increase both the number of trials and the number of repeated within-group measures in each task as well as altering the time between them, thereby allowing more individualized training and testing.

However, this protocol would need modifying if testing what type of materials (e.g., bouba or kiki, filled or outlined tactile/visual shape, and/or auditory sound or word) best ensures the learning of shape-audio regularities. A possible way of modifying this protocol could involve presenting the different materials in blocks: For example, filled tactile/visual shapes and auditory sound and/or outlined tactile/visual shapes and auditory word (Experiment 1). These blocks could be presented in random order to all the participants in one experimental group; or alternatively, each one of the four blocks to separate experimental groups (Experiment 1). This protocol would also need modifying if testing even further the robustness of the visual imagery and/or learned shape-audio regularities within and across tasks. The modification could involve adding repeated within-group measures with different tactile/visual shape and/or audio than those included in the training; for example, with the auditory sound for ‘maa-boo-maa’ and ‘tuh-kee-tee’ after training with tactile/visual bouba and kiki shapes and auditory words (Experiment 1)¹^,³^,⁴^,¹⁰^,¹¹^,¹²^,¹³^,¹⁴^,¹⁵^,¹⁶. Another option could involve presenting a third tactile/visual shape which is not mainly curved/angular together with the tactile/visual bouba and kiki shapes in Experiment 1¹⁰^,¹¹^,¹²^,¹³^,¹⁴^,¹⁵^,¹⁶. An auditory nonsense word and a sound could be created for this tactile/visual shape to produce a third shape-word pair¹⁴^,¹⁵^,¹⁶ and a third shape-sound pair¹⁷ (Experiment 1). The shape-sound pair could then be included in Experiment 2 as catch-trials; or alternatively as regular congruous trials and with corresponding incongruous trials. Finally, the sound (from Experiment 2 and Experiment 1) and/or the nonsense word (from Experiment 1) could be presented in Experiment 3. A third option could involve not conducting Experiment 2 and Experiment 3 at all, but rather including them as tasks in the repeated within-group measures in Experiment 1, in random order.

The presented protocol is unique in that it keeps under control both the visual imagery and the sensory information during training and testing. It is flexible: Large and small samples and specific individuals can be tested against themselves (cf. the within-group testing) and/or other groups of individuals (cf. the between-group testing). It is possible to test both accuracy and exploration time, change the number of trials and repeated within-group measures in each experiment and the time between them, and change or even collapse the order of trials and/or experiments.

開示

The authors have nothing to disclose.

Acknowledgements

Funders:
This work was funded by the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie Actions (grant agreement number 657440) and the Norwegian Association of the Blind and Partially Sighted.

Film makeup:
Makeup artist Rachael Parrey.

Materials

Bouba/kiki filled			30 mm horizontal, 23 mm Vertical and 0.5 mm thickness m vertical, and 0.5 mm line height, and 0.5 mm embossed embossed height
Bouba/kiki outlined			30 mm horizontal, 23 mm Vertical and 0.5 mm thickness m vertical, and 0.5 mm line height, and 0.5 mm embossed embossed height
Bouba/kiki sound (sine wave)
Bouba/kiki word (human voice)
Carton plate/foam board for covering bouba/kiki (white)	Ryman	230461860	297 mm horizontal x 210 vertical 0 mm vertical
Copy paper [white (80 g/m2 )]	Ryman	250030000	210 mm horizontal x 148 mm vertical
Foam board for bouba/kiki picture cards [white (5 mm thick)]	Ryman	230461860	50 mm horizontal x 50 mm vertical
MP3-player (smartphone with the VLC app for iOS installed)	Carphone Warehouse
Plastic embossing film	RNIB Shop	ZM04	210 mm horizontal x 148 mm vertical
Rollerball pen [black (1.0 mm tip)]	Ryman	827134001
Rubber mat	RNIB Shop	LC177	230 mm horizontal x 150 mm vertical
Saddleback style	RNIB Shop	B511
Swell paper	Zychem Ltd	Zytex2 paper	50 mm horizontal x 50 mm vertical

参考文献

Ramachandran, V. S., Hubbard, E. M. Synaesthesia – A Window Into Perception, Thought and Language. Journal of Consciousness Studies. 8 (12), 3-34 (2001).
Bremner, A. J., Caparos, S., Davidoff, J., de Fockert, J., Linnell, K. J., Spence, C. Bouba’’and ‘‘Kiki’’ in Namibia? A remote culture make similar sound-shape matches, but different taste-shape matches to Westerners. Cognition. 126 (2), 165-172 (2013).
Davis, R. The fitness of names to drawings. A cross-cultural study in Tanganyika. British Journal of Psychology. 52 (3), 259-268 (1961).
Maurer, D., Pathman, T., Mondloch, C. J. The shape of boubas: sound-shape correspondences in toddlers and adults. Developmental Science. 9 (3), 316-322 (2006).
Ozturk, O., Krehm, M., Vouloumanos, A. Sound-symbolism in infancy: Evidence for sound-Shape cross-modal correspondences in 4-months-olds. Journal of Experimental Child Psychology. 114 (2), 173-186 (2013).
Bar, M., Neta, M. Humans Prefer Curved Visual Objects. Psychological Science. 17 (8), 645-648 (2006).
Bertamini, M., Palumbo, L., Gheorghes, T. N., Galatsidas, M. Do observers like curvature or do they dislike angularity?. British Journal of Psychology. 107 (1), 154-178 (2016).
Quinn, P. C., Brown, C. R., Streppa, M. L. Perceptual Organization of Complex Visual Configurations by Young Infants. Infant Behavior and Development. 20 (1), 35-46 (1997).
Silvia, P. J., Barona, C. M. Do people prefer curved objects? Angularity, expertise, and aesthetic preference. Empirical Studies of the Arts. 27 (1), 25-42 (2009).
Nielsen, A., Rendall, D. The Sound of Round: Evaluating the Sound-Symbolic Role of Consonants in the Classic Takete-Maluma Phenomenon. Canadian Journal of Experimental Psychology. 65 (2), 115-124 (2011).
D’Onofrio, A. Phonetic Detail and Dimensionality in Sound-shape Correspondences: Refining the Bouba-Kiki Paradigm. Language and Speech. 57 (3), 367-393 (2014).
Westbury, C., Hollis, G., Sidhu, D. M., Pexman, P. M. Weighing up the evidence for sound symbolism: Distributional properties predict cue strength. Journal of Memory and Language. 99, 122-150 (2018).
Oberman, L. M., Ramachandran, V. S. Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: The mirror neuron hypothesis. Social Neuroscience. 3 (3-4), 348-355 (2008).
Köhler, W. . Gestalt psychology. , (1929).
Monaghan, P., Mattock, K., Walker, P. The Role of Sound Symbolism in Language Learning. Journal of Experimental Psychology: Learning, Memory, and Cognition. 38 (5), 1152-1164 (2012).
Fort, M., Martin, A., Peperkamp, S. Consonants are More Important than Vowels in the Bouba-kiki Effect. Language and Speech. 58 (2), 247-266 (2015).
Graven, T., Desebrock, C. Bouba or kiki with and without vision: Shape-audio regularities and mental images. Acta Psychologica. 188, 200-212 (2018).
Fryer, L., Freeman, J., Pring, L. Touching words is not enough: How visual experience influences haptic-auditory associations in the “Bouba-Kiki” effect. Cognition. 132 (2), 164-173 (2014).
Fontana, F. Association of Haptic Trajectories to Takete and Maluma. Lecture Notes in Computer Science. 7989, 60-68 (2013).
Graven, T. How blind individuals discriminate braille characters: An identification and comparison of three discrimination strategies. British Journal of Visual Impairment. 33 (2), 80-95 (2015).
Graven, T., Bello, F., Kajimoto, H., Visell, Y. How attention is allocated when using haptic touch: Shape feature distinction and discrimination strategy. EuroHaptics 2016, Part 1. , 380-393 (2016).
Heller, M. A., Calcaterra, J. A., Burson, L. L., Tyler, L. A. Tactual picture identification by blind and sighted people: Effects of providing categorical information. Perception & Psychophysics. 58 (2), 310-323 (1996).
Heller, M. A., Gentaz, E. . Psychology of Touch and Blindness. , (2014).
Pathak, K., Pring, L. Tactual picture recognition in congenitally blind and sighted children. Applied Cognitive Psychology. 3 (4), 337-350 (1989).

Play Video

PDF

DOI

DOWNLOAD MATERIALS LIST

記事を引用

Graven, T., Desebrock, C. Investigating the Effect of Visual Imagery and Learning Shape-Audio Regularities on Bouba and Kiki. J. Vis. Exp. (151), e59954, doi:10.3791/59954 (2019).

Investigating the Effect of Visual Imagery and Learning Shape-Audio Regularities on Bouba and Kiki