Auditory and visual integration in onomatopoeia

ASSANEO M. F.; N ELISEI; M TRÍPODI; J SITT; L COHEN ; M TREVISAN

Huerta Grande, Córdoba

Congreso; Annual Meeting of the Argentine Society of Neuroscience; 2013

Socidad Argentina de Neurociencia

Onomatopoeias transform events as collisions, bursts and strikes into words. We asked naive participants to produce onomatopoeic sounds from audio (A), visual (V) and audiovisual (AV) stimuli of different moving objects. As expected, onomatopoeias strongly correlate with the sounds of A and AV stimuli. More interestingly, our analysis also revealed consistent associations between the shape of the objects in the V stimuli and the place of articulation of the pronounced phonemes and between the type of movement and the sound source of the phonemes. We also used machine learning techniques to calculate the performance of the onomatopoeias at predicting the movement type of the stimulus. Our results are compatible with fmri responses to V, A and AV stimuli at the multisensory cortex. Our results support the idea that onomatopoeias result from the tension between acoustic imitation and cross-modalassociations, which contributes to build a global view of word formation from mimetic forces.