The modality-switch effect: New insights from language processing Scerrati E., Baroni G., Lugli L., Galatolo R., Nicoletti, R.
University of Bologna
As Meteyard and Vigliocco (2008) pointed out, the relation between perceptual and conceptual information is complex. The behavioural literature shows a cost for performance in terms of speed and accuracy when two different sensory modalities alternate, compared to when the same modality is presented. This effect, known as the Modality-Shifting effect or Modality-Switch effect (MSE) was initially found in a pure perceptual study (Spence, Nicholls & Driver, 2001) and then interestingly replicated both using a conceptual paradigm (Pecher, Zeelenberg & Barsalou, 2003) and across perceptual and conceptual tasks (van Dantzig, Pecher, Zeelenberg & Barsalou, 2008).
The present study aims at investigating the MSE by priming conceptual processing with a linguistic stimulus that described a light or a sound’s perceptual property (e.g., “the light is flickering”; “the sound is echoing”). In other words, participants were first primed with a unimodal sentence (see Lynott & Connell, 2009 for this aspect) that was related to the visual or auditory modality and then they were presented with a target sentence (e.g., “the butter is yellowish”, “the brushwood crackle”) upon which a property verification judgment was to be made. Crucially, the prime sentence could share (i.e., visual - visual, auditory - auditory; compatible condition) or not (i.e., visual - auditory; auditory - visual, incompatible condition) the target modality.
As expected, participants showed a better performance for the compatible compared to the incompatible condition, indicating a facilitation for the processing of those target sentences whose sensory modality was formerly primed through a unimodal perceptual sentence. These results showed that the modality of concepts can be primed through language, supporting the embodied and grounded cognition view, which claims that sensory information and conceptual representation are closely related.
References
Lynott, D., Connell, L. (2009). Modality exclusivity norms for 423 object properties. Behavior Research Methods, 41 (2), 558-564.
Meteyard, L. Vigliocco, G. (2008). The role of Sensory and Motor Information in Semantic Representation: a review. Ch. 15 in P. Calvo & A. Gomila (Eds) Handbook of Cognitive Science: An Embodied Approach. Academic Press, Elsevier, London.
Pecher, D., Zeelenberg, R., & Barsalou, L.W. (2003). Verifying the properties of object concepts across different modalities produces switching costs. Psychological Science, 14, 119-124.
Spence, C., Nicholls, M.E.R., & Driver, J. (2001). The cost of expecting events in the wrong sensory modality. Perception & Psychophysics, 63, 330-336.
van Dantzig, S., Pecher, D., Zeelenberg, R., Barsalou, L.W. (2008). Perceptual processing affects conceptual processing. Cognitive Science , 32, 579-590.
