Naturally Biased Associations
Between Music and Poetry
Liliana Albertazzi
Department of Humanities, University of Trento, Italy
Luisa Canal and Rocco Micciolo
Department of Psychology and Cognitive Sciences, University of Trento, Italy
Fulvio Ferrari and Sebastiano Sitta
Department of Humanities, University of Trento, Italy
Iacopo Hachen
Center for Mind/Brain Sciences, University of Trento, Italy
Abstract
The study analyzes the existence of naturally biased associations in the general population between a series of musical selections and a series of quatrains. Differently from other studies in the field, the association is tested between complex stimuli involving literary texts, which increases the load of the semantic factors. The stimuli were eight quatrains taken from the same poem and eight musical clips taken from a classical musical version of the poem. The experiment was conducted in two phases. First, the participants were asked to rate 10 couples of opposite adjectives on a continuous bipolar scale when reading a quatrain or when listening to a musical clip; then they were asked to associate a given clip directly with the quatrains in decreasing order. The results showed the existence of significant associations between the semantics of the quatrains and the musical selections. They also confirmed the correspondences experienced by the composer when writing the musical version of the poem. Connotative dimensions such as rough or smooth, distressing or serene, turbid or clear, and gloomy or bright, characterizing both the semantic and the auditory stimuli, may have played a role in the associations. The results also shed light on the accomplishment of the two diverse methodologies adopted in the two different phases of the test. Finally, the role of specific musical components and their combinations is likely to have played an important role in the associations, an aspect that shall be addressed in further studies.
Keywords
naturally biased associations, auditory perception, semantics, subjective judgments, music, and emotion
Corresponding author:
Liliana Albertazzi, Universita` degli Studi di Trento, Centro Interdipartimentale Mente/Cervello, Corso Bettini 31, Rovereto 38068, Italy.
Email: liliana.albertazzi@unitn.it
Perception 2016, 0(0) 1–22 !The Author(s) 2016 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/0301006616673851 pec.sagepub.com
Introduction
Starting from the pioneering studies on synesthesia (Cytowic, 1995; Marks, 1974; Melara & O’Brien, 1987; Wicker, 1968), in recent years, research on cross-modality has burgeoned and extended the analysis to the general population, not necessarily synesthetes. In fact, a large body of research has already shown that the general population exhibits consistent naturally biased associations between stimuli in different sensory modalities (auditory, visual, gustatory, olfactory, and tactile; Deroy & Spence, 2013). Naturally biased associations are correspondences that are not to be considered as either necessarily innate or learned (see Spector & Maurer, 2008, 2010). The phenomenon has been shown to be consistent between several possible pairings of sensory modalities, although the attention has concentrated on the associations between visual and acoustic modalities, and more specifically those between auditory pitch, loudness, musical modes, and visual size and color (for a review, see Spence, 2011). Most of the initial research has been conducted on rather simple stimuli (color or grapheme, color or sound, note or odor, taste or flavor or sound, etc.), and only recently has it been extended to stimuli of higher order complexity (Albertazzi, Canal, & Micciolo, 2015; Albertazzi, Bacci, Canal & Micciolo, 2016; Ku¨ssner, 2013a, 2013b).
As regards cross-modal associations in the adult general population involving musical stimuli, studies have been conducted in relation to visual shape (Ku¨ssner, 2013a, 2013b; Tan & Kelly, 2004) imagery (Jakubowski, Farrugia, Halpern, Sankarpandi and Stewart, 2015; Walker, 1981), emotions (Collier & Hub, 2001; Crowder, 1985; Murari et al., 2015; Palmer & Schloss, 2010; Palmer, Schloss, Xu, & Prado-Leo´n, 2013; Ward, Huckstep, & Tsakanikos, 2006), taste (Mesz, Trevisano, & Sigman, 2011), taste and flavors (Crisinel & Spence, 2010), odors (Belkin, Martin, Kemp, & Gilbert, 1997), and paintings (Albertazzi et al., 2015). In general, these studies shed light on the role played by pitch and musical modes (minor and major) in the emotional connotations attributed by subjects to the musical selections. On the other hand, the analysis of cross-modal associations between acoustic and visual stimuli (such as color and shape) has highlighted some aspects of musical knowledge, such as discriminating the different dimensions concerning the pitch, the rhythm, and the timbre of a sound (Ku¨ssner, 2013a; Ward et al., 2006). The difference between experts and nonexperts in music has also been considered (Albertazzi et al., 2015; Ku¨ssner, 2013a), and likewise the role of performance (Vines, Krumhansl, Wanderley, & Levitin, 2006), but with different results in the different tasks and studies. The methods used in experiments vary from Osgood semantic differential (see Bigand, Vieillard, Madurell, Marozeau and Dacquet, 2005; Osgood, 1956), through psychophysical methods (such as reaction times, implicit association test, etc.), to methods of nonverbal sensory scales (Murari et al., 2015), magnetic resonance imaging scanning, and methods of direct associations between stimuli based on subjective judgments in first person account (Albertazzi et al., 2015).
The growing interest of scientists in phenomena of this kind and the rapid development of experimental research in the field have also promoted the debate on the nature of the associations (whether they occur bottom-up or top-down, whether they are to be related to synesthesia or not), on the different methods and tasks adopted in studies (e.g., the choice of speeded or unspeeded classification tasks: see Marks, 2004), and on the appropriateness of the terminology used to classify the phenomena such as ‘‘synaesthetic correspondences’’ (Martino & Marks, 2000), ‘‘similarities’’ (Marks, 1987a, 1989b), and ‘‘mapping’’ (Evans & Treisman, 2010), and so forth. Analysis of the factors responsible for the cross-modal associations has concerned in particular the spatial and temporal factors modulating multisensory integration (Shore, Bernes, & Spence, 2006; Spence & Driver, 2004), semantic
congruency (Chen & Spence, 2010), and synesthetic congruency (Gallace & Spence, 2006; Makovac & Gerbino, 2010).
As regards the nature of cross-modal phenomena—that is, whether these phenomena should be considered related to a synesthesia (Maurer & Mondloch, 2005) or as specific phenomena (Simner, 2012; Ward, 2013)—and if their presence may be semantically influenced or even induced (Mroczko-Wasowicz & Nikolic´, 2014), in a recent review of the state of affairs, Spence (2011) proposed a qualitative division among different types of modalities. He distinguished between statistical phenomena (due to an adaptive response of the brain to the regularities of the environment), structural phenomena (due to the characteristics of the neural systems used to code sensory information), and semantically mediated phenomena (mainly explained in terms of the linguistic influence in stimuli description, e.g., the use of features such as ‘‘low’’ and ‘‘high’’ in referring to pitch variations). These tripartite explanation should not be necessarily considered either exhaustive or exclusive.
In particular, the hypothesis that synesthetic and cross-modal associations might be explained more in semantic than sensory terms has reevaluated the possible role of cognitive factors in the associations (Gallace & Spence, 2006; Martino & Marks, 1999). The semantic interpretation has seemed more appropriate to explain the associations between inducers such as the days of the week or the months, whose conceptual nature might influence their association with color, or cases of color sequence (Simner et al., 2006; Tomson, Narayan, Allen & Eagleman, 2013) and spatial sequence synesthesia (Eagleman, 2012; Sagiv, Simner, Collins, Butterworth & Ward, 2006), involving the names of time units and ordinal categories. This interpretation, lately named ‘‘ideaesthesia’’ (Meier, 2013; Mroczko-Wasowicz & Nikolic´, 2014), has raised the question of whether a full account of synesthesia should not go beyond the standard sensory-sensory approach (Dixon, Smilek, Duffy, Zanna, & Merikle, 2006; Eagleman, 2012; Ju¨rgens & Nikolic, 2012; Mroczko-Wasowicz & Werning, 2012; Simner, 2012; Ward, 2013). In his review of the interpretations of cross-modal correspondences, Spence (2011) has stressed the complexity of the endeavor (how to categorize, e.g., all of the cross-modal correspondences that have been reported to date) and of explaining the link between language and perception, which may not be arbitrary (as shown by sound symbolism research). Indeed, the point is crucial because, for example, arguing in favor of the semantic interpretation of the cross-modal associations would require identification of the qualitative factors that share the same dimensions at both perceptual and linguistic level. For the time being, phenomena of naturally biased associations between musical selections and complex literary texts have not been tested. Our study aims at identifying the qualitative factors that share the same dimensions at perceptual, linguistic, and semantic levels. Incidentally, the study is also relevant to the field of music and emotion, even though not as a principal focus, since its purpose is to analyze a broader variety of dimensions, especially perceptual ones.
The Study
We analyzed the existence of naturally biased associations in the general population between a series of classical musical selections and a series of classical poetic quatrains. Differently from other studies addressing the relationships between visual and acoustic modalities, the association was tested by involving the semantic tenor of poetry—that is, the synthetic meaning related to the entire quatrains or the whole poem, not the meaning of the single word or sentence—which increased the load of the semantic factors. It was decided to use semantic stimuli such as poetic texts in order more closely and clearly to analyze the role
of cognitive factors in the association. The prediction was that the subjects would make systematic associations between the quatrains of the poem and the series of selections of classical piano music; and that the associations would be due to the presence of similar dimensions in both, sharing semantic and perceptual traits, as also evidenced by the Osgood semantic differential. Because of the complexity of the stimuli, we tried to maintain the maximum amount of uniformity possible, choosing as stimuli the quatrains of a poem by Goethe (Erlko¨nig) and the strophes of its musical transcription by Schubert (Op. 1—D.328, for voice and piano) in its posterior piano solo rendition by Liszt (S. 558). Both stimuli showed a high level of complexity due to the contents of each of the quatrains and of the musical choices of the composer when writing the musical strophes. For example, in the musical piece, the same material often appears in different variations and contexts within the piece itself, and several different harmonic modulations occur during the piece, characterizing the strophes with different major and minor keys. The nature of each of the eight quatrains is also complex, often implying the copresence of positive and negative aspects of the narration, and more than one actorship and viewpoint. That is to say, the contents of each quatrain were not homogeneous, and sometimes expressed the viewpoints of the different characters involved in the narration (i.e., the narrator, the father, the child, and the Erlko¨nig because of the different behavior of the Erlko¨nig himself in the different strophes).
As mentioned, the purpose of using stimuli pertaining to a single poem and its musical transcription was to maintain as much uniformity as possible and to have a supplementary meter of comparison: In fact, if the associations made by the participants confirmed the musical choices made by the composer, this would have further supported the existence of natural associations between the perceptual and semantic stimuli. Intentionally, in our research, we did not directly test either individual preferences or the emotional dimension (Chatterjee, 2003; Cowles, 1935; Di Dio, & Gallese, 2009; Juslin & Sloboda, 2001; Krumhansl & Lerdahl, 2011; Ku¨ssner, 2013a; Langlois, Peterson, & Palmer, 2013; Palmer et al., 2013; Zaidel, 2010) because it was not our primary interest and because our specific goal was to test which perceptual and semantic dimensions were responsible for the associations, trying to highlight, when possible, the role played by cognitive factors.
Methods
Participants
A total of 67 participants (not synesthetes) volunteered for the two experiments: 31 women and 36 men (mean age: 26.8 years; SD: 9.9; median: 24 years). All participants were recruited by email from students in the Department of Psychology and Cognitive Science, University of Trento, Italy. The address list of the students was provided by the student office. We first sent an email asking the students to take part in the experiment, mentioning that we were looking for people with a generic background in music and nonexperts as well. We did not ask for persons with professional experience, although 10 subjects had received musical training at a conservatory or from private lessons. The questionnaire reported this information. The subjects were also asked about possible conscious synesthesia (Albertazzi et al., 2015; Palmer & Schloss, 2010; Palmer et al., 2013). The only exclusion criterion was self-reported acoustic impairment. After the experiment, the subjects were asked whether they had previously known the poem and the musical clips that they evaluated. For all the subjects, the stimuli were totally new. The subjects were also asked to give their comments on the associations that they produced, which were recorded by the investigator.
The first experiment was performed using the semantic differential on a bipolar rating scale of adjectives (Osgood, 1956). The adjectives were chosen among those dimensions characterizing both the semantics of the narration and the music transcription. The second experiment evaluated the association between semantic and auditory stimuli. All the subjects signed an informed consent form. The experiments reported here complied with the ethical guidelines of the University of Trento.
Procedure
The experiment was performed in the computer room of the Cognitive Psychology Department of Trento University. Visual stimuli appeared on LCD Dell P2214H 21.500
monitors. Participants were seated at a desk. The distance from the center of the screen to the eye was about 65 cm. As to the software, the Platform was an application written in Flask, a micro web application framework written in Python and based on the Werkzeug toolkit and Jinja2 template engine. It is BSD licensed. The Platform for the test was installed on Ubuntu Server 14.04 on a virtual machine given by the University of Trento. Each student entered the test with his or her PIN, which identified him or her in the subsequent sessions.
General Materials
The stimuli consisted of eight quatrains (text) from Goethe’s poem Erlko¨nig translated into Italian (by Roberto Fertonani, see Supplementary Materials); eight musical clips (audio) taken from a piano performance (by Cyprien Katsaris in the album ‘‘Viennese Connections’’ [PIANO21 033], see Supplementary Materials) of Schubert-Liszt’s musical version, where each clip corresponded to a single quatrain of Goethe’s poem; and a list of 20 adjectives, presented (in Italian) in couples of contraries: passive or active, agitated or calm, unpleasant or pleasant, grave or acute (see section The first task), distressing or serene, gloomy or bright, dissonant or consonant, repulsive or seductive, rough or smooth, and turbid or clear.
The Italian translation of the ballad does not make use of a consistent metrical pattern; the translator has chosen to use free verse instead. The main problem in translating a German ballad into Italian is that the Italian language has very few words which end with a stressed syllable; the whole metrical construction of the Erlko¨nig is thus impossible to recreate in Italian verse. A ‘‘singable’’ translation would of course have been possible by using a traditional Italian metre, but in this case, part of the semantic content would have been lost. The piano performance of the piece was selected because of its similarity with the original transcription of the poem by Schubert, for voice and piano. We chose a solo piano version to avoid interferences with the German language, which we could not use in our test. As to Liszt’s musical transcription of the poem, it is well known for being faithful to the original version, while Katsaris’s performance is very faithful to the score.
Task 1 (Osgood semantic differential). Participants were asked to rate 10 couples of opposite adjectives on a continuous bipolar scale (ranging from 0 to 1024) when reading a given quatrain (32 subjects) or when listening to a musical clip (35 subjects). The participants were told that they were going to be shown a random set of quatrains, one at a time, appearing on the left half of the screen. The task was to evaluate the overall semantic content (i.e., the ‘‘meaning’’) of each quatrain (or of each music clip) according to the 10 pairs of adjectives randomly shown on the right half of the screen. The evaluation was given along a continuous scale; at one end of the line, there was one of the two adjectives
of the couple and at the other end, the remaining adjective. Using a mouse, the subject had to place the pointer indicating his or her degree of agreement with the two adjectives with regard to the semantic content of the quatrain or of the music clip. The software stored the choice with two scores, one for each member of the couple. Participants were allowed to change their choices at any time until they proceeded to the next form. Only after an answer had been given to every line, totaling 10 choices, did a button with the inscription ‘‘Proceed’’ appear at the right top of the screen. When the button was pressed, the answers were registered and a new quatrain or musical clip was presented.
Task 2 (direct association expressed by subjective judgments). Thirty-four subjects were asked to associate a given quatrain directly with four different music clips, among which only one was ‘‘correct’’ (i.e., corresponding to its musical version as written by Schubert), and its position in the association (first, second, third, and fourth) was verified.
Thirty-one subjects were asked to associate a given music clip directly with four quatrains, among which only one was ‘‘correct,’’ also verifying its position in the association (first, second, third, or fourth).
The choice of limiting the association of the quatrain to only four of the eight music clips was due to the amount of cognitive load and the length of time that the experiment required. To balance the design, the association of each music clip was limited to four of the eight quatrains.
Statistical Methods. Descriptive statistics (mean and standard deviation) were calculated. The exact test for a binomial proportion was employed to evaluate if the choice of the quatrain or clip was random.
Results
Task 1
There were 32 subjects who rated the adjectives when reading the quatrains, and 35 subjects who rated the adjectives when listening to the musical clips. The mean value of the ratings was calculated for each adjective, for each quatrain and each music clip, and it is reported in Figures 1 and 2.
In these figures, the results relative to each strophe (i.e., a music clip and the correspondent quatrain) are shown. For each adjective pair, two horizontal lines are displayed; the first of these two lines (i.e., the line on the top of the adjective) refers to the music clip, while the second one (i.e., the line on the bottom of the adjective) refers to the corresponding quatrain. On each line, a black dot indicates the mean rating; the proximity of the dot to one of the two adjectives indicates the intensity of the rating.
For example, when considering the Quatrain 8 and the pairs of adjectives ‘‘gloomy or bright’’ (see Figure 2, Strophe 8), the dot is located close to the ‘‘gloomy’’ extreme, since the mean rating for this adjective was 867. As another example, when considering Clip 3 and the pairs of adjectives ‘‘unpleasant or pleasant’’ (see Figure 1, Strophe 3), the dot is located close to the ‘‘pleasant’’ extreme, since the mean rating for this adjective was 831. As a final example, when considering Clip 1 and the pairs of adjectives ‘‘dissonant or consonant’’ (see Figure 1, Strophe 1), the dot is located in the middle of the line, since the mean rating for the adjective dissonant was 513 and the mean rating for the adjective consonant was 511. The plot of each strophe reports two vertical lines. They delimit the acceptance region relative to a null hypothesis that the true mean rating is 512 (i.e., the middle point of the rating scale) at a significance level of .05.
At the end of this procedure, it was possible to classify each mean rating as falling in the central zone between the two vertical lines (the adjective pair could be considered ‘‘neutral’’) or not (one of the adjective characterizes the quatrain or the music clip). The most striking feature is displayed by Strophe 3: For all the ten pairs of adjectives, the ratings fall outside the ‘‘neutral’’ zone.
It is interesting to note that, when considering, for a given pair of adjectives, the rating given to Quatrain 3 and to Clip 3, both the mean ratings are similar. Furthermore, all these ratings are located on the semantically positive pole of the scale. A similar result was found for Strophe 8, where almost all the ratings fell outside the ‘‘neutral’’ zone. Moreover, when considering, for a given pair of adjectives, the rating given to Quatrain 8 and to Clip 8, both the mean ratings were generally similar and located on the semantically negative pole of the scale.
Figure 1. Results for the Strophes 1 to 4. The line on the top of the adjective refers to the music clip, while the line on the bottom of the adjective refers to the corresponding poetic quatrain. Black dots indicate the mean ratings; the proximity of a dot to an adjective indicates the intensity of the rating. Dashed lines delimit nonsignificant deviations from the middle point.
Strophe 5 shows seven out of ten pairs of adjectives for which the mean ratings for both the poetic quatrain and the music clip fell outside the ‘‘neutral’’ zone and were located on the semantically positive pole of the scale.
When considering the mean ratings given when listening to the eight music clips, the pair passive or active was always outside the neutral zone, while the pairs unpleasant or pleasant and agitated or calm were outside the neutral zone for seven out of eight music clips. On the other hand, the pair distressing or serene was outside the neutral zone for six out of eight music clips, while three pairs (repulsive or seductive, gloomy or bright, grave or acute) were outside the neutral zone for five out of eight music clips.
When considering the mean ratings given when reading the eight quatrains, the results were less extreme. The pair gloomy or bright was outside the neutral zone for seven out
Figure 2. Results for the Strophes 5 to 8. The line on the top of the adjective refers to the music clip, while the line on the bottom of the adjective refers to the corresponding poetic quatrain. Black dots indicate the mean ratings; the proximity of a dot to an adjective indicates the intensity of the rating. Dashed lines delimit nonsignificant deviations from middle point.
of eight quatrains. There were three pairs (distressing or serene, unpleasant or pleasant, repulsive or seductive) outside the neutral zone for five out of eight quatrains.
Task 2
Thirty-four subjects were asked to associate a given quatrain directly with four different music clips; 31 subjects were asked to associate a given music clip directly with four quatrains. Among the four choices, only one was correct, and its position in the association (first, second, third, or fourth) was verified.
Figure 3 shows the percentage of subjects who associated the given music clip with the correct quatrain as first choice. In seven out of eight music clips, this percentage was numerically greater than 25%, that is, the percentage expected in the case of a random choice. If one considers this overall result, a binomial test rules out the possibility of a overall random choice (p ¼ .0008). Furthermore, for six of the eight music clips, the number of participants associating the given music clip with the correct quatrain as their first choice, exceeded the chance level in a statistically significant manner (p < .05); in the remaining two cases, a chance association cannot be ruled out (see Figure 3).
Figure 4 shows the percentage of subjects who associated the given quatrain with the correct music clip as first choice. Also in this case for seven out of eight quatrains, this percentage was numerically greater than 25%. If one considers this overall result, a binomial test rules out the possibility of a overall random choice (p ¼ .0008). Furthermore, for four of the eight quatrains, the number of participants associating the given quatrain with the correct music clip as their first choice, exceeded the chance level in a statistically significant manner (p < .05); in the case of the remaining four quatrains, a chance association cannot be ruled out (see Figure 4).
Figure 3. Percentage of subjects who associated the given music clip with the correct quatrain as first choice. The dashed line indicates chance level.
Therefore in four strophes (3, 5, 7, and 8), music and poetry seem to have been correlated in both directions, while in two strophes (4 and 6), they appear to have been unrelated. This latter finding may partially reflect the structural similarity between Strophes 4 and 6.
General Discussion
We tested the existence of naturally biased associations between complex stimuli, introducing the novelty of poetic texts. Natural associations in the phenomenological sense are qualitatively perceived by anyone because they pertain to the organization of the perceptual field in the awareness. The purpose of introducing semantic stimuli was to verify the presence, the behavior, and the possible role of cognitive factors in the association. The prediction was that the subjects would make systematic associations between the series of quatrains of the poem and a series of selections of classical music; and that the associations would be due to the presence of similar dimensions in both, as also evidenced by the semantic differential. As mentioned, the aim of the experiment was challenging for several reasons because the contents of each quatrain were not homogeneous and expressed the viewpoints of the different characters involved in the narration, because of the different behavior of the Erlko¨nig himself in the different strophes and because of the highly sophisticated musical transcription of the poem made by the composer(s).
The First Task
The first task of the experiment consisted in evaluating the overall semantic content of the quatrains and of the musical selections according to 10 couples of opposite adjectives (passive
Figure 4. Percentage of subjects who associated the given quatrain with the correct music clip as first choice. The dashed line indicates chance level.
or active, agitated or calm, unpleasant or pleasant, grave or acute, distressing or serene, gloomy or bright, dissonant or consonant, repulsive or seductive, rough or smooth, turbid or clear). Note that in Italian, the translation of the term ‘‘grave,’’ that is, grave, has the additional meaning of low pitch.
The rating of the couples of contraries. As regards the ratings given to the specific couples of contraries, we found the following. In the evaluation of the music, the couple unpleasant or pleasant was never displaced toward the ‘‘unpleasant’’ extreme; vice versa it was always displaced toward the ‘‘pleasant’’ extreme in all but one case (Strophe 8). In the evaluation of the poem, the couple was displaced toward the ‘‘unpleasant’’ extreme only in two strophes (Strophes 7 and 8). In a similar way, the couple seductive or repulsive was never displaced toward the ‘‘repulsive’’ side in the evaluation of the music clips; vice versa it was displaced toward the ‘‘seductive’’ extreme in correspondence to most music clips (Strophes 1, 3, 4, 5, and 7). The couple was also never displaced toward the ‘‘repulsive’’ side in the quatrains, with the exception of Strophe 8. To account for these results, it is possible that, when rating the couples pleasant or unpleasant and seductive or repulsive, at least some subjects evaluated their personal enjoyment of the stimuli per se rather than the meaning evoked by the music clips and the quatrains, and in so doing added an amount of bias to the results.
It seems also that this effect was stronger for the music than for the text, probably because in the poem the negative meanings are more explicit. As to the couple grave or acute (the Italian equivalent for the couple being grave or acuto, which can also be used to refer to ‘‘high or low’’ musical pitches), it might be that some subjects attempted to evaluate pitch aspects in some of the music clips, following the musical range of the main lines with their judgments; however, as the subjects were not musical experts, this finding has to be considered ‘‘natural.’’ The same holds for the couple dissonant or consonant, where it is possible that some subjects tried to evaluate the music clips with respect to the musical connotation (culturally shaped) of the adjectives (for a detailed description of these aspects, see Terhardt, 1984 and Tramo, Cariani, Delgutte, & Braida, 2001). This interpretation is supported also by the comments given by some subjects with some musical background.
Regarding the couples active or passive and agitated or calm, the ratings of the music clips, in all strophes but 3 and 5, were displaced toward the ‘‘agitated’’ and ‘‘active’’ extremes, while in the evaluation of the quatrains they seemed better to reflect the heterogeneity of the dialogs. This behavior may principally reflect the overall fast tempo, but it also correlates strongly with the choices of specific musical features by the composer(s), most notably a predominance of the minor mode, the accompaniment’s pattern (which for most of the piece consists in ostinato chords played in triplets, giving the impression of a horse ride), and strong dynamics, which undergo the most significant alterations in Strophes 3 and 5.
The single items: music clips and quatrains. As regards the behavior, analytically considered, of the single items (music clips and quatrains), the results show the following.
Strophe 1 (The narrator describes the scene, that is, the father riding with his son in his arms). As to Strophe 1, the music clip was evaluated as in a more extreme position than the corresponding quatrain; specifically, it was evaluated as rough, distressing, agitated, grave, gloomy. This result may be explained by the fact that the story settings are clearly characterized in the musical version by the choice of specific elements, such as the opening in a minor key, with triplets at a fast tempo (mimicking a horse’s gallop), and strong dynamics. These elements seem to accentuate a few aspects, like the overall negative tenor, the accelerated riding, and perhaps the wind. In particular, there are measures dedicated
by Schubert to the piano introduction (Clip 1, from the beginning to Second 19, see Supplementary Materials), in which, most notably, the main theme is presented in minor mode, apparently to anticipate the tragic outcome of the whole story or at least its surroundings.
On the other hand, as mentioned, it would be impossible for a reader without prior knowledge of the poem to understand the main setting of the story from the very first strophe because the strophe itself shows a high degree of ambiguity: For example, it is not said why and where the child is being carried during the night by his father and what their emotional feelings are; there are no explicit devices of anxiety or stress, and the reader may even have a positive impression when accessing the contents of the first strophe. This may have led to a greater variability in the subjects’ associations, and consequently in less polarized overall ratings. The ratings of the turbid or clear couple were positioned in the neutral area in both the music clip and the quatrain, although for the clip and the quatrain, respectively, the trend points numerically more toward ‘‘turbid’’ and ‘‘clear.’’ The same elements used to account for the divergences explained earlier can be considered as a reason for this result.
The quatrain was rated as consonant, and the music clip as equally consonant or dissonant. It is possible that some subjects tried to evaluate the music clips with respect to the musical connotation of these adjectives (see earlier section The rating of the couples of contraries).
Both the clip and the quatrain of Strophe 1 were rated as pleasant, most probably reflecting the subjects’ enjoyment of the stimuli (see earlier section The rating of the couples of contraries).
The active or passive couple showed quite similar ratings in both the clip and the quatrain, with both ratings directed toward ‘‘active.’’ This may be explained by the presence in the music clip of elements such as the fast tempo, strong dynamics, and endless triplets, and by the description of the quatrain of actions such as the father riding and taking care of his child.
Strophe 2 (The father asks his son why he is afraid; the child describes the appearance of the Erlko¨nig; the father reassures the child). Strophe 2 shows 4 couples of adjectives out of 10 with concordant ratings, although no couple shifted to opposite extremes.
Both clip and quatrain were rated as equally rough and smooth, seductive and repulsive, dissonant and consonant. In the music clip, this is well compatible with the presence of contrasting elements, such as minor key in correspondence to the father’s first line (Clip 2, from the beginning to Second 06, see Supplementary Materials), played in the low register, and conversely the high register of the child’s line (from Second 07 to 19), followed by a modulation to the relative major key as the father replies again (though again in low register; from Second 21 to the end of the clip). Moreover, the harmony encompasses both strong cadential episodes (in both the father’s lines) and more static harmonic passages (e.g., pedals) in correspondence to the child’s line. In the quatrain, this is again compatible with the presence of contrasting elements in the strophe: In fact, in the narration, the father, who initially appears to be very worried, toward the end of the strophe seems to try to minimize his concerns, making the child’s fears appear unfounded. The fourth concordant couple was graveor acute, with a shift toward ‘‘grave’’ for both the quatrain and the music clip. For the remaining six couples of adjectives, there was no concordance between music clip and quatrain. The couple distressing or serene showed a shift toward ‘‘distressing’’ in both music clip and quatrain.
Overall, the music clip was rated as less distressing, turbid, and gloomy with respect to the quatrain, but more active and agitated. The music clip was also rated as more pleasant than the quatrain (see earlier section The rating of the couples of contraries).
Strophe 3 (The Erlko¨nig invites the child to follow him to a beautiful place). In Strophe 3, all the couples of adjectives were in agreement between music clip and quatrain. Moreover, they were all shifted toward one of the two extremes of the couple (i.e., there were no ‘‘in between’’ ratings). The strophe was uniformly rated as smooth, serene, seductive, pleasant, consonant, clear, calm, bright, acute, and active. Interestingly, however, all the ratings of the music clip were closer to the extremes of the couples than the ratings of the quatrain. The concordance for the couples passive or active and agitated or calm might be explained by the choice of very clear (perhaps almost stereotypical) musical elements by Schubert, such as the sole use of major key, very conventional and predictable harmonic changes, softer dynamics, and embellishments. Also stylistic elements introduced by Liszt in his transcription may have weighted in the associations: For example, arpeggios in both the principal melody and the accompaniment, higher flexibility in the tempo, and high register for the lead voice. As to the quatrain, it does not have a negative connotation and consists only in the list of the Erlko¨nig’s sweet promises to the child, being therefore very uniform.
Strophe 4 (The child asks the father if he too hears the Erlko¨nig’s voice; the father replies reassuring the child). Strophe 4 showed six agreements between clip and quatrain, five of which were in the neutral area in between the extremes of the couples. The strophe was evaluated as equally rough and smooth, distressing and serene, consonant and dissonant, turbid and clear, grave and acute. This result, as for Strophe 2, well correlates with the presence of contrasting elements in the strophe, both in the clip and in the quatrain. Among the elements in the clip were minor or major key alternance obtained by sudden key modulations, consonant or dissonant musical intervals, change in the tempo and in the articulation. In the quatrain, for example, Strophe 4 begins with the moaning of the child and ends with the reassuring words of the father, without giving any insight as what is really happening. This uncertainty in the dialog may have caused the quatrain to be rated as gloomy. On the other hand, the music clip was rated as neither gloomy nor bright, suggesting a similar relevance and contrasting effect of both the first (Clip 4, from the beginning to Second 11, see Supplementary Materials) and the second part (from about Second 11 to the end) of the music clip (the latter perhaps gaining more salience because of its concluding position). The musical elements introduced by Schubert, in fact, seem intended to bring the listener first into the father’s, and then into the child’s, perspective. Again, the music clip was rated as more agitated and active than the quatrain, possibly because of the fast tempo, endless ostinatos, and strong dynamics (see earlier section The rating of the couples of contraries).
The music clip was also rated as more pleasant than unpleasant, differently from the quatrain, which appeared in the neutral zone for this couple (see earlier section The rating of the couples of contraries).
Strophe 5 (The Erlko¨nig again asks the child to follow him, alluding to the attentions he would receive from the Erlko¨nig’s daughters). Strophe 5 showed six agreements between the ratings of the clip and the quatrain, all of them directed toward one extreme (positive side of the scale): smooth, serene, seductive, pleasant, consonant, and bright. However, the quatrain was rated as equally turbid or clear, and equally grave or acute (although both at threshold-level with their right extremes); while the music clip was rated as certainly clear and acute. Moreover, the ratings for serene and bright were closer to the neutral zone (equally serene-distressing
and equally bright-gloomy, respectively) in the quatrain than in the music clip. These results show that the music clip was more efficacious in rendering the Erlko¨nig’s seduction than the literary sentences in the quatrain. In other words, it seems that the musical version by Schubert was so powerful as to induce the listeners to be seduced in the same way as the child, to the point that they virtually adopted the same viewpoint. Also in Strophe 5, the quatrain was rated as equally passive and active, although close to threshold level of active, while the music clip was rated as active. Similarly, the quatrain was rated as calm, while the music clip was rated as equally agitated and calm. These results (the quatrain is evaluated as less active and more calm) may be related to the subject’s awareness in reading the quatrain, where the roles of the seducer and of the victim are clearly defined, while in the music only the gorgeous character of the Erlko¨nig is highlighted, mainly by means of melodically and rhythmically elaborate lines, clearly defined harmonic passages, and broken chords in the accompaniment.
Strophe 6 (The child asks the father if he too can hear the Erlko¨nig speaking, and the father replies ambiguously). In Strophe 6, there were five agreements in the ratings of the clip and the quatrain. Four of them fell in the neutral area of the couples. This is similar to the results for Strophe 4, which indeed shares major structural similarities in both the quatrain and the clip. The quatrain starts with the child complaining about the Erlko¨nig and ends with the father’s reply, as in Strophe 4; however, differently from Strophe 4, there is an ambiguity of behavior. It is not clear, in fact (a) whether the father still does not understand the danger of the situation, (b) the father understands but wants to reassure the child, and (c) the father understands and expresses his concern or worry. In the music clip, the first part (Clip 6, from the beginning to Second 11, see Supplementary Materials) is essentially identical to the first part of Clip 4 (same seconds). However, the second part (Clip 6, from Second 11 to the end), corresponding to the father’s reply, is different from the one of Clip 4 (same seconds, see Supplementary Materials), most notably because it ends with a cadence in minor key. These differences with Strophe 4 may have been reflected in the ratings: In fact, both the music clip and the quatrain of Strophe 6 were rated as equally rough and smooth, repulsive and seductive, dissonant and consonant, turbid and clear; but it was also rated as gloomy, differently from Strophe 4 (where only the quatrain was considered gloomy). Additionally, the music clip was rated as equally grave-acute as in Strophe 4, while the quatrain was rated as grave; in the music clip, this may again reflect the use of contrasting musical elements (most importantly, melody ranges), but in this quatrain (6), there is no trace of the child’s reassurance by the father.
The music clip was also rated as distressing, while the quatrain was rated as equally serene-distressing but almost at threshold level with serene-distressing. This overall shift toward the ‘‘distressing’’ extreme with respect to Strophe 4 reflects the differences highlighted earlier, with a stronger effect in the music clip and a more ambiguous one in the quatrain. A shift occurs also for the unpleasant or pleasant couple, in which the rating for the quatrain was still in the neutral area, but closer to the unpleasant than to the pleasant extreme. The rating of pleasantness given for the music clip, however, may be due to the same effect considered earlier (see section The rating of the couples of contraries), reflecting the subjects’ personal interpretation of the adjectives.
Similar to Strophe 4, Strophe 6 was rated as active in its music clip, while equally passive or active in the quatrain.
Strophe 7 (The Erlko¨nig makes explicit his desire to have the child for himself, even by using force, and the child expresses his despair because the Erlko¨nig has gripped him). In Strophe 7, there were only three agreements, it being agitated, equally grave or acute, and distressing
(with the rating of the music clip extremely close to threshold level). Three adjectives were shifted toward opposite extremes between the quatrain and the clip: The quatrain was evaluated as passive, unpleasant, and repulsive, and the music clip as active, pleasant, and seductive. The other disagreements between clip and quatrain concerned the couples rough or smooth (the clip was neutral while the quatrain was rough), dissonant or consonant (the clip was neutral while the quatrain was dissonant), turbid or clear (the clip was neutral while the quatrain turbid), and gloomy or bright (the clip was neutral while the quatrain was gloomy). All these disagreements very strongly reflect that, while what is occurring is clear in the sentences of the quatrain, in the music clip the situation is musically contrasting: In other words, what occurs is clearly and unitarily presented in the quatrain, while the music clip is clearly divided into apparently diversified sections.
In particular, the first seconds of the music clip (from Second 01 to 06, see Supplementary Materials), corresponding to the Erlko¨nig’s last declaration of his desire, are written in major key. A second section (from Second 06 to 11) is written in minor key, corresponding to the Erlko¨nig’s threat to use force, revealing his evil nature. A third section of the music clip (from Second 12 to the end), corresponding to the child’s last moanings, makes use for the third time of the musical content that appears at the beginning of Strophes 4 and 6, but now ending with a perfect minor cadence after a crescendo.
Strophe 8 (The narrator describes the father’s last efforts to rescue the child, which in the last sentence prove to be useless). In Strophe 8, there were seven agreements and three disagreements. All the agreements occurred at one extreme of the couples, and, together with Strophes 3 and 5 (but in an opposite way), the ratings of Strophe 8 showed the highest degree of displacement from the center. Vice versa, among the disagreements, none of them contained ratings on the opposite extremes of the scale in the same couple (pleasant being in the neutral zone). Strophe 8 was uniformly rated as rough, distressing, turbid, agitated, gloomy, grave, and active. In most of the cases (rough, distressing, turbid, agitated, and gloomy), the ratings of the poem seem to be more extreme than the ratings of the musical version. However, the musical version seems to have been judged as more grave and active than the poem.
Among the most relevant characteristics of the musical version of Strophe 8 are the notable shifts in the tempo (accelerando, rallentando, and recitativo), the unaccompanied melody in the recitativo (Clip 8, from Second 21 to 27, see Supplementary Materials), which entails an unexpected or abrupt melodic and harmonic motion (Clip 8, from Second 28 to 34), and the final cadenza in G minor (Second 35). Given the peculiarity of both the quatrain and the clip, the agreements in Strophe 8 are not surprising. Among the disagreements, it is interesting to note that not only was the quatrain considered unpleasant but also the clip (the couple being in the neutral area), and this was the only case for all the strophes. This result suggests that the aforementioned musical elements weighted so strongly and negatively on the evaluations as to produce this effect, even assuming the presence of a positive bias in respect to the music (see earlier section The rating of the couples of contraries). Similarly, also the couple seductive or repulsive was in the neutral area for the music clip (while it was displaced toward repulsive in the quatrain). Interestingly, this was also the only case in which the music clip was considered to be dissonant, most probably because of the unexpected melodic or harmonic motion before the final musical cadenza.
The Second Task
The second task of the experiment consisted in a direct association between quatrains and clips. In six out of eight cases, participants associated a given music clip with the
corresponding correct quatrain as their first choice. In four out of eight cases, participants associated a given quatrain with the corresponding correct music clip as their first choice.
Therefore in four strophes (3, 5, 7, and 8), clip and quatrain seemed to be correlated in both directions, hence consistently associated, while in two strophes (4 and 6), they appeared to be unrelated. This latter finding may partially reflect the structural similarity between quatrains 4 and 6. Intriguingly, the remaining two strophes (1 and 2) seem to have been correlated only when subjects listened to the music and had to choose a quatrain, but the opposite does not hold.
In Strophe 1, the association between the music clip and the corresponding quatrain was significant, while the contrary did not hold. This result may be related to the highly characterizing musical elements introduced by Schubert or Liszt in their transcription of the strophe (such as introduction in a minor key, with triplets at a fast tempo, and strong dynamics), that may suggest the overall negative tenor, the riding, and the wind. These elements seem to be sufficient for a straightforward association of the music with the text, on the basis of the given stimulus set (where there are no competing alternatives). Vice versa, the quatrain presents different possible interpretations, of which several may easily match some of these alternatives. In fact, as mentioned, it would be impossible for a reader without prior knowledge of the poem to understand the main setting of the story from the very first strophe because of the high degree of ambiguity of the strophe itself, which can lead to several different associations with the set of musical clips; in particular, the interpretation could be determined by the relative importance attributed by the subject to the first and second half of the strophe, respectively more negative or positive (see Supplementary Materials).
Also in Strophe 2, the association between the music clip and the corresponding quatrain was significant, but not the opposite. The effect may be similar to the one that happened with Strophe 1. Also Clip 2 was written mainly in minor key, and it entails specific musical elements that create a dramatic effect (e.g., a modulation from G minor to C minor in the first father’s line, and several diminished seventh chords during the entire clip). Most importantly, what happens is that the music clearly mirrors the evolution of events, mainly because it starts in a minor key and, passing through a series of modulations, ends in a major key—a fact that constrains the freedom of interpretation. Vice versa, the strophe offers more different possible interpretations, among which there are some overall negative, or positive ones (especially because one does not know the father’s tone of voice in his reply, which may be reproachful, preoccupied, playful, or reassuring, depending on his own mental state). The quatrain may therefore match also different alternatives among those given (see the stimuli themselves in the Supplementary Materials).
In Strophe 3, both clip and quatrain were reciprocally associated in both the directions. Musical elements were uniform and clearly defined (major key, easily predictable harmonic changes, lighter dynamics, see earlier section The single items: music clips and quatrains). The quatrain did not have a negative connotation and consisted only in the list of the Erlko¨nig’s sweet promises to the child. It was therefore very uniform.
In Strophe 4, there was no association between clip and quatrains in any of the directions, probably because 4 and 6 had similar structures and contents (see earlier section The single items: music clips and quatrains), and were therefore confused by the subjects.
In Strophe 5, both clip and quatrain were reciprocally associated in both the directions. Strophe 5 had elements similar to Strophe 3 in being different from the rest of the strophes (e.g., a general predominance of the major key and canonical harmonic movements, as well as a smoother articulation). However, it was not confused with Strophe 3. This result is interesting and may be due to the repetition of the peculiar melodic pattern in the music clip (Clip 5, from Second 9 to 15, see Supplementary Materials), corresponding to the
sentence in the poem ‘‘ti cullano, ballano, ti cantano la ninna nanna’’ (‘‘they’ll dance thee, and rock thee, and sing thee to sleep’’), despite the fact that the sentence was not ‘‘singable’’ along with the music in this Italian translation.
In Strophe 6, like in Strophe 4, there was no association between clip and quatrains in any of the directions. Here, the same considerations hold as those for Strophe 4.
In Strophe 7, both clip and quatrain were reciprocally associated in both the directions. This is probably due to the fact that Strophe 7 had a very peculiar evolution in both the clip and the quatrain (see earlier section The single items: music clips and quatrains).
In Strophe 8, both clip and quatrain were reciprocally associated in both the directions. It is possible that the most characterizing elements were the final recitativo followed by the cadence, and also the crescendo that corresponded to the final ride of the father. Again, it is worth noting that in most cases the subjects did not recognize the different strophes as parts of a single work (and thus did not necessarily regard Strophe 8 as the final one in the set), as shown in the comments (e.g., subjects often referred to ‘‘songs’’ and ‘‘poems’’ when interviewed about the stimuli), and that the associations were made relatively to the individual strophes.
Conclusions
In our study, we tested complex semantic and perceptual stimuli: The subjects visually read and understood the quatrains having embedded a complex conceptualization (they did not perceive a series of graphemes in succession), and they listened to and understood the music embedding a complex meaningful structure as well (the subjects did not perceive a series of frequencies at the sensory level, but highly organized qualitative wholes). Furthermore, both the stimuli conveyed emotions, perceived as occurring in and shared by the two stimuli themselves.
On the one hand, it is not possible to exclude the presence in the associations of conceptual aspects, such as understanding of the meaning conveyed by the language (usually considered to be a top-down cognitive process); on the other hand, perceptual phenomena in visual and acoustic perception—though not at bottom-up sensory level—are involved in the association. Overall, the results found in the two tasks appear consistent, since participants associated most of the quatrains of the poem with their corresponding music clips; and multisensory features expressed by adjectives such as rough or smooth, distressing or serene, turbid or clear, and gloomy or bright played a major role in the associations, while a few adjectives were less predictive (pleasant or unpleasant, consonant or dissonant, and seductive or repulsive).
Even though our design did not allow selective testing of the role of specific musical elements in the associations (as this was not the aim of the study), the results seem also to suggest that alternance of major and minor keys, the selection of harmonic movements, performance dynamics, melodic features (including ranges), and rhythmic patterns in the accompaniment played an important role, which shall be addressed in further studies.
The presence of correspondences in the semantic and the perceptual domains, experienced and evaluated as such by the subjects (Albertazzi et al., 2013; Albertazzi, Canal, Dadam and Micciolo, 2014), supports also the hypothesis of a nonarbitrary relationship among language, meaning, and perception (Pinna & Albertazzi, 2010). However, an interesting feature emerged from the comparison between the methods used in the first and in the second task. The evaluation based on the Osgood semantic differential (First task) showed in specific occurrences of the strophes a partial dissimilarity in the results between the semantic and the musical features. This result is interesting because it highlights the
different kinds of information conveyed by the linguistic and the acoustic stimuli, which is however minimal in cases such as Strophe 8, where no ambiguity seems to appear.
Vice versa, the direct association between the stimuli (Second task) showed a greater uniformity in the results: As said, in six out of eight cases, participants associated a given music clip with the corresponding correct quatrain as their first choice, and in four out of eight cases, participants associated a given quatrain with the corresponding correct music clip as their first choice. This partial diversity between the two tasks may reveal a structural difference between the perceptual and the linguistic mediums adopted in the evaluations, which is important to bear in mind for future studies in the field of cross-modality. It seems, in fact, that the different methodologies are able to highlight specific aspects explaining the general agreement found in the associations. On the other hand, both the modalities show the presence of qualitative aspects sharing the same dimensions at the semantic and the perceptual level.
In this study, as in previous ones, we use the term ‘‘natural association’’ in the phenomenological sense as qualitatively perceived by anyone because it pertains to the organization of the perceptual field in the awareness. We hope that the experiment could be repeated in other cultures to support the conclusions of our results. For example, using the Italian translation of the poem in free verse, we lost important top-down cultural information conveyed by the metrics of Goethe’s poem. That notwithstanding, the associations were strong. Moreover, the Italian subjects did not have any previous knowledge (past experience) either of the poem or of the musical transcription, and they also belonged to a cultural environment very different from those in which the poet and the musician worked (the latter two being anyway set apart by at least 30 years). This aspect has been evidenced also by the interviews with the subjects, where none of them revealed significant musical expertise (especially in classical music). We expect that by repeating the experiment with German subjects, for example, the results will be not only confirmed but also reinforced. This last idea would be particularly interesting since, although the metrical structure of the ballad is quite irregular, the rhythm is always determined by the sequence of iambs and anapaests, which is particularly suited to conveying a sense of urgency and of anxiety (the feature is explicit in the last line of the seventh quatrain).
Because of the nature of the experimental task, our results may be interesting also for the research field of music and emotion. In fact, some of the dimensions used to highlight the associations are related to the emotional sphere (in particular, couples such as calm or agitated, distressing or serene, and repulsive or seductive). However, an important difference between our study and canonical research in music and emotion is that we have made explicit use of adjectives that are usually considered to evaluate perceptual aspects (such as rough or smooth, turbid or clear, and gloomy or bright) that played a major role in the associations. Among the emotional couples, only distressing or serene proved to be relevant in the associations.
To be more specific, in a review of research on music and emotion, Juslin and Laukka (2004) define emotions as brief and intense reactions to goal-relevant changes in the environment. They consist of a number of components: subjective feeling, cognitive appraisal, emotional expression, action tendency, and emotion regulation (p. 218). In our study, we focused on the component of subjective feelings only, although these can be related to the above other components as well.
A notable finding is that some dimensions lacking an explicitly positive or negative connotation (such as rough or smooth) were effective for the associations, while not all positively or negatively characterized adjective couples were (distressing or serene and gloomy or bright, but not repulsive or seductive and unpleasant or pleasant). This suggests
the usefulness of employing a multidimensional scale to characterize musical and literary experiences. However, as already mentioned, the ineffectiveness of some positive-negative couples may be explained by subjects’ confusion between feelings related to the contents of the strophes and their possible enjoyment of the stimuli—or, to refer again to the field of music and emotion, between so-called perceived, and induced or felt components (as in Gabrielsson, 2002; Juslin, 2013; Juslin & Laukka, 2004), even though both were considered as conscious feelings in our framework.
Finally, one may also confirm that the musical version of the poem, based on the composer’s choice of specific musical features, is in agreement with the associations made in the general population. As previously mentioned, this study also adds to an important line of research addressing the issue of music expression in terms of listeners’ agreement, and accuracy in matching composers’ expressive intentions on the emotional plane (Juslin & Laukka, 2004, p. 219). In particular, the accuracy aspect has generally been much less investigated because of the difficulty of ‘‘obtaining reliable indices of composers’ expressive intentions.’’ On the contrary, the design of our study made it possible to evidence both aspects, that is, listeners’ agreement (because subjective judgments among subjects were consistent), and accuracy (because subjective judgments were consistent across the music and the poem). In particular, the consistency of the subjective judgments across the music and the poem confirms the composer’s intentions to convey an experience coherent with the literary text.
On the other hand because the consistency occurred also with not directly emotional dimensions (such as rough or smooth, gloomy or bright, and turbid or clear), the role of perceptual associations in the semantic interpretation of music is highlighted as well. This result may also suggest that, in order to describe musical aesthetics, theoretical models would profit by considering perceptual dimensions at the same level of emotional dimensions, and not as separate entities (as, e.g., in Juslin, 2013).
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
References
Albertazzi, L., Canal, L., Dadam, J., & Micciolo, R. (2014). The semantics of biological forms. Perception, 43, 1365–1376. doi: 10.1068/p7794
Albertazzi, L., Da Pos, O., Canal, L., Micciolo, R., Malfatti, M., & Vescovi, M. (2013). The hue of shapes. Journal of Experimental Psychology: Human Perception and Performance, 39, 37–47. doi:10.1037/ a0028816
Albertazzi, L., Bacci, F., Canal, L., & Micciolo, R. (2016). The tactile dimensions of abstract paintings: A cross-modal study. Perception, 45, 805–822. doi:10.1177/0301006616643660
Albertazzi, L., Canal, L., & Micciolo, R. (2015). Cross-modal associations between materic painting and classical Spanish music. Frontiers of Psychology. Retrieved from http://dx.doi.org/10.3389/ fpsyg.2015.00424
Belkin, K., Martin, R., Kemp, S. E., & Gilbert, A. N. (1997). Auditory pitch as a perceptual analogue to odor quality. Psychological Science, 8, 340–342.
Bigand, E., Vieillard, S., Madurell, F., Marozeau, J., & Dacquet, A. (2005). Multidimensional scaling of emotional responses to music: The effect of musical expertise and of the duration of the excerpts. Cognition and Emotion, 19, 1113–1139 .
Chatterjee, A. (2003). Prospects for a cognitive neuroscience of visual aesthetics. Bulletin of Psychology and the Arts, 4, 55–60.
Chen, Y.-C., & Spence, C. (2010). When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures. Cognition, 114, 389–404. Collier, W. G., & Hub, T. L. (2001). Musical scales and evaluation of happiness and awkwardness:
Effects of pitch direction and scale mode. American Journal of Psychology, 114, 355–375.
Cowles, J. T. (1935). An experimental study of the pairing of certain auditory and visual stimuli. Journal of Experimental Psychology, 18, 461–469.
Crisinel, A. S., & Spence, C. (2010). As bitter as a trombone: Synesthetic correspondences in non-synesthetes between tastes/flavours and musical notes. Attention, Perception & Psychophysics, 72, 1994–2002.
Crowder, R. G. (1985). Perception of the major/minor distinction: III. Hedonic, musical, and affective discriminations. Bulletin of the Psychonomic Society, 23, 314–316.
Cytowic, R. E. (1995). Synesthesia: A union of the senses (2nd ed.). Cambridge, MA: MIT Press. Di Dio, C., & Gallese, V. (2009). Neuroaesthetics: A review. Current Opinion in Neurobiology, 19, 682–
687.
Dixon, M. J., Smilek, D., Duffy, P. L., Zanna, M. P., & Merikle, P. M. (2006). The role of meaning in grapheme-colour synesthesia. Cortex, 42, 243–252.
Gabrielsson, A. (2002). Emotion perceived and emotion felt: Same or different? Musicae Scientiae, 5, 123–147.
Eagleman, D. M. (2012). Synesthesia in its protean guises. British Journal of Psychology, 103, 16–19. doi:10.1111/j.2044-8295.2011.02020.x
Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10, 1–12.
Gallace, A., & Spence, C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size. Perception & Psychophysics, 68, 1191–1203.
Jakubowski, K., Farrugia, N., Halpern, A. R., Sankarpandi, S. K., & Stewart, L. (2015). The speed of our mental soundtracks: Tracking the tempo of involuntary musical imagery in everyday life. Memory and Cognition, 43, 1229. doi:10.3758/s13421-015-0531-5
Ju¨rgens, U. M., & Nikolic´, D. (2012). Ideaesthesia: Conceptual processes assign similar colours to similar shapes. Translational Neuroscience, 3, 22–27.
Juslin, P. N. (2013). From everyday emotions to aesthetic emotions: Toward a unified theory of musical emotions. Physics of Life Reviews, 10, 235–266.
Juslin, P. N., & Laukka, P. (2004). Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of New Music Research, 33, 217–238.
Juslin, P. N., & Sloboda, J. A. (2001). Music and Emotion: Theory and Research. Oxford: Oxford University Press.
Krumhansl, C. H., & Lerdahl, F. (2011). Musical tension. In F. Bacci, & D. Melcher (Eds.), Art and the Senses(pp. 311–328). Oxford: Oxford University Press.
Ku¨ssner, M. B. (2013a). Shaping music visually. In A. C. Lehmann, A. Jeßulat, & C. Wu¨nsch (Eds.), Kreativita¨t – Struktur und emotion (pp. 203–209). Wu¨rzburg, Germany: Ko¨nigshausen & Neumann. Ku¨ssner, M. B. (2013b). Music and shape. Literary and Linguistic Computing, 28, 472–479.
Langlois, T., Peterson, J., & Palmer, S. (2013). Visual texture, music and emotion. Journal of Vision. Abstract retrieved from VSS.
Makovac, E., & Gerbino, W. (2010). Sound-shape congruency affects the multisensory response enhancement. Visual Cognition, 18, 133–137.
Marks, L. E. (1974). On associations of light and sounds: The mediation of brightness, pitch and loudness. The American Journal of Psychology, 87, 173–188.
Marks, L. E. (1987a). On cross-modal similarity: Auditory-visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception and Performance, 13, 384–394.
Marks, L. E. (1987b). On cross-modal similarity: Perceiving temporal patterns by hearing, touch and vision. Perception & Psychophysics, 42, 250–256.
Marks, L. E. (2004). Cross-modal interactions in speeded classification. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 85–105). Cambridge, MA: MIT Press. Martino, G., & Marks, L. E. (2000). Cross-modal interaction between vision and touch: the role of
synesthetic correspondence. Perception, 29, 745–754.
Maurer, D., & Mondloch, C. (2005). Neonatal synesthesia: A revaluation. In L. Robertson, & N. Sagiv (Eds.), Perspective from cognitive neuroscience (pp. 193–213). Oxford, England: Oxford University Press.
Meier, B. (2013). Semantic representation of synesthesia. Theoria et Historia Scientiarum, 10, 125–134. Melara, R. D., & O’Brien, T. P. (1987). Interactions between synesthetically corresponding dimensions.
Journal of Experimental Psychology. General, 116, 323–336.
Mesz, B., Trevisano, M. A., & Sigman, M. (2011). The taste of music. Perception, 40, 209–219. Mroczko-Wasowicz, A., & Nikolic´, D. (2014). Semantic mechanism may be responsible for developing
synesthesia. Frontiers in Human Neuroscience, 8, 1–13.
Mroczko-Wasowicz, A., & Werning, M. (2012). Synesthesia, sensory-motor contingency, and semantic emulation: How swimming style-color synesthesias challenges the traditional view of synesthesia. Frontiers in Psychology, 3, 279. doi:10.3389/fpsych.2012.00279
Murari, M., Roda`, A., Da Pos, O., Canazza, S., De Poli, G., & Sandri, M. (2015). Is Vivaldi smooth and takete? Non-verbal sensory scales for describing music qualities. Journal of Music Research. Retrieved from http://www.tandfonline.com/doi/full/10.10180/09298215.2015.1101475
Osgood, C. E. (1956). Method and theory in experimental psychology. Oxford, England: Oxford University Press.
Palmer, S. E., & Schloss, K. B. (2010). An ecological valence theory of color preferences. Proceedings of the National Academy of Sciences, 107, 8877–8882.
Palmer, S. E., Schloss, K. B., Xu, Z., & Prado-Leo´n, L. R. (2013). Music-colour associations are mediated by emotion. Proceedings of the National Academy of Sciences, 110, 8836–8841. doi:10.1073/pnas.1212562110
Pinna, B., & Albertazzi, L. (2010). From grouping to meaning. In L. Albertazzi, G. van Tonder, & D. Vishwanath (Eds.), Perception beyond inference. The information content of perceptual processes (pp. 287–344). Cambridge, MA: MIT Press.
Sagiv, N., Simner, J., Collins, J., Butterworth, B., & Ward, J. (2006). What is the relationship between synaesthesia and visuo-spatial number forms? Cognition, 101, 114–128.
Shore, D. I., Barnes, M. E., & Spence, C. (2006). The temporal evolution of the crossmodal congruency effect. Neuroscience Letters, 329, 96–100.
Simner, J. (2012). Defining synaesthesia. British Journal of Psychology, 103, 1–15.
Simner, J., Mulvenna, C., Sagiv, N., Tsakanikos, E., Witherby, S. A., & Fraser, C. (2006). Synaesthesia: The prevalence of atypical cross-modal experiences. Perception, 35, 1024–1033.
Spector, F., & Maurer, D. (2008). The color of Os: Naturally biased associations between shape and color. Perception, 37, 841–847. doi: 10.1068/p5830
Spector, F., & Maurer, D. (2011). The colors of the alphabet: Naturally biased associations between shape and color. Journal of Experimental Psychology: Human Perception and Performance, 37, 484– 495. doi: 10.1037/a0021437
Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73, 971–995. doi:10.3758/s13414-010-0073-7
Spence, C., & Driver, J. (Eds.). (2004). Crossmodal space and crossmodal attention. Oxford, England: Oxford University Press.
Tan, S., & Kelly, M. (2004). Graphic representations of short musical compositions. Psychology of Music, 32, 191.
Terhardt, E. (1984). The concept of musical consonance: A link between music and psychoacoustics. Music Perception: An Interdisciplinary Journal, 1, 276–295.
Tomson, S. N., Narayan, M., Allen, G. I., & Eagleman, D. (2013). Neural networks of colored sequence synesthesia. The Journal of Neuroscience, 33, 14098–14106.
Tramo, M. J., Cariani, P. A., Delgutte, B., & Braida, L. D. (2001). Neurobiological foundations for the theory of harmony in western tonal music. Annals of the New York Academy of Sciences, 930, 92–116. doi:10.1111/j.1749-6632.2001.tb05727.x
Vines, B. W., Krumhansl, C. L., Wanderley, M. M., & Levitin, D. J. (2006). Cross-modal interactions in the perception of musical performance. Cognition, 101, 80–113.
Walker, A. R. (1981). The presence of internalized images of musical sounds. Council for Research in Music Education, 66–67, 107–112.
Ward, J. (2013). Synesthesia. Annual Review of Psychology, 64, 49–75.
Ward, J., Huckstep, B., & Tsakanikos, E. (2006). Sound-colour synaesthesia: To what extent does it use cross-modal mechanism common to us all? Cortex, 42, 264–280.
Ward, J., Tsakanikos, E., & Bray, A. (2006). Synaesthesia for reading and playing musical notes. Neurocase, 12, 27–34.
Wicker, F. W. (1968). Mapping the intersensory regions of perceptual space. The American Journal of Psychology, 81, 178–188.
