For Peer Review
Cross-modal perceptual organization in works of art
Journal: Perception Manuscript ID Draft Manuscript Type: Article Date Submitted by the
Author: n/a
Complete List of Authors: Albertazzi, Liliana; Universita degli Studi di Trento Dipartimento di Lettere e Filosofia, Dipartimento di Lettere e Filosofia
Micciolo, Rocco; Università degli Studi di Trento Dipartimento di Psicologia e Scienze Cognitive Corso Bettini 31 Rovereto, Trento, IT 38068
Canal, Luisa; Università degli Studi di Trento Dipartimento di Psicologia e Scienze Cognitive Corso Bettini 31 Rovereto, Trento, IT 38068
Hachen, Iacopo; SISSA (International School for Advanced Studies), Trieste, Italy
Keywords: Abstract, Concrete, Cross-modality, Gestalt grouping
Abstract:
This study investigated in two experiments the existence of cross-modal correspondences between a series of Kandinsky’s paintings and a series of Schönberg’s musical selections. The first experiment verified the similarity between images and musical excerpts as regard perceptual characteristics relative to 11 pairs of adjectives tested on a continuous bipolar scale. The second verified a direct association between paintings and musical excerpts. The results show that certain pairs of
painting/musical excerpts were evaluated as semantically more similar, while others as semantically more different. Direct association between musical excerpts and paintings showed both attractions and repulsions among the stimuli. The semantic differential method highlighted consistent associations between specific stimuli in the two modalities, although only some of them were shared in both tasks. This result suggests that the semantic differential methods might be insufficient to fully account for cross-modal correspondences and in particular for Gestalt effects between the two modalities. In the direct association, the first abstract period of Kandinsky appears to be associated with the first abstract period of Schönberg. Kandinsky's second period and
Schönberg's second period seem to be associated, although to a lesser extent. The results shed new light on the process of perceptual grouping in cross-modal phenomena.
For Peer Review
1
2 Cross-modal perceptual organization in works of art 3
4 Liliana Albertazzi 1, Luisa Canal 2, Iacopo Hachen 3, and Rocco Micciolo2
5 1LabExP, Department of Humanities, University of Trento, Italy
6 2Department of Psychology and Cognitive Sciences, University of Trento, Italy
7 3SISSA (International School for Advanced Studies), Trieste, Italy
8
9 Corresponding author: Liliana Albertazzi 10 liliana.albertazzi@unitn.it 11 12 13 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
14 1.Introduction
15 Cross-modality is one of the most promising fields of research in perception studies. In recent 16 years, the existence of naturally biased associations among shape, color, sound, taste, touch, and 17 olfactory perception has been largely shown, though mainly for simple stimuli, with different 18 paradigms and by different methodologies (see, for example, Albertazzi et al. 2012, 2014; Gilbert, 19 Martin, & Kemp, 1996; Hanson-Vaux, Crisinel, & Spence, 2013; Kemp & Gilbert, 1997; Lau, 20 Schloss, Eagelman, & Palmer, 2009; Marks, 1987a, b; Osterbauer et al., 2005; Sagiv & Ward, 21 2006; Spector & Maurer, 2008; Spence, Ngo, Percival, & Smith, 2012). See also Bremner et al., 22 2012; Stein, 2012). The debate on cross-modal phenomena touched on several points, such as 23 the nature, the explanation, and the proper terminology to define them (Deroy & Spence, 2013a, 24 b; Spence, 2011; Spence & Deroy, 2013). Cross-modal correspondences can be defined as 25 naturally biased associations or congruency effects, between attributes or dimensions of stimuli 26 in different sensory modalities (Spector & Maurer 2008; Spence, 2011). For example, it has been 27 shown that correspondences exist between auditory pitch and the size and the shape of 28 perceived visual objects (for a replication of a series of studies and further developments see 29 Parise & Spence, 2009). It has also been shown that the perceived correspondences between 30 dimensions of stimuli in different modalities are mapped by scales of antonyms (Osgood, 1960; 31 for a review see Oyama et al., 1998). Culture may play a role in the choice of the antonyms: for 32 example, Western music construes pitch according to an up-down spatial relationship (Pratt, 33 1930), while in other languages pitches are construed according to a size scale (small and large, 34 as in Bali and Java), or to an ageing scale (young and old, as in the Amazonian basin) (Seeger, 35 1987; Albertazzi, Koenderink, & van Doorn, 2015). The most interesting novelty in the field of 36 cross-modality, however, has been the shift of attention from the associations verified between 37 simple stimuli of different modalities (a review in Stein & Meredith, 1993) to processes of 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
38 perceptual organization (Sanabria et al., 2005; Bayne & Spence, 2014; Spence & Chen, 2012; 39 Spence, Sanabria, & Soto-Franco, 2007). Over the years, a series of stimuli of increasing 40 complexity have been tested providing evidence of the presence of cross-modal Gestalten 41 organizing multisensory grouping. For example, cross-modal correspondences have been verified 42 between abstract concepts and color mapping (Albertazzi et al., 2013), between conceptual 43 literary meanings and classical music (Albertazzi et al., 2016c), between contents of 44 contemporary expressionist painting and Spanish flamenco music (Albertazzi, Canal, & Micciolo, 45 2015), and between contents of abstract paintings and tactile perceptions (Albertazzi et al., 46 2016b). The last four studies show an increasing relevance of semantic factors, they do not make 47 use of methodologies such as reaction times (RTs) or implicit association tests (IAT), and they 48 focus on the phenomenological and subjective perceptual experience in first person account, 49 without recurring to sensory-to-sensory or top-down inferential processes. These studies pertain 50 to the Gestalt tradition and explain cross-modal phenomena in terms of grouping (Marks, 1989; 51 Metzger, 1934; Spence, 2015; Watanabe & Shimojo, 2001). Moreover, they do not understand 52 the difference between concrete and abstract in perceptual structures in classical empiricist 53 terms (Locke, 1690/1975).
54 For Gestaltists, concepts are neither sets of attributes defining classes of objects (such as cat, 55 bicycle, cathedral) (Kass, Harland, & Donnelly, 2015) nor disjunctive concepts (Arnheim, 1954, 56 Ch. 2; 1969, Ch. 9, 10; Ash, 1961; Wertheimer, 1923. See also Pinna & Albertazzi, 2010), and 57 abstraction is seen as the ability to grasp the structural elements of a type of object by its 58 perceived patterns. This explains, for example, the ability to see a ‘house’ in a configuration of
59 toothpicks, a ‘face’ on a wall, or a ‘roof’ in a triangle (Arnheim, 1954, Ch. 2; Goldstein & Scheerer, 60 1941). Arnheim termed this process ‘sensible abstraction’ (Arnheim 1966, 1969). Structural 61 patterns are qualitative properties of phenomena. By no means are they metrical cues. In other 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
62 words, for Gestalt psychologists ‘concrete’ does not denote physical objects, and ‘abstract’ does 63 not denote symbolic entities.
64
65 1.2. The field of art provides an important insight into the boundary and the relationship between 66 concrete and abstract concepts in cross-modal phenomena (Actis Grosso et al., 2017; Albertazzi 67 et al., 2012, 2014; Duthie, 2013; Nikolić, 2016; Noguchi, 2007; van Campen, 2010; Vergo, 2012). 68 For example, the Western avant-garde of the twentieth century was the scenario of multifarious 69 inquiries into the perception of shape, color, and light; their expressive characteristics, and their 70 representations in drawing and painting, in an attempt to identify and analyze their abstract 71 dimensions (Albertazzi et al., 2016a; Stelzer, 1964). The switch to abstractionism and the 72 detachment from the figurative in painting was particularly apparent in the works of Kandinsky, 73 Mondrian, Delaunay and Klee. The process resulted in a new conception of the theory of 74 composition. The new tendency toward abstractionism was defined by the proponents 75 themselves as concrete art (see similarities and differences of the concept in van Doesburg (1930) 76 and Kandinsky (1912)). The concrete/abstract couplet referred to the analysis of the subjective 77 and emotional experience of shape and color in their rich appearances and expressivity in the 78 visual field, devoid of any trait of objectivity (on the concept of abstract see Kandinsky, 1912, 79 1926). The concrete experience expressed in painting and considered to be the essence of 80 abstract art (Kandinsky, 1912) had no logical, syntactic, or formalized meaning: Kandinsky termed 81 it the ‘spiritual’ (das Geistige), by which he meant the subjective qualitative content (Inhalt) of 82 awareness, and what he called its inner sonority/sound (Klang). Kandinsky’s first abstract 83 painting (Untitled watercolor) dated back to 1910, and may be considered the starting point of 84 the process (although it may appear unrelated to other paintings of the same period, see Brisch, 85 1955, 255; see also Robbins, 1963).
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
86
87 1.3. Kandinsky’s turn to abstractionism was prompted by a series of impressions on the nature 88 and the interaction of color and light: for example, the connotative characteristics of colors 89 (being rowdy, pompous, pensive, dreamy, absorbed, serious, mischievous, etc.) emitting a 90 musical call (Kandinsky, 1912). Then, most of the cross-modal associations between vision and 91 acoustics, aroused by listening to Wagner’s Lohengrin in St. Petersburg (where, he wrote, the 92 violins, bass, grave and wind instruments merged synaestetically with his memory of the sunset 93 in Moscow), and to Schönberg’s Opera 10 and Opera 11 in Munich (Göbel, 2013). These 94 experiences were the grounds for Kandinsky’s series of Impressions, Improvisations, and 95 Compositions paintings, starting in the 1910s. With the title of Impressions Kandinsky meant
96 paintings directly aroused by the environment (called by him the external nature) and coming to 97 expression by pictorial means; Improvisations, instead, are paintings motivated by the sudden 98 expressions of inner processes (called by him the inner nature); Compositions, as he observes, 99 are those works developed on the basis of the previous ones, and consciously, rationally, and 100 intentionally expressing their contents in abstract form. In these paintings, however the direct 101 sentiment or emotional experience (Gefühl) is still expressed, although in formal, cross-modal, 102 essential relationships between forms, colors, and space (Kandinsky, 1912).
103 The preparatory studies for the final works are examples of the procedure to represent the visual 104 object in abstract form. Consider, for example, the studies for Sketch for Composition 2, where 105 the process of sensible abstraction emerges in the horse-rider pattern (repeatedly drawn during 106 the 1909 in different paintings), and Improvisation. Gorge (1914), representing a chaotic 107 environment in red and yellows. Kandinsky’s first steps into abstractionism, although embedded 108 with symbolism (such as the themes of Saint George and the dragon, the Horsemen of the 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
109 Apocalypse, or the Biblical deluge), is deeply iconic (on the Gestalt process of abstraction see 110 above).
111
112 1.4. The contact and the following acquaintance and correspondence between Kandinsky and 113 Schönberg (a painter himself) arose from the above-mentioned concert in Munich and show the 114 parallel development of inquiry into the cross-modality of color, shape, and sound in both 115 painting and music composition (Hahl-Koch, 1984; Maur, 1985). It was Kandinsky’s opinion 116 that Schönberg’s music takes us into a new region where musical experiences are not acoustic, 117 but psychic (Kandinsky, 1912, Ch. 3). The affinity between Kandinsky and Schönberg is most 118 evident during the initial period that led them to abstractionism in arts. However, while 119 synaesthesia seems to be a distinguishing trait in the painter (Ione & Tyler, 2003, p. 223-224; van 120 Campen, 2010, 56) and probably genuine and inborn if considered at the light of contemporary 121 diagnostic criteria (Just, 2017); (for a contrary opinion see Harrison, 2001, p. 129; Baron-Cohen 122 & Harrison, 1997, p. 10; Dann, 1998, p. 47), it is less traceable in the composer, with a few 123 exceptions as in the case of his Op. 18, the Die Glückliche Hand (1910-1913). Therefore, the 124 comparison between the two artists primarily concerns the similarity of cross-modal dimensions 125 expressed by their works of art. Kandinsky represented the concrete experience lived by listening 126 to Schönberg’s Streichquartett, Op. 10, and the Klavierstücke, Op. 11, in his painting Impression 127 3 (Concert). It was Kandinsky’s opinion that Schönberg’s works, in music and painting as well,
128 were dictated solely by the interest in the rules of organization of the inner sound (Kandinsky, 129 1912, 169 ff, 162).
130 The comparison between the two artists from a cross-modal viewpoint is possible when 131 considering the works of the early Schönberg, which have been traditionally associated with the 132 expressionist movement in music. This association is further supported by his own activity as a 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
133 painter. In fact, although Schönberg never declared any intention to represent his music by visual 134 art, his painting, that aroused Kandinsky’s interest, displays many expressionistic characteristics 135 (Adams, 1995).
136 Schönberg’ second String quartet (Op. 10), Erwartung (Op. 17) and Pierrot Lunaire (Op. 21) 137 belong to those years. The urge to communicate subjective feelings directly, which characterizes 138 expressionistic ideals, clearly emerges from the contents and the lyrics of these works (Covach, 139 1996). To communicate such feelings, Schönberg renounces conventional techniques in favor of 140 a free atonality. This choice makes his musical works from the 1908/1920s even closer to 141 expressionism than his painting. One element that gives coherence to Schönberg’s expressionist 142 period has often been identified in his devastating personal situation starting from 1908. Its 143 traces can be found in the iconic and emotional content of his works, and the often ‘sharp’ means 144 employed to directly express such content (among which, in music, dissonance is the most 145 evident).
146 However, if the need to convey subjective experience directly was the motivation that led 147 Kandinsky until the final stage of his abstractionism, a similar parallelism cannot be drawn in 148 Schönberg. His most famous innovation (i.e. the later development of the twelve-tone method 149 made explicit in the early 1920s) corresponds to a rejection of the previous expressionistic ideals 150 by privileging musical-technical aspects over subjective feelings (Steuermann, 1989; How, 2009). 151 (Incidentally, this rejection is also accompanied by the end of the close relation between the 152 composer and Kandinsky during the war). And indeed, the accessibility of meaning in 153 dodecaphonic works through perceptual means has been strongly questioned by critics 154 (Taruskin, 2004).
155 Nevertheless, the dodecaphonic period of Schönberg is commonly associated with the later and 156 most abstract Kandinsky (e.g. Weiss, 1975), although a similar concept of abstract (mirroring 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
157 Kandinsky’s idea of the inner sound, originally inspired by listening to Schönberg’s earlier music) 158 is difficult to find in Schönberg. However, despite divergent ideological (as well as personal) 159 aspects, the later periods of both artists seem to share at least one element in common, that is, 160 a new rigor in the method. In Kandinsky, this is mostly evident in the Compositions, where, for 161 example, the search for spatial primitives results in the employment of highly geometrical 162 elements. In Schönberg, the adoption of a rigorous method for composing based on the 163 employment of a single twelve-tone row and its symmetrical transformations, results inevitably 164 in a change of compositional approach that can be perceived even by the naïve listener. 165 Schönberg was clearly aware of the perceptual side-effects of employing stricter rules of 166 composition. For example, in the Suite for piano (Op. 25), the first work entirely developed on a 167 single twelve-tone row, Schönberg sometimes seems to compensate for the increased 168 constraints in the method by the use of repeated notes, to create new melodic possibilities 169 without compromising the order of the series.
170 For Schönberg, the development of the twelve-tone technique coincided with an attempt to 171 introduce a systematic order into atonal music (Taruskin, 2004), as part of a personal 172 neoclassicist effort to get closer to the roots of Western music (How, 2009). In the Suite for piano, 173 Schönberg’s choice of reprising pre-classical forms is evident in the structure of the baroque 174 Suite, as well as in the strong polyphonic character of the pieces. This choice likely reflects a 175 rejection of the referentialistic ideas in which atonality was shaped, from the early nineteenth 176 century’s program music to expressionism. In this way, the Suite separates atonality from 177 referentialism, i.e. the opinion that music is supposed to refer to something concrete (e.g. a 178 specific concept, episode or place) to convey its meaning to the listener. Vice versa, according to 179 the absolute viewpoint, music is supposed to be able to directly evoke feelings without entering 180 external domains (see also Hachen & Albertazzi, 2018). Even if the theoretical background of 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
181 Schönberg’s twelve-tone approach lacks (if not even rejects) any explicit reference to feelings, 182 Schönberg’s ideas are fully compatible with a conception of an absolute music in semantic terms. 183 By getting closer to the roots of Western music, and by doing so rejecting any form of 184 referentialism (be it expressionistic or romantic in character), according to Schönberg the 185 structural aspects of music would become independent of external domains, a process that we
186 can compare to abstraction.
187 Whether the two different paths pursued by the two artists towards their personal form of 188 abstractionism on similar cross-modal dimensions would be ultimately perceivable (i.e. not only 189 understandable in purely intellectual terms) and/or associable by naïve participants was part of 190 our research question. The occurrence of systematic associations between abstract stimuli 191 presented in both visual and acoustic modalities, as well as their patterns, if confirmed, would 192 shed light on the relation between the different concepts of abstraction employed by the two 193 artists; most of all, this result would contribute to the current debate on the presence of cross-194 modal Gestalten in highly complex stimuli.
195 196
197 2.The study
198 The study tests in two experiments the existence of naturally biased associations among mostly 199 naïve participants between a series of Kandinsky’s paintings, and a series of Schönberg’s musical 200 selections. The reason for choosing these stimuli is partially the strong cross-modal experience 201 motivating Kandinsky’s painting and its declared relationship with Schönberg’s musical 202 innovations. The research focused on the different levels of abstraction and cross-modal 203 associations embedded in the stimuli. The driving question was whether Kandinsky’s paintings 204 and Schönberg’s musical pieces (those tested in the experiment) shared parallel paths towards 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
205 abstractionism, as has sometimes been speculated on the basis of the facts explained in the 206 introduction: the painter through Impressions to Improvisations to Compositions; the composer 207 through atonality to dodecaphony. The parallel path toward abstractionism in the two artists 208 cannot be mapped on exactly the same time span (as mentioned, Kandinsky’s change of 209 perspective is visible since the 1910s, Schönberg’s from the 1920s), but on a few common 210 structural patterns expressing the new viewpoint in the two artistic fields. Also, as mentioned, 211 Kandinsky’s reported synaesthetic experience is only slightly comparable to Schönberg’s 212 (Schönberg, 1950). Nevertheless, for the reasons explained in the Introduction, it is reasonable 213 to assume that works from the expressionist periods of both artists would be associated. As to 214 the later periods, it is interesting to test whether the parallel drawn between the abstraction 215 process of the two artists is accessible to participants without a theoretical knowledge of the 216 issue. Furthermore, to study whether or not specific fragments from Schönberg’s Op. 25 can be 217 systematically associated with Kandinsky’s paintings may shed light on the issue of the 218 accessibility of twelve-tone music by listeners, as well on the idea of abstraction applied to 219 Schönberg’s method.
220
221 2.1. As regards Kandinsky’s work, we analyzed a series of paintings dating to the years of his first 222 Compositions, and his progressive detachment from the figurative representation of objects
223 characterizing the previous Murnau phase, which resulted in painting the pure relationships 224 among the elements of visual, cross-modal appearances. We first analyzed 25 paintings, dated 225 from 1910 to 1923, from which we chose 15, where the progression towards abstractionism was 226 more visible: precisely, 14 paintings belonging to the years between 1910 and 1914 (the so-called 227 abstract period) and one in 1923 (the so-called Bauhaus period) (see below). The choice fell on 228 these pictures for the following reasons. As to the shape configuration, we chose paintings 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
229 showing the transition from figurality to abstractionism (representations of elementary 230 components of pictorial space such as lines and color spots - as occurs, for example, in 231 Improvisation 19, Improvisation. Gorge, and Lake boat trip. The paintings of this period are
232 characterized by the dominance of part/whole organization according to color spots, or lines 233 grouping, orientation and organization) (as in Composition 5, Composition 6, Improvisation 26; 234 horizontality and verticality (as in Improvisation 19, Improvisation 14, Black spot); rhythm, such 235 as static or dynamic (as in Improvisation. Gorge, Improvisation 14, Composition 7), and 236 connotative dimensions of color, such as bright/gloomy and fiery/faded, attractive/repulsive (as 237 in Composition 7, Lake boat trip, Improvisation 14), etc. These dimensions were considered in 238 accordance with a series of observations made by Kandinsky himself in his works (Kandinsky, 239 1913, 3; 1912, Ch. V): for instance, the structural presence of contraries (Gegensätze) ruling the 240 configurations in these paintings concern mass vs. line, shape vs. color, warm vs. cold, dark vs. 241 bright, thin vs. thick, calm vs. agitated, silent vs. noisy, etc. Kandinsky observes, for example, that 242 in his Composition 4 (1911) light-sweet-cold is the main opposition of contraries ruling the 243 configuration of the painting (Kandinsky, 1913, 35).
244
245 2.2. As regards Schönberg’s work, a total of 14 musical excerpts were extracted from his Op. 11 246 (Three Piano Pieces; Schönberg, 1990a) and Op. 25 (Suite for Piano; Schönberg, 1990b). Eight 247 musical excerpts were extracted from Op. 11, and six musical excerpts from Op. 25. Musical 248 excerpts were selected with the purpose of choosing the most representative musical passages 249 from the two works of the composer (partly drawing on the existing literature, see comments on 250 the individual musical excerpts in the Supplemental materials), and in particular where the 251 stylistic differences between the two different periods are most evident. For example, we 252 selected the musical excerpts from Op. 11 trying to choose some of those passages where 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
253 reminiscences of tonal harmony can be heard, and where the articulation and the agogics 254 (intentional deviations from the basic tempo) can be distinctively attributed to the expressionist 255 period. Similarly, we selected passages from Op. 25 where the dance features of the Baroque 256 Suite are mostly evident in rhythm and tempo, and/or the ones that included the most 257 remarkable presentations of the twelve-tone material. Importantly, we tried to select musical 258 excerpts so that each would be a complete fragment (i.e. with a perceivable beginning and end), 259 and by trying to diversify the character of the musical excerpts as much as possible. In order to 260 keep the stimuli as homogeneous as possible, the musical excerpts were extracted from 261 recordings of the same performer (Maurizio Pollini), and were also selected to have similar 262 durations. Among the ones from Op. 11, three musical excerpts were chosen from the first piece 263 (Mäßig), three from the second piece (Sehr langsam), and two from the last one (Bewegt). 264 Among those from Op. 25, one musical excerpt was chosen from each movement of the Suite, 265 except for the Praeludium (because of the impossibility to isolate a fragment from this movement 266 that would correspond to our requirements).
267 Since we were not testing a one-to-one correspondence between paintings and musical excerpts, 268 it was not necessary to have an identical number of visual (15) and auditory stimuli (14).
269 Among Kandinsky’s production there were many paintings that could fit our selection criteria, 270 and we tried to include as many of them as possible (15) within reasonable task efficiency/sample 271 size limits. Instead, the limited extension of the chosen Schoenberg’s works, combined with our 272 selection criteria, allowed the extraction of no more than 14 stimuli (one less than the paintings). 273 274 275 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
276 2.3. The putative characteristics attributed to paintings and musical excerpts were tested in the 277 first experiment by means of the Osgood semantic differential (Osgood, 1956). The use of the 278 Osgood semantic differential to test the associations was also partially suggested by Kandinsky’s 279 experience of the pervasive presence of contraries in visual appearances. Kandinsky, in fact, 280 speaks of “fights among tones” [sic], “contraries of the contradictions (Widersprüche)”, such as 281 light/dark and their cross-modal associations (light and sweet; cold and bitter) as founding the 282 color harmony (Kandinsky, 1913). The same opposition, for example, is represented in Marc’s 283 famous painting Fighting forms, i.e. red and black swirling shapes. The specific choice of the pairs 284 of contraries for the Osgood semantic differential, instead, besides Kandinsky’s observations, 285 was due to features characterizing both the visual and acoustic stimuli. For example, we assumed 286 that the calm/agitated couple might be present in a few Schönberg’s musical excerpts, such as 287 no. 3, no. 5, no. 6, no. 7 (agitated), and the attribute calm in the other musical excerpts taken 288 from his Op. 11, such as musical excerpt no. 1 or no. 4. Similarly, these characteristics might be 289 perceivable in Kandinsky’s paintings Lake boat trip (calm), Improvisation. Gorge (agitated). The 290 grave/acuto (left in Italian) couple was chosen to associate register and color lightness; the pair 291 hot/cold was chosen for its potential to map color to timbres and/or harmonies; the pair 292 consonant/dissonant could map subject's perception of dissonance and/or roughness (see 293 Tramo et al., 2001) of the harmony, and contrasting colors in the paintings; the pair 294 pleasant/unpleasant as general affordances of the concrete experience of the stimuli (and not as 295 an individual preference), as perceivable in Black spot, Improvisation 3 (Concert), Composition 8, 296 etc.
297 The prediction was that the participants would make systematic associations between the 298 paintings and the series of musical selections; and that the associations would be due to the 299 presence of similar dimensions in both, as also to be evidenced by the Osgood semantic 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
300 differential. Needless to say, both the stimuli showed a high level of complexity due to the 301 contents of the paintings and of the musical pieces. We were also aware that, for this reason, the 302 associations could have been conveyed by different components of the artistic works, and that 303 it was very unlikely that we would find a one-to-one correspondence between the individual 304 elements of the pictorial and acoustic compositions. Finally, we did not test individual 305 preferences: our goal was to highlight, as far as possible, the role played by the (Gestalt) concrete 306 and abstract dimensions in perceiving.
307
308 3. Methods 309 3.1. Participants
310 A total of 32 participants volunteered for the two Experiments: 14 women (mean age: 33.5 years; 311 standard deviation: 16.2 years; median: 23 years) and 18 men (mean age: 40.7 years; standard 312 deviation: 17.4 years; median: 40 years).
313 Almost all participants were recruited from students in the Departments of Humanities, 314 Psychology and Cognitive Science, Information Engineering and Computer Science, and 315 Mathematics, University of Trento, Italy, although we also invited some colleagues of those 316 departments to participate in the experiment. In so doing we had participants of different 317 background, age and expertise. The address list of the students was provided by the student 318 office. We first sent an email asking the students to take part in the experiment, mentioning that 319 we were not looking for persons with professional experience in painting and music, although 4 320 participants had received musical training at a conservatory or from private lessons, and 2 of 321 them were musicians. The questionnaire reported this information. The participants were also 322 asked about possible conscious synaesthesia (Albertazzi et al., 2013, 2015; Palmer & Schloss, 323 2010; Palmer et al., 2013) and only one declared to be a synaesthete. We did not include 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
324 synesthesia among the exclusion criteria, since the aim was to evaluate the existence of naturally 325 biased associations in the general population. According to Sagiv & Ward, 2006, the prevalence 326 of synesthesia could be one in twenty. The only exclusion criterion was visual impairment and 327 self-reported acoustic impairment. Before the test, the participants carried out the Ishihara test 328 for color. After the experiment, the participants were asked whether they had previously known 329 the paintings and the musical excerpts that they evaluated. For all the participants (with 330 exception of two) the musical stimuli were totally new. A few of them were generally acquainted 331 with one painting only (Composition 8).
332 The first experiment was performed using the semantic differential on a bipolar rating scale of 333 adjectives (Osgood, 1956). The second experiment directly evaluated the association between 334 visual and auditory stimuli. All the participants signed an informed consent form. The two 335 experiments reported here complied with the ethical guidelines of the University of Trento. 336
337 3.2. Stimuli and apparatus
338 The two experiments was carried on in the Experimental Phenomenology Laboratory (LabExP) at 339 the Department of Humanities, Trento University. The laboratory had constant and controlled 340 lighting conditions (ca. 10 lx on average in the room, given by a halogen lamp). The colors were 341 produced on a monitor Eizo Color edge mod. CG276 (68x27 cm), P7N OFTD1846 75Q, Mfd. 342 2013.05.24, S/N 23816053 A (resolution 2560x1440).
343 The software used to calibrate the monitor was Eizo Corporation, Color Navigator 6 v.6.4.0.5. The 344 calibration guaranteed a white D65, a 2.2 gamma, 120 cd/m2 luminosity, maximum contrast. The
345 monitor was recalibrated at the beginning of each session.
346 The two experiments were performed with an interval of about 30 minutes in which the 347 participants left the laboratory for a short rest.
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
348 Experiment 1
349
350 3.3. Task 1 (evaluations by Osgood semantic differential)
351 The first experiment aimed to test whether complex images and musical excerpts with varying 352 perceptual characteristics led to consistent associations relative to a series of adjectives. 353 Participants were asked to rate eleven pairs of contraries (shown in random order) on a pseudo-354 continuous bipolar scale (ranging from 0 to 100) when looking at a painting or when listening to 355 a musical excerpt (both presented in random order). The participants were told that they were 356 going to be shown a set of paintings, one at a time, appearing on the left half of the screen, that 357 could be freely zoomed by using a magnifier. The task was to evaluate the overall content (i.e. 358 the meaning) of each painting (or of each musical excerpt) according to the eleven pairs of 359 contraries (see below, Procedure). Pairs were presented on the right half of the screen, and 360 randomized with respect to their order of presentation in the couple (for example, 361 pleasant/unpleasant might be presented as unpleasant/pleasant).
362 The two contraries were placed at the extremes of a continuous line. Using a mouse, the 363 participant had to place the pointer indicating his/her degree of agreement with the two 364 adjectives with regard to the semantic content of the painting or of the musical excerpt. The 365 software stored the choice with two scores, one for each member of the pair. Participants were 366 allowed to change their choices at any time until they proceeded to the next stimulus. After each 367 answer, confirmed by pressing the button “Confirm” placed below the screen, a button with the 368 inscription “Proceed” appeared at the right top of the screen; by pressing it, the answer was 369 registered and a new image or musical excerpt was presented.
370 371 3.4. General materials 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
372 The stimuli consisted of fifteen paintings by Kandinsky and fourteen musical excerpts from 373 Maurizio Pollini’s performance extracted from Schönberg’s work.
374 Kandinsky’s paintings were: no. 1. Improvisation 10 (1910); no. 2. First abstract watercolor 375 (1910); no. 3. Lake boat trip (1910); no. 4. Improvisation 14 (1910); no. 5. Sketch for Composition 376 2 (1910); no. 6. Composition 5 (1911); no. 7. Improvisation 19 (1911); no. 8. Impression 3 (Concert)
377 (1911); no. 9. Black spot (1912); no. 10. Landscape with red spots (1913); no. 11. Improvisation 378 26 (Rowing) (1912); no. 12. Picture with a white border (1913); no. 13. Composition 6 (1913); no.
379 14. Improvisation. Gorge (1914); no. 15. Composition 8 (1923). The images were taken from the 380 website https://www.wassilykandinsky.net/work-150.
381 Schönberg’ s musical pieces were musical excerpts from Maurizio Pollini’s performance of 382 Schönberg’s Op. 11 and Op. 25, and specifically:
383 Three Piano Pieces, Op. 11:
384 no. 1. Mäßig (bars 1-8); no. 2. Mäßig (bars 34-41); no. 3. Mäßig (bars 53-58); no. 4. Sehr langsam 385 (bars 2-5); no. 5. Sehr langsam (bars 10-13); no. 6. Sehr langsam (bars 43-47); no. 7. Bewegt (bars 386 1-5); no. 8. Bewegt (bars 32-35).
387 Suite for Piano, Op. 25:
388 no. 9. Gavotte (bars 8-16); no. 10. Musette (bars 1-9); no. 11. Intermezzo (bars 1-3); no. 12. 389 Menuett (bars 1-7); no. 13. Trio (bars 34-42); no. 14. Gigue (bars 1-13).
390 (The musical excerpts were produced by Iacopo Hachen).
391 The musical excerpts were heard by the participants for about 60 seconds. 392
393 The eleven pairs of adjectives, presented in Italian were: piacevole/spiacevole 394 (pleasant/unpleasant), sereno/angoscioso (serene/distressing), grave/acuto (left in Italian, see 395 also Albertazzi et al., 2016c), consonante/dissonante (consonant/dissonant), caldo/freddo 396 (warm/cold), calmo/agitato (calm/agitated), acceso/spento (fiery/faded), morbido/duro 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
397 (soft/hard), luminoso/cupo (bright/gloomy), statico/dinamico (static/dynamic), and dolce/aspro 398 (sweet/bitter). Judgments were given on a continuous scale as described below.
399
400 3.5. Procedure
401 Participants read the following instructions on the screen:
402 This experiment consists in the evaluation of visual stimuli (pictures) and auditory stimuli (musical
403 excerpts).
404 On the left of the screen are the stimuli (see a painting or listen to a musical excerpt), while on
405 the right there is a rating scale between two opposing adjectives. The cursor can be dragged to
406 the right or left between the two adjectives, bringing it closer to one of the two. The task is to
407 evaluate which of the two adjectives is associable with the stimulus presented, and to what
408 extent, by positioning the cursor at the point that you consider most appropriate. You should
409 prefer accuracy to promptness of response.
410 Once you have made your choice, press the "Confirm" button.
411 If you decide that neither of the two attributes is associated with the stimulus, you can leave the
412 cursor at the center and confirm the choice. By pressing the "Continue" button you can move to
413 the evaluation of the stimulus on other pairs of opposing adjectives.
414 At the end of the session there will be a 30-minute break, after which the second session will start
415 with instructions for the task to be performed.
416 When you are ready, press the "Start" button.
417
418 Experiment 2
419
420 3.6. Task 2 (direct association between paintings and musical excerpts) 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
421 The aim of the second experiment was to verify whether some images of Kandinsky’s paintings, 422 with varying perceptual characteristics and contents, led to consistent cross-modal associations 423 with musical excerpts taken from the repertoire of Schönberg’s works. Each participant saw a 424 series of images of paintings in thumbnail on the screen (anytime the position of the paintings 425 were randomly selected). The participant clicked on a specific image, which thus appeared in full 426 screen mode, and likewise with the other images, without any constrained order. The participant 427 viewed the images while simultaneously listening to a musical excerpt (presented in random 428 order). The participant had to choose the image(s) that s/he most naturally associated with that 429 music. S/he could list up to three images associated with the musical excerpt by arranging them 430 in three different boxes at the bottom of the screen. The participant could go back to view the 431 images again, and s/he could also listen repeatedly to the musical excerpt. Once the association 432 had been performed, the three selected images were moved down, each into one of three boxes. 433 Once the choice had been confirmed, it could not be changed, and the task continued with the 434 presentation of all the images and the new musical excerpt, and so on until the musical excerpts 435 were exhausted.
436
437 3.7. General materials
438 The stimuli consisted of the same fifteen paintings by Kandinsky and fourteen musical excerpts 439 extracted from Schönberg’s work.
440
441 3.8. Procedure
442 Participants read the following instructions on the screen:
443 You will see a series of images of paintings in thumbnail on the screen. Click on one of them, which
444 will appear in full screen mode, and then do likewise with the other images. At the same time,
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
445 you will hear a musical excerpt. Select which image(s) you most naturally associate with the
446 music. You can go back to re-view images already seen, and also to hear the musical excerpt
447 again. You can list up to three images associated with the music, placing them in three different
448 boxes at the bottom of the screen. Once you have confirmed your choice, it cannot be changed,
449 and the task will continue with further musical excerpts until there are none left. You should prefer
450 accuracy to promptness of response.
451
452 4. Statistical methods
453 Descriptive statistics were calculated based on the rating values given by the participants. 454 Euclidean distances between paintings and musical excerpts based on mean rating values for 455 each adjective were calculated. The direct association between painting and musical excerpts 456 was evaluated by employing the chi-square test and the associated standardized residuals. 457 Analyses were performed with R 3.3.1 software (R Core Team, 2016).
458 While Task 1 is mainly descriptive, Task 2 evaluates (using the chi square test of significance) 459 the presence of a direct association between paintings and musical clips. We expected that 460 such an association existed and the approximate number of subjects was calculated in the 461 following way:
462 The contingency table which collect the results of Task 2 has 15 rows and 14 columns, and 463 therefore 210 cells. Participants were requested to choose 3 paintings for each clip that they 464 heard, and therefore there are 42 observations for each participant. The statistic to test the 465 association between the variables ‘painting’ and ‘clip’ is asymptotically distributed (under the 466 null hypothesis) as a chi square distribution (with 182 degrees of freedom). The approximation 467 to the true chi square distribution could be considered good when the lowest expected
468 frequency is higher than 5. To get, on the average, 6 observations in each cell, are therefore 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
469 necessary 210 times 6 = 1260 observations. Since each participant contributes with 42 470 observations, the minimum number of participants is 1260/42 = 30.
471
472 4.1. Results: First Experiment
473 Table 1 reports the mean rating values for each adjective and for each painting given by the 32 474 participants (the corresponding standard deviations are reported in the supplementary 475 materials). Means range between 13 and 83; therefore, painting no. 5 (Sketch for Composition 2) 476 was considered the brightest of the 15 paintings (and therefore the least gloomy), and very fiery 477 (and therefore not faded). Similarly, painting no. 10 (Landscape with red spots), painting no. 1 478 (Improvisation 10) and painting no. 15 (Composition 8) were evaluated as very bright. Painting 479 no. 14 (Improvisation. Gorge) was considered the least static (and therefore the most dynamic); 480 similarly, painting no. 12 (Picture with a white border), painting no. 13 (Composition 6), and 481 painting no. 6 (Composition 5). Painting no. 8 (Impression III (Concert)) and painting no. 5 (Sketch 482 for Composition 2) were evaluated as the fieriest. Although weaker, painting no. 1 (Improvisation
483 10), and painting no. 11 (Improvisation 26 (Rowing)) were also evaluated as fiery.
484 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
485
486
487 Table 1. Mean rating values for each adjective and for each painting given by the 32 participants.
488 Ratings can range between 0 and 100. See section 3.4. (General materials) for the 489 correspondence between numbers and stimuli.
490
491 Table 2 reports the mean rating values for each adjective and for each musical excerpt given by 492 the same 32 participants (the corresponding standard deviations are reported in the 493 supplemental materials). Means range between 16 and 84; therefore, musical excerpt no. 1 494 (Mäßig (bars 1-8) was considered the calmest of the 14 musical excerpts (and therefore the least 495 agitated), while musical excerpt no. 14 (Gigue (bars 1-13)) was considered the least static (and 496 therefore the most dynamic).
497 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
498
499 Table 2. Mean rating values for each adjective and for each musical excerpt given by the 32
500 participants. Ratings can range between 0 and 100. See section 3.4. (General materials) for the 501 correspondence between numbers and stimuli.
502
503 Each painting (as well as each musical excerpt) is identified by 11 values, corresponding to the 11 504 adjectives. For example, painting 9 is identified by the following values (reported in column 9 of 505 Table 1):
506 39 45 55 57 55 43 49 44 46 57 55
507 However, in order to calculate the “semantic” Euclidean distance between each painting and 508 each musical excerpt, other 11 values are needed, i.e. those corresponding to the antonym, 509 calculated as the complement to 100 (when considering painting 9 these values are 61 55 45 510 43 45 57 51 56 54 43 45). These distances are shown in Table 3.
511 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
512 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 135 74 97 136 84 64 95 110 64 55 118 84 52 74 2 124 38 74 119 73 41 79 86 37 47 104 63 38 57 3 120 66 37 97 30 96 77 34 82 81 89 91 76 93 4 72 35 51 69 59 82 118 72 47 88 57 43 81 109 5 119 71 120 133 122 74 139 138 61 97 121 66 89 109 6 115 34 59 107 64 50 81 72 32 54 93 59 44 68 7 77 49 52 75 52 84 117 75 53 85 59 52 79 108 8 122 52 89 123 87 48 95 105 40 58 110 64 50 72 9 105 33 51 93 49 67 82 64 47 66 82 61 57 79 10 88 64 99 105 98 86 141 120 60 99 89 50 91 119 11 97 35 87 103 93 63 119 105 35 84 92 38 73 98 12 106 33 55 95 64 68 93 67 45 73 83 59 64 85 13 122 37 60 107 74 60 80 68 49 74 101 71 60 73 14 143 54 88 137 92 27 71 97 44 47 125 79 31 45 15 137 79 110 146 108 53 116 128 55 64 127 75 56 82 MUSICAL EXCERPT PA IN TI N G513 Table 3. Euclidean distances between paintings and musical excerpts based on mean rating
514 values for each adjective. See section 3.4. (General materials) for the correspondence between 515 numbers and stimuli.
516
517 The 6 painting/musical excerpts couples with the lowest distances, and therefore considered 518 semantically more similar, are painting no. 14 (Improvisation. Gorge) and musical excerpt no. 6 519 (Sehr langsam (bars 43-47)), painting no. 3 (Lake boat trip) and musical excerpt no. 5 (Sehr 520 langsam (bars 10-13)), painting no. 14 (Improvisation. Gorge) and musical excerpt no. 13 (Trio
521 (bars 34-42)), painting no. 6 (Composition 5) and musical excerpt no. 9 (Gavotte (bars 8-16)), 522 painting no. 12 (Picture with a white border) and musical excerpt no. 2 (Mäßig (bars 34-41), 523 painting no. 9 (Black spot) and musical excerpt no. 2 (Mäßig (bars 34-41).
524 Conversely, the painting/musical excerpt couples with the highest distances (and therefore 525 considered semantically more different) are painting no. 15 (Composition 8) and musical excerpt 526 no. 4 (Sehr langsam (bars 2-5)), painting no. 14 (Improvisation. Gorge) and musical excerpt no. 1 527 (Mäßig (bars 1-8)), painting no. 10 (Landscape with red spots) and musical excerpt no. 7 (Bewegt 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
528 (bars 5)), painting no. 5 (Sketch for Composition 2) and musical excerpt no. 7 (Bewegt (bars 1-529 5)), painting no. 5 (Sketch for Composition 2) and musical excerpt no. 8 (Bewegt (bars 32-35)). 530
531 4.2. Results: Second experiment
532 Participants were requested to choose 3 paintings for each musical excerpt that they heard. The 533 chi-square test confirmed that the association between the variables ‘painting’ and ‘musical 534 excerpt’ could not be considered random but instead systematic (chi-square=325; d.f.=182; 535 p<0.001). Given that the lowest expected frequency was less than 5, a Monte Carlo simulation 536 was performed. It confirmed the significance of the association (p<0.001).
537 Since the test did not indicate which musical excerpt was associated (positively or negatively) 538 with which painting, a residual analysis was performed. A standardized form of the residual 539 (which behaves like a normal deviate) was employed to determine whether the residual was 540 large enough to indicate a departure from a random choice. In this case, there was only about a 541 5% chance that any particular standardized residual would exceed 1.96 in absolute value. When 542 we inspected 210 cells, about 10 residuals (i.e. 5% of 210) could have been so large solely because 543 of random variation. On the other hand, as can be seen in Table 4, there were 26 residuals greater 544 than 1.96 in absolute value.
545 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
546
547 Table 4. Standardized residuals from the direct association between paintings and musical
548 excerpts. "Significant" residuals are flagged with an asterisk. See section 3.4 (General materials) 549 for the correspondence between numbers and stimuli.
550
551 Overall, there were 16 residuals greater than 1.96, and 10 residuals lower than -1.96. A positive 552 residual means that the selected musical excerpt attracted the corresponding painting (i.e. the 553 painting was associated with the musical excerpt more than the average); a negative residual 554 means that the selected musical excerpt ‘repelled’ the corresponding painting (i.e. the painting 555 was associated with the musical excerpt less than the average).
556 There were 6 musical excerpts which showed a strong attraction (a residual greater than 3). 557 Musical excerpt no. 14 (Gigue (bars 1-13)) was strongly associated with painting no. 14 558 (Improvisation. Gorge), musical excerpt no. 10 (Musette (bars 1-9)) was strongly associated with 559 painting no. 2 (First abstract watercolor), musical excerpt no. 4 (Sehr langsam (wars 2-5)) was 560 associated with painting no. 7 (Improvisation 19), musical excerpt no. 10 (Musette (bars 1-9)) was 561 associated with painting no. 15 (Composition 8), musical excerpt no. 4 (Sehr langsam (bars 2-5)) 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
563 was associated with painting no. 3 (Lake boat trip).
564 On the other hand, the negative associations were weaker. Musical excerpt no. 14 (Gigue (bars 565 1-13)) was negatively associated with painting no. 3 (Lake boat trip) and showed the lowest 566 residual (-2.76). Other three couples showed quite similar residuals (-2.5): Musical excerpt no. 8 567 (Bewegt (bars 32-35)) with painting no. 15 (Composition 8), musical excerpt no. 12 (Menuett (bars 568 1-7)) with image no. 14 (Improvisation. Gorge) and musical excerpt no. 4 (Sehr langsam (bars 2-569 5)) with painting no. 15 (Composition 8).
570
571 4.3. Comparing the results of the two tasks.
572 In Task 1, participants evaluated both paintings and musical excerpts employing the Osgood 573 semantic differential, and the Euclidean distances between paintings and musical excerpts were 574 calculated. On the other hand, in Task 2, participants directly associated three paintings with 575 each musical excerpt and residuals were employed to evaluate the association between paintings 576 and musical excerpts. To compare the results of the two tasks, the correlation coefficient 577 between distances and residuals was calculated. As expected, distances and residuals were 578 inversely associated (r = -0.357; p < 0.001), i.e., on average, when a painting was ‘near’ to a 579 musical excerpt, the corresponding residual was also low and vice versa. Therefore, some degree 580 of consistency was found between the two tasks.
581 582
583 5. General discussion
584 The research verified the existence of cross-modal associations between a series of Kandinsky’s 585 paintings, and a series of Schönberg’s music selections. Participants were for the most part (26 586 out of 32) without advanced knowledge/competence in music or art. The first experiment tested 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
587 both paintings and musical excerpts with Osgood semantic differential; the second experiment 588 tested a direct association between paintings and musical excerpts.
589 The prediction that the participants would make systematic associations between the paintings 590 and the series of musical selections, notwithstanding the diversity and high complexity of the 591 stimuli, was largely validated.
592 Specifically, as regards Experiment 1, with Osgood semantic differential, some of our 593 assumptions were confirmed: for example, that the adjective ‘agitated’ might characterize some 594 Schönberg’s musical excerpts such as no. 6, no. 7, no. 10, no. 13, no. 14 and the adjective ‘calm’ 595 in some of the other musical excerpts taken from Op. 11 such as no. 1 and no. 4. As predicted, 596 the adjective ‘agitated’ characterized the painting Improvisation. Gorge and Composition 6. 597 However, surprisingly, the painting Lake boat trip was not perceived as ‘calm’, although relatively 598 ‘calmer’ than Improvisation. Gorge, i.e. neither ‘agitated’ nor ‘calm’. However, it is interesting 599 that none of the paintings was rated as ‘calm’, reflecting the dynamic character of all the 600 paintings selected in this study. Landscape with red spots (no. 10) was rated as the calmest 601 among the paintings, although not extremely calm.
602 Regarding the assumptions that guided the choice of the couples of contraries, the results 603 showed the following. The warm/cold pair, surprisingly, did not result closely associated with the 604 paintings, probably due to the great variety of colors employed by Kandinsky in the selected 605 stimuli. As to the fiery/faded and bright/gloomy pairs, the ratings seem to be more diversified 606 with respect to the paintings, although no painting was rated neither faded nor gloomy, for 607 similar reasons as above. Also, the consonant/dissonant pair was not polarized with respect to 608 any of the paintings, reflecting perhaps the difficulty of the participants in attributing an intuitive 609 cross-modal meaning to such technical musical terms. The couple was more useful to describe 610 the musical excerpts, classified as fiery, hard, and dynamic with an understandable prevalence 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
611 of the ‘dissonant’ rating. The pleasant/unpleasant pair was not polarized towards ‘unpleasant’, 612 with respect to any of the stimuli. Therefore, although explicitly instructed not to do so, it is 613 possible that some participants based their interpretation of the pairs on their personal 614 preferences for the stimuli, instead of as general affordances of their concrete experience. 615 As regards Task 2, the results of the direct association between musical excerpts and paintings 616 showed the presence of both attractions and repulsions among the stimuli. On the other hand, 617 the negative associations were weaker, reflecting the high suitability of auditory and visual 618 stimuli to be combined together to produce different and specific Gestalten.
619 Analysis of the findings revealed that the results obtained with the two methods (the Osgood 620 semantic differential and direct association) do not always overlap; something that we already 621 found in a previous study of ours (Albertazzi et al., 2016c). Although the results are generally 622 consistent, the evaluation based on the Osgood semantic differential (Task 1) showed in specific 623 occurrences a partial dissimilarity in the results of the evaluation based on direct association 624 (Task 2).
625 This partial diversity between the two experiments may reveal a structural difference between 626 the perceptual and the linguistic mediums adopted in the evaluations, which is important to bear 627 in mind in the debate among synaesthesia, cross-modality and ideasthesia. It seems, in fact, that 628 the different methodologies are able to highlight specific aspects explaining the general 629 agreement found in the associations. The semantic differential method highlighted consistent 630 associations between specific stimuli in the two modalities, although only some of them were 631 shared in both tasks. On the other hand, both the visual and acoustic modalities show the 632 presence of common patterns at the semantic and the perceptual level. This result suggests that 633 the semantic differential methods might be insufficient to fully account for cross-modal 634 correspondences, in particular for Gestalt effects in the two modalities.
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
635 As to the aim of our study, we may conclude that the cross-modal patterns towards 636 abstractionism, manifested in the works of the two artists, were often detectable by the 637 participants and comparable in the first phases of their development (for Kandinsky, the period 638 starting in the 1910s, for Schönberg the period corresponding to his development of atonal 639 music). Participants were also able to detect the change in Schönberg’s compositional structure 640 leading to dodecaphony, which was however less associable with the figural elements still 641 remaining in the second period of Kandinsky’s painting towards abstractionism (mainly, 642 Compositions). In fact, Schönberg’s dodecaphonic construction is a product of a more explicit
643 mental operation put in place by the composer.
644 As to the debate on the opposite viewpoints in cross-modal studies, the results shed new light 645 on the concept of ideasthesia. Ideasthesia is commonly conceived as the activation of concepts 646 producing phenomenal experience, where concepts are generally understood in Fodorian terms, 647 i.e. as top-down cognitive abilities for our comprehension of the environment, and semantically 648 driven. This conception, for example, equates associations between color and temperature 649 (subjectively, red is warm, blue is cold) to associations of the type doctor and nurse (Nikolić, 650 2016). These concepts and associations, however, are categorically very different (and 651 presumably mediated by different neural pathways). As Hering and Katz observed, it is in the 652 (universal) nature of the color red to be warm and of the color blue to be cold (Da Pos & 653 Albertazzi, 2003; Da Pos & Valenti, 2007; Hering, 1964; Katz, 1935). The associations perceived 654 between color and temperature, on which there is a general agreement in the field of arts 655 (Albers, 1975; Kandinsky, 1912; Itten, 1961), are produced by concrete experience directly given 656 in awareness. Color and its experienced temperature are not disjunctive and detachable 657 components of a percept; moreover, they are not a product of top-down external associations 658 between ‘abstract’ (in the sense of formal or syntactically processed) concepts due to past 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
659 experience, according to the Humean and/or probabilistic hypothesis. They are internally related 660 as a unitary qualitative content of awareness. Color and its experienced temperature form a 661 unitary (Gestalt) concrete concept. Doctor and nurse, and similar pairs of concept associations, 662 instead, are only externally, conventionally, and functionally related. Needless to say, the 663 connotative properties of color, among which warmth/coldness, may be weakened by the 664 context inducing phenomena of assimilation (see in Supplemental materials the comment on 665 musical excerpt no. 8).
666
667 5.1. A point to be considered in the debate on ideasthesia, however, is firstly the profound 668 symbolism of which Kandinsky’s work is imbued and its impact on the final configurations of his 669 paintings. Strict supporters of ideasthesia, in fact, may argue that the strong cross-modal 670 (sometimes considered as synaesthetic) content of his paintings has been induced by his 671 theosophical or religious ideation (Kandinsky, 1912, Ch. 3), and the same would hold for 672 Schönberg. In this respect Kandinsky’s paintings and Schönberg’s music play the role of a case 673 study, because they are both culturally imbued with ideas circulating in the avant-garde of the 674 20th Century (Washton Long, 1972, 1975). However, as Kanizsa clearly stated, in perceptual
675 analyses, heuristically one should distinguish phenomena pertaining to seeing (and other 676 modalities) from phenomena pertaining to thinking (Kanizsa, 1991). As to Kandinsky’s own 677 descriptions of his cross-modal experience, analyzed with contemporary diagnostic criteria, they 678 might allow us to hypothesize that the phenomenon is genuinely synaesthetic (see for example 679 Kandinsky, 1912, p. 179). Factually, the dimensions of seeing and thinking are co-present in 680 ideation, but still distinguishable. For example, that horseman and rider in Kandinsky paintings 681 culturally represent the knight of the Apocalypse does not determine or induce the process of 682 sensible abstraction regarding the pattern itself (a rider, a horse), that can appear in totally 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
For Peer Review
683 different culturally environments and contexts, such as children’s drawings (Arnheim, 1969, Ch. 684 14). In fact, most of our participants were totally naïve as to Kandinsky’s (and Schönberg’s) works 685 of art.
686 Secondly, as regards the tendency to abstractionism prompted by the Jugendstil (Art Nouveau), 687 through ornamental patterns and their intrinsic symbolism, although it influenced Kandinsky's 688 formation during his years of training in Munich (Stelzer, 1964; Weiss, 1975), it is not what 689 grounded his path to abstractionism: as he observed, it was the contents of his subjective 690 experience that drove the process that led to the series of sketches for the creation of the 691 Impressions (see, for example, Gendarme, Impression 4 (1911) to Improvisations (see for
692 example, Improvisation 14 (1910) - grounded on the interplay between color, form, brightness, 693 and figural elements (lines, cuspids, heights) - to Compositions, during the years 19011-1914; see, 694 for example, the transition from Landscape with tower (1909) to Church in Murnau (1910) (see 695 Kandinsky, 1912, Ch. 8).
696 Similarly, as to the possible influence of theosophy on Kandinsky’s painting and theory (for 697 example on the concept of soul (Geist)), which is undeniable (Kandinsky, 1912, Ch. 3), it cannot 698 be considered the decisive top-down factor in Kandinsky’s production (for a different opinion see 699 Ringbom, 1966; on the diffuse interest in synaesthesia by artists of the same period see Besant 700 & Leadbeater, 1901). Comparable influences on his theory of color were exerted by the theories 701 of Goethe, Runge, Bezold, Wagner, Debussy, Fiedler, Swedenborg, Th. Lipps, and Worringer (for 702 example the idea of a total work of art, the concept of visibility, and of inner necessity, see 703 Kandinsky, 1912, Ch. 6). These theories, however, were diffused, discussed and influential 704 through the main avant-gardes of the time. The same holds for Schönberg.
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
