Theoretical Background

REPRESENTATION OF TIME

The spatial representation of time is not culturally equivalent. Space-time metaphors are highly conventional, apparently dependent upon 'time-moving' or 'ego-moving' perceptions of the passing of time (Gentner, Imai, & Boroditsky, 2002). However, all written languages must follow a particular direction on the page, and this aspect seems to be linked to the representation of time in two dimensions (Fuhrman & Boroditsky, 2010; Mitchell, 2004; Zwaan, 1965). In fact, evidence suggests that culturally specific spatial representations inform judgements about time even in non-linguistic tasks (Boroditsky, 2001, 2011). For example, Mandarin speakers appear to think of the passage of time as movement in a vertical direction (top-to-bottom), while speakers of English as a first language seem more likely to conceive of a horizontal, left-to-right movement. Similarly, Fuhrman and Boroditsky (2010) reported that Hebrew participants' spatial representation of time was consistent with direction of writing, passing in a horizontal right-to-left manner, while Zwaan (1965) noted that Dutch and Israeli participants located the 'past' on the left-hand side and right-hand side of the page respectively. The visual representation of time in timelines also tends to follow the cultural convention for written language (Mitchell, 2004). For example, Japanese comics, in which the story traditionally flows in a vertical right-to-left manner, are mirrored horizontally before translations are printed for European and American markets (Farago, 2007).

In contrast to visual art, music takes place in and through time. In musical performance, time is apportioned and manipulated; time can be said to be the universal raw material of musical process (Blacking, 1973, pp. 105-111). The decisions taken by an individual in order to create a visual representation of musical sounds must, then, be informed not only by culturally specific ontologies of music, but also by culturally specific notions of time.

MUSIC AND SHAPE

Ethnomusicologist Steven Feld conducted extensive research with the Papua New Guinean Kaluli, documenting and interpreting the metaphorical relationships that underpin aesthetic concepts essential to Kaluli song and poetics. Feld argues that Kaluli music theory—in terms of "logical patterns of symbolic material" (Feld, 1982, p. 16)—is inextricable from its function, namely "to activate and bring forth meaningful social relations through structural expression" (Feld, 1982, p. 16). His work reveals the extent to which cultural organization and behaviour influence both the conceptualization of, and discourse on, the abstraction of musical sound into symbolic patterns. Feld's work provides a reminder that the concept of music (and its theorisation) common to most empirical music research is not, in fact, a universal given.

However, specific acoustic parameters are commonly associated with certain visual metaphors, and empirical research supports various cross-modal matches—for example, pitch height is commonly associated with relative height on the vertical axis (Casasanto, Philips, & Boroditsky, 2003) and fast tempo may be associated with fast movement (Eitan & Granot, 2006). In other investigations, loudness has also been empirically associated with verticality (Eitan, Schupak, & Marks, 2010) and length (Carello, Anderson, & Kunkler-Peck, 1998). However, as Walker (1987b) and Sadek (1987) indicate, these mappings are not found cross-culturally. Using a forced-choice design, Walker (1987b) asked participants to match various auditory stimuli to one of four visual metaphors, predicting associations between pitch-height with vertical placement on an x—y axis; timbre with pattern-sign; loudness with size; and duration with length represented horizontally across the x—y axis. Reponses were collected from six different groups of participants, including four Canadian Indian groups, and two urban control groups comparing musically competent and musically naïve participants. Walker (1987b) — and a replication study with Egyptian participants by Sadek (1987) both indicated differences according to musical training and cultural background. A questionnaire study conducted by Prior (2010) also indicated that background culture may lead to different perceptions of musical shape. Küssner and Leech-Wilkinson (in press) found a link between participants' musical training and the visual representations that participants generated using a real-time drawing paradigm.

The latter study aside, little research to date has examined what happens when adult participants are given free rein to create visual representations of musical sounds. Undertaking just this task, Tan and Kelly (2004) used whole musical compositions as stimuli, requesting musically trained and untrained US college participants to "make any marks" in order to represent the sounds visually. The musically trained participants demonstrated a tendency to create representations aligned on an x—y axis (henceforth referred to as Cartesian), representing time in a horizontal, left-to-right fashion, and depicting aspects of the musical surface (primarily pitch) on the vertical axis, in accordance with findings described elsewhere (Athanasopoulos & Moran, 2012; Athanasopoulos, Moran, & Frith, 2011). However, Tan and Kelly (2004) also found that musically untrained participants tended to provide pictorial representations through images or pictures telling a story. Küssner (2013) obtained comparable results from British participants.

Similar to the method of Tan and Kelly (2004), Bamberger (2005) and Barrett (2005) designed studies that permitted their participants to create notational systems of their own "invention" without being restricted to the pitch versus time model which most organised notational systems follow. In these two investigations, participants (mainly children) were asked to provide a means of invented notation for a familiar tune so that one of their peers would be able to perform the tune, without further guidance from the researcher. This resulted in a number of participants mimicking familiar notational models, while others attempted to develop executive models of notation that fitted the task. These tended to prioritise aesthetic concerns: for example, participants included performance guidelines for playing techniques, but disregarded specific note duration values. In a separate free-drawing investigation with Japanese children as participants (Adachi, 1997), a considerable proportion of responses included onomatopoeia and linguistic script to depict information about attack rate.

Beyond the investigations mentioned above, other studies to have deployed free representation techniques have focused on questions related to cognitive development, examining children's responses (Reybrouck, Verschaffel, & Lauwerier 2009; Verschaffel, Reybrouck, Janssens, & Van Dooren, 2010). As far as we are aware, no existing cross-cultural research has used a free-drawing task with adult musicians. It seems inevitable that cultural difference will result in varied depictions of time and shape in response to musical sound stimuli, but in what ways? What is the effect of cultural background on an individual's two-dimensional representation of musical sound?

Research Design

Using a quasi-experimental design, three participant groups heard twelve musical stimuli that varied in pitch contour (Up, Down, Peak or Valley), as detailed in the Method section below, and drew responses in which they depicted the presented sound events.

The groups were as well-matched as was possible in the circumstances of remote fieldwork, taking account of approximate number, experience as a performing musician, and gender split. The age of participants varied, but all participants were adults in this study and so acted as mature representatives of their particular musical tradition. The very concept of a dictation exercise—which in some ways describes the task—is very common when learning western music and WSN, but it does not exist in Japanese traditional music, nor for the BenaBena. The groups cannot be considered to be matched with their counterparts in terms of experience in representing sound visually. However, the open-ended nature of the task was designed to compensate for some of these limitations, the payback being the richness and diversity of the data.

DEVELOPMENT OF THE SOUND STIMULI

The sound stimuli were developed according to certain constraints, in order that the same stimuli could reasonably be used for all three groups. Auditory stimuli using unfamiliar or 'unmusical' timbres or patterns were likely to disrupt participants' responses. A synthesized flute sound was selected to represent pitch variations, as flutes in some form are common to all three musical cultures.

The musical examples consisted of very basic melodic patterns. Papua New Guinean music from the Highland provinces does not follow elaborate melodic patterns other than the pentatonic scale, favouring rhythmical complexity instead. Thus the pitch relations of the melodic examples were limited to fourths, fifths and octaves and used simple contours: in our terms, either rising (Up), rising-falling (Peak), falling-rising (Valley), or falling (Down), as illustrated in Figure 1.

The stimuli were developed using Sibelius 6 software (Avid, 2009), exported as MIDI files at a tempo of 60 beats per minute, and produced using Digidesign Pro Tools 8 (DigiDesign, 2009). They were recorded in MP3 format and replayed to the participants with a Samsung K5 MP3 player through the in-built (slide-out) stereo speakers. Headphones were not used at all for any group, since they were likely to be an obstacle to participation for the BenaBena cohort, who view all music-related activities as communal and would not be familiar with using them. The participants were seated approximately one metre from the sound source, and the stimuli were reproduced at 52 dB SPL. The British and Japanese groups participated indoors, but by necessity the study took place outdoors with the BenaBena.

Fig. 1. Test stimuli consisted of 12 sound sequences representing four different pitch contours: Up, Down, Peak, and Valley. The sequences were presented in the same fixed, quasi-random order to participants in all three groups.

Method

PARTICIPANTS

The first group (Group A) consisted of 25 musicians of British nationality and cultural background (mean age = 23.5 years; SD = 4.2 years; range: 19-37 years, 10 males, 15 females; 21 right-handed, 4 left-handed). The mean age for starting a musical instrument was 6.7 years (SD = 1.7 years, r: 5-11 years), and mean duration of performing a musical instrument was 15.7 years (SD = 4.4 years, r: 11-32 years). All participants were acquainted with WSN. 28% were also acquainted with guitar tablature, 20% with jazz chord notation and 12% had performed or composed music using graphic scores. One participant (4%) also knew a traditional notational system used to transcribe Irish folk melodies.

The second group (Group B) consisted of 24 Japanese musicians with minimal or no knowledge of WSN (mean age = 47.6; SD = 26.4 years; r: 18-87 years; 11 males, 13 females; 23 right-handed, 1 left-handed). The mean age for starting a musical instrument was 18.6 years (SD=12.3, r: 3-40 years), while the mean duration of performing a musical instrument was 32.6 years (SD = 18.5 years, r: 8-63 years). All participants were acquainted with a form of JTN, while 28% claimed to be in a position to recognize WSN as a form of notation when they saw it but were unable to use it, and one participant (4%) was aware of the existence of graphic scores, though he had never used one in performance. It should be noted that it is practically impossible to find Japanese musicians with no exposure at all to WSN.

The third group (Group C) consisted of BenaBena tribesmen and women of the Eastern highlands region of Papua New Guinea, unfamiliar with any literary or notational script. 26 musicians (mean age by estimation = 57.1 years; SD = 10.5 years; r: 35-80 years; 15 males, 11 females; 26 right-handed, none left-handed) participated in the investigation. Handedness of the participants was established by asking them with which hand they used common farming tools amongst the Highlands of Papua New Guinea, such as machetes or a cangkul (hoe). The mean age for starting a musical instrument (from self-estimated responses) was 15.5 years (SD = 4.2 years, r: 10-25 years); while the mean age of performing a musical instrument was 42.2 years (SD = 10.5 years, r: 23-62 years). The numbers may not be accurate; performers might not have been actively performing music throughout this span. Participants provided responses for their engagement in music-making and participation in "sing-sings" and traditional community ceremonies. All but one participant reported having taken part in sing-sings since they were very young children. Recruited participants were from the BenaBena tribe and came from six hamlets (Keni, Logo, Sifu, Opeks, Siopeks, Moweto). It should be noted that music among the BenaBena tribe is a highly communal activity, which does not separate active performers from a non-participating audience. Some individuals are regarded as the best singers or dancers, and may be called out to perform in group activities involving music and dance, but this does not exclude others from participating. Therefore, all BenaBena are considered musicians for the purposes of this study.

The first author conducted this fieldwork and worked with Groups B and C with the assistance of local translators. For Group B, translators were recruited with the assistance of the Kyoto City University of the Arts and the Tokyo Geijutsu Daigaku. For Group C, the local schoolteacher, one tribe member who was a college student at the University of Goroka, and the first author's host, were all proficient in English and assisted with translations. For more detail, see Athanasopoulos (2013).

PROCEDURE

Participants were exposed to 20 trials, of which 12 varied according to the four pitch contour categories (Up, Down, Peak, and Valley: see Figure 1). 4 The remaining eight stimuli featured combinations of the four categories presented here. Each sound event had a total duration of four seconds, and was repeated after a four-second pause before the next trial. Participants could start drawing after the first stimulus presentation. The overall time-limit to provide a visual response to the sound event was 16 seconds. If participants were not able to provide a response, they proceeded to the next trial after this time elapsed and the next trial began. In preparation, the order of presentation of all 20 stimuli was initially randomised; this order was then held constant in its presentation to participants.

Participants were asked to represent the sound on paper in such a way that if another member of their community saw their marks they should be able to connect them with the sound. Responses were drawn on A4 graph paper using ball-point pens. Before data collection began, participants were offered up to four trial runs using three sound stimuli drawn randomly from the database. On average, participants made use of two of the potential four trials.

The BenaBena participants were not accustomed to the particular fine-motor skills associated with holding a pen and providing responses on paper. In order to communicate the concept behind the task it was necessary to draw a comparison between the acts of etching (with which all individuals were familiar) and drawing. As preparation for the study, the group were invited to draw any image from their surrounding environment on plain white paper using thick coloured markers, and then with ball-point pens. After the free-drawing investigation took place, all participants were interviewed about their experience of the task so as to record information that could contribute to the interpretation of their drawn responses.

The participants' responses were classified according to three categories, based on their method of internal organisation and reference, and on the apparent representation of events in time (i.e. use of Cartesian pitch-versus-time axes) (Athanasopoulos, Moran, & Frith, 2011; Küssner & Leech-Wilkinson, in press; Tan & Kelly, 2004):

• Symbolic Cartesian (SC). Reference to sound events through abstract symbols. Pitch contour and time represented spatially on orthogonal axes. In this system, one symbol relates to another, and need not signify anything beyond this internal system. For example, note heads in WCN are abstract, symbolic representations of the sound event.
• Iconic Cartesian (IC). Reference to sound events through drawings that attempt to imitate the event in some respect as stand-alone icons. As opposed to abstract symbols, icons (in this classification) attempt to indicate directly and analogously some aspect of the sound event for which they stand. Icons and time represented spatially on orthogonal axes.
• Iconic (I). Reference to sound events through drawings that attempt to imitate the event in some respect as stand-alone icons.

Further to these classifications, responses were separated according to their method of representing events in time:

• Left-to-Right (L-R), imitating WSN and script.
• Top-to-Bottom (T-B), imitating a majority of JTN systems and script.
• Neither L-R nor T-B.

Classification was carried out by the first author in consultation with two independent raters. Where no majority decision was reached, the first author's decision was taken forward. The classification terms were not disclosed to participants, who drew their responses freely without such instruction.

Results

Responses were classified according to the apparent internal organisation as described above, as either SC, IC or I. Cartesian responses were further classified as either L-R (left to right), T-B (top to bottom), or as neither L-R nor T-B.

The results for each individual were examined to find the degree of reliability in their representational approach, both within contour-type and across all the stimuli. Results were then compared across groups. The classifications are illustrated below in Figures 2, 3 and 4, showing examples of each category.

Fig. 2. Examples of three different Symbolic Cartesian (SC) systems: i. Drawn by a participant from Group A (British); ii, iii. Drawn by participants from Group B (Japanese). For i and ii, time is represented horizontally L-R, and pitch variation is shown through vertical variation. No elements of WSN have been used. Example iii represents the passing of time using a vertical axis, T-B. Pitch variations are represented by the inclinations of the strokes.

Fig. 3. Examples of Iconic Cartesian (IC) representation by a British participant from Group A (iv), and by a Japanese participant (Group B) (v). Although the drawings follow Cartesian representation (L-R), they are not internally consistent, and are therefore classed here as Iconic Cartesian (IC), not Symbolic Cartesian (SC).

Fig. 4. This example (vi) illustrates the category of Iconic (I). The BenaBena participant has used this single image to represent all pitch stimuli (Up, Down, Peak and Valley). The depiction of the sound events does not involve a timeline. According to the participant at the follow-up interview, all of the flute sounds were represented by the lines drawn above.

RELIABILITY OF RESPONSES

According to the prescribed categories, individuals were entirely consistent in their own approach to the depiction of the sound events, both within and between contour types. For all groups (British, Japanese and BenaBena), individual participants maintained one method of representation throughout: see Table 1.

Table 1. Individual participants' preferred style of representation for the four different types of pitch contour: Symbolic Cartesian (SC), Iconic Cartesian (IC) or Iconic (I). NB: Responses were entirely congruent for the three different stimuli used within each contour group.

All but one of the British participants (24 out of 25) and the majority of Japanese participants (19 out of 24) used a consistent symbolic mode of representation of the pitch stimuli, presenting abstract symbols on a timeline. Of the British participants, one-quarter employed WSN in some way: three participants provided responses entirely through WSN, and a further three incorporated some WSN elements. Two of the British participants used written text directions in addition to their drawn response.

One Japanese participant apparently experienced a dilemma with a small number of his responses, which he subsequently crossed out. Had these been included in the analysis, these would have been classed as iconic. The reason he gave for changing his mind was that since the directives were to represent the sound so that if another member of their community saw them they should be able to connect them with the sound, his responses would not be clear.

The BenaBena participants provided the largest variety across the three categories of response. The majority of responses (19 out of 26) were classified as iconic, using stand-alone icons depicting sound events and no obvious (linear) portrayal of time. However, four responses were classed as consistently symbolic and used a timeline, and three were classed as consistently iconic and also used a timeline. Drawing on the post-test interview data, the nineteen iconic responses could be further separated into two categories. Twelve responses (slightly less than half of all the BenaBena responses) showed some internal consistency as a system: in terms of organisation, participants attempted to draw iconic representations of the sound event's source (a flute), rather than the sound event itself. They deemed pitch variations to be unimportant; rather, according to self-reports from the post-task interviews, they focused on elements of timbre or perceived variations in loudness (though we attempted to minimize the latter by keeping the amplitude constant (Athanasopoulos, 2013, p. 238). The remaining seven iconic respondents (approximately one-quarter of all BenaBena participants) did not appear to demonstrate this level of organisation.

DIRECTION OF TIME IN THE RESPONSES

All Cartesian (SC or IC) responses are set out in Table 2 below. Of the British participants (Group A), all used an SC approach to represent events in time. The unanimously preferred timeline was horizontal L-R. Regarding the traditional Japanese group (Group B), two participants who provided SC responses opted for a vertical representation of time. One Japanese participant provided responses that did not follow Cartesian representation.

Table 2. Directionality of time as represented by SC and IC responses by group. Direction is either Left-to-Right (L-R) or Top-to-Bottom (T-B).

Finally, the BenaBena (Group C) presented the majority of their responses through iconicity and not symbolism, and without apparently depicting the sound events as occurring in time. However, seven respondents did deploy a Cartesian system and all chose to use a horizontal timeline.

Further findings

Other findings beyond these predictions revealed that all participants, regardless of cultural background, literacy or type of musical training, were able to provide coherent responses with varying levels of organisation in a free-drawing investigation; and regardless of group, all participants who completed these tasks were highly consistent in their manner of depicting the sound events and variations.

All musicians were able to provide invented notational systems in order to depict sound. This suggests that everyone has the ability to associate musical sounds with some form of visual representation. Although many participants were sceptical about the value of the task presented to them, all (with three exceptions due to mistranslation) were able to provide graphic representations of sound, even where the idea to represent sound visually was an alien concept, as was the case with the BenaBena.

The resulting representations are surprisingly similar between groups of different cultural backgrounds, due to the wide adoption of the SC method of representation. A large number of the entire participant population (regardless of origin or type of musical training) tended to represent sound in a linear, left-to-right axial representation resembling analogue notational systems, with time located horizontally. The linear representation of music may have its roots in literacy, since the latter provides participants with a timeline of reference and an axis on which to put responses. However, this does not account for the small number of BenaBena who used the IC mode of representation, depicting the passage of time horizontally. One could argue that this minority may have been responding to what they thought that the Western investigator perceived as 'correct', mimicking a script that they may have seen elsewhere (perhaps even by watching the hands of the investigator as he took occasional notes). An alternative theory may examine the idea of cross-cultural tendencies towards the linear representation of musical sounds. Further investigation is required in this area. From the results of this study, then, the conclusion should be drawn that the visual representation of music is surprisingly consistent across cultures, and that Cartesian representation is the dominant mode.

However, we return to the notable proportion of responses which were iconic in nature and did not make use of a linear timeline. The majority of participants who did not follow the left-to-right horizontal path of representation were the non-literate BenaBena, who opted for an abstract pictorial method which did not depict time. Additionally, we have the responses of the two Japanese master musicians unfamiliar with WSN, whose interviews suggest that they are ideologically opposed to what they consider a threatening expansion of western culture (Athanasopoulos, 2013, pp. 167-169). This suggests that sociological factors may well influence—or overrule—participants' first responses to the task. The responses and interview with the BenaBena thus provide a unique insight into the opposing perspective, with the example of a non-literate community's first encounter with a novel form of musical communication. (This was a benign encounter whose potential consequences the researcher considered and attempted to mitigate (see Ethical Considerations in Athanasopoulos, 2013)).

Responses from the United Kingdom and Japan classified as iconic were idiosyncratic in nature and represent a minority in relation to the number of symbolic responses within the same groups. The BenaBena's pictorial approach was more predictable. We attribute the consistency of their responses to their approach to the task. First, after the initial introduction of the idea to the BenaBena, the participants would discuss the notion of sound representation among themselves trying to reach some sort of consensus before the task took place. Though they were not permitted to look at each other's responses, participants would openly discuss their answers and debate the appropriateness of their responses. This mode of collaborative engagement was as open to Groups A and B as it was to C, but it only took place—it only seemed to be required—for the BenaBena.

Subsequent interviews revealed that the participants were not attempting to indicate variations within the stimuli, but rather attempted to adhere strictly to the instruction given to them: "Produce responses so that if another member of the community saw them they would be able to link them to the sound events". Thus, they completed the task for its communicative function. Usually after discussion with their peers, they invented—collaboratively—an iconic method of representation appropriate for the task. Participants attempted to indicate that a flute was playing through iconic representation of the sound, without attempting to indicate specific variations between the sound events (see Figure 4). When questioned if all the sound events were the same (regardless of contour variation), the participant responded, "Yes, it is still a flute playing" (Athanasopoulos, 2013, p. 239). When asked about perceived differences regarding the values of specific sound events that followed a pattern (up, down, peak or valley) the participant replied by asking why these differences should matter, since it is always a flute playing, regardless of these variations (Athanasopoulos, 2013, p. 239). This particular style of response was, in fact, replicated throughout further sections of the task (attack rate and duration), whose results are not discussed in this particular article. For example, in order to depict variations in attack rate, participants deployed circles to stand for drums, while lines next to or on the circles indicated the action of hitting the drum, or the number of strokes (Athanasopoulos & Moran, 2012). Small variations in the length of lines next to the circles indicated perceived variations in loudness.