The expressive role of non-technical movements

Although referred to as ancillary movements (Cadoz & Wanderley, 2000; Jensenius et al., 2010), due to the fact that they are non-sound-producing or -facilitating, these non-technical motions are not of little importance, as the term ancillary implies. As observers of a live performance, we derive meaning from the audiovisual combination of technical and non-technical performance movements. When considering the meaning conveyed audibly by the performer alongside these non-sound-producing movements and the audiovisual experience of the observer, these movements become anything but ancillary. It has been suggested that changes in non-technical body motion aid communication with co-performers (Keller & Appel, 2010; King & Ginsborg; Goebl & Palmer 2009; Clayton, 2007), are involved in memory storage and retrieval (Munoz, 2007; Godøy, 2010), and can display intended expressive (Davidson, 1993; Thompson & Luck, 2008; Clarke & Davidson, 1998) and structural qualities of a performance (Vines 2006), influencing audience interpretations of these expressed qualities (Davidson, 1994; Broughton & Stevens, 2006; Dahl & Friberg, 2007; Nusseck & Wanderley, 2009). As a result, rather than being considered as ancillary, such performance motions will be referred to here as non-technical (not related to the production or modulation of sound 2) or concurrent gestures, describing observed body motion that occurs alongside technical and facilitating movements that, although non-sound-affecting, may be considered meaningful and repetitive.

Non-technical performance movements, while notably reduced, have been seen to persist when a performer alters their intended expressivity from exaggerated to restricted or deadpan (Davidson, 1993; Thompson & Luck, 2008). For pianists and violinists Davidson (1993) showed that the level of intended expressivity is related to the level of a performer's non-technical body motion. Performers altered the amount of body movement according to whether their performance was intended to be deadpan, expressive or exaggerated. Although the detailed relationship of performance motion and structural intention was not studied, Davidson's research indicates that it may be the magnitude and corporeal location of the motion that conveys expressivity, and not the motions' shape or regularity of occurrence.

There have been various suggestions as to the cause and function of non-technical performance gestures. Patterns of motion have been seen to persist throughout repeated performances by the same musician (Wanderley, 2002), with individuals finding playing uncomfortable when asked to restrain or restrict these movements (Wanderley, Vines, Middleton, McKay, & Hatch, 2005). Wanderley and colleagues also noted that some movements corresponding to structural events within the musical excerpts were similar among different performers while other movements remained particular to an individual (Wanderley, 2002; Wanderley et al., 2005). It is not clear from these studies, however, whether there were causal relationships between structural events evoking patterns of motion (regardless of whether the location in the body of these movements and their shapes were generalisable or individualised across performers) and a performer's individual performance style, interpretation or expertise.

These non-technical or concurrent movements are often repeated and appear meaningful not only to performers but also to their audience and co-performers (Dahl et al., 2010; Jensenius et al., 2010; Platz & Kopiez, 2012). Movement patterns have been seen to refer to more than simply the degree of intended emotion and expressivity persisting when little to no expression is intended, suggesting that the presence of such movements may be related to some other feature of music performance, such as the expression of structural features, memory retrieval or experience and expertise. Studies of relationships between non-technical performance movements and the structural features of scores have yielded conflicting results, identifying differing body regions as central to the movement observed (Clarke & Davidson, 1998; Davidson, 2007). Motions of the head and torso have been identified as meaningful in piano performance, relating to meter and rhythm as well as structural intent. Davidson (2007) indicated that movement shapes of the head and upper torso were recurrent for expressive locations in the score previously identified as salient by six independent judges. Davidson describes these repeated motion shapes as arising from the performer's expressive 'centre of moment', the physical core from which expressive information about the performance is extended and dispersed to other regions of the body (Davidson, 2007, p. 386). This is based upon theories of dispersal of locomotion by Cutting, Proffitt, and Kozlowski (1978), who demonstrated that any part of the motion cycle of walkers could reveal similar information regarding the target expression (in these instances the recognition of a walker's identity, whereas Davidson applies this theory to levels of expression within performance). In examining the relationship of head motion to phrasing structure, Camurri (2004) found that the temporal patterning of a solo pianist's head motion did not match the two-bar phrasing structure of the Scriabin Étude they were performing. By contrast, Thompson & Luck (2008, 2011) found that the head movements of their pianists did appear to follow the phrasing structure of the pieces performed. Similar findings have been reported for clarinettists, with performers' non-sound related movements, such as bending at the knees and waist and tilting or circling of the clarinet bell, appearing to correspond to features within the music's structure (Wanderley et al., 2005). Due to a lack of comparative repeated measures within analyses or between methods the extent to which such motions are responses to musical structure or are matters of individual performance style remains unclear.

Investigations into the communicative role of these non-technical movements have indicated that not only can an audience infer the intended mood and character of a performance and make social judgements about a performer's expertise, but they are also capable of inferring complex structural relations and segmentations from performance motion alone (Platz & Kopiez, 2012). This implies a relationship between the perception of structural features and their expression by a performer (Palmer, 1989), although the influence of interpretation and intended expression has yet to be considered with regard to visible non-technical performance motion. Palmer (1989) demonstrated that performers' specific interpretations of structure effected audible characteristics of their performance, such as melody lead and expressive timing. This is extended by Rink's (2002) assertion that interpretation of a score influences expressed performance style. We propose that there will be a similar effect to those noted for the audible aspect of performance evident within a performer's non-technical performance motions, particularly with respect to phrase structure.

Identification of performance gesture

Following theoretical approaches to gesture categorisation in speech and social interaction, such as those by Kendon (2004), McNeill (2005) and Goldin-Meadow (1999), several proposals have been made for the detection and identification of instrumentalists' performance gestures, leading to taxonomies of the nuances exhibited by individual performers (Davidson, 2007). These describe actions such as nodding and shaking the head, leaning and bending the body, flicking and circling of the wrists and bending of the knees at salient events within a performance. However, the precise nature and shape of these movements has not previously been fully examined, nor has any consistency been observed in this regard between different performers or performances.

Investigations into particular music-related shapes have concentrated on using Becking curves to analyse patterns of movements made by performers, audience members and conductors. First analysed and published by Sievers (1924), the movements made by highly experienced conductors using lighted batons in response to various classical works were classified in qualitative, observational studies (Sievers, 1924; Repp, 1993). The pattern traced by the baton connects "points of metric gravity that vary in relative weight", which were thought to represent a composers' written intentions. These notated intentions are represented in the conductors' movements, projected through performers' audible expressions, and would then be reflected within a listener's music-induced movement patterns (Repp, 1993). 3 Van Noorden (2010) presents a review of research offering a more quantitative approach to the recording and representation of performance gestures, including those using samba dancers (Nevada & Leman, 2007) and studies of participants' ability to express synchrony in time to a marching fragment using extended joysticks (Styns & Van Noorden, 2006). However, gesture patterns are identified from visual observation and subjective analysis of audiovisual-recorded stimuli, and the authors themselves note a need to refine methodology in order to qualify results. That said, the patterns arising from the samba dancers' hand and foot motion imply that the dancers embody different metrical levels of the music within different body regions. Although a useful tool for the investigation of music-induced movement, the assumptions of predefined motion patterns may sometimes mask the retrieval of gestures as they are produced in real time. To counter this, the current paper concentrates on mapping the actual route of performance motion to reveal underlying patterns that can later be applied to gesture-identification and segmentation models.

Applications of segmentation models for the detection and parsing of performance motions have recently been proposed by Desmet, Nijs, Demey, Lesaffre, Martens, and Leman (2012), Caramiaux, Wanderley, and Bevilacqua (2012), and Rasamimanana (2012) to overcome a lack of recognised types and shapes of gestures within music research, with a view to computational modelling and categorisation of performance-gesture types. In the examination of a solo clarinettist's performance, Desmet and colleagues (2012) assess the relationship between the performer's expressive body movements and certain structural features within the score identified by the performer as being focal points of intended expression. Using the performance of a musical excerpt unfamiliar to the performer, and through the temporal modelling of intended targets and motion acceleration, Desmet et al. (2012) found that a relation did exist between the cognitive focus of their performer on musical targets and exhibited body movement.

Caramiaux et al. (2012) apply segmentational hidden markov models (SHMM) to the angular trajectories of four clarinettists' bell movement during repeated performances of Brahms's First Clarinet Sonata (Op.120). This was to determine the shapes of performance motions in relation to different time structures within the piece. Their study is based upon the earlier assertion by Wanderley and colleagues (Wanderley, 2002; Wanderley et al., 2005) that clarinettists' expressive intentions may be characterised by the degree of bell movement within a performance. Caramiaux and colleagues' model assumes a dictionary of primitive base shapes, with their analysis determining that performers exhibited a tendency to form circular shapes with the clarinet bell relative to structural segmentations. Sequences of shorter motions were seen to occur at phrasing boundaries, with longer, smoother base-shapes being formed throughout phrases. Patterns of motion were seen to be repeated by performers across subsequent performances, with the direction of circular motion varying between performers while the timing and length of circular gestures appeared structurally consistent. Although the dictionary of base shapes offers an exhaustive range of motion possibilities, the proposed model appears to consider the direction of motion between beginning and end positions of movement patterns as equivalent (parameters appear limited in the ABBA vector equivalence). As a result the identified shapes of motion do not appear to consider the actual route of the motion vector, potentially masking significant repetitions and trajectories.

Rasamimanana (2012) proposes a comprehensive approach to the modelling and parsing of violinists' continuous expressive gestures during performance based upon hierarchical hidden markov models (HHMM). This model is based upon the physiological, experiential and instrumental constraints placed upon a performer in order to determine the shape of exhibited performance gestures. Rasamimanana describes the motions as falling within a performer's 'space of possibilities' (SoP). The study focuses only on those actions technically necessary for the production of sound, with the proviso that these motions are central to definitions of actions in interaction with electronic sounds. The proposal to use actual observed motion patterns for the modelling and analysis of performance gesture, rather than a set of assumptions concerning predetermined base shapes, appears highly successful, with model parameters capable of adjusting to fit isolated motion patterns of similar shapes but of different forms within the input performance. As a result the HHMM seems to identify instances of repeated gesture patterns within a particular performance by a given performer quite accurately.

Despite such proposals for the segmentation and parsing of performance gestures, no studies to date have quantifiably linked any particular motion patterns to intended or notated structural and emotive expressions of meaning across multiple music performances (Platz & Kopiez, 2012). Before a model may be effectively applied to different motion sequences, it is important to discover the form such gestural patterns may take so as to remain objective in gesture detection and analysis, reducing the risk of empirically over- or under-estimating the relevance and size of meaningful patterns. It is the premise of this article that before performance gestures may be successfully and consistently computationally detected, they must first be identified over multiple instances, performers and performances as being repeatedly produced and conceptually meaningful.

The authors have previously identified motion patterns as recurrent and idiosyncratic within nine highly skilled performers' individual motion profiles, generated through principal component analysis (PCA) of overall upper-body and head performance motions (MacRitchie, Buck & Bailey, 2013). A motion profile, as defined in this previous paper, is a weighted combination of the first few principal components such that the total variance explained is above 90%. These motion profiles are taken as a 2D representation of the movement that best describes the overall patterns of motion described over the 28 markers in three-dimensional space. The motion profiles produced across the group of pianists indicated globalised variations of motion size and direction that fluctuate repeatedly and consistently in accordance with performed phrasing structure for Chopin's Prelude in A major. This article extends previous research, examining the performances and interpretations of Chopin's Prelude in B minor and comparing these back to performances of the stricter rhythmic structure identified by performers within the A major Prelude.

It is hypothesised that the nature and form of these shapes, traced through non-technical, concurrent performance motions, will be related to each performer's intended expression of structural hierarchy as they interpret it within the compositions. The following sections demonstrate, through the use of polar coordinate projections, the visible shapes traced by performers' head and body motions around their central gravitational pole. These are presented as spherical analyses of motion. Repeated patterns of movements corresponding to phrasing structure can be termed as phrasing gestures as they have been seen to be meaningful both in their temporal generation and consistency as well as in their ability to convey meaning to an observing audience (for discussion see Buck, 2013). The patterns elucidated by means of PCA and spherical analyses will be assessed here with respect to the hypothesis that structural performance gestures may be highly individualised and potentially dependent on a performer's personalised style and interpretation.

Participants, Apparatus & Design

Nine highly trained solo pianists, each with a minimum of 10 years performance experience, were paid a one-off sum of £25 for their participation. Each pianist performed two selected Chopin Preludes (Op. 28, Nos. 7 & 6) on a Roland RD-150 weighted keyboard 4 and was recorded using a 12-camera infrared Vicon 3D-motion capture system. 5 Twenty-four retro-reflective markers were attached to performers' upper bodies fitting the model described by Cutti, Paolini, Troncossi, Cappello, and Davalli (2005), to obtain detailed upper-body performance motion recordings. Four additional markers were positioned on the head to include these movements within the data (see Figure 1). Scores for the selected Preludes were obtained from mutopiaproject.org and provided to pianists one month prior to recording. This ensured that all performers began with the same notated directions regarding tempo and dynamic instruction. Score examples are provided in Appendix A.

Performers were instructed to play the selected pieces as they would in a typical live performance scenario. No other instructions regarding performance style were issued to any of the participants. Performers were applauded into the recording room when ready so as to maintain the feeling of a live performance setting. The two selected pieces were played as part of a three-piece programme (comprising Chopin's Preludes in A major, B minor and E minor, Op. 28, Nos. 7, 6, & 4 respectively) in the order of their choosing. Those pianists who wished to perform with the aid of the scores were able to do so, as all nine performers were capable of performing the pieces from memory and so were considered to have formed concrete mental representations of their interpretations within their performances.

Fig. 1. Marker placement model for upper-body 3D-motion capture. Trajectories of all 28 markers, depicted (represented by circular points positioned at key musculature and joint locations across the performers' torso, head and arms, were used for the calculation of PCA motion profiles. Spherical analyses demonstrate movement patterns of the head and torso regions. These are depicted centrally in the above figure and represented in orange/a lighter hue. Motion of the clavicle marker was also examined to consider respiratory effects. This marker (CLAV) is positioned at the top of the sternum, just below the centre of the clavicle bones of each performer.

Selected Preludes

Examples of the full score for Chopin's Prelude in B minor issued to participating pianists and the opening phrase of the Prelude in A major are provided in the Appendix A. Performers' structural interpretations of phrasing were in agreement for the eight two-bar phrases of the A major Prelude and for the opening five phrases of the B minor Prelude, although some performers did group these phrases into different higher-order structures. Also depicted in Appendix A, Figure b, are indicators representing phrasing boundaries as taken from theoretical analyses. All reference to phrasing segmentation for the two preludes will relate to these markings as depicted unless stated as otherwise within the text. For more detail on the precise nature of phrasing for the A major Prelude see MacRitchie et al. (2013).

Chopin's Prelude in A major was selected for performance analysis to discern whether performers do make repeated phrasing gestures across analogous phrases (shown previously in MacRitchie et al., 2013) and to identify the shapes of any repeated gestures or movement patterns observed. Chopin's Prelude in B minor was selected to further examine movement patterns across the two pieces by multiple performers. This extends analyses of the A major Prelude performances to consider movements generated during phrases varying not only in structural importance but also in duration, rhythm, pitch and harmony.

The B minor Prelude begins with a two-bar motif that is repeated several times throughout the 26-bar piece. The first eight bars of the piece can be considered as the first section of the piece (or the antecedent phrase —Dodson, 2011), which includes repetitions of this initial two-bar motif, altering in pitch, with the second repetition extending into a longer phrase duration. Of note is the entry in the second half of this longer phrase of a right-hand (RH) auxiliary melody (Rink, 2012). The second section of the piece (or the consequent phrase, as marked out by Dodson, 2011) begins with a strict repetition of the opening material, the consequent repetition here changing in harmony to bring us towards the C major modulation that stretches until the end of bar 14. This is considered by Dodson (2011) to constitute a written out fermata. At bar 15 we notice the re-entry of the auxiliary melody previously heard in the RH at bar 7. This time the melody continues for 4 bars ending on a G major chord. The auxiliary melody is repeated again in bars 19-22, this time finishing on the tonic, with the last 4 bars acting as a small coda. When examining different first editions of this piece (available via www.ocve.org.uk ) one finds differences in the phrase markings at bars 15-19. In the French and English first editions, along with a select number of later editions, bar 15 is phrased as being separate from the next three bars. However, when this material is repeated in bars 19-22 it is marked as one four-bar phrase. In other editions, such as the German first edition and the score provided to the participating pianists, these two sets of four bars (15-18, 19-22) are phrased identically, as two four-bar phrases (see Appendix A, Figure b).

Post-Performance Interviews

Each pianist was interviewed after their performances with respect to their music-analytical interpretations of the selected pieces and their views regarding the role of movement within music performance. Performers were presented with the audio-visual recordings of their own performances and ask to mark on the scores their intentions with regard to structural segmentation of phrasing. A short semi-structured interview was conducted to ask performers whether they viewed movement as having a general purpose for any or all performers and were then asked to describe any particular movements they themselves had made and whether these were intended to convey particular information, such as phrasing, or to facilitate or emphasise audible aspects of their performance.

Fig. 2. A depiction of how spherical, or polar, coordinates translate into traditional Cartesian 3-dimensional space. The asterisk denotes the location of the coordinate, with rho (typically 'ρ' but also represented as 'r' above in Fig. 2a) representing the magnitude of vector being examined, in this case motion. Theta (θ) and phi (φ) then describe the angulature of the vector relative to its original position or central gravitational pole. Theta describes the variation of vertical angle within the z-dimension, for example referring to the tilt or inclination of a performer's torso or head. Phi then describes the rotation of the object within the horizontal plane, also referred to as the azimuth angle. (Images have been reproduced from Wikimedia Commons under the public domain licence use. 2a: Andeggs / Wikimedia Commons / Public Domain. 2b: Inductiveload / Wikimedia Commons / Public Domain).

Spherical analyses were conducted for all nine pianists' performances of the A major Prelude to identify whether performers make similar use of gesture shapes and patterns when they play a score. Motion coordinates were normalised so that Cartesian (x,y,z) (0,0,0) represented the coordinates at the start time of each performance. Each ordinate was then normalised to the range of its own axis, offsetting the data so that point 0 reflects the centre of motion for each performer. All data coordinates maintained their original distance from one another and the shape of motion was retained. The result, therefore, is to extend the event horizon to reveal the true shape of the motion in spherical space (as depicted in Figures 2a & b). These are compared to the motion profiles of MacRitchie et al. (2013) to further explore the comprehensive and integrated patterning of performance motion made by these performers. In doing so we explore the local movement of these regions in relation to the global movement patterns of the upper-body, examining the interdependence of the 28 measured markers and their 3-dimensional trajectories. Body movements are then assessed for performances of the B minor Prelude to provide deeper investigation into the effects of varying compositional structure and complexity, and into the influence individual interpretations of structure have upon each musician's performance motion.

Identifying Motion Patterns: A major Prelude

SPHERICAL ANALYSES:

Having found no specific leading markers in the PCA motion profiles, performers' head and torso movements were examined relative to phrasing to investigate claims within the literature that these regions hold information specifically relating to rhythm and structural segmentation. Spherical (or polar) coordinates referencing the changing locations of markers within these regions were calculated over time to show performers' spherical performance motion patterns relative to their individual central gravitational poles. This was achieved by normalising the Cartesian data points within each of the ordinates' dimensional ranges and calculating spherical coordinates, rho, theta and phi, using the following calculations (see Figures 2a & 2b):

Rho = ℘(x² + ^y2 + ^z2)
Theta = arccos(z - r)
Phi = arctan(y /x)

By transforming the coordinate space from Cartesian (x,y,z) to Spherical (r,θ,φ) it is often possible to reveal information that may otherwise be hidden beyond the event horizon. One of the novel benefits of using spherical coordinates to map performers' gestural movements is that it becomes possible to visualise the precise shapes of motion that were previously masked in Cartesian time-space. This enables recurring patterns of motion to become clear. The mapping of spherical coordinates relative to Cartesian 3-dimensional space is presented in Figure 2.

Figure 3 demonstrates the spherical motion patterns for each of the nine performers' head motion throughout their A major Prelude performances. For brevity, spherical plots for three exemplar performers (Performers 2, 3 & 9 respectively) are presented in Figure 4, depicting the changing shapes of embodied structure across the head, clavicle (or upper sternum - see Fig. 1) and torso movements. While the progression of time itself is not indicated in the spherical mapping, the plots are intended to be an illustration of the shapes of repeated patterns arising within motion that have been identified through the motion profiles. 8 By comparing the patterns produced through spherical analyses (Figures 3 & 4) with the findings of the PCA motion analyses (Appendix B) it becomes evident that performers are making repeated shapes of motion relating to phrasing structure, and that these patterns or shapes of motion are present in varying forms throughout the different regions of the performers' body.

Each performer generates a pattern of motion particular to their individual performance style. This pattern, typically appearing as lemniscate (∞), annodated (S) or crescent (C) shaped movements, and less often appearing as cruciform (t), sigmate (Σ) and 'L' shaped motions, is traced repeatedly by the given body region in accordance with each phrase. Individual performers are also seen to represent motion patterns differently across the three body regions examined. For example, Performer A's phrasing motion shows linear sweeping motions along the length of the keyboard in the torso motion, while generating more rounded motions with the head. In contrast, Performer B shows no change in the general shape of phrasing motions across body regions, instead demonstrating a broadening of the range and spatial location of pivotal points within the motion. Alternatively, Performer C shows a variation in the direction of phrasing gestures across the three body regions, with diagonal upwards motions of the head away from the keyboard at phrase endings varying from the parallel motions of the torso. Additional analyses have shown that performers' phrasing movements sometimes appear counter-productive to those movements required to produce sound (Buck, 2013). Performer C is an example of this, with the direction of head and torso motion appearing contrary to the direction of the performed melody.

Additional patterns are evident within the individual spherical plots (Figure 4), indicating variations in either the size (Performer A), end positioning (Performer B) or directionality (Performer C) of phrasing movements. Closer analysis of the time periods of these seemingly anomalous patterns reveals that performers are varying the shape, size or directionality (and often a combination of these) at points where higher-order structural features occur, such as the climax of bar 12 in phrase 6. This is particularly evident in the additional loop to the right of the keyboard in the clavicle and head motion of Performer C, for example (Figure 4, plot a), corresponding to her interpretation that this phrase is of particular importance compared to the preceding phrases.

Fig. 3. (Enlarged view) Spherical analyses describing head motion of all nine performers during performances of the A major Prelude

Fig. 4. (Enlarged view) Spherical analyses for the head, clavicle and torso motion of Performers A, B & C during performances of the A major Prelude (corresponding to motion profiles b, c & i in Appendix B).

Examining Impacts of Varying Interpretation: B minor Prelude

DISCUSSION OF B MINOR PRELUDE RESULTS

Post-performance interviews revealed that performers' interpretations differed at points of structural ambiguity within the B minor Prelude. These points included potential sub-phrasing within phrases 5 to 7. Patterns of repeated motion evident within the PCA motion profiles reflected these differences of opinion, with patterns of globalised motion variance corresponding to individual performers' structural interpretations of phrasing segmentation and sectional grouping. Spherical analysis demonstrated that motion patterns adhere to temporal patterns of phrasing, as they appear within the performed audio stream [7], across the examined body regions. These findings are supported by the spherical analysis of the A major Prelude. The consistency of temporal patterning of movements throughout these regions goes a long way to explaining the absence of any leading markers within motion profiles. While the different regions of the body are often (but not universally) seen to trace different shapes or trajectories from one another, all of the movements created follow a timing pattern relative to performers' personal interpretations of phrasing structure. The shape, size and direction of these movements alter at points of key structural importance. In the case of the B minor performances, these points of gestural divergence also correspond to points of structural ambiguity between interpretations. It is plausible, then, that performers alter the shape (lemniscate etc.) and form (size and direction) of phrasing movements in order to draw attention to points of structural importance as well as to points where personal interpretation may vary. Changes in the spatial patterning of performance gestures within individual performances may then be seen to represent both expressions of individual interpretations of structure and performer individuality, as each performer is idiosyncratic in the temporal or spatial relation between their repetitive movements and their structural segmentation of the piece.

Fig. 5. (Enlarged view) PCA motion profiles for the nine performances of the B minor Prelude*. Individual motion profiles represent combinations of principal components explaining >90% of performance movements exhibited by each of the nine performers (Performers 1-9 represented as plots a-i in figure). (*Reproduced with permission from Musicae Scientiae).

Fig. 6. (Enlarged view) Spherical analyses for the head motion of all nine performers during performances of the B minor Prelude.

Fig. 7. (Enlarged view) Spherical analyses for the head, clavicle and torso motion of Performers A, B & C during performances of the B minor Prelude (corresponding to motion profiles b, c & i in Fig. 5 and Appendix B).