Mapping Musical Features and Shape

Sonomotiongrams belong to the larger domain of sonification, a fast growing field of research and development. Sonification is based on the idea that sound may be an efficient transducer of information in various contexts of human life (Hermann, Hunt, & Neuhoff, 2011). In principle, any information, or any sets of data, from such diverse fields as geology, climate research, medical research, or financial markets, can be efficiently transmitted by sonification, meaning that instead of studying large tables and graphs, we can just listen to the data, and provided well-designed schemes for mapping, quickly perceive important information in the data. Sonification can also be done on the fly, making possible interactive scrutiny of various kinds of data, or can be used for feedback in various control tasks, in what is called interactive sonification. Such interactive sonification ranges from the classical example of the Geiger counter (making a sparkling sound reflecting the level of radiation) to beeps in modern cars, computers and household devices, to more elaborate schemes of using sound feedback for orientation and navigation.

The underlying principle in all sonifications is that of mapping from some domain (numerical, visual, kinematic, etc.) to the auditory domain. Needless to say, the crucial point will be how this mapping is done. Given the possibility of arbitrary mapping, that is, that any data can in principle be mapped to any sound, the challenge is that of designing mapping schemes that are somehow perceived as meaningful and useful, and importantly, also as aesthetically well-designed. The underlying mental scheme here is what we like to call shape-cognition, the belief that human perception and cognition in general is intimately linked with notions of shape and space, as has been suggested by the ideas of image schemata in morphodynamics (Thom, 1983; Godøy, 1997), cognitive linguistics (Johnson, 1987), as well as auditory perception research (Schaeffer, 1966; Godøy, 2006).

As for music and shape-cognition, we have good reason to claim that notions of shape are ubiquitous in Western musical culture. Expressions of shape (words and graphics) are encountered in innumerable contexts of production (shapes of sound-producing body motion and postures), perception (shapes of rhythmic, melodic, timbral, and dynamic features) and theoretical discourse (both notation-based and signal-based such as waveforms and spectrograms). Notions of shape are at the very heart of Western music theory, with the development of notation being clearly shape-oriented already with neumes, however later evolving into more abstract representations. But in spite of this more symbol-oriented evolution of Western music notation, we could say that the performance of a score in a sense is a sonification of the graphics of the score, and, this is an essential point, the graphics of the score are transduced to sound by the musicians' sound-producing body motion.

We have in the latter half of the 20^th century seen various projects of graphical notation that are less dependent on discrete symbols and rather try to represent continuous trajectories of sound and body motion. At the same time readily available technology has increasingly enabled the study of continuous sound and motion without being limited to a symbol-based system such as the Western music notation. This opens up for study more direct feature-mappings between sound, body motion, and vision in music. The essential element here is that shapes are intrinsically holistic, whereas symbols are not. What we call shape-sonifications may thus capture the ephemeral elements of musical experience, both of sound and music-related body motion, so that we may systematically explore qualitative features and nuances of sound and body motion, and also higher-level affective and aesthetic features.

One important element here is that body motion seems to play a role as translator between modalities. This was (albeit implicitly) suggested by Wolfgang Köhler's famous Maluma vs. Takete distinction (Köhler, 1947), however a more broad basis for such cross-modal translation has been provided by recent findings in neuroscience suggesting that both visual and sound perception relate closely to motor sensations and simulations (Kohler, Keysers, Umiltà , Fogassi, Gallese, & Rizzolatti, 2002), something that has for several decades been claimed by motor theories of perception (Galantucci, Fowler, & Turvey, 2006). Keeping this in mind, we believe there is the triangular relationship of sound, motion, and vision that can be exploited in the design and use of sonomotiongrams.

Motiongram

A motiongram is a spatiotemporal display of motion in a video recording (Jensenius, 2006; Jensenius, 2013). As the overview in Figure 1 shows, the process starts by reading a video stream and converting it into a greyscale image. In future research it would be interesting to use the colour information as well. It may also be useful to do some simple image adjustments at this stage, such as changing the brightness and contrast, so that the video used for further analysis is as clear as possible (Figure 1.2). The next step involves producing the motion image by calculating the absolute difference between subsequent video frames (Figure 1.3). Dependent on the quality of the original image, and the noise level in the image due to video compression, lighting and so on, it may be necessary to filter the motion image (Figure 1.4). This can be done through simple thresholding or by applying a noise-removal algorithm to remove groups containing few pixels. The motiongram is created by calculating the normalised mean value for each row in the motion image (Figure 1.5). This means that for each image matrix of size MxN, a 1xN matrix is calculated. Drawing these 1-pixel wide stripes next to each other over time results in a horizontal motiongram (Figure 1.6).

The sonomotiongram technique is based on the concept of treating the motiongram as if it were a spectrogram of an audio recording. Technically, however, a motiongram is very different from a spectrogram. A spectrogram of an audio recording displays the energy level of the frequency bands resulting from doing a Fourier transform on the audio. A motiongram, on the other hand, is a reduced display of a series of motion images. There is no analysis being done when creating a motiongram: It is only based on a simple reduction algorithm. This reduction approach may be seen as problematic in some contexts. It is, for example, not possible to separate motion in the foreground of the image from motion in the background. On the other hand, the simplicity of the approach also means that it can easily be applied to widely different video material, everything from music and dance performances to the motion of children with cerebral palsy (Jensenius, 2007).

Fig. 1. (Enlarged view) The steps involved in creating a motiongram: (1) accessing the original video stream, (2) greyscale conversion, (3) frame differencing, (4) filtering, (5) averaging each row, (6) drawing the average matrices over time.

It is worth mentioning that a single motiongram will only display motion in one spatial dimension. Thus a horizontal motiongram visualises only vertical motion, since all information about the spatial distribution of motion in the horizontal plane is represented by 1 pixel for each row. When creating motiongrams it is therefore necessary to evaluate in which plane(s) the motion is occurring, before deciding whether to create a horizontal or a vertical motiongram (or both).

From Motiongram to Sound

Even though motiongrams and spectrograms represent different features, they share one property: the temporal unfolding of shapes of either motion or sound. Furthermore, the Y-axis in a motiongram represents vertical motion, which is often associated with pitch. We were therefore curious to hear what would happen if it were treated as a spectrogram and turned into sound, as illustrated in Figure 2.

Fig. 2. (Enlarged view) A sketch of the direct mapping from motiongram to spectral audio data.

The sonomotiongram technique has been implemented in the graphical programming environment Max/MSP/Jitter, and the technical details have been presented in Jensenius (2012). The implementation is based on passing data from the motiongram matrix to an interpolated oscillator bank performing the additive sound synthesis. The result is a direct sonification of the motion, where lower sound frequencies are controlled by movement in the lower part of the image, and vice versa.

The sonification algorithm was implemented in the module jmod.sonifyer~ of the open framework Jamoma for Max (Place & Lossius, 2006). The modular approach of Jamoma makes it possible to create complex audio and video patches in Max quickly, with the added benefit of extensive preset, mapping and cueing functionality (Place, Lossius, Jensenius, & Peters, 2008). An example of how the sonifyer module may be used in conjunction with other Jamoma modules is shown in Figure 3, and a video tutorial of the functionality of the module can be seen in Video 1.

Fig. 3. (Enlarged view) An example of how a set of Jamoma modules can be used to sonify a video stream in realtime. The video input module in the upper left corner is connected to a module creating the motion image and then to the motiongram module. Finally, the output motiongram is sent to the sonifyer module for the creation of sound.

Sonification of Basic Motion Patterns

Before presenting some examples of the sonification of complex motion, it may be useful to investigate some basic motion patterns. Video 2 shows examples of moving the hand up and down, sideways, diagonally and in a circle. These examples effectively display the largest problem with the proposed sonification approach: the motiongram's ability to display motion in only one direction. Thus the up and downward motion is clearly visualised in the motiongram (Figure 4), and heard in the sound. The other motion patterns (sideways, diagonal and circular) are not represented equally well because the motion is only partly happening in the horizontal direction, is therefore not properly visualised in the motiongram, and hence is not audible.

Fig. 4. (Enlarged view) Screenshots from Video 2 showing: (a) the motion pattern of a hand moving up and down, (b) the motion pattern of a hand moving sideways. The arrows are added manually to indicate the direction of the motion. Notice how the motiongram in (a) effectively visualises the motion, while the motiongram in (b) only visualises the (small) vertical displacement of the motion.

One attempt at sonifying the two motion axes at the same time is shown in Video 3. Here both horizontal and vertical motiongrams are created from the same video recording, and the sonifications of the two motiongrams have been mapped to the left and right audio channel respectively. While this method gives some impression of the motion along the two axes, understanding the relationship between the two may be quite confusing. Clearly, more experimentation is needed to find a better solution for sonifying the two dimensions independently yet simultaneously.

Effect of Filtering

One of the most important decisions to make when creating a motiongram, and hence its sonification, is the level of filtering and thresholding to apply to the image. Setting a high threshold value will remove more noise from the image but it may also remove valuable parts of the motion to be sonified. This can be seen in an example of a high-speed recording (200 fps) of a hand in motion in Video 4 and Figure 5. Here three different types of filtering have been applied to the motion image to show the visual affect on the motiongram and how it influences the sonic results. When there is no thresholding and no noise reduction, all the details of the hand motion are visible and audible. Adding a low-pass filter removes a substantial quantity of pixels in the motiongram, and hence makes a cleaner sonification. Finally, adding a noise reduction algorithm further reduces the number of pixels, leading to a sonification of only the most important parts of the motion sequence.

Fig. 5. (Enlarged view) Examples of how filtering the motion image influences the motiongram: (a) screenshot from the original video recording, (b) no filtering, no noise reduction, (c) low-pass filtering, (d) low-pass filtering and noise reduction.

Sonification of violin performance

Sonifying the motion of a musician may seem like a paradox, since the sound-producing and sound-modifying actions of a performer are in themselves sound-producing. In these examples, however, we will look at the overall body motion of a performer, not only the sound-producing actions. Video 5 and Figure 6 show an example of the sonification of a short violin improvisation going from slow legato bowing to more rapid and energetic strokes. One confusing element here is that the first part of the sonification resembles the original sound of the performance, since it is mainly the bowing arm that is sonified in the beginning of the sequence. In the second half of the sonification, however, motion in the rest of the body is also sonified. As such, sonifying musicians' motion with such a generic sonification technique may lead to results that are somewhat tricky to interpret, since both sound-producing and non-sound-producing motion are sonified. But this may also be seen as one of the strengths of the technique, namely the ability to put into sound all the visible elements of a performance.

Sonification of Clogging

While the sonomotiongram technique is not based on any particular type of analysis or musical knowledge, it is possible to improve the sonification results manually by adjusting the visual input to the system. An example of this can be seen in the sonification of the motion of a French-Canadian fiddler in Video 6 and Figure 7. Instead of sonifying the full body motion of the performer, we are here focusing on the clogging pattern of his feet. This makes it possible to listen to, for example, the rhythmic structure of the clogging pattern, and pick up changes such as the change of rhythmic figure and tempo halfway throughout the excerpt. This change is not easy to see in the original video, nor in the motiongram, but is clearly audible in the sonification. (See Schoonderwaldt & Jensenius (2011) for a rationale and a more detailed analysis of this performance.)

Sonification of Dance Motion

An example of the sonification of dance motion is shown in Figure 8. This recording is from an experiment in which a group of dancers were asked to improvise freely to short excerpts of music. Video 7 starts by showing the original recording of one of the dancers, and how the dancer moves spontaneously to the sound of music. Then follows the same video but overlaid with the sonification of her motion. It is interesting to hear how the sonification of the motion shows some clear similarities to the sonic qualities of the original sound. This, however, is a special case of a good correspondence between the original sound and the sonification result. Although such correspondences may happen, we find that it is often more interesting to use sonification as a method to reveal motion features that contrast with the original audio, or to reveal motion features that are not easily visible to the naked eye. In a non-realtime version of our software, such parts of a sonomotiongram may be manually investigated by scrubbing through the motiongram with the mouse.

Fig. 8. (Enlarged view) Sonification of free dance motion (64 seconds).

Sonification in Live Performance

Sonification of body motion is usually seen as something different than musical mapping of the same body motion (Winters, Savard, Verfaille, & Wanderley, 2012). That is, the aim of a sonification is to convey information about the data, not to create musically and aesthetically interesting sound. But that does not close the door to using a sonification, or a sonification technique, in a musical context. While the sonomotiongram technique was originally developed for analytic applications, we have found that its greatest potential may, in fact, be for creative applications. Since the Jamoma sonification module runs in real time, it can easily be included in a performance setup. Over the last couple of years the module has been used in several concerts, both in solo performance (Video 8) and with ensembles (Video 9). Figure 9 shows the performance patch for the piece Soniperforma, which uses the sonification module as the main sound maker. The processing is based on applying various video effects to the input video, such as changing the colours, size and orientation of the image, and applying image filters and video feedback. Since the sonification is based on the input image, such video effects will in practice end up as sound effects. For example, adding a motion blur effect to the video image will result in a delay in the sound. We have found this to be creatively inspiring as a performer, and also engaging for audiences.

Fig. 9. (Enlarged view) A performance patch using various video effects to modify the motiongram, and hence the output sound. The visual result from changing various video parameters can be seen in the motiongram at the top.