Pattern recognition and style recognition

Grasping the gist of musical notation in less time than is required for a performance obviously relies on the ability to recognize familiar patterns. The recall of brief visual displays of notation was studied tachistoscopically even before the advent of cognitive psychology, showing effects of what would now be called pattern recognition or perceptual chunking. Ortmann (1934) presented vertical chords (written in staff notation without a clef) for 400 ms, asking advanced music students to write on staff paper what they had seen. An error analysis allowed him to conclude that chord forms corresponding to regular triads or seventh chords (or their inversions) were read as "higher order groupings." In another early tachistoscopic study, Bean (1938) used a presentation time of 187 ms and a piano performance task, interpreting the results of an error analysis with a distinction between "pattern readers" and "part readers." In Bean's interpretation, pattern readers use "the entire group of notes as a cue to recognition," whereas part readers "see only one or two notes of a group and fill in the remainder subjectively" (Bean, 1938, p. 78).

In their classic chess research, de Groot (1965) and Chase & Simon (1973) demonstrated large differences in the ability of players on different skill levels to reproduce briefly exposed chess game positions. In a similar study concerning musical notation, Sloboda (1976) showed that with exposure times of 150 ms or more, musicians could reproduce arrays of noteheads on clefless staves better than non-musicians. Using longer presentation times (4 s), Halpern and Bower (1982) explored the effects of expertise and musical structure on the memory of notated melodies. Parallel to research on auditory recall (Deutsch, 1980), they showed that the well-formedness of melodic structure facilitated participants' ability to write down the melodies. Notably, however, while the expertise effect in chess vanished with random game positions (de Groot, 1965; Chase & Simon, 1973; Reingold et al., 2001), it seems that even randomly structured notated music is better recognized (Waters, Underwood, & Findlay, 1997; Wong et al., 2020) and reproduced (Sloboda, 1976; Halpern & Bower, 1982) by musically experienced individuals than by novices. This indicates that musicians can attribute some musical meaning even to randomly generated sequences of musical symbols—perhaps by using their skills of notational audiation (see Brodsky et al., 2003; Brodsky et al., 2008). Kalakoski (2007) demonstrated that musicians' advantage in recall over non-musicians indeed persisted even in sequential visual presentation of either note heads or written note names, suggesting an advantage gained from pre-learned musical knowledge and skills.

In the above-mentioned studies of quick recognition of musical notation, the focus has been on the accurate memory and reproduction of note symbols. From such a perspective, experts' processing of music notation has also been shown to involve holistic effects, defined as relative inability to selectively attend to part of an object (Wong & Gauthier, 2010). We mention this to point out the difference to how we used the term "holistic" above. In the present study, we are concerned not with the accurate reproduction of notes, but with the visual impression concerning musical style. We understand style as "replications of patterning" (Meyer, 1989) that take place on a more global level in the musical score, carrying cultural meanings beyond the sum of the individual notes. Musical style cannot necessarily be pinned down to any restricted features of notation, as it involves several musical parameters such as melody, rhythm, harmony, texture, dynamics, as well as higher-level patterning in terms of musical gestures, phrases, and so on.

The holistic aspect involved in the recognition of style from musical notation might be compared to chess experts' recognition of chess game positions as meaningful, strategic configurations, or to expert radiologists' rapid viewing of medical images. Studies in medical image perception have shown that experts in that field can rapidly extract diagnostically relevant information from their initial glimpse of the image, finding a lesion in the tissue in just hundreds of milliseconds (Kundel & Nodine, 1975; for a review, see Sheridan & Reingold, 2017). Here, the experts' visual processes are assumed to commence in a "global" (Nodine & Kundel, 1987), "nonselective" (Drew et al., 2013), or "holistic" (Kundel et al., 2007) mode of perception. Such global analysis involves using parafoveal vision (Brams et al., 2019), but it also arguably involves making appropriate judgements regarding the visual display on the basis of higher-order configurations. In the case of musical notation, experts can similarly be expected to see not just note symbols, but rather music, allowing them to apply appropriate stylistic or aesthetic concepts to what they see. The music might be seen as a "chorale," a "lyrical melody," or a "fugato."

Apart from musicians preparing for performance, quickly grasping the qualities of written music can also come in handy for composers, conductors, or even just music teachers browsing through collections of music in search for suitable material for their students. Arguably, the relevant expert skills at play cannot very well be studied by letting participants choose genre labels from among given alternatives. In real life, musicians would more typically have to search for appropriate concepts from their own prior experience, and their quick previews might even result in mutually different preliminary understandings of the same score. It should also be noted that the common-practice classification of style periods is not objectively present in the music: labels such as "Baroque," "Classicism," or "Romanticism" are interpretative products of later musicological work living in our culture, and might also be challenged.

The rationale for the study

In this study, we assumed that an appropriate way to study something akin to quick auditory genre recognition in the visual domain would be to let musicians freely speak of what they see after being exposed to quick glimpses of musical notation. To take a first exploratory step towards understanding this issue, we chose to work with retrospective verbal protocols (see Ericsson & Simon, 1993) and to focus on pianists—an especially interesting group of music readers because of their typical access to the whole notated structure of the music, and thus to much of its aesthetic qualities. Our first research question was an exploratory one:

RQ1: To what extent do professional pianists gain stylistic information from brief glances at written piano music?

Supposing that pianists would be able to recognize musical style periods even from brief visual displays, it should be of further interest to ask how they are able to do it. Following Ericsson and Simon (1993), we assumed that verbalization about a task may reflect the structure of the thought processes involved, and thus we were also interested in studying the relative timing of spoken statements and the sequences of contents in the protocols. This seemed especially appropriate from the point of view of dual-process views of cognition that make a distinction between (1) intuitive processes that are rapid, automatic, and parallel, and (2) reflective ones that are slow, controlled, and serial (see Betsch, 2008; Betsch & Glöckner, 2010; Evans, 2010a; 2010b; Evans & Stanovich, 2013). On the one hand, style recognition might be based on rapid intuitive processes, where the individual quickly provides a response without consciously inferring it from other information. On the other hand, it could also possibly be based on reflective processes relying on working memory—that is, on step-by-step analytical inferences from the combined cues that are available to conscious reflection. These considerations suggest that the differences between the two kinds of cognitive processes might become evident from the latency of the recognition responses and/or from the amount of information evident in the protocols. Thus, we asked:

RQ2: Based on the latency of style attributions and other spoken contents in verbal protocols, is there evidence that correct recognition of musical style takes place through intuitive thought processes?

Inspired by the research on medical image viewing (see above), we hypothesized that experts' style recognition from musical notation would represent more intuitive than reflective processing. If so, style recognition could be expected to take place (1) more rapidly than responses that are not based on recognition, and (2) without as much available music-analytical content as in cases of non-recognition. When intuitive, effortless recognition fails, however, we could expect reflective, inferential processes to intervene (see Evans & Stanovich, 2013). Such processes might take more time than quick, intuitive recognition, and given the potentially scant evidence from the quick visual display, the resulting inferences might also more often lead to incorrect style attributions. Hypothesizing that style recognition takes place in an intuitive manner, we thus expected that incorrect style attributions might take more time than correct ones, and that that verbal protocols preceding incorrect style attributions might show more music-analytical content than is found in cases of correct recognition. The alternative scenario would be that style recognition takes place in a reflective, analytical manner. In this case, correct recognitions (1) should not be faster than incorrect ones, and (2) participants' verbal protocols could be expected to show more music-analytical content prior to correct recognitions than otherwise.

Participants

25 pianists, professionally educated in the classical tradition, volunteered in the experiment. Four participants had master's degrees in music, 11 bachelor's degrees, 8 participants were bachelor-level students, and 2 had vocational upper secondary qualification (the lowest level of professional training in music in Finland). The participants' mean age was 29.6 years (SD = 8.8), and their professional careers as pianists and piano teachers ranged between 4 to 27 years (M = 10.5 years).

In the Finnish national system of higher music education, students of classical piano performance are typically encouraged to explore repertoire from stylistic periods ranging from the Baroque, through Classicism and Romanticism, to 20^th century Modernism. The curricula do not include a compulsory core repertoire of works. Hence pianists have a liberty of creating their personalized repertoires, and, in our experience, they are actively encouraged to study less well-known composers as well. In their own free written descriptions of their piano repertoire, the participants indicated their familiarity with the above-mentioned stylistic periods. At the same time, they clearly did not limit themselves to solo piano repertoire, but worked on a daily basis on other individual interests such as chamber music, improvisation, popular styles, accompaniment, children's music, or original compositions.

Stimuli

The visual stimuli consisted of nine score extracts from the keyboard works of J. S. Bach, L. v. Beethoven (of the "Classical" period), and F. Chopin, three from each composer (see Table 1). The works were chosen to represent canonical piano literature that classically trained pianists would be likely to be acquainted with, but avoiding the opening bars with the most characteristic features of the pieces. Each of the extracts comprised three consecutive systems of keyboard notation, and they were all digitally scanned from Hänle Verlag's Urtext editions to obtain consistency in the visual outlook of the stimuli. An example is provided in Figure 1 below.

In silent-reading tasks, experienced music readers' average fixation times have been reported to lie around 200–300 ms (Waters et al, 1997; Penttinen et al., 2013). The exposure time of 500 ms was chosen to allow at least two fixations. This would not suffice for a complete search of the score, but would potentially allow for a holistic recognition process, and a saccade to some location of interest (Kundel et al., 2007).

Procedure

The second author (a professional piano teacher) met the participants for individual peer discussions. The participants were seated at an electronic keyboard, with a computer screen serving as a music stand. The idea was to facilitate musician-to-musician discussions where the participants could freely talk about what they saw on the screen and demonstrate the music at will on the keyboard. For each stimulus, the participants were asked to "describe in your own words everything that you had time to perceive in the notated example," and to "give your impression about the musical style" in question. In order to help the facilitator orient in the discussion (so as not to miss which score excerpt had been shown), we chose to use a single presentation order for all participants (see Table 1), and check for possible order effects afterwards (see below). This seemed justified as our main interest was not in effects due to specific stimulus features, but rather more generally in the kinds of "style talk" that would ensue after brief glances at notation.

Figure 1. An example stimulus: J. S. Bach: Sinfonia Nr.13 in A minor (BWV 799), bars 13–30.

The stimuli were presented on a computer screen, using the software Microsoft PowerPoint. First, the participants were familiarized to the procedure with two test scores (by Clementi and Schumann). Participants controlled the appearance of the stimuli by using a mouse. On the first click of the mouse, a white square appeared on the screen, outlining the area where the following stimulus would appear. On the second click, the stimulus appeared for 500 ms, followed by a blank screen. On the screen, the size of the stimuli (20 cm in width) matched the printed edition. With approximately 60 cm viewing distance, each stimulus horizontally subtended a visual angle of circa 19°. After each stimulus, the facilitator first allowed the participant to respond alone. When it seemed that the participant did not have more to say, the facilitator took part in the discussion, helping the participant with "content-empty" questions (i.e., ones devoid of information about the musical content, such as "What did you see then?"; see Petitmengin, 2006). The discussion parts of the sessions will not be reported in this article. The sessions were recorded with a video camera.

Content Coding

For this article, we chose to analyze only the participants' initial verbal responses after each stimulus, until the first utterance of the experimenter. The median length of these spoken responses was 52 s (M = 58.1, SD = 29.6). The responses were transcribed word for word and subjected to content analysis as follows.

We devised a coding system involving separate codes for recognition and misattribution of Style (i.e., style period), Composer, and Composition. Each of these recognition categories received two possible codes, with suffixes "C" and "I" indicating correct and incorrect responses, respectively. For instance, mentioning "Baroque" for a Bach piece was coded as Style-C, and mentioning "Bach" for a Beethoven piece would be coded as Composer-I. We also initially included the code Performance for recognizably playing some of the musical piece in question on the keyboard. By a process of constant comparison (Glaser & Strauss, 1967), the second author extended the coding scheme to cover also other spoken contents in the responses. The scheme involved content categories concerning (i) pitch organization, (ii) rhythmic and metric organization, (iii) texture, registral aspects, and the role of hands, (iv) articulation, and (v) experiential qualities. Both authors independently coded all 225 responses using these content categories, yielding a total of 1,602 shared or individual code assignments, including 1.5% discrepancies between the two codings (Cohen's κ = 0.98).

A discussion of the coding led to some modifications regarding the third content category, texture. First, statements regarding compositional type (e.g., "it was a fugue"; "it looked like a minuet") which had first been coded as indicating texture, were given a separate category, type. Second, we realized that statements regarding registral range or the role of hands were always combined with content concerning pitch organization (e.g., "there were triads in the right hand"), temporal organization (e.g. "there were eighth-notes in the bass"), or texture (e.g., "a waltz accompaniment in the left hand"). To avoid counterintuitive alternation between codes in these passages, we removed registral aspects and the role of hands from the texture category, leaving them uncoded. Third, the distinction between statements concerning articulation and texture often seemed hard to draw, and thus we subsumed statements concerning articulation into the texture category.

Apart from these changes, we realized that the recognition category Performance was only needed four times (twice for Beethoven's Sonata Pathétique, twice for Chopin's Minute waltz), and hence we combined this category with the recognition category Composition. A preliminary analysis showed that despite the brief presentation time of the visual stimuli, the right composition was identified in a number of trials (Bach: 9.3%, Beethoven: 5.3%, Chopin: 8.0%). However, given that these recognition percentages were relatively low, we further decided to collapse the Composition codes into the category Composer. With these changes, the coding scheme was revised to the final form shown in Table 2 below. According to the coding, participants' responses included a mean of 5.8 code instances (SD = 2.5), and 4.1 unique codes, on average (SD = 1.4).

We demonstrate our procedure by a coding example of one participant's (Nr. 2) response regarding Bach's Sinfonia in A minor (BWV799; see Figure 1). In this case, the participant recognized the composer after first making some observations regarding Texture, Experience, and Time. Most of the content statements are also quite appropriate, apart from the mistake regarding key (perhaps a Bb note in the score had suggested F major to the participant):

Well, it seemed like [Texture] a polyphonic piece. It occurred to me that it could be [Experience] a rather fast piece. [Texture] In the right hand, simultaneously some kind of melody, [Time] quarter or eighth notes, and at the same time in the right hand a sixteenth-note pattern there. And the left hand might have had eighth notes again. It could be something by [Composer-C] Bach. It brings to mind [Texture] a three-voice… what do you call them… are they [Type] Sinfonias or what. [Pitch] The key was perhaps F major, or maybe something with only few signs in the key signature. This could well be [Time] in three time, perhaps. Maybe 3/4 or 3/8. That's what came to mind.

Statistical Analysis

The quantitative analyses were carried out by using only the first code instances of each code type that was present in each trial. For the timing analysis, these first instances were given a time index, measuring the time from the flash of the visual stimulus onscreen to the beginning of the utterance in question (rounded to the nearest second). The analyses were carried out in the R statistical environment (R Core Team, 2019).

For a preliminary check regarding possible effects of the presentation order on the responses, we used Spearman correlation as well as repeated measures correlation. The latter, implemented in the R package "rmcorr" (Bakdash & Marusich, 2021), is a correlational method accounting for non-independence among observations (Bakdash & Marusich, 2017). According to Spearman correlation, the total number of participants recognizing the Style or the Composer of an excerpt was not associated with the trial number of the presentation order (Style: ρ = –0.13; Composer: ρ = –0.08; both ps > .1). According to repeated measures correlations, there was also no common within-individual association between the trial number and the latency of correct recognition (Style: r_rm = –0.07; Composer: r_rm = 0.14; both ps > .1). Hence, on the group level, the presentation order did not affect either the level of recognition or the latency of giving correct answers.

For estimating participants' sensitivity of recognizing the composers Bach, Beethoven, and Chopin, we used Zhang and Mueller's (2005) non-parametric estimate of sensitivity. However, our main analyses concerned the timing of correct and incorrect recognitions and the contents of the responses between correct and incorrect recognitions. For the timing analysis, we used Wilcoxon rank-sum tests for clustered data (Rosner, Glynn, & Lee, 2003), as implemented in the R package "clusrank" (Jiang, Lee, Rosner, & Yan, 2020). This allowed taking into account that each participant provided responses in several trials. For the content analysis, we used paired t-tests of participant means.

Recognition

Our first research question asked to what extent pianists could extract stylistic information from the brief visual exposure. It turned out that pianists did recognize either musical Style and/or the Composer in an average of 4.4 (out of 9) trials per participant (SD = 2.1), corresponding to almost half (48.9%) of trials in the whole participant group. Style-C was recognized in 30.7% of all trials, or in a mean of 2.8 out of nine trials per participant (SD = 1.7). Composer-C was recognized in 28.4% of all trials, or in 2.6 trials per participant (SD = 1.5). Overall, only two participants did not recognize Style in any of the trials, three did not recognize any Composer, and only one did not recognize either of these aspects in any of the trials. Using a median split, we may note that the 14 best participants recognized either Style-C or Composer-C in 5.9 trials (SD = 2.5), on average.

It is difficult to formally estimate participants' recognition sensitivity, because no response alternatives were announced beforehand, and because responses could contain several guesses, sometimes including both a correct and one or more incorrect responses. It might be noted that the observed recognition of Style-C was lower than one could have achieved by mechanically responding, say, "Baroque" in each trial. However, our participants were much more cautious in their response strategies, only giving one or more Style-I responses in 15.1% of the trials and one or more Composer-I responses in 28.0% of them. While some verbal protocols contained up to four suggested composer names, the mean number of composer names mentioned in a protocol was only 0.67 (SD = 0.85), and no less than 47.8% of the composer names mentioned by an individual participant were right answers, on average (SD = 29.2).

We further estimated participants' sensitivity of recognizing the composers Bach, Beethoven, and Chopin by treating all Composer-C responses as "hits" and all inappropriate attributions of these three composer names as "false alarms." It was assumed that providing one wrong Composer response along with the right one would diminish the recognition success to 0.5 of a hit, two accompanying wrong responses would give a 0.33 hit, and so on. For each participant, we then summed up the number of such modified hit scores across the 9 trials, and similarly counted their individual sums of false alarms (the two other composer names incorrectly used). Excluding two participants with no Composer recognition, Zhang and Mueller's (2005) non-parametric estimate of sensitivity yielded the participant mean value A = 0.78 (SD = 0.09) which is above the midpoint between perfect recognition (1) and chance performance (0.5). Hence, the observed recognition sensitivity would count as relatively good even assuming that the participants would have known beforehand which composers to look for—which they in fact did not know.

Table 3 breaks down the percentages of recognition by actual composer. Here, as in the rest of the table, instances of the same code were only counted once per response (e.g., "Composer-I" indicates the presence of one or more incorrect responses of this kind). For Style recognition, both correct and incorrect responses were rather evenly divided between the composers (albeit with slightly lower Style recognition for Beethoven), but for recognition of Composer, Chi²-tests indicated larger differences between the three composers. Thus, while Composer-C was provided in about third of the trials for Bach and Chopin, the corresponding recognition rate for Beethoven was about half of this. As for the content codes, we see that statements regarding musical structure dominated the code strings: across the three composers, the majority of responses included information on Texture, Time, and Pitch. A significant difference in the content codes emerged only for Experience which received an elevated percentage in Chopin.

The Time of Recognition

As explained above, we hypothesized that correct style recognitions would take place in a rapid, intuitive manner, and that slower inferential processes might be triggered if such a rapid recognition is not forthcoming. In order to address this possibility, we compared the timing of the first correct and the first incorrect answers given in the verbal protocols. For each code type, Figure 2 shows the times of first occurrence from the flash of the score onscreen. It appears that the different codes, indeed, tended to appear at slightly different times. Thus, content concerning Texture and Time often occurred early on in the responses, while information about Type and Experience was typically given later (see Figure 2b).

Figure 2. Participant means for the time of first code occurrence in verbal protocols, measured from the flash of the visual stimulus on the screen: (a) Recognition codes (with gray lines connecting individual participants "correct" and "incorrect" responses) and (b) Content codes. (See Table 2 for an explanation of the codes.)

For the recognition codes, the participant means in Figure 2a suggest some tendency for the correct recognitions to take place earlier than the incorrect ones. For a more formal statistical analysis, we used the original timing values (instead of means). The timings of the first occurrences of the four recognition codes were non-normally distributed (Shapiro-Wilk: all ps < .001, except for Style-I, p < .01). Thus, we analyzed the differences in the timing of the first correct and incorrect answers using the Wilcoxon rank-sum test for clustered data. Regarding Style, the test showed a significant difference between the timing of the first correct and incorrect responses available in the protocols (Z = –3.04, p = .002; n = 103, 24 clusters). The participants' grand mean for the first appearance of Style-C was 22.2 s after the flash of the stimulus (SD = 12.9) whereas the first Style-I codes occurred almost 15 s later, on average—at 36.7 s after the stimulus (SD = 19.1). No significant difference was found between the timing of Composer-C (grand mean = 25.6, SD = 19.8) and Composer-I (grand mean = 26.5, SD = 15.1; Z = –0.71, p = .546, n = 126, 25 clusters). In other words, correct recognitions did appear earlier than incorrect ones, but only in the case of generic statements concerning style period—not for statements regarding the assumed composer.

Richness of Spoken Content Preceding Recognition

In the introduction, we hypothesized that if correct recognitions result from an intuitive process, the responses indicating such recognition might also include less reflective content than in cases where the intuitive recognition fails. To address this issue, we carried out an analysis of "richness of content" in the verbal protocols, comparing the spoken contents preceding correct recognition and spoken contents when recognition did not take place. Richness of content was here operationalized as the number of different content codes occurring in recognition protocols before recognition (e.g., before the code Style-C), and in comparable stretches of the non-recognition protocols (where the aspect in question was not correctly recognized).

Our coding system had five different content codes (see Table 2), and thus the number of different types of content codes in a given response could vary between 0 and 5. In the case of recognition protocols, we took into account the number of different content codes preceding correct recognition (of Style or Composer). For a fair comparison with non-recognition protocols, we truncated each participant's non-recognition protocols at the average time of the first recognition codes in the same participant's recognition protocols, and considered only the content codes appearing before this point of time. For instance, participant Nr. 1 recognized Style-C in four trials, at the mean time of 20 s after the stimulus. For this participant, the remaining five trials lacking Style-C were truncated at 20 s, and only content codes occurring before this point of time were considered. A similar procedure was followed for determining appropriate amount of content to be considered in responses lacking Composer-C. In these analyses, we thus ignored all codes for incorrect recognition.

For comparing the richness of content between recognition and non-recognition protocols, we calculated the mean numbers of different content codes for each participant. Participants without any correct recognitions were removed from the respective analyses (two in the Style analysis, three in the Composer analysis). The remaining participants' mean numbers of different content codes are depicted in Figure 3. As the boxplots suggest, the number of different content codes was slightly higher in the non-recognition protocols than in the recognition protocols. According to paired t-tests, the difference was significant for Composer recognition (t(21) = –2.84, p < .01), but not for Style (t(22) = –1.60, p > .1; two-tailed tests). We may thus conclude that at least for Composer recognition, the participants tended to use more restricted descriptive contents before correct recognition than in comparable time periods when no correct recognition was forthcoming.

Total Spoken Content

Above, we saw that neither Composer-C nor Style-C recognition were supported by greater richness of spoken contents before the time of recognition. It remains to be seen whether such supportive content could have been available to the participants from the visual stimulus, even though only mentioned later in their protocols, after the recognition had already taken place. To check this option, we repeated the analysis regarding richness of content with all content codes available in the protocols (up to the first utterance of the interviewer). T-tests did not reveal significant differences in the numbers of different content codes, either between recognition and non-recognition of Style-C (t(22) = 1.13, p > .1) or between recognition and non-recognition of Composer-C (t(21) = –0.89, p > .1). Consequently, there was no evidence to support the idea that correct recognitions would have been accompanied by richer overall contents than were available when recognition did not take place.

Figure 3. Richness of spoken content preceding correct recognition in the recognition protocols vs. in equally long stretches of the non-recognition protocols: (a) before recognition of Style, N = 23, and (b) before recognition of Composer, N = 22. Individual participants' means are connected with gray lines.