![\"FMP\"](\"../data/FMP_Teaser_Cover.png\")
\n",
"![\"AudioLabs\"](\"../data/Logo_AudioLabs_Long.png\"){kind=logo}
\n",
"\n",
"\n",
"![\"C4\"](\"../data/C4_nav.png\")
\n",
"\n", "Following Section 4.1 of [Müller, FMP, Springer 2015], we discuss in this notebook general principles for segmenting and structuring music recordings. For an overview, we also refer to the following literature.\n", "\n", "
![\"FMP_C4_F02\"](\"../data/C4/FMP_C4_F02.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Segmentation and Structure Analysis\n",
"\n",
"The tasks of segmenting and structuring multimedia documents are of fundamental importance not only for the processing of music signals but also for general audio-visual content. **Segmentation** typically refers to the process of partitioning a given document into multiple segments with the goal of simplifying the representation into something that is more meaningful and easier to analyze than the original document. For example, in image processing the goal is to partition a given image into a set of regions such that each region is similar with respect to some characteristic such as color, intensity, or texture. Region boundaries can often be described by contour lines or edges at which the image brightness or other properties change sharply and reveal discontinuities. In music, the segmentation task is to decompose a given audio stream into acoustically meaningful **sections** each corresponding to a continuous time interval that is specified by a start and end **boundary**. At a fine level, the segmentation may aim to find the boundaries between individual notes or to find the beat intervals specified by beat positions. At a coarser level, the goal may be to detect changes in instrumentation or harmony or to find the boundaries between verse and chorus sections. Also, discriminating between silence, speech, and music, finding the actual beginning of a music recording, or separating the applause at the end of a performance are typical segmentation tasks.\n",
"\n",
"Going beyond mere segmentation, the goal of **structure analysis** is to also find and understand the relationships between the segments. For example, certain segments may be characterized by the instrumentation. There may be sections played only by strings. Sections played by the full orchestra may be followed by solo sections. The verse sections with a singing voice may be alternated with purely instrumental sections. Or a soft and slow introductory section may precede the main theme played in a much faster tempo. Furthermore, sections are often repeated. Most events of musical relevance are repeated in a musical work in one way or another. However, repetitions are rarely identical copies of the original section, but undergo modifications in aspects such as the lyrics, the instrumentation, or the melody. One main task of structure analysis is to not only segment the given music recording, but to also group the segments into musically meaningful categories (e.g., intro, chorus, verse, outro). \n",
"\n",
"The challenge in computational music structure analysis is that structure in music arises from many different kinds of **relationships** including **repetition**, **contrast**, **variation**, and **homogeneity**. In view of the various principles that crucially influence the musical structure, a large number of different approaches to music structure analysis have been developed. In the following, we want to roughly distinguish three different classes of methods. \n",
"\n",
"* First, **repetition-based** methods are used to identify recurring patterns. \n",
"* Second, **novelty-based** methods are employed to detect transitions between contrasting parts. \n",
"* Third, **homogeneity-based** methods are used to determine passages that are consistent with respect to some musical property. \n",
"\n",
"Note that novelty-based and homogeneity-based approaches are two sides of a coin: novelty detection is based on observing some surprising event or change after a more homogeneous segment. While the aim of novelty detection is to locate the changes' time positions, the focus of homogeneity analysis lies in the identification of longer passages that are coherent with respect to some musical property. The following figure illustrates that similar segmentation and structuring principles apply for other domains such as image and 3D data.\n",
"\n",
"![\"FMP_C4_F03_text\"](\"../data/C4/FMP_C4_F03_text.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Musical Structure\n",
"\n",
"To specify musical structures, we now introduce some terminology. First of all, we distinguish between a **piece of music** (in an abstract sense) and a particular **audio recording** (an actual performance) of the piece. The term **part** is used in the context of the abstract music domain, whereas the term **segment** is used for the audio domain. Musical parts are typically denoted by the capital letters $A,B,C,\\ldots$ in the order of their first occurrence, where numbers (often written as subscripts) indicate the order of repeated occurrences. As an example, we consider the Hungarian Dance No. 5 by Johannes Brahms. This dance has been arranged for a wide variety of instruments and ensembles, ranging from piano versions to versions for full orchestra. The following figure shows a sheet music representation for the violin voice of an arrangement for full orchestra. \n",
"\n",
"\n",
"![\"FMP_C4_F05_Sibelius.png\"](\"../data/C4/FMP_C4_F05_Sibelius_annotated.png\")
\n",
"\n",
"\n",
"The musical structure is $A_1A_2B_1B_2CA_3B_3B_4D$, which consists of three repeating $A$-parts, four repeating $B$-parts, as well as a $C$-part and a short closing $D$-part. The $A$-part has a substructure consisting of two more or less repeating subparts. Furthermore, as becomes apparent when looking at the musical score, the middle $C$-part may be further subdivided into a substructure that may be described by $d_1d_2e_1e_2e_3e_4$. In music notation, such subparts are often indicated using small letters $a,b,c,\\ldots$. \n",
"\n",
"![\"FMP_C4_F28\"](\"../data/C4/FMP_C4_F28.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Audio Structure Analysis\n",
"\n",
"Given a recording of a piece of music, the goal of **audio structure analysis** (as considered in this chapter) is to find the segments within the recording that correspond to the various parts of a musical structure. The following examples show that different performances not only may differ with regard to aspects such as instrumentation and tempo, but also with regard to the global musical structure.\n",
"\n",
"Orchestral version (Ormandy, $A_1A_2B_1B_2CA_3B_3B_4D$) ![\"C0\"](\"../data/C0_nav.png\") | \n",
" ![\"C1\"](\"../data/C1_nav.png\") | \n",
" ![\"C2\"](\"../data/C2_nav.png\") | \n",
" ![\"C3\"](\"../data/C3_nav.png\") | \n",
" | \n",
" ![\"C5\"](\"../data/C5_nav.png\") | \n",
" ![\"C6\"](\"../data/C6_nav.png\") | \n",
" ![\"C7\"](\"../data/C7_nav.png\") | \n",
" ![\"C8\"](\"../data/C8_nav.png\") | \n",
"