![\"FMP\"](\"../data/FMP_Teaser_Cover.png\")
\n",
"![\"AudioLabs\"](\"../data/Logo_AudioLabs_Long.png\"){kind=logo}
\n",
"\n",
"\n",
"![\"C2\"](\"../data/B_nav.png\")
\n",
"\n", "In this notebook, we introduce some tools that can be used for the sonification of feature representations and annotations. \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "\n", "In the FMP notebooks, we use a multitude of figures with concrete examples to communicate the main ideas of music processing approaches. In particular, the **visualization** of feature representations such as [spectrograms](../C2/C2_STFT-Basic.html), [chromagrams](../C3/C3S1_SpecLogFreq-Chromagram.html), or [tempograms](../C6/C6S2_TempogramFourier.html) not only deepens the understanding of signal processing concepts, but also provides valuable insights into musical and acoustic properties of the underlying audio signals. Furthermore, the [combined visualization of extracted features and reference annotations](../B/B_Annotations.html) allows for an in-depth inspection of algorithmic approaches on a piece-wise level. Such qualitative evaluations are, besides [quantitative evaluations based on suitable metrics](../C4/C4S5_Evaluation.html), essential for understanding the benefits and limitations of algorithmic approaches as well as the suitability and validity of the underlying model assumptions. \n", "\n", "As an alternative or complement to **data visualization**, one may also use **data sonification** as a means for providing acoustic feedback on the nature of extracted or annotated information. This particularly holds for music, where humans are trained to perceive even slight deviations in frequency and timing of sound events. For example, one can immediately recognize small rhythmic deviations when listing to a pulse track sonified in form a sequence of click sounds. In this notebook, we introduce three sonification methods that are helpful in analyzing annotations as well as audio features extracted from a music recording. \n", "\n", "* **Sonification of time positions** using click sounds. This technique is useful for inspecting the accuracy of [onset, beat, and measure positions](../C6/C6S1_OnsetDetection.html). \n", "* **Sonification of frequency trajectories** using sinusoidal models. This method yields insights for tasks such as [melody estimation](../C8/C8S2_FundFreqTracking.html) or [singing voice separation](../C8/C8S2_MelodyExtractSep.html).\n", "* **Sonification of chroma features** using [Shepard tones](../C1/C1S1_ChromaShepard.html). This technique provides an acoustic representation of the information encoded by [chromagrams](../C3/C3S1_SpecLogFreq-Chromagram.html).\n", "\n", "Obviously, there are many more ways for sonifying acoustic properties of music signals. In particular, a superposition of the original music signal and a suitable sonification of specific features often leads to fascinating and surprising insights." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sonification of Time Positions\n", "\n", "In the first scenario, we assume that we are given a music recording as well as a list of time positions that indicate the presence of certain musical events. For example, the musical events may refer to [onset positions](../C6/C6S1_OnsetDetection.html) of certain notes, to [beat positions](../C6/C6S2_TempoBeat.html), or [structural boundaries](../C4/C4S1_MusicStructureGeneral.html) between musical parts. Then, the goal of sonification is to generate a succinct acoustic stimulus at each of the time positions, thus giving the listener a precise temporal feedback. Ideally, the stimuli should be perceivable also when being superimposed with the original music recording. Often the time positions are further classified according to different categories (e.g., downbeat and upbeat positions). Therefore, it is useful to have a \"coloration\" method for generating distinguishable stimuli that can easily be associated with the different categories.\n", "\n", "The [LibROSA](https://librosa.org/doc/latest/index.html) python package provides the function [`librosa.clicks`](https://librosa.org/doc/latest/generated/librosa.clicks.html?highlight=click#librosa.clicks) for generating a family of distinguishable stimuli in form of click sounds that can be placed at the time positions specified. The function allows for adjusting the frequency (with a default of $1000~\\mathrm{Hz}$) as well as the duration (with a default of $100~\\mathrm{ms}$) of the click signal. \n", "\n", "As an example, we consider a short excerpt of an orchestra recording of the Waltz No. 2 from the Suite for Variety Orchestra No. 1 by Dimitri Shostakovich. In the annotation file, we have marked the beat positions (given in seconds). Being in the $3/4$ meter, every third beat corresponds to a downbeat or measure position. In the following code cell, we generate a sonification, where we use a long click sound of low frequency to indicate measure positions and a short click sound of high frequency to indicate beat positions. While, in the visualization, the annotated time positions are superimposed with the waveform plot, the generated click sounds are superimposed with the original music recording.\n", "\n", "\n", "![\"FMP_B_Sonify_Beat_Shostakovich_Waltz.png\"](\"../data/B/FMP_B_Sonify_Beat_Shostakovich_Waltz.png\")
\n",
"\n",
"![\"FMP_C8_F10a\"](\"../data/B/FMP_B_Sonify_F0_Weber_Freischuetz.png\")
\n",
"\n",
"![\"FMP_C8_F10a\"](\"../data/B/FMP_B_Sonify_Pitch_Schubert_D911-11_D.png\")
\n",
"\n",
"![\"C0\"](\"../data/C0_nav.png\") | \n",
" ![\"C1\"](\"../data/C1_nav.png\") | \n",
" ![\"C2\"](\"../data/C2_nav.png\") | \n",
" ![\"C3\"](\"../data/C3_nav.png\") | \n",
" ![\"C4\"](\"../data/C4_nav.png\") | \n",
" ![\"C5\"](\"../data/C5_nav.png\") | \n",
" ![\"C6\"](\"../data/C6_nav.png\") | \n",
" ![\"C7\"](\"../data/C7_nav.png\") | \n",
" ![\"C8\"](\"../data/C8_nav.png\") | \n",
"