International Audio Laboratories Erlangen

Towards Leitmotif Activity Detection in Opera Recordings

This is the accompanying website for the following paper:

1. Michael Krause, Meinard Müller, and Christof Weiß
   Towards Leitmotif Activity Detection in Opera Recordings
   Transactions of the International Society for Music Information Retrieval (TISMIR), 4(1): 127–140, 2021. Bibtex PDF Details DOI

   @article{KrauseMW21_TowardsLeitmotifDetection_TISMIR,
   title       = {Towards Leitmotif Activity Detection in Opera Recordings},
   author      = {Michael Krause and Meinard M{\"u}ller and Christof Wei{\ss}},
   journal     = {Transactions of the International Society for Music Information Retrieval ({TISMIR})},
   volume      = {4},
   number      = {1},
   year        = {2021},
   pages       = {127--140},
   doi         = {10.5334/tismir.116},
   url         = {https://transactions.ismir.net/articles/10.5334/tismir.116/},
   url-pdf     = {2021_KrauseMW_LeitmotifDetection_TISMIR_ePrint.pdf},
   url-details = {https://www.audiolabs-erlangen.de/resources/MIR/2021-TISMIR-TowardsLeitmotifDetection}
   }

Abstract

This paper approaches the automatic detection of musical patterns in audio recordings with a particular focus on leitmotifs, which are specific types of patterns associated with certain characters, places, items, or feelings occurring in an opera or movie soundtrack. The detection of such leitmotifs is particularly challenging since their appearance can change substantially over the course of a musical work. In our case study, we consider a self-contained yet comprehensive scenario comprising 16 recorded performances of Richard Wagner’s four-opera cycle Der Ring des Nibelungen, which is a prime example for the use of leitmotifs. Within this scenario, we introduce and formalize the novel task of leitmotif activity detection. Based on a dataset of 200 hours of audio with over 50000 annotated leitmotif instances, we explore the benefits and limitations of deep-learning techniques for detecting leitmotifs. To this end, we adapt two common deep-learning strategies based on recurrent and convolutional neural networks, respectively. To investigate the robustness of the trained systems, we test their sensitivity to different modifications of the input. We find that our deep-learning systems work well in general but capture confounding factors, such as pitch distributions in leitmotif regions, instead of characteristic musical properties, such as rhythm and melody. Thus, our in-depth analysis demonstrates some challenges that may arise from applying deep-learning approaches for detecting complex musical patterns in audio recordings.

Dataset

The occurrence and instance positions of the twenty leitmotifs considered in this paper are made publicly available as a dataset for further research.

This dataset constitutes a strict superset of the data found on this page.

In this dataset, we use the following naming scheme to refer to different parts of the Ring:

Occurrences

Occurrence positions are found in the .csv-files in the "Occurrences" subfolder of the zip-archive. For example, "Occurrences/B-2.csv" contains all motif occurrences in Die Walküre, Act 2. In these files, each line corresponds to a motif occurrence. Start and end positions are given in measures. For example, the line

Ring;778.5;780.25

in "Occurrences/B-2.csv" signifies that there is an occurrence of the Ring motif starting at 778.5 (a half measure after 778) and ending at 780.25 (a quarter measure after measure 780). Measure numbers correspond to the piano score from Richard Kleinmichel, available at IMSLP.

Instances

Instance positions are found in the .csv-files inside the 16 directories of the "Instance" subfolder of the zip-archive. For example, "Instances/P-Ba/B-2.csv" contains all motif instances in the Daniel Barenboim performance of Die Walküre, Act 2. In these files, each line corresponds to a motif instance. Start and end positions are given in seconds (for this, all CD tracks for a particular performance of an act have been cut and concatenated to form one continuous audio file for that act). For example, the line

Ring;2130.6;2137.2

in "Instances/P-Ba/B-2.csv" signifies that there is an instance of the Ring motif starting at second 2130.6 and ending at 2137.2.

A table of the performances used in this study, including ID, conductors, years of recording and length, can be found in the paper. The following table provides helpful information for identifying the exact CD releases:

Thus, the performance by Wolfgang Sawallisch (P-Sa) was released by EMI in 2012.

Acknowledgements

We thank Julia Zalkow and Vlora Arifi-Müller for their assistance in preparing the data. This work was supported by the German Research Foundation (DFG MU 2686/7-2). The International Audio Laboratories Erlangen are a joint institution of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) and Fraunhofer Institut für Integrierte Schaltungen (IIS).

References

1. Richard Wagner
   On the Application of Music to the Drama
   In: Prose Works, Broude Brothers, New York: 175–191, 1966. Bibtex

   @incollection{Wagner66_MusicDrama_BOOK,
   author      = {Richard Wagner},
   title       = {On the Application of Music to the Drama},
   booktitle   = {Prose Works},
   year        = {1966},
   translator  = {William A. Ellis},
   publisher   = {Broude Brothers, New York},
   pages       = {175--191}
   }

2. Matthew Bribitzer-Stull
   Understanding the Leitmotif
   Cambridge University Press, 2015. Bibtex

   @book{Bribitzer15_Leitmotif_BOOK,
   author    = {Matthew Bribitzer-Stull},
   title     = {Understanding the Leitmotif},
   publisher = {Cambridge University Press},
   year      = {2015}
   }

3. Frank Zalkow, Christof Weiß, and Meinard Müller
   Exploring Tonal-Dramatic Relationships in Richard Wagner's Ring Cycle
   In Proceedings of the International Conference on Music Information Retrieval (ISMIR): 642–648, 2017. Bibtex PDF

   @inproceedings{ZalkowWM17_WagnerHarmony_ISMIR,
   author    = {Frank Zalkow and Christof Wei{\ss} and Meinard M{\"u}ller},
   title     = {Exploring Tonal-Dramatic Relationships in {R}ichard {W}agner's {R}ing Cycle},
   booktitle = {Proceedings of the International Conference on Music Information Retrieval ({ISMIR})},
   pages     = {642--648},
   address   = {Suzhou, China},
   year      = {2017},
   url-pdf   = {http://ismir2017.smcnus.org/wp-content/uploads/2017/10/132_Paper.pdf}
   }

4. Frank Zalkow, Christof Weiß, Thomas Prätzlich, Vlora Arifi-Müller, and Meinard Müller
   A Multi-Version Approach for Transferring Measure Annotations Between Music Recordings
   In Proceedings of the AES International Conference on Semantic Audio: 148–155, 2017. Bibtex

   @inproceedings{ZalkowWPAM17_MeasureTransfer_AES,
   author    = {Frank Zalkow and Christof Wei{\ss} and Thomas Pr{\"a}tzlich and Vlora Arifi-M{\"u}ller and Meinard M{\"u}ller},
   title     = {A Multi-Version Approach for Transferring Measure Annotations Between Music Recordings},
   booktitle = {Proceedings of the {AES} International Conference on Semantic Audio},
   pages     = {148--155},
   address   = {Erlangen, Germany},
   year      = {2017}
   }

5. Richard Wagner
   Der Ring des Nibelungen. Vollständiger Text mit Notentafeln der Leitmotive
   Schott Music, 2013. Bibtex

   @book{Burghold13_Leitmotive_Schott,
   title     = {{D}er {R}ing des {N}ibelungen. {V}ollständiger {T}ext mit {N}otentafeln der {L}eitmotive},
   author    = {Richard Wagner},
   editor    = {Julius Burghold},
   address   = {Mainz},
   publisher = {Schott Music},
   note      = {Reprint of the original edition from 1913 (Ed. Julius Burghold)},
   year      = {2013}
   }