Towards Leitmotif Activity Detection in Opera Recordings

This is the accompanying website for the following paper:

  1. Michael Krause, Meinard Müller, and Christof Weiß
    Towards Leitmotif Activity Detection in Opera Recordings
    Transactions of the International Society for Music Information Retrieval (TISMIR), 4(1): 127–140, 2021. PDF Details DOI
    @article{KrauseMW21_TowardsLeitmotifDetection_TISMIR,
    title       = {Towards Leitmotif Activity Detection in Opera Recordings},
    author      = {Michael Krause and Meinard M{\"u}ller and Christof Wei{\ss}},
    journal     = {Transactions of the International Society for Music Information Retrieval ({TISMIR})},
    volume      = {4},
    number      = {1},
    year        = {2021},
    pages       = {127--140},
    doi         = {10.5334/tismir.116},
    url         = {https://transactions.ismir.net/articles/10.5334/tismir.116/},
    url-pdf     = {https://transactions.ismir.net/articles/10.5334/tismir.116/galley/95/download/},
    url-details = {https://www.audiolabs-erlangen.de/resources/MIR/2021-TISMIR-TowardsLeitmotifDetection}
    }

Abstract

This paper approaches the automatic detection of musical patterns in audio recordings with a particular focus on leitmotifs, which are specific types of patterns associated with certain characters, places, items, or feelings occurring in an opera or movie soundtrack. The detection of such leitmotifs is particularly challenging since their appearance can change substantially over the course of a musical work. In our case study, we consider a self-contained yet comprehensive scenario comprising 16 recorded performances of Richard Wagner’s four-opera cycle Der Ring des Nibelungen, which is a prime example for the use of leitmotifs. Within this scenario, we introduce and formalize the novel task of leitmotif activity detection. Based on a dataset of 200 hours of audio with over 50000 annotated leitmotif instances, we explore the benefits and limitations of deep-learning techniques for detecting leitmotifs. To this end, we adapt two common deep-learning strategies based on recurrent and convolutional neural networks, respectively. To investigate the robustness of the trained systems, we test their sensitivity to different modifications of the input. We find that our deep-learning systems work well in general but capture confounding factors, such as pitch distributions in leitmotif regions, instead of characteristic musical properties, such as rhythm and melody. Thus, our in-depth analysis demonstrates some challenges that may arise from applying deep-learning approaches for detecting complex musical patterns in audio recordings.

Dataset

The occurrence and instance positions of the twenty leitmotifs considered in this paper are made publicly available as a dataset for further research.

This dataset constitutes a strict superset of the data found on this page.

In this dataset, we use the following naming scheme to refer to different parts of the Ring:

Name Description
A Das Rheingold
B-1 Die Walküre, Act 1
B-2 Die Walküre, Act 2
B-3 Die Walküre, Act 2
C-1 Siegfried, Act 1
C-2 Siegfried, Act 2
C-3 Siegfried, Act 3
D-0 Götterdämmerung, Vorspiel
D-1 Götterdämmerung, Act 1
D-2 Götterdämmerung, Act 2
D-3 Götterdämmerung, Act 3

Occurrences

Occurrence positions are found in the .csv-files in the "Occurrences" subfolder of the zip-archive. For example, "Occurrences/B-2.csv" contains all motif occurrences in Die Walküre, Act 2. In these files, each line corresponds to a motif occurrence. Start and end positions are given in measures. For example, the line

Ring;778.5;780.25

in "Occurrences/B-2.csv" signifies that there is an occurrence of the Ring motif starting at 778.5 (a half measure after 778) and ending at 780.25 (a quarter measure after measure 780). Measure numbers correspond to the piano score from Richard Kleinmichel, available at IMSLP.

Instances

Instance positions are found in the .csv-files inside the 16 directories of the "Instance" subfolder of the zip-archive. For example, "Instances/P-Ba/B-2.csv" contains all motif instances in the Daniel Barenboim performance of Die Walküre, Act 2. In these files, each line corresponds to a motif instance. Start and end positions are given in seconds (for this, all CD tracks for a particular performance of an act have been cut and concatenated to form one continuous audio file for that act). For example, the line

Ring;2130.6;2137.2

in "Instances/P-Ba/B-2.csv" signifies that there is an instance of the Ring motif starting at second 2130.6 and ending at 2137.2.

A table of the performances used in this study, including ID, conductors, years of recording and length, can be found in the paper. The following table provides helpful information for identifying the exact CD releases:

ID in Paper Label Year of release Conductor, Orchestra, Choir
P-Ba WC 2009 Daniel Barenboim, Chor und Orchester der Bayreuther Festspiele
P-Ha EMI 2008 Bernard Haitink, Symphonieorchester und Chor des Bayrischen Rundfunks
P-Ka DG 1998 Herbert von Karajan, Berliner Philharmoniker, Chor der Deutschen Oper Berlin
P-Sa EMI 2012 Wolfgang Sawallisch, Bayrisches Staatsorchester, Chor der Bayrischen Staatsoper
P-So DECCA 2012 Georg Solti, Wiener Staatsopernchor, Wiener Philharmoniker
P-We OEHMS 2013 Sebastian Weigle, Frankfurter Opern- und Museumsorchester, Chor und Herren des Extrachores der Oper Frankfurt
P-Bo PHILIPS 2006 Pierre Boulez, Chor und Orchester der Bayreuther Festspiele
P-Bö DECCA 2008 Karl Böhm, Chor und Orchester der Bayreuther Festspiele
P-Fu EMI 2011 Wilhelm Furtwängler, Orchestra Sinfonica della Radio Italiana, Coro della Radio Italiana
P-Ja SONY 2012 Marek Janowski, Staatskapelle Dresden, Männer des Staatsopernchores Leipzig, Staatsopernchor Dresden
P-Ke ZYX 2012 Joseph Keilberth, Chor und Orchester der Bayreuther Festspiele; Wilhelm Furtwängler, Wiener Philharmoniker
P-Kr ORFEO 2010 Clemens Krauss, Chor und Orchester der Bayreuther Festspiele
P-Le DG 2012 James Levine, The Metropolitan Opera Orchestra, The Metropolitan Opera Chorus
P-Ne MEMBRAN 1995 Günther Neuhold, Badische Staatskapelle, Badischer Staatsopernchor
P-Sw PROFIL 2013 Hans Swarowsky, Grosses Symphonieorchester mit Mitgliedern der Tschechischen Philharmonie und des Orchesters des Nationaltheaters Prag, Chor der Wiener Volksoper
P-Th DG 2013 Christian Thielemann, Chor und Zusatzchor der Wiener Staatsoper, Orchester der Wiener Staatsoper, Bühnenorchester der Wiener Staatsoper

Thus, the performance by Wolfgang Sawallisch (P-Sa) was released by EMI in 2012.

Acknowledgements

We thank Julia Zalkow and Vlora Arifi-Müller for their assistance in preparing the data. This work was supported by the German Research Foundation (DFG MU 2686/7-2). The International Audio Laboratories Erlangen are a joint institution of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) and Fraunhofer Institut für Integrierte Schaltungen (IIS).

References

  1. Richard Wagner
    On the Application of Music to the Drama
    In: Prose Works, Broude Brothers, New York: 175–191, 1966.
    @incollection{Wagner66_MusicDrama_BOOK,
    author      = {Richard Wagner},
    title       = {On the Application of Music to the Drama},
    booktitle   = {Prose Works},
    year        = {1966},
    translator  = {William A. Ellis},
    publisher   = {Broude Brothers, New York},
    pages       = {175--191}
    }
  2. Matthew Bribitzer-Stull
    Understanding the Leitmotif
    Cambridge University Press, 2015.
    @book{Bribitzer15_Leitmotif_BOOK,
    author    = {Matthew Bribitzer-Stull},
    title     = {Understanding the Leitmotif},
    publisher = {Cambridge University Press},
    year      = {2015}
    }
  3. Frank Zalkow, Christof Weiß, and Meinard Müller
    Exploring Tonal-Dramatic Relationships in Richard Wagner's Ring Cycle
    In Proceedings of the International Conference on Music Information Retrieval (ISMIR): 642–648, 2017. PDF
    @inproceedings{ZalkowWM17_WagnerHarmony_ISMIR,
    author    = {Frank Zalkow and Christof Wei{\ss} and Meinard M{\"u}ller},
    title     = {Exploring Tonal-Dramatic Relationships in {R}ichard {W}agner's {R}ing Cycle},
    booktitle = {Proceedings of the International Conference on Music Information Retrieval ({ISMIR})},
    pages     = {642--648},
    address   = {Suzhou, China},
    year      = {2017},
    url-pdf   = {http://ismir2017.smcnus.org/wp-content/uploads/2017/10/132_Paper.pdf}
    }
  4. Frank Zalkow, Christof Weiß, Thomas Prätzlich, Vlora Arifi-Müller, and Meinard Müller
    A Multi-Version Approach for Transferring Measure Annotations Between Music Recordings
    In Proceedings of the AES International Conference on Semantic Audio: 148–155, 2017.
    @inproceedings{ZalkowWPAM17_MeasureTransfer_AES,
    author    = {Frank Zalkow and Christof Wei{\ss} and Thomas Pr{\"a}tzlich and Vlora Arifi-M{\"u}ller and Meinard M{\"u}ller},
    title     = {A Multi-Version Approach for Transferring Measure Annotations Between Music Recordings},
    booktitle = {Proceedings of the {AES} International Conference on Semantic Audio},
    pages     = {148--155},
    address   = {Erlangen, Germany},
    year      = {2017}
    }
  5. Richard Wagner
    Der Ring des Nibelungen. Vollständiger Text mit Notentafeln der Leitmotive
    Schott Music, 2013.
    @book{Burghold13_Leitmotive_Schott,
    title     = {{D}er {R}ing des {N}ibelungen. {V}ollständiger {T}ext mit {N}otentafeln der {L}eitmotive},
    author    = {Richard Wagner},
    editor    = {Julius Burghold},
    address   = {Mainz},
    publisher = {Schott Music},
    note      = {Reprint of the original edition from 1913 (Ed. Julius Burghold)},
    year      = {2013}
    }