International Audio Laboratories Erlangen

Tools for Semi-Automatic Bounding Box Annotation of Musical Measures in Sheet Music

This is the accompanying website for the following paper:

  1. Frank Zalkow, Angel Villar Corrales, TJ Tsai, Vlora Arifi-Müller, and Meinard Müller
    Tools for Semi-Automatic Bounding Box Annotation of Musical Measures in Sheet Music
    In Demos and Late Breaking News of the International Society for Music Information Retrieval Conference (ISMIR), 2019. PDF Details 
    @inproceedings{2019_ZalkowVTAM_MeasureAnnotation_ISMIR-LBD,
author      = {Frank Zalkow and Angel Villar Corrales and TJ Tsai and Vlora Arifi-M{\"u}ller and Meinard M{\"u}ller},
title       = {Tools for Semi-Automatic Bounding Box Annotation of Musical Measures in Sheet Music},
booktitle   = {Demos and Late Breaking News of the International Society for Music Information Retrieval Conference ({ISMIR})},
address     = {Delft, The Netherlands},
year        = {2019},
url-pdf     = {2019_ZalkowVTAM_BoundingBox_ISMIR-LBD.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/2019-ISMIR-LBD-Measures}
}

Abstract

Teaser_BoundingBox

In score following, one main goal is to highlight measure positions in sheet music synchronously to audio playback. Such applications require alignments between sheet music and audio representations. Often, such alignments can be computed automatically in the case that the sheet music representations are given in some symbolically encoded music format. However, sheet music is often available only in the form of digitized scans. In this case, the automated computation of accurate alignments poses still many challenges [1]. In this contribution, we present various semi-automatic tools for solving the subtask of determining bounding boxes (given in pixels) of measure positions in digital scans of sheet music—a task that is extremely tedious when being done manually.

Data Set

We provide measure annotations for several hundred pages of sheet music, including the complete cycle Der Ring des Nibelungen by Richard Wagner, selected piano sonatas by Ludwig von Beethoven, the complete cycle Winterreise by Franz Schubert, as well as selected pieces from the Carus publishing house.

Download (248 MB)

The data set also contains a Jupyter notebook with Python code for demonstrating how to parse the data and how to visualize the bounding box annotations on top of a sheet music image. The following HTML export shows the notebook.

Jupyter notebook

Score Following Demos

  • Selected Pieces from Carus: Chorissimo! Blue
    • No. 44: Abends, will ich schafen gehn No. 56: Der Ring No. 79: Schwesterlein, wann gehn wir nach Haus No. 81: Un poquito cantas No. 105: Greensleeves
  • Selected Movements from Ludwig van Beethoven's Sonatas
    • No. 12, Opus 26, 1st Movement No. 27, Opus 90, 1st Movement No. 28, Opus 101, 1st Movement
  • Franz Schubert: Winterreise
    • No. 1: Gute Nacht No. 2: Die Wetterfahne No. 3: Gefror’ne Thränen No. 4: Erstarrung No. 5: Der Lindenbaum No. 6: Wasserfluth No. 7: Auf dem Flusse No. 8: Rückblick No. 9: Irrlicht No. 10: Rast No. 11: Frühlingstraum No. 12: Einsamkeit No. 13: Die Post No. 14: Der greise Kopf No. 15: Die Krähe No. 16: Letzte Hoffnung No. 17: Im Dorfe No. 18: Der stürmische Morgen No. 19: Täuschung No. 20: Der Wegweiser No. 21: Das Wirtshaus No. 22: Muth No. 23: Die Nebensonnen No. 24: Der Leiermann

    Acknowledgements

    The International Audio Laboratories Erlangen are a joint institution of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) and Fraunhofer Institute for Integrated Circuits IIS. This work was supported by the German Research Foundation (MU 2686/12-1, MU 2686/7-1, MU 2686/7-2). We thank Johannes Graulich from Carus publishing house for providing sheet music and audio examples for our demonstrators.

    References

    1. Matthias Dorfer, Andreas Arzt, and Gerhard Widmer
      Towards Score Following In Sheet Music Images
      In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 789–795, 2016. 
      @inproceedings{DorferAW16_ScoreFollowDNN_ISMIR,
author    = {Matthias Dorfer and Andreas Arzt and Gerhard Widmer},
title     = {Towards Score Following In Sheet Music Images},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
address   = {New York, USA},
pages     = {789--795},
year      = {2016}
}
    2. Meinard Müller
      Fundamentals of Music Processing
      Springer Verlag, ISBN: 978-3-319-21944-8, 2015. 
      @book{Mueller15_FMP_SPRINGER,
author    = {Meinard M{\"u}ller},
title     = {Fundamentals of Music Processing},
type      = {Monograph},
year      = {2015},
isbn      = {978-3-319-21944-8},
publisher = {Springer Verlag}
}
    3. Thitaree Tanprasert, Teerapat Jenrungrot, Meinard Müller, and Timothy Tsai
      MIDI—Sheet Music Alignment Using Bootleg Score Synthesis
      In Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2019. 
      @inproceedings{TanprasertJTM19_SheetMusic_ISMIR,
author = {Thitaree Tanprasert and Teerapat Jenrungrot and Meinard M{\"u}ller and Timothy Tsai},
title = {{MIDI}--Sheet Music Alignment Using Bootleg Score Synthesis},
booktitle = {Proceedings of the International Conference on Music Information Retrieval ({ISMIR})},
address   = {Delft, The Netherlands},
month     = {November},
year      = {2019},
}
    4. Nils Werner, Stefan Balke, Fabian-Robert Stöter, Meinard Müller, and Bernd Edler
      trackswitch.js: A Versatile Web-Based Audio Player for Presenting Scientific Results
      In Proceedings of the Web Audio Conference (WAC), 2017. 
      @inproceedings{WernerBSME17_TrackswitchJSPlayer_WAC,
title     = {{trackswitch.js}: A Versatile Web-Based Audio Player for Presenting Scientific Results},
author    = {Nils Werner and Stefan Balke and Fabian-Robert St{\"o}ter and Meinard M{\"u}ller and Bernd Edler},
booktitle = {Proceedings of the Web Audio Conference ({WAC})},
pages     = {},
address  = {London, UK},
year      = {2017}
}