AudioLabs - ISMIR 2019 Tutorial: Fundamentals of Music Processing

ISMIR 2019 Tutorial: Fundamentals of Music Processing

This is the accompanying website for the tutorial Fundamentals of Music Processing: An Introduction Using Python and Jupyter Notebooks offered at the International Society for Music Information Retrieval Conference 2019.

Lecturers: Meinard Müller, Frank Zalkow
Date: November 4th 2019, 9.00am–12.30am
Place: Congress Center of Delft University of Technology
ISMIR Website: Tutorials@ISMIR
FMP Notebooks: www.audiolabs-erlangen.de/FMP
Text book: www.music-processing.de
Overview Article: PDF

Content

This tutorial will give an easy-to-understand introduction to music processing with a particular focus on audio-related analysis and retrieval tasks. In particular, the tutorial is aimed at non-experts and researchers who are new to the field. Based on well-established topics in Music Information Retrieval (MIR) as motivating application scenarios, we present fundamental techniques and algorithms that apply to a wide range of analysis and retrieval problems. We intend to explain the main ideas and techniques in an intuitive fashion using various figures and sound examples. Besides the theory, we also show how these techniques can be implemented going through specific Python code examples. All material, including the introduction of MIR scenarios, illustrations, sound examples, technical concepts, mathematical details, and code examples, are integrated into a comprehensive framework based on Jupyter notebooks. The notebooks are organized along with the eight chapters of the textbook on Fundamentals of Music Processing (FMP) (Springer 2015, http://www.music-processing.de). Another important goal of this tutorial is to show how the notebooks can be used to generate educational material for lectures and presentations. The notebooks (as well as HTML exports and multimedia examples) can be accessed via https://www.audiolabs-erlangen.de/FMP.

Topics and Slides

Overview
Slides (PDF), Handouts (6 slides per page) (PDF)
FMP Notebooks: Overview
Music Representations and Retrieval
Slides (PDF), Handouts (6 slides per page) (PDF)
FMP Notebooks: Sheet Music Representations, Piano Roll Representations, Symbolic Format MIDI, Log-Frequency Spectrogram and Chromagram, Dynamic Time Warping, Music Synchronization
Audio Structure Analysis
Slides (PDF), Handouts (6 slides per page) (PDF)
FMP Notebooks: General Principles, Self-Similarity Matrix, Scape Plot Representation, Novelty-Based Segmentation
Tempo and Beat Tracking
Slides (PDF), Handouts (6 slides per page) (PDF)
FMP Notebooks: Onset Detection, Spectral-Based Novelty Fourier Tempogram Predominant Local Pulse
Audio Decomposition
Slides (PDF), Handouts (6 slides per page) (PDF)
FMP Notebooks: Harmonic–Percussive Separation, Harmonic–Residual–Percussive Separation Nonnegative Matrix Factorization (NMF) NMF-Based Spectrogram Factorization NMF-Based Audio Decomposition

Further Links

There are a number of comprehensive and well-document toolboxes that provide modular source code for processing and analyzing music and audio signals. Some of the code examples have been inspired or are based on code provided by other code collections. In particular, we want to mention the following excellent sources:

Audio Content Analysis (hands-on experience in audio and music processing)
Audio Degradation Toolbox (data augmentation)
Chroma Toolbox (functions chroma feature extraction)
Constant-Q Toolbox (time-frequency transforms)
essentia library
The course Fundamentals of Music Processing by Eran Egozy comprises many Python code examples.
JAMS (A JSON Annotated Music Specification for Reproducible MIR Research)
jAudio toolbox (programs for audio feature extraction)
The python package LibROSA maintained by Brian McFee provides many building blocks to create music information retrieval systems. It also contains a gallery of more advanced examples.
madmom toolbox (MIR approaches that employ deep learning techniques)
Marsyas toolbox (Music Analysis, Retrieval and Synthesis for Audio Signals)
MIR toolbox (computational approaches to MIR)
mir_eval library (evaluation of MIR approaches)
MSAF toolbox (audio structure analysis)
muda library (data augmentation)
The website Notes on Music Information Retrieval maintained by Steve Tjoa is a collection of instructional materials for music information retrieval (MIR).
SM Toolbox (audio structure analysis)
Tempogram Toolbox (tempo and pulse tracking)
TSM Toolbox (time-scale modification)

Literature

The FMP notebooks offer a collection of educational material closely following the textbook Fundamentals of Music Processing (FMP).

Meinard Müller
Fundamentals of Music Processing
Audio, Analysis, Algorithms, Applications
ISBN: 978-3-319-21944-8
Springer, 2015

In the following list, one finds additional pointers to books and articles related to the tutorial:

Meinard Müller and Frank Zalkow
FMP Notebooks: Educational Material for Teaching and Learning Fundamentals of Music Processing
In Proceedings of the International Conference on Music Information Retrieval (ISMIR), 2019. PDF

@inproceedings{MuellerZ19_FMP_ISMIR,
author    = {Meinard M{\"u}ller and Frank Zalkow},
title     = {{FMP} Notebooks: Educational Material for Teaching and Learning Fundamentals of Music Processing},
booktitle = {Proceedings of the International Conference on Music Information Retrieval ({ISMIR})},
address   = {Delft, The Netherlands},
month     = {November},
year      = {2019},
url-pdf   = {2019_MuellerZalkow_FMP_ISMIR.pdf},
}

Patricio López-Serrano, Christian Dittmar, Yigitcan Özer, and Meinard Müller
NMF Toolbox: Music Processing Applications of Nonnegative Matrix Factorization
In Proceedings of the International Conference on Digital Audio Effects (DAFx), 2019. PDF Demo

@inproceedings{LopezDOM19_ToolboxNMF_DAFx,
author    = {Patricio L{\'o}pez-Serrano and Christian Dittmar and Yigitcan {\"O}zer and Meinard M{\"u}ller},
title     = {{NMF} Toolbox: Music Processing Applications of Nonnegative Matrix Factorization},
booktitle = {Proceedings of the International Conference on Digital Audio Effects ({DAFx})},
address   = {Birmingham, UK},
year      = {2019},
pages     = {},
url-pdf     = {2019_LopezSerranoDOM_NMF_DAFx.pdf},
url-demo    = {https://www.audiolabs-erlangen.de/resources/MIR/NMFtoolbox/}
}

Meinard Müller, Bryan Pardo, Gautham J. Mysore, and Vesa Välimäki
Recent Advances in Music Signal Processing
IEEE Signal Processing Magazine, 36(1): 17–19, 2019. Details DOI

@article{MullerPMV19_Editorial_IEEE-SPM,
author    = {Meinard M{\"u}ller and Bryan Pardo and Gautham J. Mysore and Vesa V{\"a}lim{\"a}ki},
title     = {Recent Advances in Music Signal Processing},
journal   = {{IEEE} Signal Processing Magazine},
volume    = {36},
number    = {1},
pages     = {17--19},
year      = {2019},
url       = {https://doi.org/10.1109/MSP.2018.2876190},
doi       = {10.1109/MSP.2018.2876190},
url-details={https://ieeexplore.ieee.org/document/8588415/}
}

Brian McFee, Matt McVicar, Oriol Nieto, Stefan Balke, Carl Thome, Dawen Liang, Eric Battenberg, Josh Moore, Rachel Bittner, Ryuichi Yamamoto, Dan Ellis, Fabian-Robert Stöter, Douglas Repetto, Simon Waloschek, CJ Carr, Seth Kranzler, Keunwoo Choi, Petr Viktorin, Joao Felipe Santos, Adrian Holovaty, Waldir Pimenta, and Hojin Lee
librosa 0.5.0
2017. DOI

@misc{McFeeEtAl17_librosa_zenodo,
author       = {Brian McFee and Matt McVicar and Oriol Nieto and Stefan Balke and Carl Thome and Dawen Liang and Eric Battenberg and Josh Moore and Rachel Bittner and Ryuichi Yamamoto and Dan Ellis and Fabian-Robert St{\"o}ter and Douglas Repetto and Simon Waloschek and CJ Carr and Seth Kranzler and Keunwoo Choi and Petr Viktorin and Joao Felipe Santos and Adrian Holovaty and Waldir Pimenta and Hojin Lee},
title        = {librosa 0.5.0},
month        = feb,
year         = 2017,
doi          = {10.5281/zenodo.293021},
url          = {https://doi.org/10.5281/zenodo.293021}
}

Sebastian Böck, Filip Korzeniowski, Jan Schlüter, Florian Krebs, and Gerhard Widmer
madmom: A New Python Audio and Music Signal Processing Library
In Proceedings of the ACM International Conference on Multimedia (ACM-MM): 1174–1178, 2016.

@inproceedings{BoeckKSKW16_madmom_ACM-MM,
author    = {Sebastian B{\"{o}}ck and Filip Korzeniowski and Jan Schl{\"{u}}ter and Florian Krebs and Gerhard Widmer},
title     = {madmom: {A} New Python Audio and Music Signal Processing Library},
booktitle = {Proceedings of the {ACM} International Conference on Multimedia ({ACM-MM})},
address   = {Amsterdam, The Netherlands},
pages     = {1174--1178},
year      = {2016}
}

Peter Knees and Markus Schedl
Music Similarity and Retrieval
Springer Verlag, ISBN: 978-3-662-49720-3, 2016.

@book{KneesS16_MusicSimilarityRetrieval_SPRINGER,
author    = {Peter Knees and Markus Schedl},
title     = {Music Similarity and Retrieval},
type      = {Monograph},
year      = {2016},
isbn      = {978-3-662-49720-3},
publisher = {Springer Verlag}
}

Oriol Nieto and Juan Pablo Bello
Systematic Exploration Of Computational Music Structure Research
In Proceedings of the International Conference on Music Information Retrieval (ISMIR): 547–553, 2016.

@inproceedings{NietoB16_MSAF_ISMIR,
author    = {Oriol Nieto and Juan Pablo Bello},
title     = {Systematic Exploration Of Computational Music Structure Research},
booktitle = {Proceedings of the International Conference on Music Information Retrieval ({ISMIR})},
address   = {New York City, USA},
year      = {2016},
pages     = {547--553}
}

Claus Weihs, Dietmar Jannach, Igor Vatolkin, and Guenter Rudolph
Music Data Analysis: Foundations and Applications
CRC Press, ISBN: 978-1-498-71956-8, 2016.

@book{WeihsJVR16_MusicDataAnalysis_CRC,
author    = {Claus Weihs and Dietmar Jannach and Igor Vatolkin and Guenter Rudolph},
title     = {Music Data Analysis: Foundations and Applications},
type      = {Monograph},
year      = {2016},
isbn      = {978-1-498-71956-8},
publisher = {CRC Press}
}

Brian McFee, Eric J. Humphrey, and Juan Pablo Bello
A Software Framework for Musical Data Augmentation
In Proceedings of International Society for Music Information Retrieval Conference (ISMIR): 248–254, 2015.

@inproceedings{McFeeHB15_DataAug_ISMIR,
author    = {Brian McFee and Eric J. Humphrey and Juan Pablo Bello},
title     = {A Software Framework for Musical Data Augmentation},
booktitle = {Proceedings of International Society for Music Information Retrieval Conference ({ISMIR})},
pages     = {248--254},
address   = {M{\'a}laga, Spain},
year      = {2015}
}

Brian McFee, Colin Raffel, Dawen Liang, Daniel P.W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto
Librosa: Audio and Music Signal Analysis in Python
In Proceedings the Python Science Conference: 18–25, 2015.

@inproceedings{McFeeRLEMBN15_librosa_Python,
author    = {Brian McFee and Colin Raffel and Dawen Liang and Daniel P.W. Ellis and Matt McVicar and Eric Battenberg and Oriol Nieto},
title     = {Librosa: Audio and Music Signal Analysis in Python},
booktitle = {Proceedings the Python Science Conference},
pages     = {18--25},
year      = {2015}
}

Meinard Müller
Fundamentals of Music Processing — Audio, Analysis, Algorithms, Applications
Springer Verlag, ISBN: 978-3-319-21944-8, 2015. Details

@book{Mueller15_FMP_SPRINGER,
author    = {Meinard M\"{u}ller},
title     = {Fundamentals of Music Processing -- Audio, Analysis, Algorithms, Applications},
type      = {Monograph},
year      = {2015},
isbn      = {978-3-319-21944-8},
publisher = {Springer Verlag},
url-details={http://www.music-processing.de}
}

Jonathan Driedger and Meinard Müller
TSM Toolbox: MATLAB Implementations of Time-Scale Modification Algorithms
In Proceedings of the International Conference on Digital Audio Effects (DAFx): 249–256, 2014. PDF Details

@inproceedings{DriedgerM14_TSM_DAFx,
author    = {Jonathan Driedger and Meinard M{\"u}ller},
title     = {{TSM} {T}oolbox: {MATLAB} Implementations of Time-Scale Modification Algorithms},
booktitle = {Proceedings of the International Conference on Digital Audio Effects ({DAFx})},
address   = {Erlangen, Germany},
year      = {2014},
pages     = {249--256},
url-pdf   = {2014_DriedgerMueller_TSM-Toolbox_DAFX.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/TSMtoolbox}
}

Eric J. Humphrey, Justin Salamon, Oriol Nieto, Jon Forsyth, Rachel M. Bittner, and Juan Pablo Bello
JAMS: A JSON Annotated Music Specification for Reproducible MIR Research
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 591–596, 2014.

@inproceedings{HumphreySNBB14_JAMS_ISMIR,
author    = {Eric J. Humphrey and Justin Salamon and Oriol Nieto and Jon Forsyth and Rachel M. Bittner and Juan Pablo Bello},
title     = {{JAMS:} {A} {JSON} Annotated Music Specification for Reproducible {MIR} Research},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
address   = {Taipei, Taiwan},
month     = {October},
pages     = {591--596},
year      = {2014},
}

Meinard Müller, Nanzhu Jiang, and Harald Grohganz
SM Toolbox: MATLAB implementations for computing and enhancing similarity matrices
In Proceedings of the Audio Engineering Society (AES) Conference on Semantic Audio, 2014. PDF Details

@inproceedings{MuellerJG14_SM-Toolbox_AES,
author    = {Meinard M{\"u}ller and Nanzhu Jiang and Harald Grohganz},
title     = {{SM} {T}oolbox: {MATLAB} implementations for computing and enhancing similarity matrices},
booktitle = {Proceedings of the Audio Engineering Society ({AES}) Conference on Semantic Audio},
address   = {London, UK},
year      = {2014},
url-pdf   = {2014_MuellerJiangGrohganz_ToolboxSM_AES.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/SMtoolbox}
}

Colin Raffel, Brian McFee, Eric J. Humphrey, and Justin Salamon and
MIR _EVAL: A Transparent Implementation of Common MIR Metrics
In Proceedings of the International Conference on Music Information Retrieval (ISMIR): 367–372, 2014.

@inproceedings{RaffelMHSNLE14_MirEval_ISMIR,
author    = {Colin Raffel and Brian McFee and Eric J. Humphrey and Justin Salamon and
Oriol Nieto and Dawen Liang and Daniel P. W. Ellis},
title     = {{MIR{\_}EVAL}: {A} Transparent Implementation of Common {MIR} Metrics},
pages     = {367--372},
booktitle = {Proceedings of the International Conference on Music Information Retrieval ({ISMIR})},
address   = {Taipei, Taiwan},
year      = {2014},
}

Dmitry Bogdanov, Nicolas Wack, Emilia Gómez, and Sankalp Gulati and
Essentia: An Audio Analysis Library for Music Information Retrieval
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 493–498, 2013.

@inproceedings{BogdanovWGGHMRSZS13_essentia_ISMIR,
author    = {Dmitry Bogdanov and Nicolas Wack and Emilia G{\'{o}}mez and Sankalp Gulati and
Perfecto Herrera and Oscar Mayor and Gerard Roma and Justin Salamon and
Jos{\'{e}} R. Zapata and Xavier Serra},
title     = {Essentia: An Audio Analysis Library for Music Information Retrieval},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
pages     = {493--498},
address   = {Curitiba, Brazil},
year      = {2013},
}

Matthias Mauch and Sebastian Ewert
The Audio Degradation Toolbox and Its Application to Robustness Evaluation
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 83–88, 2013.

@inproceedings{MauchE13_AudioDegradationToolbox_ISMIR,
author    = {Matthias Mauch and Sebastian Ewert},
title     = {The Audio Degradation Toolbox and Its Application to Robustness Evaluation},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
pages     = {83--88},
year      = {2013},
address   = {Curitiba, Brazil}
}

Peter Grosche and Meinard Müller
Tempogram Toolbox: MATLAB Tempo and Pulse Analysis of Music Recordings
In Late-Breaking and Demo Session of the Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2011. PDF Details

@inproceedings{GroscheM11_TempogramToolbox_ISMIR-lateBreaking,
author    = {Peter Grosche and Meinard M{\"u}ller},
title     = {{T}empogram {T}oolbox: {MATLAB} Tempo and Pulse Analysis of Music Recordings},
booktitle = {Late-Breaking and Demo Session of the Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
address   = {Miami, Florida, USA},
year      = {2011},
month     = {October},
url-pdf = {2011_GroscheMueller_TempogramToolbox_ISMIR-LateBreaking.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/tempogramtoolbox}
}

Alexander Lerch
An Introduction to Audio Content Analysis
John Wiley & Sons, ISBN: 9781118266823, 2012.

@book{Lerch12_AudioAnalysis_Wiley,
author    = {Alexander Lerch},
title     = {An Introduction to Audio Content Analysis},
type      = {Monograph},
year      = {2012},
isbn      = {9781118266823},
publisher = {John Wiley & Sons}
}

Meinard Müller and Sebastian Ewert
Chroma Toolbox: MATLAB implementations for extracting variants of chroma-based audio features
In Proceedings of the International Conference on Music Information Retrieval (ISMIR): 215–220, 2011. PDF Details

@inproceedings{MuellerEwert11_ChromaToolbox_ISMIR,
author     = {Meinard M{\"u}ller and Sebastian Ewert},
title      = {{C}hroma {T}oolbox: {MATLAB} implementations for extracting variants of chroma-based audio features},
booktitle  = {Proceedings of the International Conference on Music Information Retrieval ({ISMIR})},
address    = {Miami, Florida, USA},
year       = {2011},
pages      = {215--220},
url-pdf    = {2011_MuellerEwert_ChromaToolbox_ISMIR.pdf},
url-details = {https://www.audiolabs-erlangen.de/resources/MIR/chromatoolbox}
}

Christian Schörkhuber and Anssi P. Klapuri
Constant-Q transform toolbox for music processing
In Proceedings of the Sound and Music Computing Conference (SMC): 3–64, 2010.

@inproceedings{SchoerkhuberK10_ConstantQTransform_SMC,
title     = {Constant-{Q} transform toolbox for music processing},
author    = {Christian Sch\"{o}rkhuber and Anssi P. Klapuri},
booktitle = {Proceedings of the Sound and Music Computing Conference {(SMC)}},
year      = {2010},
address   = {Barcelona, Spain},
pages     = {3-64},
}

George Tzanetakis
Music analysis, retrieval and synthesis of audio signals MARSYAS
In Proceedings of the ACM International Conference on Multimedia (ACM-MM): 931–932, 2009.

@inproceedings{Tzanetakis09_MARSYAS_ACM-MM,
author    = {George Tzanetakis},
title     = {Music analysis, retrieval and synthesis of audio signals {MARSYAS}},
booktitle = {Proceedings of the {ACM} International Conference on Multimedia ({ACM-MM})},
address   = {Vancouver, British Columbia, Canada},
pages     = {931--932},
year      = {2009},
}

Olivier Lartillot and Petri Toiviainen
MIR in MATLAB (II): A Toolbox for Musical Feature Extraction from Audio
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 127–130, 2007.

@inproceedings{LartillotT07_MirToolbox_ISMIR,
author    = {Olivier Lartillot and Petri Toiviainen},
title     = {{MIR} in {MATLAB} {(II):} {A} Toolbox for Musical Feature Extraction from Audio},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
address   = {Vienna, Austria},
year      = {2007},
pages     = {127--130}
}

Anssi P. Klapuri and Manuel Davy
Signal Processing Methods for Music Transcription
Springer, ISBN: 0-387-30667-6, 2006.

@book{KlapuriD06_SPforMusic_BOOK,
author    = {Anssi P. Klapuri and Manuel Davy},
title     = {Signal Processing Methods for Music Transcription},
publisher = {Springer},
address   = {New York},
year      = {2006},
isbn      = {0-387-30667-6}
}

Daniel McEnnis, Cory McKay, Ichiro Fujinaga, and Philippe Depalle
jAudio: An Feature Extraction Library
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 600–603, 2005.

@inproceedings{McEnnisKFD05_jAudio_ISMIR,
author    = {Daniel McEnnis and Cory McKay and Ichiro Fujinaga and Philippe Depalle},
title     = {jAudio: An Feature Extraction Library},
pages     = {600--603},
booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})},
address   = {London, UK},
year      = {2005}
}

International Audio Laboratories Erlangen

ISMIR 2019 Tutorial: Fundamentals of Music Processing

Content

Topics and Slides

Further Links

Literature